Compare commits

..
Author SHA1 Message Date
Léo bec4d1805d fix(docker): let searxng boot when the settings migration fails
The migration runs under `set -eu`, so any settings file it cannot parse or
rewrite took the container down instead of merely going unmigrated. A symlinked
/etc/searxng/settings.yml is enough: the migration refuses a non-regular file
and searxng, which reads through the symlink perfectly well, never got to start.

Guard the call with `|| true` in all three Compose variants. The failure still
prints its reason on stderr, and searxng is left to report anything genuinely
wrong with the file.
2026-08-16 04:06:38 +02:00
Léo 54d794e8de fix(docker): chmod the settings temp file before chowning it
The Compose cap set is `cap_drop: ALL` plus CHOWN/SETGID/SETUID/DAC_OVERRIDE
and carries no FOWNER, and searxng's own entrypoint chowns /etc/searxng to
searxng:searxng, so every retained settings file belongs to that user by the
second boot. Chowning the temporary file first left root unable to chmod it,
so the migration exited 1 and `set -eu` killed the container before
`exec /usr/local/searxng/entrypoint.sh` — SearXNG never started and odysseus
blocked on its healthcheck.

Swap the two calls so the chmod lands while the temporary file is still
root-owned, and cover the ordering with a test that refuses the chmod once
the chown has happened, the way the kernel does.
2026-08-16 03:53:42 +02:00
RaresKeY 3cd6cdb638 fix(docker): migrate retained SearXNG settings
Retained nonempty SearXNG settings can miss defaults required by newer pinned images while bypassing the entrypoint's narrow regeneration checks.

Add an atomic PyYAML-aware migration to all Compose variants. Preserve existing inheritance choices, custom content, secrets, ownership, and mode while inserting only the missing top-level default-inheritance key.

Validated with 39 focused and adjacent tests, compile checks, and fresh and retained pinned-image HTTP 200 gates. Full repository CI remains for the PR.
2026-08-15 10:52:51 +00:00
329 changed files with 3637 additions and 32684 deletions
-2
View File
@@ -30,8 +30,6 @@ secrets.env~
.idea/
dev-docs/
docs/
website/
assets/branding/
*.md
*.db
*.sqlite
+2 -29
View File
@@ -76,24 +76,12 @@ SEARXNG_INSTANCE=http://localhost:8080
# Change this if another local service already uses 7000 (macOS AirPlay often does).
# APP_PORT=7000
# Optional HTTP address advertised in companion/mobile pairing codes. Set this
# when Docker would otherwise advertise a container address or loopback. Use a
# LAN or Tailscale IPv4 address, a single-label hostname, or an mDNS *.local
# name that the phone can reach. HTTPS and public hostnames are not supported
# by the current companion client. Do not include credentials, a path, query,
# or fragment.
# COMPANION_BASE_URL=http://192.168.1.50:7000
# Development-only auth bypass for loopback requests.
# Keep false for Docker, LAN, reverse proxy, and any shared deployment.
# LOCALHOST_BYPASS=false
# Mark session cookies Secure. Left unset, this follows the request scheme:
# an HTTPS login gets a Secure cookie, a plain-HTTP one does not. Set true to
# force it on, or false to force it off while you still serve plain HTTP.
# Upgrading: this used to default to false. Drop a leftover SECURE_COOKIES=false
# from your .env unless you still need that escape hatch — it keeps HTTPS logins
# on a non-Secure cookie.
# Mark session cookies Secure. Set true when Odysseus is served through HTTPS
# by a trusted reverse proxy or private access gateway.
# SECURE_COOKIES=true
# Optional: pre-seed the first admin password during setup.
@@ -163,21 +151,6 @@ SEARXNG_INSTANCE=http://localhost:8080
# Local HTTP setups may use the callback URL inferred by the application.
# GOOGLE_OAUTH_REDIRECT_URI=https://your-domain.com/api/email/oauth/google/callback
# Origin the MCP OAuth callback is sent back to, for remote (Streamable HTTP)
# MCP servers that register it dynamically. Defaults to http://localhost:$APP_PORT,
# which is right only when you reach Odysseus directly on that port. Set it for
# HTTPS, reverse-proxy, hosted, and Docker installs — inside the container the
# app always listens on 7000 and cannot see the host port map, so the default is
# wrong there whenever APP_PORT is not 7000.
#
# Not for Google MCP servers. Those use Desktop App credentials, and Google only
# accepts loopback redirect URIs for that client type, so a public origin here is
# rejected with redirect_uri_mismatch. Leave it unset for a Google-only install:
# the loopback default is what Google wants, and remote users finish through the
# paste-back page, which never has to load the redirect.
# https://developers.google.com/identity/protocols/oauth2/native-app
# OAUTH_REDIRECT_BASE_URL=https://your-domain.com
# ============================================================
# Misc
# ============================================================
-7
View File
@@ -15,13 +15,6 @@ docker/entrypoint.sh text eol=lf
*.cmd text eol=crlf
*.bat text eol=crlf
# Vendored third-party bundles in static/lib/ are published minified artifacts
# and must stay byte-identical to what npm ships — stripping trailing whitespace
# to satisfy `git diff --check` would desync them from the upstream release. Turn
# the whitespace check off for that tree instead, and keep the bundles out of
# GitHub's language statistics.
static/lib/** -whitespace linguist-vendored
# Binary assets — never normalize.
*.png binary
*.jpg binary
+1 -1
View File
@@ -6,4 +6,4 @@
# A per-area ownership map (security/auth, CI, frontend, agent internals, with
# multiple named owners per line) is being worked out in issue #593; once
# agreed it replaces this file. Until then, required reviews and the security
# CI gate (website/security-ci.md) remain in force via branch protection.
# CI gate (docs/security-ci.md) remain in force via branch protection.
-12
View File
@@ -26,18 +26,6 @@ body:
- label: I am running the latest code from the `dev` branch (the default branch you get on clone, where fixes land first) and the bug still reproduces there. Please `git pull` the latest `dev` before filing.
required: true
- type: input
id: revision
attributes:
label: Odysseus Revision
description: |
From the repository root (on the host when using Docker), run
`git show -s --abbrev=12 --format='%h (%cs)' HEAD`
and paste the output exactly.
placeholder: "1fef4929cf1d (2026-08-11)"
validations:
required: true
- type: dropdown
id: install-method
attributes:
-1
View File
@@ -28,7 +28,6 @@ Fixes #
- [ ] This PR targets `dev`
- [ ] My changes are limited to the scope described above — no unrelated refactors or whitespace changes mixed in.
- [ ] I actually ran the app (`docker compose up` or `uvicorn app:app`) and verified the change works end-to-end. Type-checks and unit tests are not enough.
- [ ] I did not run the app/runtime validation and stated that gap in **How to Test**. Leave this unchecked when the app-run box above is checked.
## How to Test
@@ -41,14 +41,6 @@ module.exports = async ({ github, context, core }) => {
break;
case 'bug': {
const revisionText = section('Odysseus Revision');
if (!/^[0-9a-f]{12} \(\d{4}-\d{2}-\d{2}\)$/i.test(revisionText)) {
failures.push(
'**Odysseus Revision** — paste the 12-character commit SHA and date, ' +
'for example `1fef4929cf1d (2026-08-11)`',
);
}
if (!section('Install Method')) {
failures.push('**Install Method** — select how you installed Odysseus');
}
+32 -142
View File
@@ -21,11 +21,11 @@ module.exports = async ({ github, context, core }) => {
return strip(m?.[0].replace(new RegExp(`#+\\s+${heading}`, 'i'), '') ?? '');
}
const descriptionProblems = [];
const problems = [];
// 1. Summary must be filled in.
if (section('Summary').length < 20) {
descriptionProblems.push('**Summary** is empty or too short — describe what changed and why.');
problems.push('**Summary** is empty or too short — describe what changed and why.');
}
// 2. Linked Issue must reference a real issue. Accept a bare #NNN, a closing
@@ -34,18 +34,18 @@ module.exports = async ({ github, context, core }) => {
const linkedSection = section('Linked Issue');
const hasIssueRef = /#\d+\b/.test(linkedSection) || /\/issues\/\d+/.test(linkedSection);
if (!linkedSection || !hasIssueRef) {
descriptionProblems.push('**Linked Issue** — add a reference like `Fixes #NNN`, a bare `#NNN`, or a link to the issue.');
problems.push('**Linked Issue** — add a reference like `Fixes #NNN`, a bare `#NNN`, or a link to the issue.');
}
// 3. At least one Type of Change box must be checked.
const typeBlock = body.match(/##\s+Type of Change[\s\S]*?(?=\n##\s|$)/i)?.[0] ?? '';
if (!/- \[x\]/i.test(typeBlock)) {
descriptionProblems.push('**Type of Change** — check at least one box.');
problems.push('**Type of Change** — check at least one box.');
}
// 4. Duplicate-search checklist item must be checked.
if (!/- \[x\] I searched/i.test(body)) {
descriptionProblems.push('**Checklist** — check the duplicate-search box to confirm you searched existing issues and PRs.');
problems.push('**Checklist** — check the duplicate-search box to confirm you searched existing issues and PRs.');
}
// 5. How to Test must contain enough real detail for a reviewer to act on.
@@ -53,83 +53,7 @@ module.exports = async ({ github, context, core }) => {
// code block — so we only require non-trivial content, not a specific shape.
const howTo = section('How to Test');
if (howTo.length < 30) {
descriptionProblems.push('**How to Test** — explain how a reviewer can verify this change. Numbered steps, the commands you ran, or a short code block all work — give a sentence or two of real detail (not just "tested locally").');
}
// Classify paths from GitHub's API. This workflow runs in the privileged base
// context, so it must never check out or execute code from the PR branch.
const changedFiles = await github.paginate(github.rest.pulls.listFiles, {
owner, repo, pull_number: prNum, per_page: 100,
});
const changedPaths = changedFiles.map(file => file.filename);
function isUiSensitivePath(filename) {
const path = filename.toLowerCase();
return path.startsWith('static/')
|| path.startsWith('templates/')
|| /\.(?:html?|css|svg)$/.test(path);
}
function isDocsOnlyPath(filename) {
const path = filename.toLowerCase();
return /\.(?:md|mdx|rst|adoc|txt)$/.test(path)
|| (path.startsWith('docs/') && !isUiSensitivePath(path));
}
function isRuntimeSensitivePath(filename) {
const path = filename.toLowerCase();
if (isUiSensitivePath(path)) return false;
if (path.startsWith('tests/') || path.startsWith('.github/')) return false;
return /^(?:app\.py|routes\/|services\/|src\/|core\/|mcp_servers\/|scripts\/|docker\/)/.test(path)
|| /^(?:dockerfile|docker-compose.*\.ya?ml|requirements(?:-optional)?\.txt|pyproject\.toml|setup\.py)$/.test(path)
|| /\.(?:py|sh|ps1|bat)$/.test(path);
}
let classification = 'tooling';
if (changedPaths.some(isUiSensitivePath)) {
classification = 'UI-sensitive';
} else if (changedPaths.some(isRuntimeSensitivePath)) {
classification = 'backend/runtime';
} else if (changedPaths.length > 0 && changedPaths.every(isDocsOnlyPath)) {
classification = 'docs-only';
}
const appRan = /- \[x\]\s+I actually ran the app\b/i.test(body);
const appNotRun = /- \[x\]\s+I did not run the app\/runtime validation\b/i.test(body);
// Anchor on the wording, not the template's emphasis: a ticked box the author
// retyped without the surrounding ** renders identically on the PR page, so
// treating it as unchecked is invisible from their side. Matches the two
// attestations above, which already ignore formatting.
const screenshotChecked = /- \[x\]\s+[*_]{0,2}Screenshot or short clip[*_]{0,2}/i.test(body);
const screenshotSection = section('Screenshots / clips');
const hasVisualEvidence = /!\[[^\]]*\]\([^)]+\)|<(?:img|video|source)\b[^>]*(?:src|href)=|https?:\/\/[^\s)]+/i.test(screenshotSection);
const evidenceGaps = [];
let needsRuntimeValidation = false;
let needsVisualEvidence = false;
if (classification === 'backend/runtime' || classification === 'UI-sensitive') {
if (appRan && appNotRun) {
needsRuntimeValidation = true;
evidenceGaps.push('The app-run and explicit not-run boxes are both checked. Select the one state that is true.');
} else if (!appRan) {
needsRuntimeValidation = true;
if (appNotRun) {
evidenceGaps.push('The author explicitly reports that app/runtime validation was not performed.');
} else {
evidenceGaps.push('App/runtime validation is not author-attested. Check the run box only after running it, or check the explicit not-run box and describe the gap.');
}
}
}
if (classification === 'UI-sensitive') {
if (!screenshotChecked) {
needsVisualEvidence = true;
evidenceGaps.push('The screenshot/clip checkbox is not checked for this UI-sensitive change.');
}
if (!hasVisualEvidence) {
needsVisualEvidence = true;
evidenceGaps.push('The Screenshots / clips section does not contain an actual attachment or link.');
}
problems.push('**How to Test** — explain how a reviewer can verify this change. Numbered steps, the commands you ran, or a short code block all work — give a sentence or two of real detail (not just "tested locally").');
}
// ── Comment ──────────────────────────────────────────────────────────────
@@ -138,43 +62,22 @@ module.exports = async ({ github, context, core }) => {
});
const existing = comments.find(c => (c.body ?? '').includes(MARKER));
if (descriptionProblems.length === 0 && evidenceGaps.length === 0) {
if (problems.length === 0) {
if (existing) {
await github.rest.issues.deleteComment({ owner, repo, comment_id: existing.id });
}
} else {
const commentLines = [MARKER];
if (descriptionProblems.length > 0) {
commentLines.push(
'⚠️ **PR description — action needed**',
'',
'The following required sections are missing or incomplete. Please update the PR description to address them:',
'',
descriptionProblems.map(problem => `- ${problem}`).join('\n'),
);
} else {
commentLines.push(
'⚠️ **PR description is complete; validation evidence is still outstanding**',
'',
`Changed-file classification: **${classification}**.`,
);
}
if (evidenceGaps.length > 0) {
commentLines.push(
'',
'**Author-reported runtime / visual state**',
'',
evidenceGaps.map(gap => `- ${gap}`).join('\n'),
'',
'Checkboxes are author attestations. GitHub Actions results remain the execution evidence for CI; this check does not prove that a local command ran.',
);
}
commentLines.push(
const commentBody = [
MARKER,
'⚠️ **PR description — action needed**',
'',
'The following required sections are missing or incomplete. Please update the PR description to address them:',
'',
problems.map(p => `- ${p}`).join('\n'),
'',
'---',
'_This comment updates automatically when the description or changed files change._',
);
const commentBody = commentLines.join('\n');
'_This comment is deleted automatically once all sections are complete._',
].join('\n');
if (existing) {
await github.rest.issues.updateComment({ owner, repo, comment_id: existing.id, body: commentBody });
@@ -194,47 +97,34 @@ module.exports = async ({ github, context, core }) => {
return true;
} catch (e) {
if (e.status === 404) return false;
if (e.status === 403) {
core.warning(`Could not inspect label "${name}" — token lacks label read access; skipping.`);
return false;
}
throw e;
}
}
async function setLabel(name, wanted) {
if (wanted && await labelExists(name)) {
async function swapLabel(num, add, remove) {
if (await labelExists(add)) {
try {
await github.rest.issues.addLabels({ owner, repo, issue_number: prNum, labels: [name] });
await github.rest.issues.addLabels({ owner, repo, issue_number: num, labels: [add] });
} catch (e) {
// Fail soft on a token that can't write labels so a label permission
// problem never masks the actual description verdict.
if (e.status !== 403 && e.status !== 404) throw e;
core.warning(`Could not add "${name}" — label is unavailable or the token lacks label write access; skipping.`);
if (e.status !== 403) throw e;
core.warning(`Could not add "${add}" — token lacks label write here; skipping.`);
}
} else if (wanted) {
core.warning(`Label "${name}" does not exist in the repo — skipping. Create it once to enable labelling.`);
} else {
try {
await github.rest.issues.removeLabel({ owner, repo, issue_number: prNum, name });
} catch (e) {
if (e.status !== 404 && e.status !== 410 && e.status !== 403) throw e;
}
core.warning(`Label "${add}" does not exist in the repo — skipping. Create it once to enable labelling.`);
}
try {
await github.rest.issues.removeLabel({ owner, repo, issue_number: num, name: remove });
} catch (e) {
if (e.status !== 404 && e.status !== 410 && e.status !== 403) throw e;
}
}
const descriptionComplete = descriptionProblems.length === 0;
const evidenceComplete = evidenceGaps.length === 0;
const isDraft = Boolean(context.payload.pull_request.draft);
await setLabel(
'ready for review',
descriptionComplete && evidenceComplete && !isDraft,
);
await setLabel('needs work', !descriptionComplete);
await setLabel('needs runtime validation', needsRuntimeValidation);
await setLabel('needs visual evidence', needsVisualEvidence);
if (!descriptionComplete) {
core.setFailed(`PR description has ${descriptionProblems.length} issue(s) — see bot comment for details.`);
if (problems.length === 0) {
await swapLabel(prNum, 'ready for review', 'needs work');
} else {
await swapLabel(prNum, 'needs work', 'ready for review');
core.setFailed(`PR description has ${problems.length} issue(s) — see bot comment for details.`);
}
};
+10 -11
View File
@@ -21,7 +21,7 @@ jobs:
runs-on: ubuntu-latest
continue-on-error: true
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
fetch-depth: 0
persist-credentials: false
@@ -73,10 +73,10 @@ jobs:
name: Python syntax (compileall)
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
- uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
with:
python-version: "3.11"
# Byte-compile sources — catches syntax errors without installing deps.
@@ -86,10 +86,10 @@ jobs:
name: JS syntax (node --check)
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
- uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
with:
node-version: "20"
# Syntax-check our own JS (skip vendored libs in static/lib).
@@ -105,12 +105,12 @@ jobs:
runs-on: ubuntu-latest
# Make Python test validation authoritative for the configured scope.
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
fetch-depth: 0
persist-credentials: false
# Detect whether this PR only touches repository prose outside the Pages site.
# Detect whether this PR only touches documentation files.
# If so, skip the expensive pytest run while still reporting a passing check.
- name: Check for docs-only changes
id: docs-check
@@ -122,10 +122,9 @@ jobs:
BASE="${{ github.event.before }}"
HEAD="${{ github.sha }}"
fi
# Keep website/ and assets/branding/ out of this bypass: pytest owns
# regression guards for their published-file and orphan-asset contracts.
# List all changed files; if every file matches docs/markdown patterns, skip pytest.
changed=$(git diff --name-only "$BASE" "$HEAD" 2>/dev/null || git diff --name-only HEAD~1 HEAD)
non_docs=$(echo "$changed" | grep -Ev '^(docs/|[^/]+\.md$|\.github/[^/]+\.md$)' || true)
non_docs=$(echo "$changed" | grep -Ev '^(docs/|.*\.md$|\.github/[^/]+\.md$)' || true)
if [ -z "$non_docs" ]; then
echo "docs_only=true" >> "$GITHUB_OUTPUT"
echo "Docs-only change detected — skipping pytest."
@@ -133,7 +132,7 @@ jobs:
echo "docs_only=false" >> "$GITHUB_OUTPUT"
fi
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
- uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
if: steps.docs-check.outputs.docs_only != 'true'
with:
python-version: "3.11"
+3 -3
View File
@@ -27,15 +27,15 @@ jobs:
language: [actions, javascript-typescript, python]
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
- name: Initialize CodeQL
uses: github/codeql-action/init@ff2f1c621b7f889edc0d3c761ac2e6a3f8cdb0dd # v4.37.7
uses: github/codeql-action/init@8aad20d150bbac5944a9f9d289da16a4b0d87c1e # v4.36.2
with:
languages: ${{ matrix.language }}
build-mode: none
- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@ff2f1c621b7f889edc0d3c761ac2e6a3f8cdb0dd # v4.37.7
uses: github/codeql-action/analyze@8aad20d150bbac5944a9f9d289da16a4b0d87c1e # v4.36.2
with:
category: "/language:${{ matrix.language }}"
+2 -2
View File
@@ -37,12 +37,12 @@ jobs:
contents: read
steps:
- name: Checkout repository
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
- name: Lint Dockerfile
uses: hadolint/hadolint-action@2a66e89f53d0771bb131a7fa31f3136336094aa6 # v3.4.0
uses: hadolint/hadolint-action@2332a7b74a6de0dda2e2221d575162eba76ba5e5 # v3.3.0
with:
dockerfile: Dockerfile
# DL3008: pinning apt package versions is impractical on a -slim base
+7 -11
View File
@@ -23,16 +23,12 @@ on:
paths-ignore:
- '**.md'
- 'docs/**'
- 'website/**'
- 'assets/branding/**'
- '.github/ISSUE_TEMPLATE/**'
push:
branches: [main]
paths-ignore:
- '**.md'
- 'docs/**'
- 'website/**'
- 'assets/branding/**'
- '.github/ISSUE_TEMPLATE/**'
workflow_dispatch:
@@ -56,17 +52,17 @@ jobs:
contents: read
steps:
- name: Checkout repository
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
- name: Set up Buildx
uses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e # v4.3.0
uses: docker/setup-buildx-action@d7f5e7f509e45cec5c76c4d5afdd7de93d0b3df5 # v4.1.0
# Build without pushing so a broken Dockerfile is caught here, and the
# exact image we ship is what gets scanned.
- name: Build image
uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a # v7.3.0
uses: docker/build-push-action@f9f3042f7e2789586610d6e8b85c8f03e5195baf # v7.2.0
with:
context: .
push: false
@@ -97,15 +93,15 @@ jobs:
security-events: write # upload SARIF to the Security tab
steps:
- name: Checkout repository
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
- name: Set up Buildx
uses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e # v4.3.0
uses: docker/setup-buildx-action@d7f5e7f509e45cec5c76c4d5afdd7de93d0b3df5 # v4.1.0
- name: Build image
uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a # v7.3.0
uses: docker/build-push-action@f9f3042f7e2789586610d6e8b85c8f03e5195baf # v7.2.0
with:
context: .
push: false
@@ -123,7 +119,7 @@ jobs:
TRIVY_DB_REPOSITORY: ghcr.io/aquasecurity/trivy-db:2
- name: Upload Trivy results
uses: github/codeql-action/upload-sarif@ff2f1c621b7f889edc0d3c761ac2e6a3f8cdb0dd # v4.37.7
uses: github/codeql-action/upload-sarif@8aad20d150bbac5944a9f9d289da16a4b0d87c1e # v4.36.2
with:
sarif_file: trivy-results.sarif
category: trivy-image
+3 -3
View File
@@ -36,7 +36,7 @@ jobs:
contents: read
steps:
- name: Checkout repository
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
@@ -55,12 +55,12 @@ jobs:
contents: read
steps:
- name: Checkout repository
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
with:
python-version: '3.12'
-50
View File
@@ -1,50 +0,0 @@
name: Deploy GitHub Pages
on:
push:
branches: [main]
paths:
- 'website/**'
- '.github/workflows/deploy-pages.yml'
workflow_dispatch:
permissions: {}
concurrency:
group: pages
cancel-in-progress: false
jobs:
build:
name: Package static site
runs-on: ubuntu-latest
permissions:
contents: read
pages: read
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: actions/configure-pages@45bfe0192ca1faeb007ade9deae92b16b8254a0d # v6.0.0
- uses: actions/jekyll-build-pages@44a6e6beabd48582f863aeeb6cb2151cc1716697 # v1.0.13
with:
source: website
destination: _site
- uses: actions/upload-pages-artifact@fc324d3547104276b827a68afc52ff2a11cc49c9 # v5.0.0
with:
path: _site
deploy:
name: Deploy static site
needs: build
runs-on: ubuntu-latest
permissions:
pages: write
id-token: write
environment:
name: github-pages
url: ${{ steps.deployment.outputs.page_url }}
steps:
- name: Deploy to GitHub Pages
id: deployment
uses: actions/deploy-pages@cd2ce8fcbc39b97be8ca5fce6e763baed58fa128 # v5.0.0
+8 -10
View File
@@ -14,8 +14,6 @@ on:
paths-ignore:
- '**.md'
- 'docs/**'
- 'website/**'
- 'assets/branding/**'
- '.github/ISSUE_TEMPLATE/**'
concurrency:
@@ -47,20 +45,20 @@ jobs:
arch: arm64
runner: ubuntu-24.04-arm
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
- name: Set up Buildx
uses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e # v4.3.0
uses: docker/setup-buildx-action@d7f5e7f509e45cec5c76c4d5afdd7de93d0b3df5 # v4.1.0
- name: Log in to GHCR
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
uses: docker/login-action@650006c6eb7dba73a995cc03b0b2d7f5ca915bee # v4.2.0
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Build and push by digest
id: build
uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a # v7.3.0
uses: docker/build-push-action@f9f3042f7e2789586610d6e8b85c8f03e5195baf # v7.2.0
with:
context: .
platforms: ${{ matrix.platform }}
@@ -88,7 +86,7 @@ jobs:
contents: read
packages: write
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
- name: Read APP_VERSION + short sha
@@ -105,16 +103,16 @@ jobs:
pattern: digest-*
merge-multiple: true
- name: Set up Buildx
uses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e # v4.3.0
uses: docker/setup-buildx-action@d7f5e7f509e45cec5c76c4d5afdd7de93d0b3df5 # v4.1.0
- name: Log in to GHCR
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
uses: docker/login-action@650006c6eb7dba73a995cc03b0b2d7f5ca915bee # v4.2.0
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Compute tags
id: meta
uses: docker/metadata-action@dc802804100637a589fabce1cb79ff13a1411302 # v6.2.0
uses: docker/metadata-action@80c7e94dd9b9319bd5eb7a0e0fe9291e23a2a2e9 # v6.1.0
with:
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
tags: |
@@ -14,7 +14,7 @@ jobs:
# Skip bots (Dependabot, release-drafter, etc.)
if: ${{ github.event.issue.user.type != 'Bot' }}
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
sparse-checkout: .github/scripts
persist-credentials: false
+4 -10
View File
@@ -5,11 +5,7 @@ on:
# works on fork PRs. Safe here: the checkout pins to the base branch (no fork
# code runs) and the scripts only read context.payload and call the GitHub API.
pull_request_target: # zizmor: ignore[dangerous-triggers]
types: [opened, edited, synchronize, reopened, ready_for_review, converted_to_draft]
concurrency:
group: pr-description-${{ github.event.pull_request.number }}
cancel-in-progress: true
types: [opened, edited, synchronize, reopened, ready_for_review]
# Default-deny at the workflow level; each job opts into only the scopes it needs.
# Note: modifying a PR's labels/comments needs pull-requests:write even though the
@@ -27,7 +23,7 @@ jobs:
# Skip bots: they open PRs programmatically and have their own process.
if: github.event.pull_request.user.type != 'Bot'
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
ref: ${{ github.base_ref }}
sparse-checkout: .github/scripts
@@ -63,14 +59,12 @@ jobs:
check-mergeable:
name: Flag unmergeable PRs
needs: check-description
runs-on: ubuntu-latest
permissions:
pull-requests: write
issues: write
# Run after description validation failures, but never from an obsolete
# workflow run canceled by a newer PR event.
if: ${{ !cancelled() && github.event.pull_request.user.type != 'Bot' }}
# Skip bots: they open PRs programmatically and have their own process.
if: github.event.pull_request.user.type != 'Bot'
steps:
- uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
with:
+1 -1
View File
@@ -35,7 +35,7 @@ jobs:
contents: read
steps:
- name: Checkout repository
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
# Full history so a secret committed in an earlier commit (and later
# deleted) is still caught -- deletion does not remove it from Git.
+3 -3
View File
@@ -36,7 +36,7 @@ jobs:
contents: read
steps:
- name: Checkout repository
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
@@ -61,12 +61,12 @@ jobs:
contents: read
steps:
- name: Checkout repository
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
with:
python-version: '3.12'
-18
View File
@@ -85,24 +85,6 @@ output.txt.txt
!docs/**/*.gif
!docs/**/*.webp
# …except shipped website and branding media.
!website/**/*.jpg
!website/**/*.jpeg
!website/**/*.png
!website/**/*.gif
!website/**/*.bmp
!website/**/*.webp
!website/**/*.tiff
!website/**/*.pdf
!assets/branding/**/*.jpg
!assets/branding/**/*.jpeg
!assets/branding/**/*.png
!assets/branding/**/*.gif
!assets/branding/**/*.bmp
!assets/branding/**/*.webp
!assets/branding/**/*.tiff
!assets/branding/**/*.pdf
# Reports and temp files
reports/
tasks/
+2 -10
View File
@@ -65,16 +65,6 @@ Vendored in `static/lib/` and served directly:
| [jsPDF](https://github.com/parallax/jsPDF) (bundled in html2pdf) | PDF generation | MIT |
| [html2canvas](https://github.com/niklasvh/html2canvas) (bundled in html2pdf) | DOM → canvas rasterization | MIT |
| [node-qrcode](https://github.com/soldair/node-qrcode) (`qrcode.min.js`) | QR-code rendering (2FA setup) | MIT |
| [KaTeX](https://github.com/KaTeX/KaTeX) v0.16.22 (`katex/katex.min.{js,css}` + `katex/fonts/*.woff2`) | Math typesetting | MIT ([`licenses/KaTeX-MIT-LICENSE.txt`](licenses/KaTeX-MIT-LICENSE.txt)) |
| [Mermaid](https://github.com/mermaid-js/mermaid) v11.16.1 (`mermaid.min.js`) | Diagrams from text | MIT ([`licenses/Mermaid-MIT-LICENSE.txt`](licenses/Mermaid-MIT-LICENSE.txt)) |
KaTeX and Mermaid are loaded on first use by `static/js/markdown.js` rather than
from `index.html`, so a session that renders no math and no diagram never fetches
either. Only the `.woff2` KaTeX fonts are shipped, matching `static/fonts/`; the
`.woff` and `.ttf` variants its stylesheet also lists are never requested by a
browser that supports `woff2`. The bundles are the published npm artifacts,
unmodified — `.gitattributes` turns the whitespace check off for `static/lib/`
so they can stay byte-identical to upstream.
## Front-end libraries loaded at runtime (CDN)
@@ -82,6 +72,8 @@ Referenced from `cdn.jsdelivr.net` / `cdnjs.cloudflare.com` at runtime — not v
| Library | Purpose | License |
|---|---|---|
| [KaTeX](https://github.com/KaTeX/KaTeX) 0.16.22 | Math typesetting | MIT |
| [Mermaid](https://github.com/mermaid-js/mermaid) 11 | Diagrams from text | MIT |
| [Pyodide](https://github.com/pyodide/pyodide) 0.27.5 | In-browser Python runtime | MPL-2.0 |
| [PDFObject](https://github.com/pipwerks/PDFObject) 2.1.1 | Inline PDF embedding | MIT |
+10 -15
View File
@@ -1,5 +1,5 @@
<p align="center">
<img src="assets/branding/odysseus-wordmark.png" alt="Odysseus" width="238">
<img src="docs/odysseus-wordmark.png" alt="Odysseus" width="238">
</p>
<p align="center">
@@ -8,7 +8,7 @@
<p align="center">
<a href="#quick-start">Quick Start</a> ·
<a href="website/setup.md">Setup Guide</a> ·
<a href="docs/setup.md">Setup Guide</a> ·
<a href="CONTRIBUTING.md">Contributing</a> ·
<a href="ROADMAP.md">Roadmap</a>
</p>
@@ -18,7 +18,7 @@
</p>
<p align="center">
<img src="assets/branding/odysseus-browser.jpg" alt="Odysseus interface">
<img src="docs/odysseus-browser.jpg" alt="Odysseus interface">
</p>
---
@@ -36,7 +36,7 @@ docker compose up -d --build
Open `http://localhost:7000` when the containers are healthy. The first admin password is printed in `docker compose logs odysseus`.
Native installs, GPU notes, Windows/macOS instructions, HTTPS, and configuration live in the [setup guide](website/setup.md).
Native installs, GPU notes, Windows/macOS instructions, HTTPS, and configuration live in the [setup guide](docs/setup.md).
## Features
@@ -51,7 +51,7 @@ Native installs, GPU notes, Windows/macOS instructions, HTTPS, and configuration
## Demo
A full hover-to-play tour lives on the [Odysseus landing page](https://odysseus-dev.github.io/odysseus/). Its source lives under [`website/`](website/).
A full hover-to-play tour lives on the landing page: [`docs/index.html`](docs/index.html).
## Contributing
@@ -59,20 +59,15 @@ Help is welcome. The best entry points are fresh-install testing, provider setup
## Security
Odysseus is a self-hosted workspace with powerful local tools. Keep auth enabled, keep private data out of Git, and do not expose raw model/service ports publicly.
- Keep `AUTH_ENABLED=true` for any network-accessible deployment.
- Keep `LOCALHOST_BYPASS=false` outside local development.
Deployment details are in the [setup guide](website/setup.md#security-notes).
Odysseus is a self-hosted workspace with powerful local tools. Keep auth enabled, keep private data out of Git, and do not expose raw model/service ports publicly. Deployment details are in the [setup guide](docs/setup.md#security-notes).
## Star History
<a href="https://star-history.dera.page/#odysseus-dev/odysseus&type=date&legend=top-left">
<a href="https://www.star-history.com/?repos=odysseus-dev%2Fodysseus&type=date&legend=top-left">
<picture>
<source media="(prefers-color-scheme: dark)" srcset="https://star-history.dera.page/svg?repos=odysseus-dev/odysseus&type=date&theme=dark&legend=top-left" />
<source media="(prefers-color-scheme: light)" srcset="https://star-history.dera.page/svg?repos=odysseus-dev/odysseus&type=date&legend=top-left" />
<img alt="Star History Chart" src="https://star-history.dera.page/svg?repos=odysseus-dev/odysseus&type=date&legend=top-left" />
<source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/chart?repos=odysseus-dev/odysseus&type=date&theme=dark&legend=top-left" />
<source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/chart?repos=odysseus-dev/odysseus&type=date&legend=top-left" />
<img alt="Star History Chart" src="https://api.star-history.com/chart?repos=odysseus-dev/odysseus&type=date&legend=top-left" />
</picture>
</a>
+1 -1
View File
@@ -10,7 +10,7 @@ Security fixes are handled on the default branch until formal releases are cut.
- Keep `AUTH_ENABLED=true` for any network-accessible deployment.
- Keep `LOCALHOST_BYPASS=false` outside local development.
- Leave `SECURE_COOKIES` unset unless you need to override it: session cookies are marked `Secure` whenever the request arrives over HTTPS. Set `SECURE_COOKIES=true` to force it on (for a proxy Odysseus cannot see the scheme of), or `SECURE_COOKIES=false` to force it off while you still serve plain HTTP alongside HTTPS.
- Set `SECURE_COOKIES=true` when Odysseus is served through HTTPS by a trusted reverse proxy or private access gateway.
- Use HTTPS when exposing the app beyond localhost.
- Put the authenticated Odysseus web/API entrypoint behind a trusted reverse proxy or private access layer such as Cloudflare Access, Tailscale, or a VPN.
- Keep ChromaDB, SearXNG, ntfy, Ollama, vLLM, llama.cpp, databases, and raw model/provider APIs internal-only.
+1 -1
View File
@@ -37,7 +37,7 @@ Non-admin defaults are in `core/auth.py:DEFAULT_PRIVILEGES`. Tool enforcement is
- **Sessions:** bcrypt passwords, 7-day session tokens stored atomically in `data/sessions.json` via `core/atomic_io.py`.
- **2FA:** TOTP with 8 single-use backup codes. Verified after password check, before session issuance.
- **Reserved usernames:** request sentinels and the Default/Local storage owner cannot be registered or renamed into. Defined in `core/auth.py:RESERVED_USERNAMES`.
- **Reserved usernames:** `internal-tool`, `api`, `demo`, `system` cannot be registered or renamed into. Defined in `core/auth.py:RESERVED_USERNAMES`.
- `internal-tool` is security-critical: `core/middleware.py:require_admin` treats any request where `request.state.current_user == "internal-tool"` as the in-process tool loopback and grants admin unconditionally. A real account with that name would silently pass every `require_admin` check.
- **Orphan sessions:** `validate_token` re-checks that the user record still exists on every call. A deleted user's cookie is dropped on next request rather than continuing to authenticate.
+11 -46
View File
@@ -67,13 +67,7 @@ from core.constants import (
REQUEST_TIMEOUT, OPENAI_API_KEY, AUTH_FILE,
)
from core.database import SessionLocal, ApiToken
from core.middleware import (
SecurityHeadersMiddleware,
get_application_route_path,
is_cors_preflight,
path_is_route_or_child,
with_asgi_root_path,
)
from core.middleware import SecurityHeadersMiddleware, is_cors_preflight
from core.auth import AuthManager, normalize_known_username
from core.exceptions import (
SessionNotFoundError, InvalidFileUploadError,
@@ -84,7 +78,6 @@ import bcrypt as _bcrypt
from src.app_helpers import abs_join, serve_html_with_nonce
from src.generated_images import GENERATED_IMAGE_HEADERS, resolve_generated_image_path
from src.owner_identity import auth_disabled
from starlette.responses import RedirectResponse
# ========= LOGGING =========
@@ -255,7 +248,7 @@ from routes.auth_routes import setup_auth_routes, SESSION_COOKIE
auth_manager = AuthManager()
app.state.auth_manager = auth_manager
AUTH_ENABLED = not auth_disabled()
AUTH_ENABLED = os.getenv("AUTH_ENABLED", "true").lower() != "false"
LOCALHOST_BYPASS = os.getenv("LOCALHOST_BYPASS", "false").lower() == "true"
if LOCALHOST_BYPASS:
logger.warning("LOCALHOST_BYPASS is enabled, loopback requests bypass authentication. Do not expose this instance to a network.")
@@ -291,7 +284,7 @@ if AUTH_ENABLED:
def _is_auth_exempt(path: str) -> bool:
if path in AUTH_EXEMPT_EXACT:
return True
if any(path_is_route_or_child(path, p) for p in AUTH_EXEMPT_PREFIXES):
if any(path.startswith(p) for p in AUTH_EXEMPT_PREFIXES):
return True
return any(p.match(path) for p in AUTH_EXEMPT_PATTERNS)
@@ -362,7 +355,7 @@ if AUTH_ENABLED:
class AuthMiddleware(BaseHTTPMiddleware):
async def dispatch(self, request: Request, call_next):
path = get_application_route_path(request.scope)
path = request.url.path
# A genuine CORS preflight (OPTIONS + Access-Control-Request-Method)
# carries no credentials by design and must reach CORSMiddleware to be
# answered. AuthMiddleware is the outermost middleware, so gating the
@@ -406,10 +399,7 @@ if AUTH_ENABLED:
if not auth_manager.is_configured:
# No users yet — redirect to login for first-time setup
if not path.startswith("/api/"):
return RedirectResponse(
url=with_asgi_root_path(request.scope, "/login"),
status_code=302,
)
return RedirectResponse(url="/login", status_code=302)
return JSONResponse(status_code=401, content={"error": "Setup required"})
# --- Bearer token auth (API tokens for external integrations) ---
@@ -471,10 +461,7 @@ if AUTH_ENABLED:
if not auth_manager.validate_token(token):
if path.startswith("/api/"):
return JSONResponse(status_code=401, content={"error": "Not authenticated"})
return RedirectResponse(
url=with_asgi_root_path(request.scope, "/login"),
status_code=302,
)
return RedirectResponse(url="/login", status_code=302)
# Attach current username to request state for downstream routes
request.state.current_user = auth_manager.get_username_for_token(token)
@@ -516,45 +503,23 @@ async def serve_generated_image(filename: str, request: Request):
# SECURITY: filename is the only key, so anyone who knows / guesses a
# 12-hex content hash could pull another user's image bytes. Require
# auth and verify ownership via the gallery row (when one exists).
_is_bearer = False
try:
from src.auth_helpers import (
effective_user,
get_current_user,
is_bearer_principal,
require_chat_scope,
)
from src.auth_helpers import get_current_user
from core.database import SessionLocal as _SL, GalleryImage as _GI
_is_bearer = is_bearer_principal(request)
if _is_bearer:
# Gallery JSON attributes rows to the token owner. Reuse the same
# owner/scope gate for the binary follow-up so the returned URL is
# actually readable by that bearer principal.
require_chat_scope(request)
_user = effective_user(request)
else:
_user = get_current_user(request)
_user = get_current_user(request)
if _user:
_db = _SL()
try:
_row = _db.query(_GI).filter(_GI.filename == filename).first()
# Generated-but-not-yet-imported images have no row → allow.
# A bearer gallery row must have the exact token owner; cookie
# callers retain the legacy null-owner compatibility below.
if _row is not None and (
(_is_bearer and _row.owner != _user)
or (not _is_bearer and _row.owner and _row.owner != _user)
):
# Row exists with a different owner → 404 (don't confirm existence).
if _row is not None and _row.owner and _row.owner != _user:
raise HTTPException(status_code=404, detail="Image not found")
finally:
_db.close()
except HTTPException:
raise
except Exception as _e:
if _is_bearer:
# An authenticated bearer request must not become a public file
# read because ownership lookup degraded or the DB was unavailable.
raise HTTPException(status_code=404, detail="Image not found") from _e
logger.warning("Image ownership verification failed for %r", filename, exc_info=_e)
ext = filename.rsplit('.', 1)[-1].lower()
mime = {
@@ -806,7 +771,7 @@ from src.task_scheduler import TaskScheduler
task_scheduler = TaskScheduler(session_manager)
from src.event_bus import set_task_scheduler
set_task_scheduler(task_scheduler)
from routes.task.task_routes import setup_task_routes
from routes.task_routes import setup_task_routes
app.include_router(setup_task_routes(task_scheduler))
from routes.assistant_routes import setup_assistant_routes
+4 -8
View File
@@ -27,13 +27,13 @@ echo " port: $PORT"
rm -rf "$APP"
mkdir -p "$APP/Contents/MacOS" "$APP/Contents/Resources"
# ── Icon (best effort) — center-crop the branding image to a square .icns ──
if [ -f "$REPO_DIR/assets/branding/odysseus.jpg" ] && command -v sips >/dev/null 2>&1; then
# ── Icon (best effort) — center-crop docs/odysseus.jpg to a square .icns ──
if [ -f "$REPO_DIR/docs/odysseus.jpg" ] && command -v sips >/dev/null 2>&1; then
TMPIMG="$(mktemp -d)"
# Center-crop to a square, scale to 512 (sips' icns encoder caps at 512), and
# let sips emit the .icns directly — more robust across macOS versions than
# building an .iconset by hand.
sips -c 720 720 "$REPO_DIR/assets/branding/odysseus.jpg" --out "$TMPIMG/sq.png" >/dev/null 2>&1 || cp "$REPO_DIR/assets/branding/odysseus.jpg" "$TMPIMG/sq.png"
sips -c 720 720 "$REPO_DIR/docs/odysseus.jpg" --out "$TMPIMG/sq.png" >/dev/null 2>&1 || cp "$REPO_DIR/docs/odysseus.jpg" "$TMPIMG/sq.png"
sips -z 512 512 "$TMPIMG/sq.png" --out "$TMPIMG/icon.png" >/dev/null 2>&1
if sips -s format icns "$TMPIMG/icon.png" --out "$APP/Contents/Resources/odysseus.icns" >/dev/null 2>&1; then
echo " icon: odysseus.icns"
@@ -42,7 +42,7 @@ if [ -f "$REPO_DIR/assets/branding/odysseus.jpg" ] && command -v sips >/dev/null
fi
rm -rf "$TMPIMG"
else
echo " icon: (skipped — no assets/branding/odysseus.jpg)"
echo " icon: (skipped — no docs/odysseus.jpg)"
fi
# ── Info.plist ──
@@ -73,10 +73,6 @@ cat > "$APP/Contents/MacOS/$APP_NAME.tmpl" <<'LAUNCHER'
INSTALL_DIR="__INSTALL_DIR__"
PORT="__PORT__"
URL="http://127.0.0.1:${PORT}"
# uvicorn is started with --port below, but APP_PORT is what the app itself
# reads when it needs to build a URL for this instance (internal_api_base(),
# companion pairing, the MCP OAuth callback), so export it as well.
export APP_PORT="$PORT"
export PATH="/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:$PATH"
UVICORN="$INSTALL_DIR/venv/bin/uvicorn"
-99
View File
@@ -6,14 +6,11 @@ units so the route layer stays thin and the logic is directly testable.
from __future__ import annotations
import ipaddress
import json
import os
import re
import secrets
import socket
import uuid
from urllib.parse import urlsplit
import bcrypt
@@ -23,102 +20,6 @@ PAIRING_VERSION = 1
COMPANION_SCOPE = "chat"
_COMPANION_IPV4_NETWORKS = tuple(
ipaddress.ip_network(cidr)
for cidr in (
"10.0.0.0/8",
"100.64.0.0/10",
"127.0.0.0/8",
"169.254.0.0/16",
"172.16.0.0/12",
"192.168.0.0/16",
)
)
_DNS_LABEL_RE = re.compile(r"[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?\Z")
def _valid_companion_client_host(host: str) -> bool:
"""Match the host forms supported by the current v1 Expo client."""
if not host or len(host) > 253 or not host.isascii() or "%" in host:
return False
try:
address = ipaddress.ip_address(host)
except ValueError:
labels = host.split(".")
if any(not _DNS_LABEL_RE.fullmatch(label) for label in labels):
return False
if any(label.startswith("xn--") for label in labels):
return False
# WHATWG URL parsers treat a decimal or ``0x`` single-label hostname
# as an IPv4 number even though Python's strict ``ipaddress`` parser
# rejects that spelling. The v1 client interpolates this host back
# into a URL, so accepting e.g. ``134744072`` would make the phone send
# its bearer token to public 8.8.8.8. Keep DNS labels unambiguous.
if len(labels) == 1 and (
labels[0].isdigit()
or re.fullmatch(r"0x[0-9a-f]*", labels[0]) is not None
):
return False
return len(labels) == 1 or (len(labels) >= 2 and labels[-1] == "local")
return isinstance(address, ipaddress.IPv4Address) and any(
address in network for network in _COMPANION_IPV4_NETWORKS
)
def parse_companion_base_url(value: str) -> tuple[str, int]:
"""Validate a v1 companion address and return its legacy (host, port).
The deployed client understands only HTTP plus a LAN-style host and port.
Reject anything outside that exact contract instead of advertising a URL
the client would reject, downgrade, or interpret differently.
"""
if not isinstance(value, str) or not value:
raise ValueError("COMPANION_BASE_URL must be a canonical HTTP LAN origin")
if not value.isascii():
raise ValueError("COMPANION_BASE_URL must contain only ASCII characters")
if any(
ord(char) <= 32 or ord(char) == 127 or char in {"\\", "%"}
for char in value
):
raise ValueError(
"COMPANION_BASE_URL contains a forbidden character"
)
try:
parsed = urlsplit(value)
port = parsed.port
except ValueError as exc:
raise ValueError("COMPANION_BASE_URL must be a valid HTTP LAN origin") from exc
host = parsed.hostname
if parsed.scheme.lower() != "http" or not parsed.netloc or not host:
raise ValueError("COMPANION_BASE_URL must be a canonical HTTP LAN origin")
if parsed.username is not None or parsed.password is not None:
raise ValueError("COMPANION_BASE_URL must not contain credentials")
if parsed.path or parsed.query or parsed.fragment:
raise ValueError("COMPANION_BASE_URL must not contain a path, query, or fragment")
if port is not None and not 1 <= port <= 65535:
raise ValueError("COMPANION_BASE_URL port must be between 1 and 65535")
if not _valid_companion_client_host(host):
raise ValueError("COMPANION_BASE_URL host is not supported by companion v1")
netloc = f"{host}:{port}" if port is not None else host
origin = f"http://{netloc}"
if value != origin:
raise ValueError("COMPANION_BASE_URL must be a canonical HTTP LAN origin")
return host, port or 80
def configured_companion_origin() -> tuple[str, int] | None:
"""Return the validated operator-configured v1 address, if any."""
value = os.environ.get("COMPANION_BASE_URL")
if value is None or value == "":
return None
return parse_companion_base_url(value)
def default_port() -> int:
"""Best guess at the port the server is reachable on. Callers that know the
real request port should pass it explicitly."""
+8 -23
View File
@@ -23,7 +23,7 @@ from fastapi import APIRouter, HTTPException, Request
from fastapi.responses import HTMLResponse
from core.middleware import require_admin
from src.auth_helpers import _auth_disabled, get_current_user
from src.auth_helpers import get_current_user
from companion import pairing as _pairing
@@ -113,9 +113,8 @@ def setup_companion_routes() -> APIRouter:
The stock /api/models route scopes to get_current_user, which for a
bearer token is the sandboxed pseudo-user "api" (owns nothing). Here we
scope to the token's real owner instead, plus legacy null-owner shared
rows -- the same rule as owner_filter. Explicit auth-disabled mode keeps
the stock route's single-user all-endpoints view. Read-only; never
returns api_key material.
rows -- the same rule as owner_filter. Read-only; never returns api_key
material.
"""
require_models_scope(request)
import json as _json
@@ -124,11 +123,6 @@ def setup_companion_routes() -> APIRouter:
from src.endpoint_resolver import build_chat_url
owner = token_owner(request)
single_user_mode = (
owner is None
and not getattr(request.state, "api_token", False)
and _auth_disabled()
)
out = []
db = SessionLocal()
try:
@@ -139,7 +133,7 @@ def setup_companion_routes() -> APIRouter:
if owner:
q = q.filter((ModelEndpoint.owner == owner) | (ModelEndpoint.owner == None)) # noqa: E711
for ep in q.all():
if not single_user_mode and not owner_can_see(ep.owner, owner):
if not owner_can_see(ep.owner, owner):
continue
try:
model_ids = _json.loads(ep.cached_models) if ep.cached_models else []
@@ -200,27 +194,19 @@ def setup_companion_routes() -> APIRouter:
the code works immediately, no restart. `?format=json` returns the
payload for an in-app pairing screen."""
require_admin(request)
try:
configured_origin = _pairing.configured_companion_origin()
except ValueError as exc:
raise HTTPException(500, str(exc)) from None
owner = get_current_user(request)
invalidate = getattr(request.app.state, "invalidate_token_cache", None)
token_id, raw_token = mint_pairing_token(owner, invalidate)
if configured_origin:
host, port = configured_origin
hosts = [host]
else:
hosts = _pairing.lan_ip_candidates()
host = hosts[0] if hosts else "127.0.0.1"
port = request.url.port or _pairing.default_port()
hosts = _pairing.lan_ip_candidates()
host = hosts[0] if hosts else "127.0.0.1"
port = request.url.port or _pairing.default_port()
payload = _pairing.pairing_payload(host, port, raw_token)
qr = _pairing.pairing_qr_png_data_uri(payload)
qr_ok = bool(qr and qr.startswith("data:image/png;base64,"))
if (request.query_params.get("format") or "").lower() == "json":
response = {
return {
"host": host,
"port": port,
"token": raw_token,
@@ -229,7 +215,6 @@ def setup_companion_routes() -> APIRouter:
"payload": payload,
"qr": qr if qr_ok else None,
}
return response
import json as _json
payload_json = _json.dumps(payload, separators=(",", ":"))
+10 -28
View File
@@ -30,20 +30,11 @@ def atomic_write_json(path: str, data: Any, *, indent: Optional[int] = None) ->
"""
os.makedirs(os.path.dirname(path) or ".", exist_ok=True)
tmp = f"{path}.tmp.{uuid.uuid4().hex}"
try:
with open(tmp, "w", encoding="utf-8") as f:
json.dump(data, f, indent=indent)
f.flush()
os.fsync(f.fileno())
os.replace(tmp, path)
finally:
# Directly unlink to avoid a check-then-act race condition.
# Swallows FileNotFoundError (on success path) and other cleanup OSErrors.
try:
os.unlink(tmp)
except OSError:
pass
with open(tmp, "w", encoding="utf-8") as f:
json.dump(data, f, indent=indent)
f.flush()
os.fsync(f.fileno())
os.replace(tmp, path)
def atomic_write_text(path: str, text: str) -> None:
@@ -51,17 +42,8 @@ def atomic_write_text(path: str, text: str) -> None:
raise TypeError("atomic_write_text expects a string")
os.makedirs(os.path.dirname(path) or ".", exist_ok=True)
tmp = f"{path}.tmp.{uuid.uuid4().hex}"
try:
with open(tmp, "w", encoding="utf-8") as f:
f.write(text)
f.flush()
os.fsync(f.fileno())
os.replace(tmp, path)
finally:
# Directly unlink to avoid a check-then-act race condition.
# Swallows FileNotFoundError (on success path) and other cleanup OSErrors.
try:
os.unlink(tmp)
except OSError:
pass
with open(tmp, "w", encoding="utf-8") as f:
f.write(text)
f.flush()
os.fsync(f.fileno())
os.replace(tmp, path)
+16 -9
View File
@@ -20,6 +20,7 @@ logger = logging.getLogger(__name__)
from core.atomic_io import atomic_write_json as _atomic_write_json # noqa: E402
from core.middleware import INTERNAL_TOOL_USER # noqa: E402
DEFAULT_PRIVILEGES = {
"can_use_agent": True,
@@ -48,18 +49,24 @@ ADMIN_PRIVILEGES["allowed_models_restricted"] = False
ADMIN_PRIVILEGES["block_all_models"] = False
from src.constants import AUTH_FILE, PASSWORD_MIN_LENGTH
from src.owner_identity import RESERVED_AUTH_USERNAMES
DEFAULT_AUTH_PATH = AUTH_FILE
TOKEN_TTL = 60 * 60 * 24 * 7 # 7 days
# Usernames the auth + middleware layer reserves for request sentinels and
# internal storage owners; they must never belong to a real login account.
# "internal-tool" is the most dangerous because `core.middleware.require_admin`
# treats it as the in-process tool loopback. "api" collides with bearer-token
# attribution. "demo"/"system" are synthetic owners already special-cased by
# scheduler/assistant/research paths. The Default/Local owner is a storage
# bucket for explicit auth-disabled no-login mode, not a login username.
RESERVED_USERNAMES = frozenset(RESERVED_AUTH_USERNAMES)
# Usernames the auth + middleware layer reserve as internal "synthetic owner"
# sentinels; they must never belong to a real account. The most dangerous is
# "internal-tool": `core.middleware.require_admin` treats any request whose
# `current_user == "internal-tool"` as the in-process tool loopback and grants
# admin, and because the cookie auth path sets `current_user` to the raw
# username, an account literally named "internal-tool" would be silently
# treated as an admin by every `require_admin`-gated route. "api" collides with
# the bearer-token owner-attribution sentinel. "demo"/"system" round out the
# synthetic-owner set the rest of the codebase already special-cases (see
# `_SYNTHETIC_OWNERS` in routes/assistant_routes.py and the matching guards in
# src/task_scheduler.py / routes/research_routes.py) — a real account with one
# of those names would be denied an assistant and inconsistently owner-scoped.
# Refuse to create or rename into any of them so the sentinels can't be
# impersonated. (Keep this in sync with that synthetic-owner set.)
RESERVED_USERNAMES = frozenset({INTERNAL_TOOL_USER, "api", "demo", "system"})
def normalize_known_username(users: Dict[str, Any], username: str | None) -> Optional[str]:
+1 -84
View File
@@ -5,7 +5,7 @@ from datetime import datetime, timezone
from pathlib import Path
from typing import Optional
from urllib.parse import unquote, urlparse
from sqlalchemy import DDL, event, create_engine, Column, String, Text, Boolean, DateTime, Integer, ForeignKey, JSON, Index, UniqueConstraint, func, inspect, text
from sqlalchemy import DDL, event, create_engine, Column, String, Text, Boolean, DateTime, Integer, ForeignKey, JSON, Index, func, inspect, text
from sqlalchemy.engine import Engine, make_url
from sqlalchemy.types import TypeDecorator
from sqlalchemy.ext.declarative import declarative_base, declared_attr
@@ -187,13 +187,6 @@ class Session(TimestampMixin, Base):
endpoint_url = Column(String, nullable=False)
model = Column(String, nullable=False)
owner = Column(String, nullable=True, index=True) # username; null = legacy/shared
# Bearer-chat sessions must retain the exact server-owned endpoint they
# were created from. Keep this reference non-cascading so endpoint
# disable/delete/owner changes remain observable as an orphan and fail
# closed at the next bearer LLM boundary.
model_endpoint_id = Column(String, nullable=True, index=True)
endpoint_provenance = Column(String, nullable=True)
# Configuration flags
rag = Column(Boolean, default=False)
@@ -287,47 +280,6 @@ class ChatMessage(Base):
Index('ix_messages_session_time', 'session_id', 'timestamp'), # Composite for efficient message retrieval
)
class ChatSessionApprovalGrant(Base):
"""Server-owned, durable approval provenance for one chat session.
A resolved tool-approval card is display/history data, not authority. This
separate row is inserted only by the interactive approval continuation and
is keyed by the real session owner plus session id. It deliberately has no
update path; deleting the owning session cascades the grant so an old id
cannot carry approval authority into a newly-created conversation.
"""
__tablename__ = "chat_session_approval_grants"
id = Column(String, primary_key=True, index=True)
session_id = Column(
String,
ForeignKey("sessions.id", ondelete="CASCADE"),
nullable=False,
index=True,
)
owner = Column(String, nullable=False, index=True)
approval_id = Column(String, nullable=False, index=True)
provenance_version = Column(Integer, nullable=False, default=1)
created_at = Column(DateTime, default=utcnow_naive, nullable=False)
__table_args__ = (
UniqueConstraint(
"session_id",
"owner",
"approval_id",
name="uq_chat_session_approval_grant",
),
Index(
"ix_chat_session_approval_grant_lookup",
"session_id",
"owner",
"provenance_version",
),
)
class Document(TimestampMixin, Base):
"""Living document that the AI can create and edit in-place."""
__tablename__ = "documents"
@@ -1006,40 +958,6 @@ def _migrate_add_owner_column():
except Exception:
pass
def _migrate_add_session_endpoint_provenance_columns():
"""Add the durable endpoint identity used by bearer session validation."""
import sqlite3
db_path = DATABASE_URL.replace("sqlite:///", "")
if not os.path.exists(db_path):
return
conn = None
try:
conn = sqlite3.connect(db_path)
columns = {row[1] for row in conn.execute("PRAGMA table_info(sessions)")}
if "model_endpoint_id" not in columns:
conn.execute("ALTER TABLE sessions ADD COLUMN model_endpoint_id TEXT")
if "endpoint_provenance" not in columns:
conn.execute("ALTER TABLE sessions ADD COLUMN endpoint_provenance TEXT")
conn.execute(
"CREATE INDEX IF NOT EXISTS ix_sessions_model_endpoint_id "
"ON sessions(model_endpoint_id)"
)
conn.commit()
logging.getLogger(__name__).info(
"Migrated: added session endpoint identity/provenance columns"
)
except Exception as e:
logging.getLogger(__name__).warning(
"Session endpoint provenance migration failed: %s", e
)
finally:
try:
conn.close()
except Exception:
pass
def _migrate_model_endpoints():
"""Recreate model_endpoints table if schema changed (url->base_url)."""
import sqlite3
@@ -2193,7 +2111,6 @@ def init_db():
_migrate_add_supports_tools_column()
_migrate_add_task_run_model_column()
_migrate_add_owner_column()
_migrate_add_session_endpoint_provenance_columns()
_migrate_add_document_archived_column()
_migrate_add_last_message_at_column()
_migrate_add_folder_column()
+3 -37
View File
@@ -3,15 +3,10 @@
import os
import secrets
from collections.abc import Mapping
from fastapi import HTTPException, Request
from starlette.middleware.base import BaseHTTPMiddleware
from starlette.responses import Response
from starlette.routing import get_route_path
from src.owner_identity import INTERNAL_TOOL_USER, auth_disabled
from src.auth_helpers import is_bearer_principal
# Per-process token that lets the in-app tool layer hit admin-gated
@@ -20,30 +15,8 @@ from src.auth_helpers import is_bearer_principal
# same value from this module. Never persisted or exposed externally.
INTERNAL_TOOL_TOKEN = os.environ.get("ODYSSEUS_INTERNAL_TOKEN") or secrets.token_hex(32)
INTERNAL_TOOL_HEADER = "X-Odysseus-Internal-Token"
def get_application_route_path(scope: Mapping[str, object]) -> str:
"""Return the application-relative path used by Starlette routing.
Uvicorn prefixes ``scope["path"]`` with a configured ASGI ``root_path``;
Starlette removes that prefix before matching routes. Middleware policy
must use the same path form or a deployment prefix can change which policy
applies to an otherwise unchanged application route.
"""
return get_route_path(scope)
def with_asgi_root_path(scope: Mapping[str, object], path: str) -> str:
"""Prefix an application path for a client-facing redirect target."""
root_path = scope.get("root_path", "")
if not isinstance(root_path, str) or not root_path:
return path
return f"{root_path.rstrip('/')}{path}"
def path_is_route_or_child(path: str, prefix: str) -> bool:
"""Return whether ``path`` is exactly ``prefix`` or below that route."""
return path == prefix or path.startswith(prefix + "/")
# Pseudo-username on in-process tool-loopback requests; require_admin trusts it and it is reserved.
INTERNAL_TOOL_USER = "internal-tool"
def is_cors_preflight(method: str, headers) -> bool:
@@ -60,13 +33,6 @@ def require_admin(request: Request):
Allows access when auth is explicitly disabled, or when the request carries
the in-process internal-tool token used by loopback agent tools.
"""
# A bearer principal never inherits admin authority, even when the token
# carries a legacy cookbook scope or auth is disabled in a direct-entry
# test. Host-control routes use this centralized gate, so rejecting here
# covers shell, model-serving, MCP, runtime, and other admin surfaces.
if is_bearer_principal(request):
raise HTTPException(403, "API tokens cannot use admin host-control surfaces")
# In-process bypass for tool-layer loopback calls. Two paths:
# (a) header-direct (caller set X-Odysseus-Internal-Token), or
# (b) the auth middleware already validated the token and stamped
@@ -81,7 +47,7 @@ def require_admin(request: Request):
pass
auth_mgr = getattr(request.app.state, "auth_manager", None)
if auth_disabled():
if os.getenv("AUTH_ENABLED", "true").lower() == "false":
return
if not auth_mgr or not auth_mgr.is_configured:
raise HTTPException(403, "Admin only")
+5 -56
View File
@@ -8,12 +8,6 @@ These are simple datacontainers. All persistence is handled by SessionManager.
from dataclasses import dataclass
from typing import Dict, List, Any, Optional, TYPE_CHECKING
from src.tool_approval_scopes import (
CHAT_SESSION_APPROVAL_CONTEXT_MARKER,
)
from src.message_metadata import sanitize_projected_message_metadata
from src.tool_approval_provenance import has_chat_session_approval_grant
if TYPE_CHECKING:
from .session_manager import SessionManager
@@ -37,18 +31,6 @@ set_session_manager = set_session_manager_instance
get_session_manager = get_session_manager_instance
def _history_grants_chat_session_approval(
history: List["ChatMessage"],
session_id: str,
) -> bool:
"""Compatibility shim: durable history is never an authority source.
Keep the old private symbol for downstream imports, but deliberately return
false. The live projection checks the separate server-owned grant table.
"""
return False
@dataclass
class ChatMessage:
"""A single chat message."""
@@ -90,8 +72,6 @@ class Session:
headers: Optional[Dict[str, str]] = None
history: List[ChatMessage] = None
owner: Optional[str] = None
model_endpoint_id: Optional[str] = None
endpoint_provenance: Optional[str] = None
is_important: bool = False
message_count: int = 0
@@ -136,42 +116,11 @@ class Session:
the model. Display/history-load paths use the raw ``history`` and are
unaffected.
"""
messages = []
for msg in self.history:
raw_metadata = getattr(msg, "metadata", None)
if isinstance(raw_metadata, dict) and raw_metadata.get("source") == "slash":
continue
projected = msg.to_dict()
metadata = projected.get("metadata")
if not isinstance(metadata, dict):
# Old or malformed durable rows must not make context
# projection fail, and non-mapping metadata has no trusted
# fields that belong in the model context.
projected.pop("metadata", None)
messages.append(projected)
continue
metadata = sanitize_projected_message_metadata(metadata)
if metadata:
projected["metadata"] = metadata
else:
projected.pop("metadata", None)
messages.append(projected)
if not has_chat_session_approval_grant(self.id, self.owner):
return messages
# Keep the grant close to the latest user request so route-neutral
# compaction/trimming preserves it. Copy the metadata instead of
# mutating the durable transcript object.
for index in range(len(messages) - 1, -1, -1):
if messages[index].get("role") != "user":
continue
message = dict(messages[index])
metadata = dict(message.get("metadata") or {})
metadata[CHAT_SESSION_APPROVAL_CONTEXT_MARKER] = True
message["metadata"] = metadata
messages[index] = message
break
return messages
return [
msg.to_dict()
for msg in self.history
if (msg.metadata or {}).get("source") != "slash"
]
def get(self, key: str, default=None):
"""Dict-like access for compatibility."""
+5 -89
View File
@@ -14,8 +14,6 @@ import logging
from datetime import datetime, timezone, timedelta
from typing import Dict, Optional
from sqlalchemy import func
from .database import Session as DbSession, ChatMessage as DbChatMessage, Document as DbDocument, SessionLocal, utcnow_naive
from .models import Session, ChatMessage
from src.attachment_refs import persistable_message_content
@@ -62,22 +60,6 @@ def _parse_msg_content(raw):
return raw
def _parse_message_metadata(raw) -> dict:
"""Decode only JSON objects from durable message metadata.
Legacy rows may contain a JSON list (including list-of-pairs) or another
scalar. Such values have no trusted message fields and must not reach the
``_db_id``/timestamp merge below or any approval projection.
"""
if not raw:
return {}
try:
parsed = json.loads(raw) if isinstance(raw, str) else raw
except (json.JSONDecodeError, TypeError, ValueError):
return {}
return dict(parsed) if isinstance(parsed, dict) else {}
class SessionManager:
"""
Manages chat sessions with database persistence.
@@ -110,28 +92,14 @@ class SessionManager:
try:
db_sessions = db.query(DbSession).filter(
DbSession.archived == False,
DbSession.messages.any(),
DbSession.message_count > 0,
).order_by(DbSession.last_accessed.desc()).limit(100).all()
# message_count is derived metadata and can drift after interrupted
# or legacy writes. Count only the bounded discovery set so startup
# remains metadata-only while lazy hydration sees an authoritative
# positive count for every discovered non-empty session.
message_counts = {}
if db_sessions:
message_counts = dict(
db.query(DbChatMessage.session_id, func.count(DbChatMessage.id))
.filter(DbChatMessage.session_id.in_([row.id for row in db_sessions]))
.group_by(DbChatMessage.session_id)
.all()
)
loaded_count = 0
for db_session in db_sessions:
try:
session = self._db_to_session_meta(db_session)
if session is not None:
session.message_count = message_counts[db_session.id]
self.sessions[db_session.id] = session
loaded_count += 1
except Exception as e:
@@ -165,8 +133,6 @@ class SessionManager:
headers=headers,
history=[],
owner=getattr(db_session, "owner", None),
model_endpoint_id=getattr(db_session, "model_endpoint_id", None),
endpoint_provenance=getattr(db_session, "endpoint_provenance", None),
is_important=getattr(db_session, "is_important", False) or False,
)
session.message_count = getattr(db_session, "message_count", 0) or 0
@@ -179,7 +145,8 @@ class SessionManager:
# Try relationship first, then direct query
if db_session.messages:
for db_msg in db_session.messages:
meta = _parse_message_metadata(db_msg.meta_data)
meta = json.loads(db_msg.meta_data) if db_msg.meta_data else {}
if meta is None: meta = {}
meta['_db_id'] = db_msg.id
meta.setdefault('timestamp', _message_timestamp_iso(db_msg.timestamp))
history.append(ChatMessage(
@@ -193,7 +160,8 @@ class SessionManager:
).order_by(DbChatMessage.timestamp).all()
for db_msg in db_messages:
meta = _parse_message_metadata(db_msg.meta_data)
meta = json.loads(db_msg.meta_data) if db_msg.meta_data else {}
if meta is None: meta = {}
meta['_db_id'] = db_msg.id
meta.setdefault('timestamp', _message_timestamp_iso(db_msg.timestamp))
history.append(ChatMessage(
@@ -223,8 +191,6 @@ class SessionManager:
headers=headers,
history=history,
owner=getattr(db_session, 'owner', None),
model_endpoint_id=getattr(db_session, 'model_endpoint_id', None),
endpoint_provenance=getattr(db_session, 'endpoint_provenance', None),
is_important=getattr(db_session, 'is_important', False) or False,
)
@@ -272,8 +238,6 @@ class SessionManager:
logger.warning("Dropping message for deleted session %s", session_id)
return
if not isinstance(message.metadata, dict):
message.metadata = None
missing_upload_id = reserve_message_upload_references(
getattr(self, "upload_handler", None),
getattr(db_session, "owner", None),
@@ -386,8 +350,6 @@ class SessionManager:
# ownership check/access touch and the replacement transaction.
# A failed reservation must leave the existing transcript intact.
for message in messages:
if not isinstance(message.metadata, dict):
message.metadata = None
missing_upload_id = reserve_message_upload_references(
getattr(self, "upload_handler", None),
getattr(db_session, "owner", None),
@@ -506,8 +468,6 @@ class SessionManager:
session.rag = db_session.rag
session.archived = db_session.archived
session.owner = getattr(db_session, "owner", None)
session.model_endpoint_id = getattr(db_session, "model_endpoint_id", None)
session.endpoint_provenance = getattr(db_session, "endpoint_provenance", None)
session.is_important = getattr(db_session, "is_important", False) or False
session.message_count = (
db.query(DbChatMessage)
@@ -608,50 +568,6 @@ class SessionManager:
finally:
db.close()
def set_session_endpoint_provenance(
self,
session_id: str,
*,
model_endpoint_id: Optional[str],
endpoint_provenance: str,
) -> bool:
"""Persist the server-owned endpoint provenance for a session.
``registered`` rows carry an exact ModelEndpoint id. ``direct`` rows
deliberately carry no endpoint id and retain direct API-key
compatibility. The values are assigned only after the durable write
succeeds so an in-memory session cannot claim provenance the database
did not accept.
"""
provenance = str(endpoint_provenance or "").strip().lower()
endpoint_id = str(model_endpoint_id or "").strip() or None
if provenance == "registered" and not endpoint_id:
raise ValueError("registered session provenance requires an endpoint id")
if provenance == "direct":
endpoint_id = None
if provenance not in {"registered", "direct"}:
raise ValueError("unsupported session endpoint provenance")
db = SessionLocal()
try:
db_session = db.query(DbSession).filter(DbSession.id == session_id).first()
if db_session is None:
raise KeyError(f"Session {session_id} not found")
db_session.model_endpoint_id = endpoint_id
db_session.endpoint_provenance = provenance
db.commit()
except Exception:
db.rollback()
raise
finally:
db.close()
session = self.sessions.get(session_id)
if session is not None:
session.model_endpoint_id = endpoint_id
session.endpoint_provenance = provenance
return True
def delete_session(self, session_id: str) -> bool:
"""Permanently delete a session and all its messages."""
db = SessionLocal()
+1 -7
View File
@@ -46,11 +46,10 @@ services:
- DATABASE_URL=${DATABASE_URL:-sqlite:///./data/app.db}
- AUTH_ENABLED=${AUTH_ENABLED:-true}
- LOCALHOST_BYPASS=${LOCALHOST_BYPASS:-false}
- COMPANION_BASE_URL=${COMPANION_BASE_URL:-}
- ODYSSEUS_ADMIN_USER=${ODYSSEUS_ADMIN_USER:-admin}
- ODYSSEUS_ADMIN_PASSWORD=${ODYSSEUS_ADMIN_PASSWORD:-}
- ALLOWED_ORIGINS=${ALLOWED_ORIGINS:-http://localhost,http://127.0.0.1}
- SECURE_COOKIES=${SECURE_COOKIES:-}
- SECURE_COOKIES=${SECURE_COOKIES:-false}
- EMBEDDING_URL=${EMBEDDING_URL:-}
- EMBEDDING_MODEL=${EMBEDDING_MODEL:-}
- EMBEDDING_API_KEY=${EMBEDDING_API_KEY:-}
@@ -75,11 +74,6 @@ services:
- GOOGLE_OAUTH_CLIENT_ID=${GOOGLE_OAUTH_CLIENT_ID:-}
- GOOGLE_OAUTH_CLIENT_SECRET=${GOOGLE_OAUTH_CLIENT_SECRET:-}
- GOOGLE_OAUTH_REDIRECT_URI=${GOOGLE_OAUTH_REDIRECT_URI:-}
# Externally reachable origin for MCP OAuth callbacks. The container
# always listens on 7000 and cannot see the host port map above, so
# remote MCP OAuth needs this set whenever the browser reaches
# Odysseus on anything other than http://localhost:7000.
- OAUTH_REDIRECT_BASE_URL=${OAUTH_REDIRECT_BASE_URL:-}
- TAVILY_API_KEY=${TAVILY_API_KEY:-}
- SERPER_API_KEY=${SERPER_API_KEY:-}
# PUID / PGID — the user/group the container drops to before
+1 -7
View File
@@ -45,11 +45,10 @@ services:
- DATABASE_URL=${DATABASE_URL:-sqlite:///./data/app.db}
- AUTH_ENABLED=${AUTH_ENABLED:-true}
- LOCALHOST_BYPASS=${LOCALHOST_BYPASS:-false}
- COMPANION_BASE_URL=${COMPANION_BASE_URL:-}
- ODYSSEUS_ADMIN_USER=${ODYSSEUS_ADMIN_USER:-admin}
- ODYSSEUS_ADMIN_PASSWORD=${ODYSSEUS_ADMIN_PASSWORD:-}
- ALLOWED_ORIGINS=${ALLOWED_ORIGINS:-http://localhost,http://127.0.0.1}
- SECURE_COOKIES=${SECURE_COOKIES:-}
- SECURE_COOKIES=${SECURE_COOKIES:-false}
- EMBEDDING_URL=${EMBEDDING_URL:-}
- EMBEDDING_MODEL=${EMBEDDING_MODEL:-}
- EMBEDDING_API_KEY=${EMBEDDING_API_KEY:-}
@@ -74,11 +73,6 @@ services:
- GOOGLE_OAUTH_CLIENT_ID=${GOOGLE_OAUTH_CLIENT_ID:-}
- GOOGLE_OAUTH_CLIENT_SECRET=${GOOGLE_OAUTH_CLIENT_SECRET:-}
- GOOGLE_OAUTH_REDIRECT_URI=${GOOGLE_OAUTH_REDIRECT_URI:-}
# Externally reachable origin for MCP OAuth callbacks. The container
# always listens on 7000 and cannot see the host port map above, so
# remote MCP OAuth needs this set whenever the browser reaches
# Odysseus on anything other than http://localhost:7000.
- OAUTH_REDIRECT_BASE_URL=${OAUTH_REDIRECT_BASE_URL:-}
- TAVILY_API_KEY=${TAVILY_API_KEY:-}
- SERPER_API_KEY=${SERPER_API_KEY:-}
# PUID / PGID — the user/group the container drops to before
+1 -7
View File
@@ -34,11 +34,10 @@ services:
- DATABASE_URL=${DATABASE_URL:-sqlite:///./data/app.db}
- AUTH_ENABLED=${AUTH_ENABLED:-true}
- LOCALHOST_BYPASS=${LOCALHOST_BYPASS:-false}
- COMPANION_BASE_URL=${COMPANION_BASE_URL:-}
- ODYSSEUS_ADMIN_USER=${ODYSSEUS_ADMIN_USER:-admin}
- ODYSSEUS_ADMIN_PASSWORD=${ODYSSEUS_ADMIN_PASSWORD:-}
- ALLOWED_ORIGINS=${ALLOWED_ORIGINS:-http://localhost,http://127.0.0.1}
- SECURE_COOKIES=${SECURE_COOKIES:-}
- SECURE_COOKIES=${SECURE_COOKIES:-false}
- EMBEDDING_URL=${EMBEDDING_URL:-}
- EMBEDDING_MODEL=${EMBEDDING_MODEL:-}
- EMBEDDING_API_KEY=${EMBEDDING_API_KEY:-}
@@ -63,11 +62,6 @@ services:
- GOOGLE_OAUTH_CLIENT_ID=${GOOGLE_OAUTH_CLIENT_ID:-}
- GOOGLE_OAUTH_CLIENT_SECRET=${GOOGLE_OAUTH_CLIENT_SECRET:-}
- GOOGLE_OAUTH_REDIRECT_URI=${GOOGLE_OAUTH_REDIRECT_URI:-}
# Externally reachable origin for MCP OAuth callbacks. The container
# always listens on 7000 and cannot see the host port map above, so
# remote MCP OAuth needs this set whenever the browser reaches
# Odysseus on anything other than http://localhost:7000.
- OAUTH_REDIRECT_BASE_URL=${OAUTH_REDIRECT_BASE_URL:-}
- TAVILY_API_KEY=${TAVILY_API_KEY:-}
- SERPER_API_KEY=${SERPER_API_KEY:-}
# PUID / PGID — the user/group the container drops to before
@@ -1,7 +1,3 @@
---
layout: default
---
# Agent migration manifests
Odysseus should be able to learn from another agent without blindly trusting
@@ -1,7 +1,3 @@
---
layout: default
---
# Attachment References and Upload Storage
Odysseus stores uploaded bytes once under the configured upload directory and
@@ -1,7 +1,3 @@
---
layout: default
---
# Backup & Restore
Odysseus keeps all of your state in the `data/` directory — the SQLite database
View File
@@ -1,7 +1,3 @@
---
layout: default
---
# Outlook / Office 365 email accounts
Odysseus email accounts currently use IMAP and SMTP with username/password

Before

Width:  |  Height:  |  Size: 185 KiB

After

Width:  |  Height:  |  Size: 185 KiB

Before

Width:  |  Height:  |  Size: 16 KiB

After

Width:  |  Height:  |  Size: 16 KiB

Before

Width:  |  Height:  |  Size: 79 KiB

After

Width:  |  Height:  |  Size: 79 KiB

@@ -1,7 +1,3 @@
---
layout: default
---
# PR Blocker Audit
`scripts/pr_blocker_audit.py` is a small, read-only triage helper for maintainers who need to inspect open pull request overlap before reviewing or starting related work.
@@ -1,7 +1,3 @@
---
layout: default
---
# Security CI guide
This project runs a set of automated security checks on pull requests and
+12 -39
View File
@@ -1,7 +1,3 @@
---
layout: default
---
# Odysseus Setup Guide
This page keeps the detailed install, deployment, troubleshooting, and configuration notes out of the front README.
@@ -19,7 +15,8 @@ On first setup, Odysseus creates an admin account (`admin` unless
For Docker installs, the same line is in `docker compose logs odysseus`.
Use that for the first login, then change it in **Settings**.
Contributing? See [CONTRIBUTING.md](https://github.com/odysseus-dev/odysseus/blob/dev/CONTRIBUTING.md) for setup, testing, and pull request guidelines.
Contributing? See [CONTRIBUTING.md](../CONTRIBUTING.md) for setup, testing, and
pull request guidelines.
### Docker (recommended)
```bash
@@ -208,11 +205,9 @@ failed to fulfil mount request: open /usr/lib/wsl/lib/libdxcore.so: no such file
Check with `snap list docker` or:
<!-- {% raw %} -->
```bash
docker info --format '{{.DockerRootDir}}'
```
<!-- {% endraw %} -->
A Docker root under `/var/snap/docker/` means snap confinement can prevent
Docker from seeing WSL2's `/usr/lib/wsl/lib` GPU libraries even when the files
@@ -446,19 +441,10 @@ A grab-bag of small gotchas that otherwise turn into long debugging sessions.
| Package | Feature unlocked |
|---------|-----------------|
| `faster-whisper` | Local speech-to-text (microphone -> text) via the "local" STT provider. |
| `kokoro`, `soundfile` | Local Kokoro-82M text-to-speech on a CUDA GPU. The pinned Kokoro release supports Odysseus installs on Python 3.11-3.12; these packages are intentionally skipped on Python 3.13+ (including the Python 3.14 container image). |
| `ddgs` | DuckDuckGo as a search provider option. |
| `PyMuPDF` | PDF page rendering in the side viewer panel and form-filling. (Note: AGPL-3.0) |
| `markitdown` | Office/EPUB document text extraction (converts .docx/.xlsx/.pptx/.xls/.epub to Markdown). |
Install the optional set only when you need these features:
```bash
pip install -r requirements-optional.txt
```
The default Docker image currently uses Python 3.14, while Kokoro 0.9.4 declares Python `>=3.10,<3.13`. Odysseus itself continues to support Python 3.11+, but this pinned optional local-TTS feature requires a native Python 3.11 or 3.12 environment. Kokoro declares `torch`, but the local provider only activates when that torch build has CUDA and a GPU is visible; install the CUDA build appropriate for your host. Browser and configured endpoint TTS remain available on Python 3.13+ and in the container image.
### Faster, reproducible installs with uv (optional)
[uv](https://docs.astral.sh/uv/) works as a drop-in replacement for the
venv + pip steps in the native install guides, no project changes are needed but this change results in faster installs along with a lockfile for reproducible environments. After [installing `uv`](https://docs.astral.sh/uv/getting-started/installation/), use:
@@ -481,7 +467,7 @@ uv pip sync requirements.lock # reproduce it exactly la
### Outlook / Office 365 email
Odysseus email accounts currently use IMAP/SMTP username-password auth. Outlook
and Microsoft 365 generally require OAuth instead, so normal Microsoft mailbox
passwords will fail. See [the Outlook email guide](email-outlook.md) for the
passwords will fail. See [docs/email-outlook.md](docs/email-outlook.md) for the
current limitation and the planned integration direction.
## Security Notes
@@ -489,7 +475,7 @@ Odysseus is a self-hosted workspace with powerful local tools: shell access, fil
- Keep `AUTH_ENABLED=true` for any network-accessible deployment.
- Keep `LOCALHOST_BYPASS=false` outside local development.
- Leave `SECURE_COOKIES` unset unless you need to override it: session cookies are marked `Secure` whenever the request arrives over HTTPS. Use `SECURE_COOKIES=true` to force it on for a proxy whose scheme Odysseus cannot see, or `SECURE_COOKIES=false` to force it off while you still serve plain HTTP alongside HTTPS.
- Use `SECURE_COOKIES=true` when Odysseus is served through HTTPS by a trusted reverse proxy or private access gateway.
- Do not expose it directly to the public internet without HTTPS and a trusted reverse proxy or private access layer.
- Keep `.env`, `data/`, `logs/`, databases, uploads, generated media, backups, auth/session files, API keys, and model/provider tokens out of Git and private shares. They are ignored by default.
- Review `data/auth.json` after first boot: disable open signup unless you intentionally want it, make only your own account admin, and keep demo/test accounts non-admin.
@@ -500,14 +486,6 @@ Odysseus is a self-hosted workspace with powerful local tools: shell access, fil
- Keep ChromaDB, SearXNG, ntfy, Ollama, vLLM, llama.cpp, databases, and raw model/provider APIs internal-only. Expose only the authenticated Odysseus web/API entrypoint through your trusted proxy or private access layer.
- Before publishing a fork, run `git status --short` and confirm no private files from `.env`, `data/`, `logs/`, uploads, backups, or local databases are staged.
> **Upgrading an existing install:** `SECURE_COOKIES` used to default to
> `false`, so an install set up before scheme derivation may still carry
> `SECURE_COOKIES=false` in its own `.env`. That explicit value stays
> authoritative, so HTTPS logins keep getting a non-`Secure` session cookie.
> Pulling this change updates the tracked Compose files, but nothing rewrites
> your `.env` — drop the line from it unless you deliberately serve plain HTTP
> alongside HTTPS and want the escape hatch.
### Private or proxied deployments
Odysseus serves plain HTTP on its app port. Docker Compose binds Odysseus and the bundled services to `127.0.0.1` by default, so a typical production/private setup is:
@@ -516,7 +494,7 @@ Odysseus serves plain HTTP on its app port. Docker Compose binds Odysseus and th
3. Put the authenticated Odysseus web/API entrypoint behind that layer.
4. Keep raw service and model ports internal-only.
Cloudflare Access, Tailscale, Caddy, nginx, and Traefik can all fit this pattern; none are required by Odysseus. If your access layer reaches Odysseus on the same host, proxy to `http://127.0.0.1:7000` and keep `AUTH_ENABLED=true` and `LOCALHOST_BYPASS=false`. Any proxy that forwards `X-Forwarded-Proto: https` gets `Secure` session cookies without configuration, so `SECURE_COOKIES` only needs setting when you want to override that — force it on for a proxy that forwards no scheme at all, or off while you still serve plain HTTP.
Cloudflare Access, Tailscale, Caddy, nginx, and Traefik can all fit this pattern; none are required by Odysseus. If your access layer reaches Odysseus on the same host, proxy to `http://127.0.0.1:7000` and keep `AUTH_ENABLED=true`, `LOCALHOST_BYPASS=false`, and `SECURE_COOKIES=true`.
`ALLOWED_ORIGINS` lists exact permitted origins for cross-origin browser/API clients; ordinary same-origin reverse-proxy access usually does not need a special CORS entry.
#### Faster over the network: HTTP/2
@@ -604,12 +582,9 @@ Odysseus's own service is unchanged; the proxy runs alongside it. Under Docker,
run the proxy as another container, or on the host pointing at the published
port.
**4. Point Odysseus at the new origin** in `.env`, then restart it.
A proxy that exposes the HTTPS request scheme to Odysseus needs no `SECURE_COOKIES` setting. Only force it on when the proxy cannot expose that scheme:
**4. Point Odysseus at the new origin** in `.env`, then restart it:
```bash
# only if the proxy cannot expose the external HTTPS scheme to Odysseus:
SECURE_COOKIES=true
# only if you use remote MCP servers with OAuth:
OAUTH_REDIRECT_BASE_URL=https://odysseus.example.com
@@ -644,12 +619,10 @@ enable it by right-clicking the column headers.
Three things bite when moving an existing install behind TLS:
- Leave `SECURE_COOKIES` unset when Odysseus can see the external HTTPS scheme;
the cookie then follows the request automatically. If your proxy cannot expose
that scheme, set `SECURE_COOKIES=true` **at the same time** you stop serving
plain HTTP, not before. An explicit `true` applies to every login, so while an
HTTP entrypoint is still reachable the browser will reject the `Secure` cookie
there and login will appear to loop.
- Set `SECURE_COOKIES=true` **at the same time** you stop serving plain HTTP,
not before. The flag is applied to every login regardless of the scheme the
request arrived on, so while an HTTP entrypoint is still reachable the
browser will reject the `Secure` cookie there and login will appear to loop.
- `OAUTH_REDIRECT_BASE_URL` defaults to `http://localhost:7000`. Unlike the
Gmail redirect URI it cannot be derived from a request — it is registered
with each MCP authorization server up front — so set it to the external
@@ -702,7 +675,7 @@ Key settings:
| `AUTH_ENABLED` | `true` | Enable/disable login |
| `LOCALHOST_BYPASS` | `false` | Development-only auth bypass for loopback requests. Keep false for shared/network deployments. |
| `ALLOWED_ORIGINS` | `http://localhost,http://127.0.0.1` | Comma-separated exact permitted origins for cross-origin browser/API clients. |
| `SECURE_COOKIES` | derived from the request scheme | Marks session cookies `Secure` on HTTPS requests. Set true to force it on, false to force it off. |
| `SECURE_COOKIES` | `false` | Set true when serving Odysseus through HTTPS at a trusted proxy or private access gateway. |
| `DATABASE_URL` | `sqlite:///./data/app.db` | Database connection string |
| `CHROMADB_HOST` | `localhost` | ChromaDB host for vector memory. Docker overrides this to `chromadb`. |
| `CHROMADB_PORT` | `8100` | ChromaDB port for manual host runs. Docker overrides this to `8000`. |
@@ -738,7 +711,7 @@ src/ llm_core, agent_loop, agent_tools, chat_processor, search/
routes/ chat, session, document, memory, model … endpoints
services/ docs, memory, search, hwfit (Cookbook) …
static/ index.html + app.js + style.css + js/ (modular front-end)
website/ landing page (index.html) + preview clips
docs/ landing page (index.html) + preview clips
```
## Data
-4
View File
@@ -163,10 +163,6 @@ if (Test-Path $cudaBase) {
}
# 7. Start the server (use `python -m uvicorn` - bare `uvicorn` may not be on PATH)
# -Port only reaches uvicorn as a flag. Everything that builds a URL for this
# instance - internal_api_base(), companion pairing, the MCP OAuth callback -
# reads APP_PORT, so set it too or they all assume 7000.
$env:APP_PORT = $Port
Write-Step ("Starting Odysseus at http://{0}:{1}" -f $BindHost, $Port)
Write-Host "Press Ctrl+C to stop."
Write-Host ""
-21
View File
@@ -1,21 +0,0 @@
The MIT License (MIT)
Copyright (c) 2013-2020 Khan Academy and other contributors
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
-21
View File
@@ -1,21 +0,0 @@
The MIT License (MIT)
Copyright (c) 2014 - 2022 Knut Sveidqvist
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
+4 -4
View File
@@ -5,13 +5,13 @@
"packages": {
"": {
"devDependencies": {
"@antithesishq/bombadil": "^0.7.0"
"@antithesishq/bombadil": "^0.6.1"
}
},
"node_modules/@antithesishq/bombadil": {
"version": "0.7.0",
"resolved": "https://registry.npmjs.org/@antithesishq/bombadil/-/bombadil-0.7.0.tgz",
"integrity": "sha512-alJmnphJ/iUoL5mCsnV3DwtajGy/sEQ3NJJCiMhgjqXshSq2BUtAs0vqdXEiiSkB8HbsOX5CLrAcaogYdwfAJg==",
"version": "0.6.1",
"resolved": "https://registry.npmjs.org/@antithesishq/bombadil/-/bombadil-0.6.1.tgz",
"integrity": "sha512-d1iufG3MI7gSMSiSmMeNdcMW+qR0yQXL2zdkVynC3n3DYgFJYlYXKUQzygmqU12m4RWlR5iOdQU1hsx5UT6+IA==",
"dev": true,
"license": "MIT",
"bin": {
+1 -1
View File
@@ -4,6 +4,6 @@
"url": "https://github.com/odysseus-dev/odysseus.git"
},
"devDependencies": {
"@antithesishq/bombadil": "^0.7.0"
"@antithesishq/bombadil": "^0.6.1"
}
}
-10
View File
@@ -12,16 +12,6 @@
# GPU-accelerated transcription — it's auto-detected, CPU is used otherwise.
faster-whisper
# Local text-to-speech via Kokoro-82M for the "local" TTS provider.
# Kokoro 0.9.4 declares Python >=3.10,<3.13; Odysseus itself requires 3.11+,
# so pip installs these extras on 3.11-3.12 and deliberately skips them on
# Python 3.13+ (including the Python 3.14 container image). Kokoro declares
# torch; the local provider still
# requires a CUDA-enabled torch build and GPU at runtime. SoundFile is separate
# in Kokoro's official install instructions and is not a transitive dependency.
kokoro==0.9.4; python_version >= "3.11" and python_version < "3.13"
soundfile; python_version >= "3.11" and python_version < "3.13"
# DuckDuckGo as a search provider option.
# Install if you want DDG in the search-provider dropdown.
# Alternatives: SearXNG, Brave, Tavily, Serper, Google PSE.
+7 -12
View File
@@ -11,12 +11,12 @@ import json
from datetime import datetime
from typing import Optional
from fastapi import APIRouter, Depends, HTTPException, Request
from fastapi import APIRouter, HTTPException, Request
from pydantic import BaseModel
from core.database import SessionLocal, CrewMember, ScheduledTask
from src.auth_helpers import require_interactive_request
from src.owner_identity import REQUEST_SENTINEL_OWNERS
from src.auth_helpers import get_current_user
from core.auth import RESERVED_USERNAMES
from src.task_scheduler import compute_next_run
@@ -78,14 +78,10 @@ def _task_to_checkin_dict(t: ScheduledTask) -> dict:
def setup_assistant_routes(task_scheduler) -> APIRouter:
router = APIRouter(
prefix="/api/assistant",
tags=["assistant"],
dependencies=[Depends(require_interactive_request)],
)
router = APIRouter(prefix="/api/assistant", tags=["assistant"])
def _owner(request: Request) -> str:
owner = require_interactive_request(request)
owner = get_current_user(request)
if not owner:
raise HTTPException(status_code=401, detail="Not authenticated")
return owner
@@ -94,12 +90,11 @@ def setup_assistant_routes(task_scheduler) -> APIRouter:
# check-in tasks seeded. Hitting any /assistant route under one of these
# used to seed a full CrewMember + Morning/Midday/Evening tasks under that
# owner, which then double-fired alongside the real user's check-ins.
# REQUEST_SENTINEL_OWNERS covers request-only identities; Default/Local is a
# reserved login name but remains a valid storage owner.
# RESERVED_USERNAMES covers the same set; the `not owner` guard handles "".
async def _get_or_create(owner: str) -> CrewMember:
"""Return the per-owner assistant CrewMember, creating it on demand."""
if not owner or owner in REQUEST_SENTINEL_OWNERS:
if not owner or owner in RESERVED_USERNAMES:
raise HTTPException(status_code=400, detail=f"Cannot seed assistant for {owner!r}")
db = SessionLocal()
try:
+1 -28
View File
@@ -86,33 +86,6 @@ class SetOpenRegistrationRequest(BaseModel):
SESSION_COOKIE = "odysseus_session"
def _secure_cookie(request: Request) -> bool:
"""Decide the ``Secure`` attribute of the session cookie.
``SECURE_COOKIES`` stays authoritative when it holds an explicit value:
``true`` always marks the cookie Secure (the documented knob for a TLS
proxy), ``false`` never does, which is the escape hatch for an install
that still answers on plain HTTP alongside HTTPS. Anything else
unset, or the present-but-empty value docker-compose injects for a
variable the host has not defined derives it from the request, so an
HTTPS login gets a Secure cookie without any configuration.
Either the connection scheme or ``X-Forwarded-Proto`` saying https is
enough, which is the same test ``core/middleware.py`` applies before it
sends HSTS. Uvicorn's proxy-headers middleware already folds that header
into the scheme for the proxies it trusts, so reading it here only adds
the case of a terminator that is not on a trusted address; the cost is
that a client talking to the app directly can set the header and lock
its own session out over plain HTTP.
"""
configured = os.getenv("SECURE_COOKIES", "").strip().lower()
if configured in ("true", "false"):
return configured == "true"
# A chained proxy sends a list — the client-facing hop comes first.
forwarded_proto = request.headers.get("x-forwarded-proto", "").split(",")[0]
return request.url.scheme == "https" or forwarded_proto.strip().lower() == "https"
def setup_auth_routes(auth_manager: AuthManager) -> APIRouter:
router = APIRouter(prefix="/api/auth", tags=["auth"])
@@ -186,7 +159,7 @@ def setup_auth_routes(auth_manager: AuthManager) -> APIRouter:
value=token,
httponly=True,
samesite="lax",
secure=_secure_cookie(request),
secure=os.getenv("SECURE_COOKIES", "false").lower() == "true",
path="/",
)
if body.remember:
+26 -269
View File
@@ -16,13 +16,7 @@ from src.llm_core import normalize_model_id
from src.endpoint_resolver import normalize_base
from src.context_compactor import maybe_compact, trim_for_context
from src.model_context import estimate_tokens, get_context_length
from src.auth_helpers import (
RequestCapability,
effective_user,
is_bearer_principal,
request_capability as build_request_capability,
)
from src.tool_approval_scopes import CHAT_SESSION_APPROVAL_CONTEXT_MARKER
from src.auth_helpers import effective_user
from src.prompt_security import untrusted_context_message
from src.attachment_refs import attachment_ref
from routes.prefs_routes import _load_for_user as load_prefs_for_user
@@ -110,33 +104,6 @@ def _append_incognito_message(session_id: str, role: str, content: Any, metadata
bundle["updated_at"] = time.time()
def _history_for_request_capability(sess, capability: RequestCapability) -> list[dict[str, Any]]:
"""Project persisted history without interactive approval authority for bearers."""
history = sess.get_context_messages()
if not capability.is_bearer:
return history
# Session.get_context_messages() derives the marker only from the separate
# server-owned grant table. A pure bearer chat may still read its owner's
# ordinary transcript, but it must not receive even that interactive
# approval signal as model context or future tool authority.
projected = []
for item in history or []:
if not isinstance(item, dict):
continue
message = dict(item)
metadata = message.get("metadata")
if isinstance(metadata, dict) and CHAT_SESSION_APPROVAL_CONTEXT_MARKER in metadata:
metadata = dict(metadata)
metadata.pop(CHAT_SESSION_APPROVAL_CONTEXT_MARKER, None)
if metadata:
message["metadata"] = metadata
else:
message.pop("metadata", None)
projected.append(message)
return projected
# ── Data containers ────────────────────────────────────────────────────── #
@dataclass
@@ -205,11 +172,6 @@ def _allowed_models_from_privileges(privs: dict) -> Optional[frozenset[str]]:
def _allowed_models_for_request(request) -> Optional[frozenset[str]]:
"""Return the caller's model allowlist, or ``None`` when unrestricted."""
# ``effective_user`` is an attribution/storage identity for bearers, not a
# browser privilege principal. In particular, an admin-owned token must
# not inherit the owner's ADMIN_PRIVILEGES map through this lookup.
if is_bearer_principal(request):
return None
try:
user = effective_user(request)
except Exception:
@@ -232,12 +194,6 @@ def _enforce_chat_privileges(request, sess) -> None:
(single-user mode). Admins receive ADMIN_PRIVILEGES from get_privileges,
which means unrestricted allowed_models / zero cap -> no-op for them.
"""
# Bearer authority is defined by the token scope at the route boundary.
# Do not turn its owner attribution back into a browser privilege lookup;
# that would make an admin-owned token inherit the admin model/cap policy.
if is_bearer_principal(request):
return
try:
user = effective_user(request)
except Exception:
@@ -450,13 +406,7 @@ def build_uploaded_file_manifest(att_ids: list, upload_handler, owner: Optional[
return manifest
def add_user_message(
sess,
chat_handler,
preprocessed: PreprocessedMessage,
incognito: bool = False,
capability: RequestCapability | None = None,
):
def add_user_message(sess, chat_handler, preprocessed: PreprocessedMessage, incognito: bool = False):
"""Add user message to session history and update session name.
Incognito messages must not mutate persistent session history, even in
memory, because a later normal turn can persist the same session object."""
@@ -464,23 +414,11 @@ def add_user_message(
return
user_meta = {"attachments": preprocessed.attachment_meta} if preprocessed.attachment_meta else None
sess.add_message(ChatMessage("user", preprocessed.user_content, metadata=user_meta))
if capability is None or capability.allow_auto_naming:
chat_handler.update_session_name_if_needed(sess, preprocessed.text_for_context)
chat_handler.update_session_name_if_needed(sess, preprocessed.text_for_context)
def fire_message_event(
request,
webhook_manager,
session_id: str,
sess,
message: str,
compare_mode: bool = False,
capability: RequestCapability | None = None,
):
def fire_message_event(request, webhook_manager, session_id: str, sess, message: str, compare_mode: bool = False):
"""Fire webhook and event_bus events for a new user message."""
capability = capability or build_request_capability(request)
if not capability.allow_message_events:
return
if webhook_manager and not compare_mode:
webhook_manager.fire_and_forget("chat.message", {
"session_id": session_id, "model": sess.model, "message": message[:2000],
@@ -514,37 +452,16 @@ def _has_auth_keys(headers) -> bool:
)
def resolve_session_auth(
sess,
session_id: str,
owner: Optional[str] = None,
*,
allow_live_probes: bool = True,
):
def resolve_session_auth(sess, session_id: str, owner: Optional[str] = None):
"""Ensure session has auth headers — resolve from endpoint DB if missing."""
if not allow_live_probes:
# Bearer chat is cache-only and request-local. Do not resolve provider
# credentials or write recovered headers/session state in this mode.
return
try:
from src.chatgpt_subscription import is_chatgpt_subscription_base
is_chatgpt_subscription = is_chatgpt_subscription_base(getattr(sess, "endpoint_url", "") or "")
except Exception:
is_chatgpt_subscription = False
provenance = (getattr(sess, "endpoint_provenance", None) or "").strip().lower()
endpoint_id = (getattr(sess, "model_endpoint_id", None) or "").strip()
has_auth = _has_auth_keys(sess.headers)
if has_auth and not is_chatgpt_subscription and provenance != "registered":
if has_auth and not is_chatgpt_subscription:
return
if provenance == "direct":
# A direct API-key session owns its request headers; a same-URL
# registered endpoint must never supply another user's credentials by
# coincidence.
return
if provenance == "registered":
# Do not carry a previously persisted key through endpoint rotation or
# an unavailable endpoint while attempting exact re-resolution below.
sess.headers = {}
try:
from src.endpoint_resolver import build_headers, resolve_endpoint_runtime
@@ -560,10 +477,6 @@ def resolve_session_auth(
# with similar endpoint URLs can borrow each other's API key.
from src.auth_helpers import owner_filter
q = owner_filter(q, ModelEndpoint, owner)
if provenance == "registered":
if not endpoint_id:
return
q = q.filter(ModelEndpoint.id == endpoint_id)
for ep in q.all():
if not _session_url_matches_endpoint(target_url, ep.base_url or ""):
continue
@@ -619,7 +532,7 @@ def _match_cached_model_id(requested: str, models) -> Optional[str]:
def _normalize_model_id_from_cache(sess) -> Optional[str]:
"""Use stored ``cached_models``/pinned IDs before a live /models probe."""
"""Use stored endpoint model IDs before falling back to a live /models probe."""
endpoint_url = getattr(sess, "endpoint_url", "") or ""
requested = getattr(sess, "model", "") or ""
if not endpoint_url or not requested:
@@ -632,12 +545,6 @@ def _normalize_model_id_from_cache(sess) -> Optional[str]:
if not session_base:
return None
provenance = getattr(sess, "endpoint_provenance", None)
endpoint_id = (getattr(sess, "model_endpoint_id", None) or "").strip()
if provenance == "direct":
# Direct API-key sessions are intentionally outside the registered
# endpoint inventory. Never borrow a same-URL endpoint's model list.
return None
db = SessionLocal()
try:
q = db.query(ModelEndpoint).filter(ModelEndpoint.is_enabled == True)
@@ -645,10 +552,6 @@ def _normalize_model_id_from_cache(sess) -> Optional[str]:
if owner:
from src.auth_helpers import owner_filter
q = owner_filter(q, ModelEndpoint, owner)
if provenance == "registered":
if not endpoint_id:
return None
q = q.filter(ModelEndpoint.id == endpoint_id)
endpoints = q.all()
for ep in endpoints:
try:
@@ -657,12 +560,11 @@ def _normalize_model_id_from_cache(sess) -> Optional[str]:
except Exception:
continue
raw_models = getattr(ep, "cached_models", None)
if not raw_models:
continue
try:
from routes.model_routes import _effective_endpoint_kind, _picker_models_for_endpoint
base_url = getattr(ep, "base_url", "") or ""
kind = _effective_endpoint_kind(ep, base_url)
models, _ = _picker_models_for_endpoint(ep, base_url, kind)
models = json.loads(raw_models) if isinstance(raw_models, str) else raw_models
except Exception:
continue
@@ -677,91 +579,6 @@ def _normalize_model_id_from_cache(sess) -> Optional[str]:
return None
def _validate_bearer_session_model(sess, owner: str | None = None) -> Optional[str]:
"""Enforce endpoint-picker authority for a bearer session model.
Direct API-key sessions intentionally have no ``ModelEndpoint`` row and
retain their documented compatibility behavior. Registered endpoint
sessions, including provider-auth-backed rows, must use the visible
server-owned inventory and never trigger a live provider lookup here.
"""
# Lightweight in-memory test doubles from older route tests do not carry
# durable provenance fields. They cannot represent a persisted bearer
# session; retain their historical seam while every SessionManager-loaded
# object (which always has both fields) takes the fail-closed path below.
if not hasattr(sess, "endpoint_provenance") and not hasattr(sess, "model_endpoint_id"):
return None
provenance = (getattr(sess, "endpoint_provenance", None) or "").strip().lower()
endpoint_id = (getattr(sess, "model_endpoint_id", None) or "").strip()
if provenance == "direct":
if endpoint_id:
raise HTTPException(400, "Direct API-key sessions cannot carry a registered endpoint")
# No registered ModelEndpoint row is consulted for this documented
# compatibility path.
return None
if provenance != "registered":
raise HTTPException(400, "Session endpoint provenance is unavailable")
if not owner:
raise HTTPException(403, "A bearer session owner is required")
if not endpoint_id:
raise HTTPException(400, "Registered session endpoint identity is unavailable")
endpoint_url = (getattr(sess, "endpoint_url", "") or "").strip()
requested = (getattr(sess, "model", "") or "").strip()
if not endpoint_url:
raise HTTPException(400, "Registered session endpoint is not configured")
if not requested:
raise HTTPException(400, "Registered session model is not configured")
db = SessionLocal()
try:
from src.auth_helpers import owner_filter
q = db.query(ModelEndpoint).filter(
ModelEndpoint.id == endpoint_id,
ModelEndpoint.is_enabled == True,
)
q = owner_filter(q, ModelEndpoint, owner)
endpoints = q.all()
if len(endpoints) != 1:
# This covers disabled/deleted/owner-mismatched rows as well as
# malformed duplicate results. Do not fall back to URL matching.
raise HTTPException(400, "Registered model endpoint is no longer available")
ep = endpoints[0]
if not _session_url_matches_endpoint(endpoint_url, getattr(ep, "base_url", "") or ""):
raise HTTPException(400, "Session endpoint provenance is stale")
from routes.model_routes import _validate_bearer_model_selection
validated = _validate_bearer_model_selection(ep, requested)
# A session may outlive an endpoint-key rotation. For bearer calls,
# use the current exact-endpoint credentials and never trust a stale
# persisted Authorization header. Provider-auth credentials are
# owner-scoped, request-local, and cache-only in this boundary.
try:
from src.endpoint_resolver import build_headers, resolve_endpoint_runtime
base, api_key = resolve_endpoint_runtime(
ep,
owner=owner,
allow_live_probes=False,
)
sess.headers = build_headers(api_key, base)
except Exception as exc:
logger.warning("Could not refresh bearer session endpoint auth: %s", exc)
sess.headers = {}
if getattr(ep, "provider_auth_id", None):
raise HTTPException(401, "Registered provider credentials are unavailable") from exc
raise HTTPException(400, "Registered endpoint credentials are unavailable") from exc
sess.model = validated
return validated
finally:
db.close()
def _session_is_research_spinoff(sess) -> bool:
"""True if this session was created via research "Discuss" spin-off.
@@ -807,17 +624,12 @@ async def build_chat_context(
agent_mode: bool = False,
allow_tool_preprocessing: bool = True,
defer_context_shaping: bool = False,
continuation_context_message: str | None = None,
persist_user_message: bool = True,
capability: RequestCapability | None = None,
) -> ChatContext:
"""Build the full context (preface + messages) for an LLM call.
This is the shared logic between /chat and /chat_stream preset extraction,
message preprocessing, memory/RAG/web injection, compaction, normalization.
"""
capability = capability or build_request_capability(request)
# Preset
preset = extract_preset(chat_handler, preset_id)
@@ -835,29 +647,15 @@ async def build_chat_context(
# Add user message to history. Nobody/incognito uses a request-local
# transcript store instead of session history so stale saved chats cannot
# bleed into context and the turn is not persisted.
if persist_user_message and incognito:
if incognito:
user_meta = {"attachments": preprocessed.attachment_meta} if preprocessed.attachment_meta else None
_append_incognito_message(session_id, "user", preprocessed.user_content, user_meta)
elif persist_user_message:
add_user_message(
sess,
chat_handler,
preprocessed,
incognito=False,
capability=capability,
)
else:
add_user_message(sess, chat_handler, preprocessed, incognito=False)
# Fire events
if persist_user_message and not incognito:
fire_message_event(
request,
webhook_manager,
session_id,
sess,
message,
compare_mode,
capability=capability,
)
if not incognito:
fire_message_event(request, webhook_manager, session_id, sess, message, compare_mode)
# Resolve owner-scoped prefs/context. Browser requests keep the cookie user;
# bearer-token chat requests use the token owner instead of the "api" sentinel.
@@ -868,12 +666,7 @@ async def build_chat_context(
getattr(chat_handler, "upload_handler", None),
getattr(sess, "owner", None),
)
context_message = (
str(continuation_context_message).strip()
if continuation_context_message
else message
)
casual_low_signal = _is_casual_low_signal(context_message)
casual_low_signal = _is_casual_low_signal(message)
# Memory enabled?
mem_enabled = not incognito and not no_memory and uprefs.get("memory_enabled", True)
@@ -910,15 +703,7 @@ async def build_chat_context(
# Build context preface
# The stream path uses enhanced_message (with CoT/preprocessing applied),
# the sync path uses text_for_context.
_ctx_msg = (
context_message
if continuation_context_message
else (
preprocessed.enhanced_message
if use_enhanced_message
else preprocessed.text_for_context
)
)
_ctx_msg = preprocessed.enhanced_message if use_enhanced_message else preprocessed.text_for_context
_preface_kwargs = dict(
message=_ctx_msg,
session=sess,
@@ -931,7 +716,6 @@ async def build_chat_context(
agent_mode=agent_mode,
incognito=incognito,
use_skills=skills_enabled,
allow_tool_preprocessing=allow_tool_preprocessing,
)
if use_rag is not None or is_research_spinoff or casual_low_signal:
_preface_kwargs["use_rag"] = use_rag_val
@@ -950,27 +734,18 @@ async def build_chat_context(
# Normalize model ID. Prefer cached endpoint models so group chat does not
# re-hit slow local /models endpoints on every participant turn.
norm = _normalize_model_id_from_cache(sess)
# Model normalization falls back to a live /models or /tags request on a
# cache miss. A bearer chat request may use the stored model as-is, but it
# must not implicitly refresh an endpoint catalogue while building context.
if norm is None and capability.allow_live_probes:
norm = normalize_model_id(
sess.endpoint_url,
sess.model,
owner=getattr(sess, "owner", None),
)
norm = _normalize_model_id_from_cache(sess) or normalize_model_id(
sess.endpoint_url,
sess.model,
owner=getattr(sess, "owner", None),
)
if norm:
sess.model = norm
# Build messages. In Nobody/incognito mode, never read saved session
# history: the session id may be a temporary wrapper or, in buggy clients, a
# stale normal session id. Only the ephemeral incognito transcript is safe.
messages = preface + (
_incognito_messages(session_id)
if incognito
else _history_for_request_capability(sess, capability)
)
messages = preface + (_incognito_messages(session_id) if incognito else sess.get_context_messages())
# Current date/time — injected as a standalone *user*-role context message
# placed immediately before the latest user turn, NOT folded into the
@@ -998,22 +773,11 @@ async def build_chat_context(
# session history before we know which route can answer and would make a
# later larger-context candidate unable to recover discarded history.
if defer_context_shaping:
context_kwargs = {}
if not capability.allow_live_probes:
context_kwargs["allow_live_probes"] = False
context_length = get_context_length(sess.endpoint_url, sess.model, **context_kwargs)
context_length = get_context_length(sess.endpoint_url, sess.model)
was_compacted = False
else:
compact_kwargs = {"owner": user}
if not capability.allow_live_probes:
compact_kwargs["allow_live_probes"] = False
messages, context_length, was_compacted = await maybe_compact(
sess,
sess.endpoint_url,
sess.model,
messages,
sess.headers,
**compact_kwargs,
sess, sess.endpoint_url, sess.model, messages, sess.headers, owner=user,
)
_before_trim_messages = len(messages)
_before_trim_tokens = estimate_tokens(messages)
@@ -1393,7 +1157,6 @@ def run_post_response_tasks(
owner: str = None,
extract_skills: bool = True,
allow_background_extraction: bool = True,
capability: RequestCapability | None = None,
):
"""Fire background tasks after a completed response: memory extraction, webhooks, auto-name, skill extraction.
@@ -1409,12 +1172,6 @@ def run_post_response_tasks(
``_queue_background_extraction`` keeps them from overlapping the *next*
turn's request too.
"""
if capability is not None and not capability.allow_deferred_work:
# Pure bearer chat is intentionally synchronous and request-bound.
# Do not schedule extraction, teacher/model work, callbacks, or
# auto-naming after the authorized request has returned/disconnected.
return
_extraction_jobs: list = []
# Memory extraction — only every 4th message pair to avoid excess LLM calls
+96 -572
View File
File diff suppressed because it is too large Load Diff
+41 -85
View File
@@ -1,9 +1,8 @@
"""Codex integration routes.
These are small HTTP surfaces intended for the Codex plugin/MCP bridge. They
reuse existing Odysseus helpers. Owner-scoped data operations support bearer
principals with the matching token scope; the Cookbook/plugin host-control
plane remains interactive-only.
reuse existing Odysseus helpers and enforce API-token scopes before touching
user data.
"""
import asyncio
@@ -13,16 +12,11 @@ from io import BytesIO
from pathlib import Path
from typing import Any
from fastapi import APIRouter, BackgroundTasks, Body, Depends, HTTPException, Request
from fastapi import APIRouter, BackgroundTasks, Body, HTTPException, Request
from fastapi.responses import StreamingResponse
from core.middleware import require_admin
from src.auth_helpers import (
require_api_token_owner,
require_authenticated_request,
require_non_bearer_request,
require_user,
)
from src.auth_helpers import require_authenticated_request, require_user
from src.tool_implementations import do_manage_notes
from src.constants import COOKBOOK_STATE_FILE
from routes._validators import validate_remote_host, validate_ssh_port
@@ -67,40 +61,9 @@ async def _as_owner(request: Request, owner: str, fn, *args, **kwargs):
"""Run an existing route handler with request.state.current_user temporarily
set to ``owner`` so its internal get_current_user/require_user calls see
the scope-gated owner (not the "api" pseudo-user the bearer middleware sets).
Temporarily hide the bearer header as well: nested legacy handlers classify
the raw header independently of ``request.state.api_token``. Restore every
request value when done. Works for sync and async handlers."""
Restores the original value when done. Works for sync and async handlers."""
orig = getattr(request.state, "current_user", None)
orig_api_token = getattr(request.state, "api_token", None)
missing = object()
scope = getattr(request, "scope", None)
original_scope_headers = missing
original_cached_headers = missing
original_mapping_headers = missing
if isinstance(scope, dict) and "headers" in scope:
original_scope_headers = scope["headers"]
scope["headers"] = [
(name, value)
for name, value in (original_scope_headers or [])
if not (
(isinstance(name, bytes) and name.lower() == b"authorization")
or (isinstance(name, str) and name.casefold() == "authorization")
)
]
request_dict = getattr(request, "__dict__", {})
if "_headers" in request_dict:
original_cached_headers = request_dict["_headers"]
request_dict.pop("_headers", None)
else:
current_headers = getattr(request, "headers", missing)
if isinstance(current_headers, dict):
original_mapping_headers = current_headers
request.headers = {
name: value
for name, value in current_headers.items()
if str(name).casefold() != "authorization"
}
request.state.current_user = owner
request.state.api_token = False
try:
@@ -117,49 +80,46 @@ async def _as_owner(request: Request, owner: str, fn, *args, **kwargs):
pass
else:
request.state.api_token = orig_api_token
if original_scope_headers is not missing:
scope["headers"] = original_scope_headers
request_dict = getattr(request, "__dict__", {})
request_dict.pop("_headers", None)
if original_cached_headers is not missing:
request_dict["_headers"] = original_cached_headers
if original_mapping_headers is not missing:
request.headers = original_mapping_headers
def _scope_owner(request: Request, allowed: set[str]) -> str:
"""Return the data owner if the caller is allowed for this Codex action."""
if getattr(request.state, "api_token", False) is True:
if getattr(request.state, "api_token", False):
scopes = set(getattr(request.state, "api_token_scopes", []) or [])
if not scopes.intersection(allowed):
required = " or ".join(sorted(allowed))
raise HTTPException(403, f"API token missing required scope: {required}")
return require_api_token_owner(request)
owner = getattr(request.state, "api_token_owner", None)
if not owner:
raise HTTPException(403, "API token has no owner")
return owner
return require_user(request)
def _scope_owner_all(request: Request, required: set[str]) -> str:
"""Return owner only when an API token has every required scope."""
if getattr(request.state, "api_token", False) is True:
if getattr(request.state, "api_token", False):
scopes = set(getattr(request.state, "api_token_scopes", []) or [])
missing = required - scopes
if missing:
raise HTTPException(403, f"API token missing required scope: {' and '.join(sorted(missing))}")
return require_api_token_owner(request)
owner = getattr(request.state, "api_token_owner", None)
if not owner:
raise HTTPException(403, "API token has no owner")
return owner
return require_user(request)
def _require_cookbook_scope(request: Request, allowed: set[str]) -> str:
"""Authorize a Codex cookbook route.
Bearer callers are rejected by the host-control boundary regardless of
legacy scope labels. Cookie-session callers additionally require admin
privileges because cookbook surfaces expose host topology, task logs, tmux
For API-token callers, enforce the given scope set.
For cookie-session callers, additionally require admin privileges
because cookbook surfaces expose host topology, task logs, tmux
commands, and model-serving controls.
"""
require_non_bearer_request(request)
owner = _scope_owner(request, allowed)
if getattr(request.state, "api_token", False) is not True:
if not getattr(request.state, "api_token", False):
require_admin(request)
return owner
@@ -191,10 +151,7 @@ def setup_codex_routes(
calendar_router: APIRouter | None = None,
document_router: APIRouter | None = None,
) -> APIRouter:
router = APIRouter(
prefix="/api/codex",
tags=["codex"],
)
router = APIRouter(prefix="/api/codex", tags=["codex"])
email_list_endpoint = _find_endpoint(email_router, "GET", "/api/email/list")
email_read_endpoint = _find_endpoint(email_router, "GET", "/api/email/read/{uid}")
email_send_endpoint = _find_endpoint(email_router, "POST", "/api/email/send")
@@ -210,7 +167,7 @@ def setup_codex_routes(
@router.get("/capabilities")
def capabilities(request: Request):
token_scopes = set(getattr(request.state, "api_token_scopes", []) or [])
has_token = getattr(request.state, "api_token", False) is True
has_token = bool(getattr(request.state, "api_token", False))
def scoped(allowed):
return bool(token_scopes.intersection(allowed)) if has_token else True
return {
@@ -258,9 +215,8 @@ def setup_codex_routes(
},
}
@router.get("/plugin.zip", dependencies=[Depends(require_non_bearer_request)])
@router.get("/plugin.zip")
def plugin_zip(request: Request):
require_non_bearer_request(request)
require_authenticated_request(request)
root = Path(__file__).resolve().parent.parent / "integrations" / "codex"
if not root.exists():
@@ -557,10 +513,15 @@ def setup_codex_routes(
return await _as_owner(request, owner, documents_create_endpoint, request, req)
# ── Cookbook surface ──
# These handlers retain their legacy scope constants for compatibility
# with callers and tests, but the bridge is an interactive-only
# host-control plane. Bearer principals are rejected before any task-list,
# tmux-output, launch, stop, or model-serving operation.
# Lets the agent run the same launch / monitor / kill loop the user
# would do by hand in the Cookbook UI: read the current task list +
# tmux output, launch a serve task, stop one. Two scopes:
# cookbook:read — list tasks + tail output + list servers
# cookbook:launch — also start/stop serves (host shell exec)
# `cookbook:launch` is genuinely powerful: /api/model/serve runs SSH'd
# commands on the user's hosts. The existing _validate_serve_cmd
# allowlist (vllm/python3/sglang/llama-server/etc., no shell metachars)
# keeps the agent inside the same sandbox the UI uses.
async def _run_shell(cmd: str, timeout: float = 15.0) -> dict:
"""Run a shell command, return {exit_code, stdout, stderr}."""
@@ -604,14 +565,14 @@ def setup_codex_routes(
if k not in ("hf_token", "_secrets")}
return clean
@router.get("/cookbook/tasks", dependencies=[Depends(require_non_bearer_request)])
@router.get("/cookbook/tasks")
async def codex_cookbook_tasks(request: Request):
_require_cookbook_scope(request, COOKBOOK_READ_SCOPES)
state = _read_cookbook_state()
tasks = state.get("tasks") or []
return {"tasks": [_redact_task(t) for t in tasks]}
@router.get("/cookbook/servers", dependencies=[Depends(require_non_bearer_request)])
@router.get("/cookbook/servers")
async def codex_cookbook_servers(request: Request):
_require_cookbook_scope(request, COOKBOOK_READ_SCOPES)
state = _read_cookbook_state()
@@ -630,7 +591,7 @@ def setup_codex_routes(
})
return {"servers": cleaned}
@router.get("/cookbook/output/{session_id}", dependencies=[Depends(require_non_bearer_request)])
@router.get("/cookbook/output/{session_id}")
async def codex_cookbook_output(request: Request, session_id: str, tail: int = 400):
_require_cookbook_scope(request, COOKBOOK_READ_SCOPES)
# Defensive: session_id must be the tmux-style id we issue
@@ -672,7 +633,7 @@ def setup_codex_routes(
"task": _redact_task(task),
}
@router.post("/cookbook/serve", dependencies=[Depends(require_non_bearer_request)])
@router.post("/cookbook/serve")
async def codex_cookbook_serve(request: Request, body: dict[str, Any] = Body(default_factory=dict)):
_require_cookbook_scope(request, COOKBOOK_LAUNCH_SCOPES)
# Wraps /api/model/serve with the SAME validation the UI uses.
@@ -711,7 +672,7 @@ def setup_codex_routes(
raise HTTPException(503, "model serve endpoint unavailable")
return await serve_endpoint(request, req)
@router.post("/cookbook/stop/{session_id}", dependencies=[Depends(require_non_bearer_request)])
@router.post("/cookbook/stop/{session_id}")
async def codex_cookbook_stop(request: Request, session_id: str):
_require_cookbook_scope(request, COOKBOOK_LAUNCH_SCOPES)
import re as _re
@@ -728,7 +689,7 @@ def setup_codex_routes(
result = await _run_shell(cmd, timeout=10)
return {"session_id": session_id, "exit_code": result.get("exit_code"), "host": host or "local"}
@router.get("/cookbook/cached", dependencies=[Depends(require_non_bearer_request)])
@router.get("/cookbook/cached")
async def codex_cookbook_cached(request: Request, host: str | None = None):
"""List cached models on a configured server (or local if host is omitted).
Mirrors `list_cached_models` from the chat agent so external agents have
@@ -790,7 +751,7 @@ def setup_codex_routes(
platform=params.get("platform") or None,
)
@router.get("/cookbook/presets", dependencies=[Depends(require_non_bearer_request)])
@router.get("/cookbook/presets")
async def codex_cookbook_presets(request: Request):
"""List saved serve presets (model + host + port + launch cmd).
Counterpart to `list_serve_presets`. Use BEFORE composing a `serve`
@@ -811,7 +772,7 @@ def setup_codex_routes(
})
return {"presets": out, "default_host": (state.get("env") or {}).get("defaultServer", "")}
@router.post("/cookbook/preset/{name}", dependencies=[Depends(require_non_bearer_request)])
@router.post("/cookbook/preset/{name}")
async def codex_cookbook_serve_preset(request: Request, name: str):
"""Launch a saved preset by name. Reuses the working cmd + host the
user already saved, avoiding the cmd-allowlist trial-and-error loop."""
@@ -861,7 +822,7 @@ def setup_codex_routes(
raise HTTPException(503, "model serve endpoint unavailable")
return await serve_endpoint(request, req)
@router.post("/cookbook/adopt", dependencies=[Depends(require_non_bearer_request)])
@router.post("/cookbook/adopt")
async def codex_cookbook_adopt(request: Request, body: dict[str, Any] = Body(default_factory=dict)):
"""Adopt an existing tmux session (one started via raw ssh+tmux) into
cookbook tracking. Needed when serve_model rejects a cmd and the
@@ -925,15 +886,10 @@ def setup_claude_routes() -> APIRouter:
this router only exists to deliver the skill zip via `/api/claude/plugin.zip`
so the user-facing setup commands stay in the Claude namespace.
"""
router = APIRouter(
prefix="/api/claude",
tags=["claude"],
dependencies=[Depends(require_non_bearer_request)],
)
router = APIRouter(prefix="/api/claude", tags=["claude"])
@router.get("/plugin.zip")
def plugin_zip(request: Request):
require_non_bearer_request(request)
require_authenticated_request(request)
# Only ship the skills/ subtree so extracting at ~/.claude/ doesn't dump
# README.md or other bundle metadata into the user's claude config dir.
+17 -87
View File
@@ -4,24 +4,19 @@ import json
import uuid
import random
from datetime import datetime
from fastapi import APIRouter, Depends, Form, HTTPException, Request
from fastapi import APIRouter, Form, HTTPException, Request
from typing import List
from pydantic import BaseModel
import logging
from core.database import Comparison, SessionLocal
from core.session_manager import SessionManager
from src.auth_helpers import effective_user, is_bearer_principal, require_chat_scope
from src.session_provenance import persist_session_endpoint_provenance
from src.auth_helpers import get_current_user
from routes.session_routes import _reject_raw_endpoint_url_for_non_admin
logger = logging.getLogger(__name__)
router = APIRouter(
prefix="/api/compare",
tags=["compare"],
dependencies=[Depends(require_chat_scope)],
)
router = APIRouter(prefix="/api/compare", tags=["compare"])
def _owned_endpoint_by_url(db, base_url, owner):
@@ -69,37 +64,6 @@ class RecordVoteRequest(BaseModel):
is_blind: bool = True
def _validate_bearer_compare_models(models, owner: str) -> list[str]:
"""Validate record-only comparison models against visible endpoint caches."""
from core.database import ModelEndpoint
from src.auth_helpers import owner_filter
from routes.model_routes import _validate_bearer_model_selection
db = SessionLocal()
try:
q = db.query(ModelEndpoint).filter(ModelEndpoint.is_enabled == True)
q = owner_filter(q, ModelEndpoint, owner)
endpoints = q.all()
selected = []
for requested in models:
matches = []
for ep in endpoints:
try:
matches.append(_validate_bearer_model_selection(ep, requested))
except HTTPException:
continue
unique = list(dict.fromkeys(matches))
if len(unique) != 1:
raise HTTPException(
400,
f"Model is not permitted by a visible server endpoint: {requested}",
)
selected.append(unique[0])
return selected
finally:
db.close()
def setup_compare_routes(session_manager: SessionManager):
"""Setup comparison routes."""
@@ -120,9 +84,7 @@ def setup_compare_routes(session_manager: SessionManager):
Returns the comparison ID and the two session IDs so the client
can fire two independent SSE streams to /api/chat_stream.
"""
require_chat_scope(request)
user = effective_user(request)
bearer = is_bearer_principal(request)
user = getattr(request.state, 'current_user', None)
comp_id = str(uuid.uuid4())
sid_a = str(uuid.uuid4())
sid_b = str(uuid.uuid4())
@@ -198,13 +160,6 @@ def setup_compare_routes(session_manager: SessionManager):
_reject_raw_endpoint_url_for_non_admin(
request, user, str(ep.id) if ep is not None else None, endpoint
)
selected_model = model
if bearer:
if ep is None:
raise HTTPException(403, "Choose a registered model endpoint")
from routes.model_routes import _validate_bearer_model_selection
selected_model = _validate_bearer_model_selection(ep, model)
# Bind the [CMP] session to the RESOLVED endpoint, not the raw
# caller-supplied string. When the URL matches a registered
# endpoint visible to the caller, use that row's own normalized
@@ -221,24 +176,15 @@ def setup_compare_routes(session_manager: SessionManager):
# `ep` is None (raw admin URL or no match), so a comparison can
# never inherit another user's key/headers.
headers = build_headers(ep.api_key, ep.base_url) if (ep and ep.api_key) else None
resolved.append(
(
sid,
selected_model,
session_endpoint_url,
headers,
str(ep.id) if ep is not None else None,
"registered" if ep is not None else None,
)
)
resolved.append((sid, model, session_endpoint_url, headers))
finally:
db.close()
# Both endpoints validated — only now create the ephemeral [CMP]
# sessions and copy any resolved headers.
for sid, model, session_endpoint_url, headers, endpoint_id, provenance in resolved:
for sid, model, session_endpoint_url, headers in resolved:
name = f"[CMP] {slot_name[sid]}" if blind else f"[CMP] {model.split('/')[-1]}"
comparison_session = session_manager.create_session(
session_manager.create_session(
session_id=sid,
name=name,
endpoint_url=session_endpoint_url,
@@ -246,14 +192,6 @@ def setup_compare_routes(session_manager: SessionManager):
rag=False,
owner=user,
)
if provenance in {"registered", "direct"}:
persist_session_endpoint_provenance(
session_manager,
sid,
comparison_session,
model_endpoint_id=endpoint_id,
endpoint_provenance=provenance,
)
if headers:
s = session_manager.sessions.get(sid)
if s:
@@ -265,8 +203,8 @@ def setup_compare_routes(session_manager: SessionManager):
comp = Comparison(
id=comp_id,
prompt=prompt,
model_a=resolved[0][1],
model_b=resolved[1][1],
model_a=model_a,
model_b=model_b,
# Record the URL the session actually dials. For URL callers this
# is their raw input; for id-only callers (empty endpoint_a/_b)
# fall back to the resolved endpoint URL so the column stays
@@ -303,8 +241,7 @@ def setup_compare_routes(session_manager: SessionManager):
winner: str = Form(...), # "left", "right", or "tie"
):
"""Record the user's vote and reveal model names if blind."""
require_chat_scope(request)
user = effective_user(request)
user = get_current_user(request)
db = SessionLocal()
try:
comp = db.query(Comparison).filter(Comparison.id == comp_id).first()
@@ -346,20 +283,15 @@ def setup_compare_routes(session_manager: SessionManager):
@router.post("/record")
def record_comparison(request: Request, body: RecordVoteRequest):
"""Lightweight endpoint to record a comparison vote from the frontend."""
require_chat_scope(request)
user = effective_user(request)
user = get_current_user(request)
comp_id = str(uuid.uuid4())
models = list(body.models or [])
if is_bearer_principal(request):
models = _validate_bearer_compare_models(models, user)
model_a = models[0] if len(models) > 0 else ""
model_b = models[1] if len(models) > 1 else ""
model_a = body.models[0] if len(body.models) > 0 else ""
model_b = body.models[1] if len(body.models) > 1 else ""
# For N>2 models, store the full list as JSON in blind_mapping
if len(models) > 2:
blind_mapping = json.dumps({"models": models})
if len(body.models) > 2:
blind_mapping = json.dumps({"models": body.models})
else:
blind_mapping = None
@@ -388,8 +320,7 @@ def setup_compare_routes(session_manager: SessionManager):
@router.get("/history")
def list_comparisons(request: Request):
"""List past comparisons."""
require_chat_scope(request)
user = effective_user(request)
user = get_current_user(request)
db = SessionLocal()
try:
q = db.query(Comparison)
@@ -415,8 +346,7 @@ def setup_compare_routes(session_manager: SessionManager):
@router.delete("/{comp_id}")
def delete_comparison(request: Request, comp_id: str):
"""Delete a comparison and its ephemeral sessions."""
require_chat_scope(request)
user = effective_user(request)
user = get_current_user(request)
db = SessionLocal()
try:
comp = db.query(Comparison).filter(Comparison.id == comp_id).first()
-35
View File
@@ -1204,41 +1204,6 @@ def _safe_env_prefix(ep: str | None) -> str | None:
return f'[ -f "{path}" ] && source "{path}" || true'
def _local_windows_bash_env_prefix(ep: str | None) -> str | None:
"""Convert a frontend PowerShell venv prefix for the local Git Bash runner."""
if not ep:
return ep
prefix = ep.strip()
if not prefix.startswith("&"):
return ep
raw_path = prefix[1:].lstrip()
if not raw_path:
return ep
if raw_path.startswith("'"):
if len(raw_path) < 2 or not raw_path.endswith("'"):
return ep
quoted_path = raw_path[1:-1]
if "'" in quoted_path.replace("''", ""):
return ep
path = quoted_path.replace("''", "'")
else:
path = raw_path.rstrip()
if "'" in path or '"' in path:
return ep
if any(c in path for c in "\r\n;&|`$<>"):
return ep
if not path.replace("\\", "/").casefold().endswith("/scripts/activate.ps1"):
return ep
bash_path = _git_bash_path(path)
if "\\" in bash_path:
return ep
bash_path = bash_path[: -len("Activate.ps1")] + "activate"
return "source " + shlex.quote(bash_path)
def _ssh_ps(host, script_path, port=None):
"""Build SSH command to run a PowerShell script on a Windows remote."""
pf = f"-p {port} " if port and port != "22" else ""
+3 -3
View File
@@ -50,7 +50,7 @@ from routes.cookbook_helpers import (
_SESSION_ID_RE, _validate_repo_id, _validate_serve_model_id, _validate_include, _validate_token,
_validate_local_dir, _validate_gpus, _shell_path,
_ps_squote, _bash_squote, _validate_serve_cmd, _parse_serve_phase, OLLAMA_MISSING_HINT,
_safe_env_prefix, _local_windows_bash_env_prefix, _local_tooling_path_export, _append_serve_preflight_exit_lines,
_safe_env_prefix, _local_tooling_path_export, _append_serve_preflight_exit_lines,
_append_serve_exit_code_lines, _append_llama_cpp_linux_accel_build_lines, _cached_model_scan_script,
load_stored_hf_token,
_append_vllm_linux_preflight_lines, _ollama_bind_from_cmd, _pip_install_fallback_chain,
@@ -1336,7 +1336,7 @@ def setup_cookbook_routes() -> APIRouter:
# Local: run hf download in the background (tmux on POSIX, a detached
# process + logfile on Windows where tmux doesn't exist).
if req.env_prefix:
lines.append(_safe_env_prefix(_local_windows_bash_env_prefix(req.env_prefix) if local_windows else req.env_prefix))
lines.append(_safe_env_prefix(req.env_prefix))
else:
lines.append("deactivate 2>/dev/null; hash -r")
# Show whether the HF token reached this run (masked) — tells a gated
@@ -2166,7 +2166,7 @@ def setup_cookbook_routes() -> APIRouter:
if req.gpus:
runner_lines.append(f"export CUDA_VISIBLE_DEVICES='{req.gpus}'")
if req.env_prefix:
runner_lines.append(_safe_env_prefix(_local_windows_bash_env_prefix(req.env_prefix) if local_windows else req.env_prefix))
runner_lines.append(_safe_env_prefix(req.env_prefix))
else:
runner_lines.append("deactivate 2>/dev/null; hash -r")
_append_venv_nvidia_library_path_lines(runner_lines, cmd=req.cmd)
+1 -10
View File
@@ -34,7 +34,7 @@ from fastapi import Query, HTTPException, Request
from pydantic import BaseModel
from typing import Optional, List
from src.auth_helpers import _auth_disabled, get_current_user, is_bearer_principal
from src.auth_helpers import _auth_disabled, get_current_user
from src.secret_storage import decrypt as _decrypt
logger = logging.getLogger(__name__)
@@ -420,15 +420,6 @@ def _require_auth(request: Request) -> str:
unconfigured mode are only honoured if they're coming from
localhost; everyone else gets 401.
"""
# The legacy email router uses one generic dependency for mailbox reads,
# drafts, AI helpers, and SMTP send. It has no per-route token-scope
# contract, so a bearer must not be allowed to enter it as the ``api``
# pseudo-user. Otherwise owner-scoped lookup can miss the token owner and
# fall through to process-wide legacy settings credentials below.
# Scope-aware integrations must use their dedicated route boundary.
if is_bearer_principal(request):
raise HTTPException(403, "API tokens must use a scope-aware email route")
u = get_current_user(request)
if u:
return u
+38 -92
View File
@@ -10,19 +10,11 @@ import uuid
from pathlib import Path
from typing import Dict, Any, Optional
from fastapi import APIRouter, Depends, HTTPException, Query, Request
from fastapi import APIRouter, HTTPException, Query, Request
from core.database import SessionLocal, GalleryImage, GalleryAlbum, ModelEndpoint
from core.database import Session as DbSession
from src.auth_helpers import (
effective_user,
get_current_user,
is_bearer_principal,
owner_filter,
require_chat_scope,
require_non_bearer_request,
require_privilege,
)
from src.auth_helpers import get_current_user, owner_filter, require_privilege
from src.upload_limits import (
read_upload_limited,
GALLERY_UPLOAD_MAX_BYTES,
@@ -41,13 +33,6 @@ _SAM_STATE: Dict[str, Any] = {}
_GROUNDING_STATE: Dict[str, Any] = {}
def _gallery_owner(request: Request) -> Optional[str]:
"""Use the token owner for bearer calls and preserve the legacy seam otherwise."""
if is_bearer_principal(request):
return effective_user(request)
return get_current_user(request)
def _b64_to_pil_image(image_b64: str, *, mode: str = "RGBA"):
if not image_b64:
raise HTTPException(400, "Missing image")
@@ -361,10 +346,7 @@ async def _fetch_result_image_b64(url: str) -> Optional[str]:
def setup_gallery_routes() -> APIRouter:
router = APIRouter(
tags=["gallery"],
dependencies=[Depends(require_chat_scope)],
)
router = APIRouter(tags=["gallery"])
# ---- POST /api/gallery/upload ----
@router.post("/api/gallery/upload")
@@ -378,7 +360,7 @@ def setup_gallery_routes() -> APIRouter:
if not file or not hasattr(file, 'filename'):
raise HTTPException(400, "No file provided")
user = _gallery_owner(request)
user = get_current_user(request)
album_id = form.get("album_id") or None
content = await read_upload_limited(file, GALLERY_UPLOAD_MAX_BYTES, "Gallery upload")
@@ -452,7 +434,7 @@ def setup_gallery_routes() -> APIRouter:
@router.post("/api/gallery/{image_id}/replace")
async def gallery_replace(request: Request, image_id: str):
"""Replace an existing gallery image file with a new one."""
user = _gallery_owner(request)
user = get_current_user(request)
db = SessionLocal()
try:
img = db.query(GalleryImage).filter(GalleryImage.id == image_id).first()
@@ -497,7 +479,7 @@ def setup_gallery_routes() -> APIRouter:
"""Rename a gallery photo. Stores the new name in the `prompt`
column (which serves as the user-facing label for uploaded
photos that have no AI prompt)."""
user = _gallery_owner(request)
user = get_current_user(request)
data = await request.json()
new_name = (data.get("name") or "").strip()
if not new_name:
@@ -534,7 +516,7 @@ def setup_gallery_routes() -> APIRouter:
if angle not in (90, -90, 180, 270):
raise HTTPException(400, "Angle must be 90, -90, 180, or 270")
user = _gallery_owner(request)
user = get_current_user(request)
db = SessionLocal()
try:
img = db.query(GalleryImage).filter(GalleryImage.id == image_id).first()
@@ -575,10 +557,7 @@ def setup_gallery_routes() -> APIRouter:
db.close()
# ---- POST /api/gallery/ai-upscale ----
@router.post(
"/api/gallery/ai-upscale",
dependencies=[Depends(require_non_bearer_request)],
)
@router.post("/api/gallery/ai-upscale")
async def gallery_ai_upscale(request: Request):
"""AI upscale using img2img with the diffusion server."""
import base64, httpx
@@ -622,10 +601,7 @@ def setup_gallery_routes() -> APIRouter:
return {"error": "Upscale request failed"}
# ---- POST /api/gallery/style-transfer ----
@router.post(
"/api/gallery/style-transfer",
dependencies=[Depends(require_non_bearer_request)],
)
@router.post("/api/gallery/style-transfer")
async def gallery_style_transfer(request: Request):
"""Style transfer using img2img with the diffusion server."""
import base64, httpx
@@ -675,7 +651,7 @@ def setup_gallery_routes() -> APIRouter:
@router.get("/api/gallery/tags")
async def gallery_tags(request: Request) -> Dict[str, Any]:
"""Return distinct tags across all active gallery images."""
user = _gallery_owner(request)
user = get_current_user(request)
db = SessionLocal()
try:
q = db.query(GalleryImage.tags).filter(
@@ -707,7 +683,7 @@ def setup_gallery_routes() -> APIRouter:
offset: int = Query(0, ge=0),
limit: int = Query(24, ge=1, le=100),
) -> Dict[str, Any]:
user = _gallery_owner(request)
user = get_current_user(request)
db = SessionLocal()
try:
# Distinct tags for filter UI
@@ -835,7 +811,7 @@ def setup_gallery_routes() -> APIRouter:
@router.get("/api/gallery/albums")
async def list_albums(request: Request):
user = _gallery_owner(request)
user = get_current_user(request)
db = SessionLocal()
try:
q = db.query(GalleryAlbum)
@@ -874,7 +850,7 @@ def setup_gallery_routes() -> APIRouter:
@router.post("/api/gallery/albums")
async def create_album(request: Request):
import uuid
user = _gallery_owner(request)
user = get_current_user(request)
data = await request.json()
name = (data.get("name") or "").strip()
if not name:
@@ -894,7 +870,7 @@ def setup_gallery_routes() -> APIRouter:
@router.get("/api/gallery/stats")
async def gallery_stats(request: Request):
user = _gallery_owner(request)
user = get_current_user(request)
db = SessionLocal()
try:
from sqlalchemy import func
@@ -918,16 +894,13 @@ def setup_gallery_routes() -> APIRouter:
finally:
db.close()
@router.post(
"/api/gallery/ai-tag-batch",
dependencies=[Depends(require_non_bearer_request)],
)
@router.post("/api/gallery/ai-tag-batch")
async def ai_tag_batch(
request: Request,
album_id: Optional[str] = Query(None),
limit: int = Query(200),
):
user = _gallery_owner(request)
user = get_current_user(request)
db = SessionLocal()
try:
q = db.query(GalleryImage).filter(
@@ -946,7 +919,7 @@ def setup_gallery_routes() -> APIRouter:
# ---- GET /api/gallery/{image_id} ----
@router.get("/api/gallery/{image_id}")
async def get_gallery_image(request: Request, image_id: str) -> Dict[str, Any]:
user = _gallery_owner(request)
user = get_current_user(request)
db = SessionLocal()
try:
row = (
@@ -967,7 +940,7 @@ def setup_gallery_routes() -> APIRouter:
# ---- PATCH /api/gallery/{image_id} ----
@router.patch("/api/gallery/{image_id}")
async def patch_gallery_image(request: Request, image_id: str, req: GalleryPatch) -> Dict[str, Any]:
user = _gallery_owner(request)
user = get_current_user(request)
db = SessionLocal()
try:
img = db.query(GalleryImage).filter(GalleryImage.id == image_id).first()
@@ -1019,7 +992,7 @@ def setup_gallery_routes() -> APIRouter:
# of a flood of individual downloads).
@router.post("/api/gallery/download-zip")
async def gallery_download_zip(request: Request):
user = _gallery_owner(request)
user = get_current_user(request)
if not user:
raise HTTPException(401, "Not authenticated")
try:
@@ -1074,7 +1047,7 @@ def setup_gallery_routes() -> APIRouter:
# AI-suggested values you never added.
@router.post("/api/gallery/clear-user-tags")
async def clear_gallery_user_tags(request: Request) -> Dict[str, Any]:
user = _gallery_owner(request)
user = get_current_user(request)
db = SessionLocal()
try:
q = db.query(GalleryImage).filter(GalleryImage.is_active == True)
@@ -1099,7 +1072,7 @@ def setup_gallery_routes() -> APIRouter:
# "woman" have leaked into the gallery and you want them gone.
@router.post("/api/gallery/clear-ai-tags")
async def clear_gallery_ai_tags(request: Request, image_id: Optional[str] = Query(None)) -> Dict[str, Any]:
user = _gallery_owner(request)
user = get_current_user(request)
db = SessionLocal()
try:
q = db.query(GalleryImage).filter(GalleryImage.is_active == True)
@@ -1126,7 +1099,7 @@ def setup_gallery_routes() -> APIRouter:
# Returns how many rows were touched + how many tags removed.
@router.post("/api/gallery/dedupe-tags")
async def dedupe_gallery_tags(request: Request) -> Dict[str, Any]:
user = _gallery_owner(request)
user = get_current_user(request)
db = SessionLocal()
try:
q = db.query(GalleryImage).filter(GalleryImage.is_active == True)
@@ -1162,7 +1135,7 @@ def setup_gallery_routes() -> APIRouter:
# ---- DELETE /api/gallery/{image_id} ----
@router.delete("/api/gallery/{image_id}")
async def delete_gallery_image(request: Request, image_id: str) -> Dict[str, str]:
user = _gallery_owner(request)
user = get_current_user(request)
db = SessionLocal()
try:
img = db.query(GalleryImage).filter(GalleryImage.id == image_id).first()
@@ -1281,10 +1254,7 @@ def setup_gallery_routes() -> APIRouter:
db.close()
# ---- POST /api/image/inpaint — proxy to diffusion server OR OpenAI ----
@router.post(
"/api/image/inpaint",
dependencies=[Depends(require_non_bearer_request)],
)
@router.post("/api/image/inpaint")
async def inpaint_proxy(request: Request):
"""Forward inpaint request. If the selected endpoint is OpenAI, re-shape
the request for /v1/images/edits (multipart, inverted mask). Otherwise
@@ -1542,10 +1512,7 @@ def setup_gallery_routes() -> APIRouter:
# scratch using the prompt", ignoring the source. Real img2img sends
# the image alongside a `strength` (denoising strength) and the model
# mixes that fraction of new noise into the existing pixels.
@router.post(
"/api/image/harmonize",
dependencies=[Depends(require_non_bearer_request)],
)
@router.post("/api/image/harmonize")
async def harmonize_image(request: Request):
"""Harmonize = img2img. The model preserves (1 - strength) of the
original and regenerates `strength` fraction. With strength ~0.4
@@ -1745,10 +1712,7 @@ def setup_gallery_routes() -> APIRouter:
"/v1/images/harmonize, /v1/images/img2img, /v1/images/variations, /sdapi/v1/img2img.")
# ---- POST /api/image/sharpen ----
@router.post(
"/api/image/sharpen",
dependencies=[Depends(require_non_bearer_request)],
)
@router.post("/api/image/sharpen")
async def sharpen_image(request: Request):
"""Apply unsharp-mask sharpening to an image."""
require_privilege(request, "can_generate_images")
@@ -1773,10 +1737,7 @@ def setup_gallery_routes() -> APIRouter:
# AI denoise via Real-ESRGAN with the realesr-general-x4v3 weights at
# outscale=1 + denoise_strength. Falls back to a "package missing"
# error so the client can prompt the user to install via Cookbook.
@router.post(
"/api/image/denoise",
dependencies=[Depends(require_non_bearer_request)],
)
@router.post("/api/image/denoise")
async def denoise_image(request: Request):
require_privilege(request, "can_generate_images")
body = await request.json()
@@ -1827,10 +1788,7 @@ def setup_gallery_routes() -> APIRouter:
# ---- POST /api/image/upscale-local ----
# Local Real-ESRGAN upscale (2× or 4×). Self-contained — no diffusion
# server required. Used by the editor's AI Upscale button.
@router.post(
"/api/image/upscale-local",
dependencies=[Depends(require_non_bearer_request)],
)
@router.post("/api/image/upscale-local")
async def upscale_image_local(request: Request):
require_privilege(request, "can_generate_images")
body = await request.json()
@@ -1876,10 +1834,7 @@ def setup_gallery_routes() -> APIRouter:
return {"error": "AI upscale failed"}
# ---- POST /api/image/remove-bg ----
@router.post(
"/api/image/mask",
dependencies=[Depends(require_non_bearer_request)],
)
@router.post("/api/image/mask")
async def smart_mask(request: Request):
"""Create a neutral segmentation mask from user-provided points or a box.
@@ -2005,10 +1960,7 @@ def setup_gallery_routes() -> APIRouter:
logger.exception("smart_mask failed")
raise HTTPException(500, f"SAM mask failed: {exc}") from exc
@router.post(
"/api/image/remove-bg",
dependencies=[Depends(require_non_bearer_request)],
)
@router.post("/api/image/remove-bg")
async def remove_background(request: Request):
"""Remove background from an image. If the client passes a `hint_mask`
(white-where-the-user-wants-the-subject PNG, same dims as the
@@ -2101,10 +2053,7 @@ def setup_gallery_routes() -> APIRouter:
return {"image": base64.b64encode(buf.getvalue()).decode()}
# ---- POST /api/image/enhance-face ----
@router.post(
"/api/image/enhance-face",
dependencies=[Depends(require_non_bearer_request)],
)
@router.post("/api/image/enhance-face")
async def enhance_face(request: Request):
"""Face/portrait enhancement. Uses GFPGAN if available, falls back to PIL."""
require_privilege(request, "can_generate_images")
@@ -2190,7 +2139,7 @@ def setup_gallery_routes() -> APIRouter:
@router.put("/api/gallery/albums/{album_id}")
async def update_album(request: Request, album_id: str):
user = _gallery_owner(request)
user = get_current_user(request)
data = await request.json()
db = SessionLocal()
try:
@@ -2211,7 +2160,7 @@ def setup_gallery_routes() -> APIRouter:
@router.delete("/api/gallery/albums/{album_id}")
async def delete_album(request: Request, album_id: str):
user = _gallery_owner(request)
user = get_current_user(request)
db = SessionLocal()
try:
album = _get_or_404_album(db, album_id, user)
@@ -2227,7 +2176,7 @@ def setup_gallery_routes() -> APIRouter:
@router.post("/api/gallery/albums/{album_id}/add")
async def add_to_album(request: Request, album_id: str):
user = _gallery_owner(request)
user = get_current_user(request)
data = await request.json()
ids = data.get("image_ids", [])
db = SessionLocal()
@@ -2245,7 +2194,7 @@ def setup_gallery_routes() -> APIRouter:
@router.post("/api/gallery/albums/{album_id}/remove")
async def remove_from_album(request: Request, album_id: str):
user = _gallery_owner(request)
user = get_current_user(request)
data = await request.json()
ids = data.get("image_ids", [])
db = SessionLocal()
@@ -2266,7 +2215,7 @@ def setup_gallery_routes() -> APIRouter:
@router.post("/api/gallery/{image_id}/favorite")
async def toggle_favorite(request: Request, image_id: str):
user = _gallery_owner(request)
user = get_current_user(request)
db = SessionLocal()
try:
img = _get_or_404_image(db, image_id, user)
@@ -2278,16 +2227,13 @@ def setup_gallery_routes() -> APIRouter:
# ---- AI auto-tag ----
@router.post(
"/api/gallery/{image_id}/ai-tag",
dependencies=[Depends(require_non_bearer_request)],
)
@router.post("/api/gallery/{image_id}/ai-tag")
async def ai_tag_image(request: Request, image_id: str):
"""Send image to vision model for auto-tagging."""
import base64, httpx
from pathlib import Path
user = _gallery_owner(request)
user = get_current_user(request)
db = SessionLocal()
try:
img = _get_or_404_image(db, image_id, user)
+54 -167
View File
@@ -6,31 +6,19 @@ import logging
import re
from typing import Dict, Any, Optional
from fastapi import APIRouter, Depends, Request, HTTPException
from fastapi import APIRouter, Request, HTTPException
from core.models import ChatMessage
from core.database import SessionLocal, ChatMessage as DbChatMessage, Session as DbSession
from src.auth_helpers import (
effective_user,
is_bearer_principal,
request_capability,
require_chat_scope,
)
from src.message_metadata import (
sanitize_client_message_metadata,
sanitize_projected_message_metadata,
normalize_client_message_role,
)
from src.auth_helpers import effective_user
from src.topic_analyzer import analyze_topics
from src.upload_handler import reserve_message_upload_references
from src.session_provenance import persist_session_endpoint_provenance
from routes.session_routes import (
_message_role,
_message_text,
_reject_compact_during_active_run,
_verify_session_owner,
)
from routes.chat_helpers import _validate_bearer_session_model
logger = logging.getLogger(__name__)
@@ -38,24 +26,6 @@ _HISTORY_INLINE_MEDIA_THRESHOLD = 200_000
_DATA_IMAGE_RE = re.compile(r"data:image/[^;,\"]+;base64,[A-Za-z0-9+/=\s]+")
def _metadata_dict(value: Any) -> dict:
"""Return only mapping-shaped message metadata.
Legacy rows and client payloads can contain JSON lists/scalars. They are
display noise, not trusted fields, and must not reach ``dict.update`` or
approval projection code.
"""
if isinstance(value, dict):
return value
if isinstance(value, str):
try:
parsed = json.loads(value)
except (json.JSONDecodeError, TypeError, ValueError):
return {}
return parsed if isinstance(parsed, dict) else {}
return {}
def _history_display_content(content: Any) -> Any:
"""Return a lightweight browser-display copy of stored message content.
@@ -131,7 +101,7 @@ def _merge_continue_rows_to_delete(db_messages, db1, db2):
def setup_history_routes(session_manager, upload_handler=None) -> APIRouter:
router = APIRouter(tags=["history"], dependencies=[Depends(require_chat_scope)])
router = APIRouter(tags=["history"])
def _reserve_message_uploads(
request: Request,
@@ -153,19 +123,14 @@ def setup_history_routes(session_manager, upload_handler=None) -> APIRouter:
f"Referenced upload is no longer available: {missing_id}",
)
def _display_metadata(value: Any, *, sanitize: bool) -> dict:
meta = _metadata_dict(value)
if sanitize:
return sanitize_projected_message_metadata(meta) or {}
return dict(meta)
def _db_history_entry(
m: DbChatMessage,
*,
sanitize: bool = False,
) -> Dict[str, Any]:
def _db_history_entry(m: DbChatMessage) -> Dict[str, Any]:
entry = {"role": m.role, "content": _history_display_content(m.content)}
meta = _display_metadata(m.meta_data, sanitize=sanitize)
meta = {}
if m.meta_data:
try:
meta = json.loads(m.meta_data) or {}
except (json.JSONDecodeError, ValueError):
meta = {}
if m.timestamp and "timestamp" not in meta:
meta["timestamp"] = m.timestamp.isoformat() + "Z"
if meta:
@@ -179,8 +144,6 @@ def setup_history_routes(session_manager, upload_handler=None) -> APIRouter:
limit: Optional[int] = None,
offset: Optional[int] = None,
) -> Dict[str, Any]:
require_chat_scope(request)
sanitize_history = is_bearer_principal(request)
_verify_session_owner(request, session_id)
if limit is not None:
page_limit = max(1, min(int(limit), 100))
@@ -208,11 +171,7 @@ def setup_history_routes(session_manager, upload_handler=None) -> APIRouter:
.all()
)
history_dict = [
entry
for entry in (
_db_history_entry(m, sanitize=sanitize_history)
for m in rows
)
entry for entry in (_db_history_entry(m) for m in rows)
if not (entry.get("metadata") or {}).get("hidden")
]
return {
@@ -238,29 +197,21 @@ def setup_history_routes(session_manager, upload_handler=None) -> APIRouter:
for msg in session.history:
if isinstance(msg, ChatMessage):
# Skip hidden messages (e.g. compaction summaries for AI context)
msg_meta = _display_metadata(
msg.metadata,
sanitize=sanitize_history,
)
if msg_meta.get("hidden"):
if msg.metadata and msg.metadata.get("hidden"):
continue
entry = {"role": msg.role, "content": _history_display_content(msg.content)}
if msg_meta:
entry["metadata"] = msg_meta
if msg.metadata:
entry["metadata"] = msg.metadata
history_dict.append(entry)
elif isinstance(msg, dict):
msg_meta = _display_metadata(
msg.get("metadata"),
sanitize=sanitize_history,
)
if msg_meta.get("hidden"):
if msg.get("metadata", {}).get("hidden"):
continue
entry = {
"role": msg.get("role", ""),
"content": _history_display_content(msg.get("content", "")),
}
if msg_meta:
entry["metadata"] = msg_meta
if msg.get("metadata"):
entry["metadata"] = msg["metadata"]
history_dict.append(entry)
# Fallback: load from DB if in-memory renders empty. Display only —
@@ -278,11 +229,7 @@ def setup_history_routes(session_manager, upload_handler=None) -> APIRouter:
)
# Response excludes hidden messages, matching the in-memory path.
history_dict = [
entry
for entry in (
_db_history_entry(m, sanitize=sanitize_history)
for m in db_messages
)
entry for entry in (_db_history_entry(m) for m in db_messages)
if not (entry.get("metadata") or {}).get("hidden")
]
except Exception as e:
@@ -299,7 +246,6 @@ def setup_history_routes(session_manager, upload_handler=None) -> APIRouter:
@router.post("/api/session/{session_id}/truncate")
async def truncate_session(request: Request, session_id: str):
require_chat_scope(request)
_verify_session_owner(request, session_id)
try:
body = await request.json()
@@ -315,15 +261,14 @@ def setup_history_routes(session_manager, upload_handler=None) -> APIRouter:
@router.post("/api/session/{session_id}/message")
async def add_message(request: Request, session_id: str):
"""Add a message to a session (for slash command persistence)."""
require_chat_scope(request)
_verify_session_owner(request, session_id)
try:
body = await request.json()
role = normalize_client_message_role(body.get("role", "assistant"))
role = body.get("role", "assistant")
content = body.get("content", "")
if not content:
raise HTTPException(400, "content is required")
metadata = sanitize_client_message_metadata(body.get("metadata"))
metadata = body.get("metadata")
_reserve_message_uploads(request, content, metadata)
msg = ChatMessage(role=role, content=content, metadata=metadata)
session_manager.add_message(session_id, msg)
@@ -334,7 +279,6 @@ def setup_history_routes(session_manager, upload_handler=None) -> APIRouter:
@router.post("/api/session/{session_id}/delete-messages")
async def delete_messages(request: Request, session_id: str):
"""Delete specific messages by DB ID (or legacy index)."""
require_chat_scope(request)
_verify_session_owner(request, session_id)
try:
body = await request.json()
@@ -398,7 +342,6 @@ def setup_history_routes(session_manager, upload_handler=None) -> APIRouter:
@router.post("/api/session/{session_id}/edit-message")
async def edit_message(request: Request, session_id: str):
"""Edit the content of a message by its database ID."""
require_chat_scope(request)
_verify_session_owner(request, session_id)
try:
body = await request.json()
@@ -421,8 +364,9 @@ def setup_history_routes(session_manager, upload_handler=None) -> APIRouter:
db_msg.content = content
meta = {}
meta = _metadata_dict(db_msg.meta_data)
meta = dict(meta)
if db_msg.meta_data:
try: meta = json.loads(db_msg.meta_data)
except (json.JSONDecodeError, ValueError): pass
meta['edited'] = True
db_msg.meta_data = json.dumps(meta)
@@ -453,7 +397,6 @@ def setup_history_routes(session_manager, upload_handler=None) -> APIRouter:
@router.post("/api/session/{session_id}/mark-stopped")
async def mark_stopped(request: Request, session_id: str):
"""Mark the last assistant message as stopped by user."""
require_chat_scope(request)
_verify_session_owner(request, session_id)
try:
session = session_manager.get_session(session_id)
@@ -462,13 +405,13 @@ def setup_history_routes(session_manager, upload_handler=None) -> APIRouter:
if (isinstance(msg, ChatMessage) and msg.role == 'assistant') or \
(isinstance(msg, dict) and msg.get('role') == 'assistant'):
if isinstance(msg, ChatMessage):
if not isinstance(msg.metadata, dict):
if not msg.metadata:
msg.metadata = {}
msg.metadata['stopped'] = True
if not msg.metadata.get('model'):
msg.metadata['model'] = session.model
else:
if not isinstance(msg.get('metadata'), dict):
if 'metadata' not in msg:
msg['metadata'] = {}
msg['metadata']['stopped'] = True
if not msg['metadata'].get('model'):
@@ -486,8 +429,11 @@ def setup_history_routes(session_manager, upload_handler=None) -> APIRouter:
)
if db_messages:
meta = {}
meta = _metadata_dict(db_messages.meta_data)
meta = dict(meta)
if db_messages.meta_data:
try:
meta = _json.loads(db_messages.meta_data)
except (json.JSONDecodeError, ValueError):
pass
meta['stopped'] = True
if not meta.get('model'):
meta['model'] = session.model
@@ -506,11 +452,10 @@ def setup_history_routes(session_manager, upload_handler=None) -> APIRouter:
@router.post("/api/session/{session_id}/update-last-meta")
async def update_last_meta(request: Request, session_id: str):
"""Merge metadata into the last assistant message (e.g. save variants)."""
require_chat_scope(request)
_verify_session_owner(request, session_id)
try:
body = await request.json()
meta_update = sanitize_client_message_metadata(body.get("metadata", {})) or {}
meta_update = body.get("metadata", {})
session = session_manager.get_session(session_id)
# Update in-memory
@@ -518,11 +463,11 @@ def setup_history_routes(session_manager, upload_handler=None) -> APIRouter:
if (isinstance(msg, ChatMessage) and msg.role == 'assistant') or \
(isinstance(msg, dict) and msg.get('role') == 'assistant'):
if isinstance(msg, ChatMessage):
if not isinstance(msg.metadata, dict):
if not msg.metadata:
msg.metadata = {}
msg.metadata.update(meta_update)
else:
if not isinstance(msg.get('metadata'), dict):
if 'metadata' not in msg:
msg['metadata'] = {}
msg['metadata'].update(meta_update)
break
@@ -538,7 +483,10 @@ def setup_history_routes(session_manager, upload_handler=None) -> APIRouter:
.first()
)
if db_msg:
meta = dict(_metadata_dict(db_msg.meta_data))
meta = {}
if db_msg.meta_data:
try: meta = _json.loads(db_msg.meta_data)
except (json.JSONDecodeError, ValueError): pass
meta.update(meta_update)
db_msg.meta_data = _json.dumps(meta)
db.commit()
@@ -555,7 +503,6 @@ def setup_history_routes(session_manager, upload_handler=None) -> APIRouter:
@router.post("/api/session/{session_id}/merge-last-assistant")
async def merge_last_assistant(request: Request, session_id: str):
"""Merge the last two assistant messages into one (for continue)."""
require_chat_scope(request)
_verify_session_owner(request, session_id)
try:
body = await request.json()
@@ -580,12 +527,8 @@ def setup_history_routes(session_manager, upload_handler=None) -> APIRouter:
merged_content = content1 + separator + content2
# Merge metadata
meta1 = dict(_metadata_dict(
msg1.metadata if isinstance(msg1, ChatMessage) else msg1.get('metadata')
))
meta2 = dict(_metadata_dict(
msg2.metadata if isinstance(msg2, ChatMessage) else msg2.get('metadata')
))
meta1 = (msg1.metadata if isinstance(msg1, ChatMessage) else msg1.get('metadata')) or {}
meta2 = (msg2.metadata if isinstance(msg2, ChatMessage) else msg2.get('metadata')) or {}
merged_meta = {**meta1, **meta2}
merged_meta.pop('stopped', None) # no longer stopped after continue
@@ -649,7 +592,6 @@ def setup_history_routes(session_manager, upload_handler=None) -> APIRouter:
@router.post("/api/session/{session_id}/fork")
async def fork_session(request: Request, session_id: str):
"""Create a new session with messages copied up to keep_count."""
require_chat_scope(request)
_verify_session_owner(request, session_id)
try:
body = await request.json()
@@ -666,15 +608,6 @@ def setup_history_routes(session_manager, upload_handler=None) -> APIRouter:
if not source:
raise HTTPException(404, "Session not found")
source_provenance = getattr(source, "endpoint_provenance", None)
source_endpoint_id = getattr(source, "model_endpoint_id", None)
if (
is_bearer_principal(request)
and (hasattr(source, "endpoint_provenance") or hasattr(source, "model_endpoint_id"))
and source_provenance not in {"registered", "direct"}
):
raise HTTPException(400, "Session endpoint provenance is unavailable")
# Create new session
new_id = str(uuid.uuid4())
fork_name = f"\u2ADD {source.name}"
@@ -686,14 +619,6 @@ def setup_history_routes(session_manager, upload_handler=None) -> APIRouter:
rag=False,
owner=getattr(source, 'owner', None),
)
if source_provenance in {"registered", "direct"}:
persist_session_endpoint_provenance(
session_manager,
new_id,
new_session,
model_endpoint_id=source_endpoint_id,
endpoint_provenance=source_provenance,
)
# Copy messages up to keep_count
msgs_to_copy = source.history[:keep_count]
@@ -704,15 +629,12 @@ def setup_history_routes(session_manager, upload_handler=None) -> APIRouter:
# in-memory messages, corrupting their _db_id and breaking
# edit/delete-by-id on the original conversation.
meta = dict(msg.metadata) if isinstance(msg.metadata, dict) else None
if is_bearer_principal(request):
meta = sanitize_projected_message_metadata(meta)
new_session.add_message(ChatMessage(msg.role, msg.content, meta))
if not is_bearer_principal(request):
try:
from src.event_bus import fire_event
fire_event("session_created", getattr(source, 'owner', None))
except Exception:
logger.debug("session_created event dispatch failed", exc_info=True)
try:
from src.event_bus import fire_event
fire_event("session_created", getattr(source, 'owner', None))
except Exception:
logger.debug("session_created event dispatch failed", exc_info=True)
return {
"status": "ok",
@@ -728,7 +650,6 @@ def setup_history_routes(session_manager, upload_handler=None) -> APIRouter:
@router.get("/api/conversations/topics")
async def get_conversation_topics(request: Request) -> Dict[str, Any]:
require_chat_scope(request)
from src.auth_helpers import require_user
user = require_user(request)
try:
@@ -744,8 +665,6 @@ def setup_history_routes(session_manager, upload_handler=None) -> APIRouter:
endpoint estimates the persisted session context so the header can show
when the whole chat is approaching compaction.
"""
require_chat_scope(request)
capability = request_capability(request)
_verify_session_owner(request, session_id)
try:
session = session_manager.get_session(session_id)
@@ -757,23 +676,16 @@ def setup_history_routes(session_manager, upload_handler=None) -> APIRouter:
messages = session.get_context_messages()
used = int(estimate_tokens(messages))
context_kwargs = {}
if not capability.allow_live_probes:
context_kwargs["allow_live_probes"] = False
ctx_len = int(get_context_length(
session.endpoint_url,
session.model,
**context_kwargs,
) or 0)
ctx_len = int(get_context_length(session.endpoint_url, session.model) or 0)
pct = round((used / ctx_len) * 100, 1) if ctx_len else 0.0
pct = max(0.0, min(100.0, pct))
visible_messages = sum(
1 for m in session.history
if not _metadata_dict(getattr(m, "metadata", None)).get("hidden")
if not (getattr(m, "metadata", None) or {}).get("hidden")
)
compacted_messages = sum(
1 for m in session.history
if _metadata_dict(getattr(m, "metadata", None)).get("compacted")
if (getattr(m, "metadata", None) or {}).get("compacted")
)
can_compact = used > 0
return {
@@ -797,8 +709,6 @@ def setup_history_routes(session_manager, upload_handler=None) -> APIRouter:
@router.post("/api/session/{session_id}/compact")
async def compact_session(request: Request, session_id: str):
"""Manually trigger context compaction for a session."""
require_chat_scope(request)
capability = request_capability(request)
_verify_session_owner(request, session_id)
from src.auth_helpers import effective_user
owner = effective_user(request)
@@ -811,21 +721,12 @@ def setup_history_routes(session_manager, upload_handler=None) -> APIRouter:
try:
from src.model_context import estimate_tokens, get_context_length
from src.llm_core import llm_call_async
from src.endpoint_resolver import resolve_endpoint
if len(session.history) < 6:
return {"status": "ok", "message": "Not enough messages to compact"}
if capability.is_bearer:
_validate_bearer_session_model(session, owner=owner)
context_kwargs = {}
if not capability.allow_live_probes:
context_kwargs["allow_live_probes"] = False
ctx_len = get_context_length(
session.endpoint_url,
session.model,
**context_kwargs,
)
ctx_len = get_context_length(session.endpoint_url, session.model)
messages_before = session.get_context_messages()
used_before = estimate_tokens(messages_before)
pct_before = round((used_before / ctx_len) * 100, 1) if ctx_len else 0
@@ -843,26 +744,15 @@ def setup_history_routes(session_manager, upload_handler=None) -> APIRouter:
for m in older
)
# Use the utility model only for interactive/live-capable callers.
# Bearer compaction remains on the selected session route.
if capability.allow_live_probes:
from src.endpoint_resolver import resolve_endpoint
util_url, util_model, util_headers = resolve_endpoint("utility", owner=owner or None)
compact_url = util_url or session.endpoint_url
compact_model = util_model or session.model
compact_headers = util_headers if util_url else session.headers
else:
compact_url = session.endpoint_url
compact_model = session.model
compact_headers = session.headers
# Use utility model if available
util_url, util_model, util_headers = resolve_endpoint("utility", owner=owner or None)
compact_url = util_url or session.endpoint_url
compact_model = util_model or session.model
compact_headers = util_headers if util_url else session.headers
from src.context_compactor import SELF_SUMMARY_SYSTEM_PROMPT, normalize_compaction_summary
compaction_count = sum(1 for m in session.history if isinstance(m, ChatMessage) and "[Conversation summary" in (m.content or ""))
sys_prompt = SELF_SUMMARY_SYSTEM_PROMPT.replace("{count}", str(len(older))).replace("{n}", str(compaction_count + 1))
compact_kwargs = {}
if not capability.allow_live_probes:
compact_kwargs["allow_live_probes"] = False
summary = await llm_call_async(
compact_url, compact_model,
[
@@ -871,7 +761,6 @@ def setup_history_routes(session_manager, upload_handler=None) -> APIRouter:
],
temperature=0.2, max_tokens=1024,
headers=compact_headers, timeout=30,
**compact_kwargs,
)
summary = normalize_compaction_summary(summary)
@@ -949,8 +838,6 @@ def setup_history_routes(session_manager, upload_handler=None) -> APIRouter:
"after": pct_after,
}
except HTTPException:
raise
except Exception as e:
logger.error(f"Manual compact error {session_id}: {e}")
raise HTTPException(500, str(e))
+6 -19
View File
@@ -5,11 +5,10 @@ import shlex
import subprocess
from copy import deepcopy
from fastapi import APIRouter, Depends, HTTPException, Request
from fastapi import APIRouter, HTTPException
from core.platform_compat import run_ssh_command
from routes._validators import validate_remote_host, validate_ssh_port
from src.auth_helpers import require_non_bearer_request
# Backends the manual hardware simulator accepts. Must stay a subset of what
@@ -181,32 +180,24 @@ def _inspect_model_path(model_path: str, host: str = "", ssh_port: str = "") ->
def setup_hwfit_routes():
router = APIRouter(
prefix="/api/hwfit",
tags=["hwfit"],
dependencies=[Depends(require_non_bearer_request)],
)
router = APIRouter(prefix="/api/hwfit", tags=["hwfit"])
@router.get("/system")
def get_system(host: str = "", ssh_port: str = "", platform: str = "", fresh: bool = False, request: Request = None):
def get_system(host: str = "", ssh_port: str = "", platform: str = "", fresh: bool = False):
"""Detect and return current system hardware info. Pass host=user@server for remote.
fresh=true bypasses the per-host cache (the Rescan button)."""
if request is not None:
require_non_bearer_request(request)
from services.hwfit.hardware import detect_system
host, ssh_port = _validate_detection_target(host, ssh_port)
return detect_system(host=host, ssh_port=ssh_port, platform=platform, fresh=fresh)
@router.get("/models")
def get_models(use_case: str = "", sort: str = "newest", limit: int = 50, search: str = "", host: str = "", quant: str = "", ctx: str = "", gpu_count: str = "", gpu_group: str = "", ssh_port: str = "", platform: str = "", fresh: bool = False, refresh_catalog: bool = False, manual_mode: str = "", manual_gpu_count: str = "", manual_vram_gb: str = "", manual_ram_gb: str = "", manual_backend: str = "", ignore_detected_gpu: bool = False, ignore_detected_ram: bool = False, fit_only: bool = False, request: Request = None):
def get_models(use_case: str = "", sort: str = "newest", limit: int = 50, search: str = "", host: str = "", quant: str = "", ctx: str = "", gpu_count: str = "", gpu_group: str = "", ssh_port: str = "", platform: str = "", fresh: bool = False, refresh_catalog: bool = False, manual_mode: str = "", manual_gpu_count: str = "", manual_vram_gb: str = "", manual_ram_gb: str = "", manual_backend: str = "", ignore_detected_gpu: bool = False, ignore_detected_ram: bool = False, fit_only: bool = False):
"""Rank LLM models against detected hardware and return scored results.
gpu_count: override GPU count (0 = CPU only, 1-N = simulate N GPUs of the
active group). gpu_group: index into system.gpu_groups (the homogeneous
pools) to target empty/auto = the largest pool. vLLM can only
tensor-parallel across identical GPUs, so we never mix pools.
fresh=true bypasses the hardware-detection cache."""
if request is not None:
require_non_bearer_request(request)
from services.hwfit.hardware import detect_system
from services.hwfit.fit import rank_models
from services.hwfit.models import get_models, model_catalog_path, refresh_dynamic_catalogs
@@ -325,7 +316,7 @@ def setup_hwfit_routes():
return payload
@router.get("/profiles")
def get_serve_profiles(model: str = "", model_path: str = "", host: str = "", ssh_port: str = "", platform: str = "", fresh: bool = False, serve_weights_gb: float = 0.0, serve_quant: str = "", request: Request = None):
def get_serve_profiles(model: str = "", model_path: str = "", host: str = "", ssh_port: str = "", platform: str = "", fresh: bool = False, serve_weights_gb: float = 0.0, serve_quant: str = ""):
"""Compute llama.cpp serve profiles (Quality/Balanced/Speed) for `model`
against the detected hardware on `host` (or local). Returns concrete
flags (n_gpu_layers, n_cpu_moe, cache_type, ctx) the serve UI can apply.
@@ -334,8 +325,6 @@ def setup_hwfit_routes():
catalog (e.g. an ad-hoc HF repo), pass enough hints via a minimal synthetic
entry isn't possible here, so we return [] and the UI keeps manual flags.
"""
if request is not None:
require_non_bearer_request(request)
from services.hwfit.hardware import detect_system
from services.hwfit.models import get_models
from services.hwfit.profiles import compute_serve_profiles
@@ -421,10 +410,8 @@ def setup_hwfit_routes():
}
@router.get("/image-models")
def get_image_models(sort: str = "fit", search: str = "", host: str = "", gpu_count: str = "", ssh_port: str = "", platform: str = "", fresh: bool = False, manual_mode: str = "", manual_gpu_count: str = "", manual_vram_gb: str = "", manual_ram_gb: str = "", manual_backend: str = "", ignore_detected_gpu: bool = False, ignore_detected_ram: bool = False, request: Request = None):
def get_image_models(sort: str = "fit", search: str = "", host: str = "", gpu_count: str = "", ssh_port: str = "", platform: str = "", fresh: bool = False, manual_mode: str = "", manual_gpu_count: str = "", manual_vram_gb: str = "", manual_ram_gb: str = "", manual_backend: str = "", ignore_detected_gpu: bool = False, ignore_detected_ram: bool = False):
"""Rank image generation models against detected hardware."""
if request is not None:
require_non_bearer_request(request)
from services.hwfit.hardware import detect_system
from services.hwfit.image_models import rank_image_models
host, ssh_port = _validate_detection_target(host, ssh_port)
+7 -13
View File
@@ -475,7 +475,7 @@ def setup_mcp_routes(mcp_manager: McpManager):
return RedirectResponse(auth_url)
else:
# Remote device — show paste-back page
return HTMLResponse(_oauth_authorize_page(auth_url, server_id, redirect_uri))
return HTMLResponse(_oauth_authorize_page(auth_url, server_id, host, redirect_uri))
finally:
db.close()
@@ -612,13 +612,15 @@ def setup_mcp_routes(mcp_manager: McpManager):
def _oauth_authorize_page(
auth_url: str,
server_id: str,
redirect_uri: str,
host: str,
redirect_uri: str = "http://localhost:7000/api/mcp/oauth/callback",
) -> str:
"""Page with Google sign-in link and URL paste-back form for remote access."""
# Escape values interpolated into the page: `server_id` comes from the OAuth
# state and is not trusted.
# Escape values interpolated into the page: `host` comes from the request
# Host header and `server_id` from the OAuth state — neither is trusted.
auth_url = html.escape(auth_url, quote=True)
server_id = html.escape(server_id, quote=True)
host = html.escape(host, quote=True)
redirect_uri = html.escape(redirect_uri, quote=True)
return f"""<!DOCTYPE html>
<html><head>
@@ -662,15 +664,7 @@ def _oauth_authorize_page(
</div>
<a class="auth-link" href="{auth_url}" target="_blank" rel="noopener">Sign in with Google</a>
<div class="divider"></div>
<!-- Relative action: the browser resolves it against the origin this page was
served from, so the form follows the user through any proxy without the
app having to know the scheme or the host. An absolute http:// action is
blocked as mixed content on exactly the HTTPS deployments that need
paste-back, and request.url.scheme cannot be trusted to spot them
uvicorn only honours X-Forwarded-Proto from a peer in
--forwarded-allow-ips, which defaults to 127.0.0.1 and excludes a proxy
arriving over the Docker bridge. -->
<form method="POST" action="/api/mcp/oauth/exchange/{server_id}">
<form method="POST" action="http://{host}/api/mcp/oauth/exchange/{server_id}">
<p>Paste the URL from your browser after signing in:</p>
<input type="text" name="callback_url" placeholder="{redirect_uri}?code=..." required>
<br><button type="submit">Connect</button>
+2 -12
View File
@@ -1,5 +1,5 @@
# routes/memory_routes.py
from fastapi import APIRouter, Depends, Form, HTTPException, Request, UploadFile, File
from fastapi import APIRouter, Form, HTTPException, Request, UploadFile, File
from typing import Dict, Any, Optional, List
import json
import os
@@ -53,19 +53,9 @@ def _load_for_update(memory_manager) -> List[Dict[str, Any]]:
def setup_memory_routes(memory_manager: MemoryManager, session_manager: SessionManager, memory_vector=None):
"""Set up memory-related routes."""
router = APIRouter(
prefix="/api/memory",
tags=["memory"],
dependencies=[Depends(require_user)],
)
router = APIRouter(prefix="/api/memory", tags=["memory"])
def _owner(request: Request) -> Optional[str]:
# Router dependencies do not run when a handler is called directly
# (including through an integration router), so keep the same bearer
# rejection at the owner-resolution seam. ``None`` is retained only
# for legacy unit callers; real ASGI requests always carry Request.
if request is not None:
require_user(request)
return get_current_user(request)
def _assert_session_owner(session_obj, user):
+25 -116
View File
@@ -29,12 +29,7 @@ from src.endpoint_resolver import (
build_models_url,
build_headers,
)
from src.auth_helpers import (
_auth_disabled,
is_bearer_principal,
owner_filter,
require_chat_scope,
)
from src.auth_helpers import _auth_disabled, effective_user, owner_filter
logger = logging.getLogger(__name__)
@@ -1356,14 +1351,14 @@ def _legacy_visible_api_models(ep) -> List[str]:
def _picker_models_for_endpoint(ep, base_url: str, kind: str):
"""Return model IDs that should appear in the picker for an endpoint.
API providers expose remote inventory from /v1/models. Default to that
visible inventory until an explicit pinned-model allow-list is saved.
Local/self-hosted endpoints keep the older hide-list behavior.
API providers expose remote inventory from /v1/models. Treat that cache as
inventory, not approval: only manually pinned API models should appear in
the picker. Local/self-hosted endpoints keep the older hide-list behavior.
"""
pinned = _normalize_model_ids(getattr(ep, "pinned_models", None))
if _picker_requires_pinning(base_url, kind):
if not _has_explicit_pinned_models(ep):
pinned = _legacy_visible_api_models(ep)
pinned = _legacy_visible_api_models(ep) if _hidden_model_ids(ep) else []
return pinned, pinned
return _visible_models(
_cached_model_ids(ep),
@@ -1372,68 +1367,6 @@ def _picker_models_for_endpoint(ep, base_url: str, kind: str):
), pinned
def _validate_bearer_model_selection(
ep,
requested_model: Optional[str],
*,
allow_empty: bool = False,
) -> str:
"""Validate a bearer-selected model against the server-owned picker.
Bearer requests cannot perform a provider model probe. Their model choice
must therefore come from the same endpoint-local cache/pin inventory that
the server exposes to the model picker. ``allow_empty`` is used only by
default-chat, where an explicitly empty inventory has a deterministic empty
result rather than an implicit provider alias.
"""
if ep is None:
if allow_empty:
return ""
raise HTTPException(400, "A registered model endpoint is required")
base_url = _normalize_base(getattr(ep, "base_url", "") or "")
kind = _effective_endpoint_kind(ep, base_url)
models, _ = _picker_models_for_endpoint(ep, base_url, kind)
models = [model for model in models if isinstance(model, str) and model.strip()]
requested = str(requested_model or "").strip()
if not requested:
if models:
return models[0]
if allow_empty:
return ""
raise HTTPException(400, "No permitted model is configured for this endpoint")
# A registered local endpoint may intentionally have no persisted
# catalog: local models are operator-controlled and bearer requests
# must not discover them live. Preserve that documented compatibility
# path, while still enforcing any inventory that the server does own
# and rejecting explicitly hidden entries below.
raw_inventory = _merge_model_ids(
_normalize_model_ids(getattr(ep, "cached_models", None)),
_normalize_model_ids(getattr(ep, "pinned_models", None)),
)
hidden = set(_normalize_model_ids(getattr(ep, "hidden_models", None)))
if (
requested
and not raw_inventory
and _classify_endpoint(base_url, kind) == "local"
and requested not in hidden
):
return requested
if requested in models:
return requested
requested_base = os.path.basename(requested.rstrip("/"))
matches = [
model for model in models
if os.path.basename(model.rstrip("/")) == requested_base
]
if len(matches) == 1:
return matches[0]
if len(matches) > 1:
raise HTTPException(400, "Model selection is ambiguous for this endpoint")
raise HTTPException(400, f"Model is not permitted for this endpoint: {requested}")
def _api_key_fingerprint(api_key: Optional[str]) -> str:
"""Stable, non-secret label for distinguishing same-URL credentials."""
key = (api_key or "").strip()
@@ -1603,12 +1536,7 @@ def setup_model_routes(model_discovery):
_refresh_inflight["v"] = False
threading.Thread(target=_do, daemon=True).start()
def _fetch_models(
owner: str = "",
is_admin: bool = False,
*,
read_only: bool = False,
):
def _fetch_models(owner: str = "", is_admin: bool = False):
"""Return model list from cached data (instant). Background refresh keeps caches fresh.
SECURITY: filters endpoints by `owner` without this the picker
@@ -1623,7 +1551,7 @@ def setup_model_routes(model_discovery):
db = SessionLocal()
try:
if not read_only and _disable_stale_cookbook_local_endpoints(db):
if _disable_stale_cookbook_local_endpoints(db):
_invalidate_models_cache()
q = db.query(ModelEndpoint).filter(ModelEndpoint.is_enabled == True)
if owner and not is_admin:
@@ -1694,7 +1622,13 @@ def setup_model_routes(model_discovery):
# Require auth; "" is the unconfigured single-user mode, treated as
# "see everything" by _fetch_models.
try:
owner = require_chat_scope(request) or ""
if getattr(request.state, "api_token", False):
scopes = set(getattr(request.state, "api_token_scopes", []) or [])
if "chat" not in scopes:
raise HTTPException(403, "API token is not scoped for chat")
if not getattr(request.state, "api_token_owner", None):
raise HTTPException(403, "API token has no owner")
owner = effective_user(request) or ""
# Reject anonymous in configured deployments — no leaking the model
# list to unauthenticated callers.
@@ -1706,17 +1640,6 @@ def setup_model_routes(model_discovery):
except Exception as e:
logger.error("Auth gate error in GET /api/models, failing closed: %s", e)
raise HTTPException(status_code=500, detail="Internal error")
bearer = is_bearer_principal(request)
if bearer and (refresh or background):
raise HTTPException(
403,
"API tokens may only read the owner-scoped cached model list",
)
if bearer:
# The bearer-compatible path is deliberately read-only: no global
# admin view, stale-row cleanup, process cache writes, background
# probes, stored endpoint credentials, or refresh state changes.
return _fetch_models(owner=owner, is_admin=False, read_only=True)
# Admins see every endpoint (they manage the global pool); regular
# users get the owner-scoped view.
_is_admin = False
@@ -2419,7 +2342,9 @@ def setup_model_routes(model_discovery):
else:
response.headers["X-Model-Refresh-Status"] = "failed"
response.headers["X-Model-Refresh-Warning"] = "Model refresh failed or returned no models; kept cached models."
_, pinned = _picker_models_for_endpoint(ep, base, kind)
pinned = _normalize_model_ids(getattr(ep, "pinned_models", None))
if picker_requires_pinning and not _has_explicit_pinned_models(ep):
pinned = _legacy_visible_api_models(ep)
pinned_set = set(pinned)
return [
{
@@ -2496,11 +2421,11 @@ def setup_model_routes(model_discovery):
# no per-user default yet, we resolve via the owner-scoped endpoint
# lookup below (last-resort: first enabled endpoint THIS user owns).
# Unauthenticated single-user mode keeps the old behavior.
# Resolve through the same owner/scope gate as the model picker. In an
# auth-disabled process there is no middleware to stamp token state, so
# raw bearer detection must still prevent a token from resolving
# global/admin defaults.
_user = require_chat_scope(request) or ""
from src.auth_helpers import get_current_user as _gcu
try:
_user = _gcu(request) or ""
except Exception:
_user = ""
# Admins resolve via the global defaults (they own them, and the
# scoped resolution was making the picker disappear for them).
# Regular users get per-user prefs with NO global fallback for the
@@ -2510,12 +2435,7 @@ def setup_model_routes(model_discovery):
_is_admin = False
try:
auth_mgr = getattr(request.app.state, "auth_manager", None)
if (
_user
and not is_bearer_principal(request)
and auth_mgr is not None
and getattr(auth_mgr, "is_admin", None)
):
if _user and auth_mgr is not None and getattr(auth_mgr, "is_admin", None):
_is_admin = bool(auth_mgr.is_admin(_user))
except Exception:
_is_admin = False
@@ -2563,13 +2483,7 @@ def setup_model_routes(model_discovery):
return {"endpoint_id": "", "endpoint_url": "", "model": ""}
base = _normalize_base(ep.base_url)
chat_url = build_chat_url(base)
if is_bearer_principal(request):
model = _validate_bearer_model_selection(
ep,
model,
allow_empty=True,
)
elif not model and (getattr(ep, "cached_models", None) or getattr(ep, "pinned_models", None)):
if not model and (getattr(ep, "cached_models", None) or getattr(ep, "pinned_models", None)):
try:
visible = _visible_models(ep.cached_models, getattr(ep, "hidden_models", None), getattr(ep, "pinned_models", None))
if visible:
@@ -2780,13 +2694,8 @@ def setup_model_routes(model_discovery):
# ── Tool management ──
@router.get("/tools")
def list_tools(request: Request):
def list_tools():
"""List all available tools with their enabled/disabled status."""
# Tool inventory is an interactive/agent capability description, not
# part of the narrow bearer chat contract. Cookie/local callers retain
# the historical response.
from src.auth_helpers import require_non_bearer_request
require_non_bearer_request(request)
from src.agent_tools import TOOL_TAGS
settings = _load_settings()
disabled = set(settings.get("disabled_tools", []))
+6 -9
View File
@@ -9,13 +9,13 @@ from datetime import datetime
from pathlib import Path
from typing import Optional
from fastapi import APIRouter, Depends, HTTPException, Query, Request
from fastapi import APIRouter, HTTPException, Query, Request
from fastapi.responses import HTMLResponse, StreamingResponse
from pydantic import BaseModel, Field
from core.middleware import INTERNAL_TOOL_USER
from src.endpoint_resolver import resolve_endpoint
from src.auth_helpers import _auth_disabled, require_interactive_request
from src.owner_identity import REQUEST_SENTINEL_OWNERS
from src.auth_helpers import _auth_disabled, get_current_user
from core.auth import RESERVED_USERNAMES
from src.constants import DEEP_RESEARCH_DIR
_SESSION_ID_RE = re.compile(r"^[a-zA-Z0-9-]{1,128}$")
@@ -207,17 +207,14 @@ def _resolve_endpoint_runtime(ep, owner=None, model: Optional[str] = None):
def setup_research_routes(research_handler, session_manager=None) -> APIRouter:
router = APIRouter(
tags=["research"],
dependencies=[Depends(require_interactive_request)],
)
router = APIRouter(tags=["research"])
def _require_user(request: Request) -> str:
"""All research endpoints require an authenticated user. Research
data isn't owner-scoped in the on-disk JSON yet, so we at least
block anonymous access. Multi-tenant deploys should additionally
verify the session belongs to this user."""
user = require_interactive_request(request)
user = get_current_user(request)
if not user:
if _auth_disabled():
return ""
@@ -499,7 +496,7 @@ def setup_research_routes(research_handler, session_manager=None) -> APIRouter:
user = require_privilege(request, "can_use_research")
if user == INTERNAL_TOOL_USER:
tool_owner = (request.headers.get("X-Odysseus-Owner") or "").strip()
if tool_owner and tool_owner not in REQUEST_SENTINEL_OWNERS:
if tool_owner and tool_owner not in RESERVED_USERNAMES:
auth_mgr = getattr(request.app.state, "auth_manager", None)
if auth_mgr is not None and getattr(auth_mgr, "is_configured", False):
try:
+4 -12
View File
@@ -3,14 +3,13 @@
import logging
from typing import Dict, Any
from fastapi import APIRouter, Depends, Request
from fastapi import APIRouter, Request
import time
from services.search import get_search_config, comprehensive_web_search, PROVIDER_INFO
from services.search.core import _call_provider
from services.search.providers import _get_provider_key, _get_search_instance
from src.auth_helpers import require_interactive_request
logger = logging.getLogger(__name__)
@@ -38,14 +37,10 @@ async def _request_values(request: Request) -> Dict[str, Any]:
def setup_search_routes(config) -> APIRouter:
router = APIRouter(
tags=["search"],
dependencies=[Depends(require_interactive_request)],
)
router = APIRouter(tags=["search"])
@router.get("/api/search/config")
async def get_search_settings(request: Request) -> Dict[str, Any]:
require_interactive_request(request)
async def get_search_settings() -> Dict[str, Any]:
return get_search_config()
@router.post("/api/search")
@@ -54,7 +49,6 @@ def setup_search_routes(config) -> APIRouter:
Used by Compare mode to pre-search once and share results across panes.
"""
require_interactive_request(request)
values = await _request_values(request)
query = str(values.get("query") or values.get("q") or "").strip()
if not query:
@@ -72,9 +66,8 @@ def setup_search_routes(config) -> APIRouter:
return {"context": "", "sources": [], "error": str(e)}
@router.get("/api/search/providers")
async def list_search_providers(request: Request):
async def list_search_providers():
"""Return available search providers with config status."""
require_interactive_request(request)
providers = []
for pid, (label, needs_key, needs_url) in PROVIDER_INFO.items():
if pid == "disabled":
@@ -94,7 +87,6 @@ def setup_search_routes(config) -> APIRouter:
@router.post("/api/search/query")
async def search_with_provider(request: Request) -> Dict[str, Any]:
"""Search using a specific provider. Used by compare search mode."""
require_interactive_request(request)
values = await _request_values(request)
query = str(values.get("query") or values.get("q") or "").strip()
provider = str(values.get("provider") or "").strip()
+69 -195
View File
@@ -4,30 +4,17 @@ import html
import json
import uuid
from datetime import datetime
from fastapi import APIRouter, Depends, Form, HTTPException, Response, Request
from fastapi import APIRouter, Form, HTTPException, Response, Request
import logging
from core.session_manager import SessionManager
from core.models import ChatMessage
from src.request_models import SessionResponse
from core.database import Session as DbSession, SessionLocal, Document, GalleryImage, utcnow_naive
from src.auth_helpers import (
effective_user,
_auth_disabled,
is_bearer_principal,
owner_filter,
request_capability,
require_chat_scope,
require_interactive_request,
)
from src.message_metadata import (
normalize_client_message_role,
sanitize_client_message_metadata,
)
from src.auth_helpers import effective_user, _auth_disabled, owner_filter
from src.session_image_cleanup import _generated_image_path_for_cleanup, session_image_refs
from src.session_actions import is_session_recently_active
from src.upload_handler import reserve_message_upload_references
from src.session_provenance import persist_session_endpoint_provenance
def _sanitize_export_filename(name: str) -> str:
@@ -137,11 +124,7 @@ def _verify_session_owner(request: Request, session_id: str, session_manager=Non
logger = logging.getLogger(__name__)
router = APIRouter(
prefix="/api",
tags=["sessions"],
dependencies=[Depends(require_chat_scope)],
)
router = APIRouter(prefix="/api", tags=["sessions"])
def _current_user_is_admin(request: Request, user: str | None) -> bool:
if not user:
@@ -170,10 +153,7 @@ def _reject_raw_endpoint_url_for_non_admin(
# Raw URLs make the server dial whatever host the request supplies. For
# non-admin users, require a saved endpoint row so normal owner scoping and
# endpoint validation have already happened.
# A bearer may be attributed to an admin owner for storage and endpoint
# visibility, but it is still not an interactive admin principal. Raw
# endpoint URLs therefore remain unavailable to every bearer request.
if is_bearer_principal(request) or (user and not _current_user_is_admin(request, user)):
if user and not _current_user_is_admin(request, user):
raise HTTPException(403, "Choose a registered model endpoint")
@@ -240,7 +220,6 @@ def setup_session_routes(
@router.get("/sessions")
def list_sessions(request: Request):
require_chat_scope(request)
user = effective_user(request)
active_incognito_id = str(request.query_params.get("active_incognito_id") or "").strip()
# Lazy purge: incognito sessions are ephemeral by design — wipe leftovers
@@ -252,37 +231,32 @@ def setup_session_routes(
# session is current and won't delete the live one — this server-side
# purge exists only to catch ghosts the frontend missed (tab close,
# crash). Only clean up rows old enough to be definitely orphaned.
# Listing is an owner-scoped read for bearer integrations. The legacy
# incognito cleanup query has no owner predicate and would otherwise
# let a chat token mutate another user's stale sessions before the
# owner-filtered result is assembled. Browser cleanup remains intact.
if not is_bearer_principal(request):
try:
from datetime import timedelta as _td
_cutoff = utcnow_naive() - _td(minutes=10)
_purge_db = SessionLocal()
try:
from datetime import timedelta as _td
_cutoff = utcnow_naive() - _td(minutes=10)
_purge_db = SessionLocal()
try:
from core.database import ChatMessage as _DbMsg
_ghosts = _purge_db.query(DbSession).filter(
DbSession.name.in_(("Nobody", "Incognito")),
DbSession.created_at < _cutoff,
).all()
for _g in _ghosts:
if active_incognito_id and _g.id == active_incognito_id:
continue
_purge_db.query(_DbMsg).filter(_DbMsg.session_id == _g.id).delete()
_purge_db.delete(_g)
if hasattr(session_manager, "delete_session"):
try:
session_manager.delete_session(_g.id)
except Exception:
pass
if _ghosts:
_purge_db.commit()
finally:
_purge_db.close()
except Exception:
pass
from core.database import ChatMessage as _DbMsg
_ghosts = _purge_db.query(DbSession).filter(
DbSession.name.in_(("Nobody", "Incognito")),
DbSession.created_at < _cutoff,
).all()
for _g in _ghosts:
if active_incognito_id and _g.id == active_incognito_id:
continue
_purge_db.query(_DbMsg).filter(_DbMsg.session_id == _g.id).delete()
_purge_db.delete(_g)
if hasattr(session_manager, "delete_session"):
try:
session_manager.delete_session(_g.id)
except Exception:
pass
if _ghosts:
_purge_db.commit()
finally:
_purge_db.close()
except Exception:
pass
user_sessions = session_manager.get_sessions_for_user(user)
# Fetch folder info from DB for each session
db = SessionLocal()
@@ -364,14 +338,10 @@ def setup_session_routes(
api_key: str = Form(""),
endpoint_id: str = Form(""),
):
require_chat_scope(request)
capability = request_capability(request)
probe_kwargs = {} if capability.allow_live_probes else {"allow_live_probes": False}
skip_val = str(skip_validation).lower() == "true"
user = effective_user(request)
endpoint_api_key = ""
endpoint_base_url = ""
endpoint_row = None
_reject_raw_endpoint_url_for_non_admin(request, user, endpoint_id, endpoint_url)
if endpoint_id and endpoint_id.strip():
from core.database import ModelEndpoint
@@ -405,14 +375,7 @@ def setup_session_routes(
from src.endpoint_resolver import build_headers
validation_headers = build_headers(effective_api_key, endpoint_base_url or endpoint_url)
if is_bearer_principal(request) and endpoint_row is not None:
# Bearer requests are cache-only, but cache-only does not mean
# caller-authorized. Validate explicit selections and choose an
# empty selection deterministically from the server-owned picker.
from routes.model_routes import _validate_bearer_model_selection
model_to_use = _validate_bearer_model_selection(endpoint_row, model_to_use)
elif skip_val:
if skip_val:
# skip_validation = trust the caller and do NOT probe /v1/models.
# Used for custom endpoints AND for bare placeholder sessions with no
# model at all (e.g. an email reply draft just needs a session to live
@@ -426,7 +389,6 @@ def setup_session_routes(
headers=validation_headers,
owner=user,
endpoint_id=endpoint_id.strip() if endpoint_id else None,
**probe_kwargs,
)
if not ids:
raise HTTPException(400, "Cannot reach /v1/models")
@@ -438,35 +400,28 @@ def setup_session_routes(
chat_ids = [m for m in ids if not any(p in m.lower() for p in _NON_CHAT)]
model_to_use = (chat_ids or ids)[0]
else:
# A bearer with an explicit model is already using an owner-scoped
# registered endpoint (raw URLs are rejected above). Do not turn
# that synchronous session-creation request into a live catalog
# probe merely to validate a value the caller supplied. Interactive
# requests retain the existing catalog-backed validation.
if capability.allow_live_probes:
from src.llm_core import list_model_ids
import os as _os
req_base = _os.path.basename(model_to_use.rstrip("/"))
avail = list_model_ids(
endpoint_url,
timeout=SESSION_MODEL_VALIDATION_TIMEOUT,
headers=validation_headers,
owner=user,
endpoint_id=endpoint_id.strip() if endpoint_id else None,
**probe_kwargs,
)
if not avail:
raise HTTPException(400, "Cannot reach /v1/models")
if model_to_use not in avail:
found = None
for a in avail:
if _os.path.basename(a.rstrip("/")) == req_base:
found = a
break
if not found:
raise HTTPException(400,
f"Model not found at server. Available: {', '.join(avail)}")
model_to_use = found
from src.llm_core import list_model_ids
import os as _os
req_base = _os.path.basename(model_to_use.rstrip("/"))
avail = list_model_ids(
endpoint_url,
timeout=SESSION_MODEL_VALIDATION_TIMEOUT,
headers=validation_headers,
owner=user,
endpoint_id=endpoint_id.strip() if endpoint_id else None,
)
if not avail:
raise HTTPException(400, "Cannot reach /v1/models")
if model_to_use not in avail:
found = None
for a in avail:
if _os.path.basename(a.rstrip("/")) == req_base:
found = a
break
if not found:
raise HTTPException(400,
f"Model not found at server. Available: {', '.join(avail)}")
model_to_use = found
sid = str(uuid.uuid4())
user = effective_user(request)
@@ -478,20 +433,6 @@ def setup_session_routes(
rag=str(rag).lower() == "true" if rag else False,
owner=user,
)
if endpoint_row is not None or request_api_key:
try:
persist_session_endpoint_provenance(
session_manager,
sid,
session,
model_endpoint_id=getattr(endpoint_row, "id", None),
endpoint_provenance=(
"registered" if endpoint_row is not None else "direct"
),
)
except Exception as exc:
logger.error("Failed to persist session endpoint provenance for %s: %s", sid, exc)
raise HTTPException(500, "Failed to persist session endpoint provenance") from exc
# Set auth headers for custom API-key endpoints
resolved_key = request_api_key
resolved_base = endpoint_url
@@ -502,17 +443,14 @@ def setup_session_routes(
from src.endpoint_resolver import build_headers
session.headers = build_headers(resolved_key, resolved_base)
_persist_session_headers(sid, session.headers)
# A bearer can create owner-attributed chat data, but must not cause
# owner lifecycle automation or webhook delivery as a side effect.
if not is_bearer_principal(request):
# Fire webhook (sync-safe)
if webhook_manager:
webhook_manager.fire_and_forget("session.created", {
"session_id": sid, "name": session.name, "model": model_to_use,
})
# Fire event for automation tasks
from src.event_bus import fire_event
fire_event("session_created", user)
# Fire webhook (sync-safe)
if webhook_manager:
webhook_manager.fire_and_forget("session.created", {
"session_id": sid, "name": session.name, "model": model_to_use,
})
# Fire event for automation tasks
from src.event_bus import fire_event
fire_event("session_created", user)
return SessionResponse(
id=sid,
name=session.name,
@@ -527,7 +465,6 @@ def setup_session_routes(
model: str = Form(None), endpoint_url: str = Form(None),
endpoint_id: str = Form(None),
):
require_chat_scope(request)
_verify_session_owner(request, sid)
try:
session = session_manager.get_session(sid)
@@ -555,7 +492,6 @@ def setup_session_routes(
_reject_raw_endpoint_url_for_non_admin(request, user, endpoint_id, endpoint_url)
endpoint_api_key = ""
endpoint_base_url = ""
endpoint_row = None
if endpoint_id:
from core.database import ModelEndpoint
from src.auth_helpers import owner_filter
@@ -571,23 +507,13 @@ def setup_session_routes(
ep = q.first()
if not ep:
raise HTTPException(400, "Model endpoint no longer exists")
endpoint_row = ep
endpoint_base_url = ep.base_url or ""
endpoint_api_key = ep.api_key or ""
endpoint_url = build_chat_url(normalize_base(endpoint_base_url))
finally:
_db.close()
if is_bearer_principal(request) and endpoint_row is not None:
from routes.model_routes import _validate_bearer_model_selection
# Validate before mutating either the in-memory or durable
# session. The same server-owned inventory is enforced again
# immediately before each bearer LLM consumer.
model = _validate_bearer_model_selection(endpoint_row, model)
session.model = model
session.endpoint_url = endpoint_url
session.model_endpoint_id = getattr(endpoint_row, "id", None)
session.endpoint_provenance = "registered" if endpoint_row is not None else None
# Update auth headers from the endpoint's stored API key
if endpoint_api_key:
from src.endpoint_resolver import build_headers
@@ -602,8 +528,6 @@ def setup_session_routes(
db_session.model = model
db_session.endpoint_url = endpoint_url
db_session.headers = session.headers or {}
db_session.model_endpoint_id = getattr(endpoint_row, "id", None)
db_session.endpoint_provenance = "registered" if endpoint_row is not None else None
db_session.updated_at = utcnow_naive()
db.commit()
finally:
@@ -615,7 +539,6 @@ def setup_session_routes(
@router.post("/session/{sid}/inject_messages")
async def inject_messages(request: Request, sid: str):
"""Bulk-inject messages into a session's history (for group chat sync)."""
require_chat_scope(request)
_verify_session_owner(request, sid)
try:
sess = session_manager.get_session(sid)
@@ -631,7 +554,7 @@ def setup_session_routes(
upload_handler,
owner,
message.get("content"),
sanitize_client_message_metadata(message.get("metadata")),
message.get("metadata"),
)
if missing_id:
raise HTTPException(
@@ -641,24 +564,18 @@ def setup_session_routes(
except (AttributeError, TypeError, ValueError) as exc:
raise HTTPException(400, "Invalid message attachment metadata") from exc
for m in messages:
sess.add_message(ChatMessage(
normalize_client_message_role(m.get("role", "user"), default="user"),
m["content"],
metadata=sanitize_client_message_metadata(m.get("metadata")),
))
sess.add_message(ChatMessage(m["role"], m["content"], metadata=m.get("metadata")))
session_manager.save_sessions()
return {"ok": True, "count": len(messages)}
@router.post("/session/{sid}/delete")
def delete_session_beacon(request: Request, sid: str):
"""Delete session via POST (for navigator.sendBeacon on page close)."""
require_chat_scope(request)
return delete_session(request, sid)
@router.post("/sessions/bulk-delete")
async def bulk_delete_sessions(request: Request):
"""Delete multiple sessions (for compare cleanup via sendBeacon)."""
require_chat_scope(request)
from core.database import ChatMessage as _CM
try:
body = await request.json()
@@ -688,7 +605,6 @@ def setup_session_routes(
@router.delete("/session/{sid}")
def delete_session(request: Request, sid: str):
"""Permanently delete a session and all its messages."""
require_chat_scope(request)
_verify_session_owner(request, sid, session_manager)
try:
# Block deletion of starred/favorited sessions
@@ -723,7 +639,6 @@ def setup_session_routes(
@router.delete("/sessions/all")
def delete_all_sessions(request: Request):
"""Admin only: permanently delete ALL sessions and their messages."""
require_chat_scope(request)
from core.middleware import require_admin
require_admin(request)
@@ -777,7 +692,6 @@ def setup_session_routes(
@router.post("/session/{sid}/archive")
def archive_session(request: Request, sid: str):
"""Archive a session, keeping its data but removing it from active sessions."""
require_chat_scope(request)
_verify_session_owner(request, sid)
try:
# First check if session exists
@@ -816,7 +730,6 @@ def setup_session_routes(
@router.post("/session/{sid}/unarchive")
def unarchive_session(request: Request, sid: str):
"""Restore an archived session back to the active session list."""
require_chat_scope(request)
_verify_session_owner(request, sid)
db = SessionLocal()
try:
@@ -847,7 +760,6 @@ def setup_session_routes(
@router.get("/sessions/archived")
def list_archived_sessions(request: Request, search: str = "", offset: int = 0, limit: int = 20, sort: str = "recent", model: str = ""):
"""List archived sessions for the archive browser."""
require_chat_scope(request)
user = effective_user(request)
db = SessionLocal()
try:
@@ -895,7 +807,6 @@ def setup_session_routes(
Supported formats: md (markdown), txt (plain text), json, html
"""
require_chat_scope(request)
_verify_session_owner(request, sid)
try:
session = session_manager.get_session(sid)
@@ -982,7 +893,6 @@ def setup_session_routes(
@router.post("/sessions/save")
def sessions_save_now(request: Request):
require_chat_scope(request)
user = effective_user(request)
if not user:
raise HTTPException(401, "Not authenticated")
@@ -996,15 +906,6 @@ def setup_session_routes(
model: str = Form("gpt-4o"),
rag: str = Form(None)
):
require_chat_scope(request)
# This legacy alias uses the server-owned OPENAI_API_KEY rather than a
# caller-selected, owner-visible ModelEndpoint. A bearer must not turn
# that credential into an owner-attributed session whose provenance
# cannot be represented as either a registered endpoint or a direct
# caller-supplied key. Registered bearer chat remains available through
# POST /api/session with endpoint_id.
if is_bearer_principal(request):
raise HTTPException(403, "Bearer callers must choose a registered model endpoint")
if not OPENAI_API_KEY:
raise HTTPException(400, "Server missing OPENAI_API_KEY")
sid = str(uuid.uuid4())
@@ -1019,15 +920,13 @@ def setup_session_routes(
)
session.headers = {"Authorization": f"Bearer {OPENAI_API_KEY}"}
session_manager.save_sessions()
if not is_bearer_principal(request):
from src.event_bus import fire_event
fire_event("session_created", user)
from src.event_bus import fire_event
fire_event("session_created", user)
return {"id": sid, "name": "", "model": model}
@router.post("/session/{session_id}/important")
async def mark_session_important(request: Request, session_id: str, important: bool = Form(True)):
"""Mark a session as important to protect it from automatic cleanup."""
require_chat_scope(request)
_verify_session_owner(request, session_id)
try:
# Validate session exists
@@ -1065,8 +964,6 @@ def setup_session_routes(
@router.post("/session/{session_id}/compact")
async def compact_session(request: Request, session_id: str):
"""Summarize older messages into one compacted history entry."""
require_chat_scope(request)
capability = request_capability(request)
_verify_session_owner(request, session_id)
try:
session = session_manager.get_session(session_id)
@@ -1086,22 +983,13 @@ def setup_session_routes(
if not older:
raise HTTPException(400, "Nothing old enough to compact")
if capability.is_bearer:
from routes.chat_helpers import _validate_bearer_session_model
_validate_bearer_session_model(session, owner=effective_user(request))
from src.context_compactor import SELF_SUMMARY_SYSTEM_PROMPT
from src.endpoint_resolver import resolve_endpoint
from src.llm_core import llm_call_async
owner = getattr(session, "owner", None) or effective_user(request)
if capability.allow_live_probes:
from src.endpoint_resolver import resolve_endpoint
url, model, headers = resolve_endpoint("utility", owner=owner)
if not url or not model:
url, model, headers = session.endpoint_url, session.model, session.headers
else:
url, model, headers = resolve_endpoint("utility", owner=owner)
if not url or not model:
url, model, headers = session.endpoint_url, session.model, session.headers
if not url or not model:
raise HTTPException(400, "No model configured for compaction")
@@ -1120,9 +1008,6 @@ def setup_session_routes(
for m in older
)
try:
compact_kwargs = {}
if not capability.allow_live_probes:
compact_kwargs["allow_live_probes"] = False
summary = await llm_call_async(
url,
model,
@@ -1131,7 +1016,6 @@ def setup_session_routes(
max_tokens=1024,
headers=headers,
timeout=60,
**compact_kwargs,
)
except Exception as e:
logger.error("Manual compaction failed: %s", e)
@@ -1157,10 +1041,7 @@ def setup_session_routes(
"message_count": len(new_history),
}
@router.post(
"/sessions/auto-sort",
dependencies=[Depends(require_interactive_request)],
)
@router.post("/sessions/auto-sort")
def auto_sort_sessions(request: Request, skip_llm: bool = False):
"""Use AI to categorize all sessions into folders.
@@ -1169,8 +1050,6 @@ def setup_session_routes(
after Phase 1 used by the "Tidy (no AI)" UI affordance so
users can clean junk without spending tokens.
"""
require_chat_scope(request)
require_interactive_request(request)
from src.llm_core import llm_call
user = effective_user(request)
single_user_mode = not user and _auth_disabled()
@@ -1451,8 +1330,6 @@ def setup_session_routes(
@router.get("/session/{session_id}/context_info")
async def get_context_info(request: Request, session_id: str):
"""Get the real context length for a session's model from the endpoint."""
require_chat_scope(request)
capability = request_capability(request)
_verify_session_owner(request, session_id)
session = session_manager.get_session(session_id)
if not session:
@@ -1461,10 +1338,7 @@ def setup_session_routes(
return {"context_length": None}
try:
from src.model_context import get_context_length
context_kwargs = {}
if not capability.allow_live_probes:
context_kwargs["allow_live_probes"] = False
ctx = get_context_length(session.endpoint_url, session.model, **context_kwargs)
ctx = get_context_length(session.endpoint_url, session.model)
return {"context_length": ctx, "model": session.model}
except Exception:
return {"context_length": None}
-6
View File
@@ -16,7 +16,6 @@ from pathlib import Path
from typing import Dict, Any
from core.platform_compat import IS_APPLE_SILICON, which_tool
from core.middleware import INTERNAL_TOOL_USER
from src.auth_helpers import is_bearer_principal
from src.host_docker_access import (
HOST_DOCKER_ACCESS_HINT,
host_docker_access_enabled as _host_docker_access_enabled,
@@ -54,11 +53,6 @@ from core.platform_compat import (
def _require_admin(request: Request):
"""Reject non-admin callers. Shell exec is admin-only — never expose to
regular users; that's RCE-after-signup."""
# This route predates the shared middleware helper and is also called
# directly by a few integration paths. Reject the credential class before
# trusting a caller-supplied current_user that might look administrative.
if is_bearer_principal(request):
raise HTTPException(403, "API tokens cannot use admin host-control surfaces")
auth_manager = getattr(request.app.state, "auth_manager", None)
if not auth_manager:
# No auth at all — only safe in fully-trusted localhost dev mode
+21 -262
View File
@@ -13,12 +13,11 @@ from typing import List, Optional
import httpx
from fastapi import APIRouter, Depends, HTTPException, Request
from fastapi import APIRouter, HTTPException, Request
from pydantic import BaseModel, Field
from services.memory.skills import SkillsManager
from src.auth_helpers import require_interactive_request
from src.prompt_security import untrusted_context_message
from src.auth_helpers import get_current_user
from core.middleware import require_admin
logger = logging.getLogger(__name__)
@@ -108,23 +107,6 @@ def _skill_test_task(skill: dict) -> str:
)
def _skill_test_messages(md: str, task: str) -> list[dict]:
"""Keep user-editable skill text out of the trusted system role."""
return [
{
"role": "system",
"content": (
"You are TESTING a skill. Follow the supplied reusable procedure "
"to complete the user's task for real, using available tools step "
"by step. If the skill is wrong, unclear, or references tools that "
"do not exist, do your best; the problems will be reviewed afterward."
),
},
untrusted_context_message("skill under test", md),
{"role": "user", "content": task},
]
async def _eval_skill_run(skill_md: str, task: str, transcript: str,
url: str, model: str, headers: Optional[dict]) -> dict:
"""LLM-as-judge: grade a skill test run from its transcript. Advisory only.
@@ -429,21 +411,7 @@ async def _eval_skill_retrieval_precision(skill_md: str, others: list,
_skill_test_jobs: dict = {}
async def _run_skill_test_job(
key,
name,
md,
task,
url,
model,
headers,
owner,
skills_manager=None,
*,
messages=None,
transcript=None,
exact_approval=None,
):
async def _run_skill_test_job(key, name, md, task, url, model, headers, owner, skills_manager=None):
"""Background coroutine: run the skill in an agent loop, capture a condensed
log + transcript, then have the judge grade it. Writes into _skill_test_jobs."""
import json as _json
@@ -453,7 +421,7 @@ async def _run_skill_test_job(
if job is None:
return
log = job["log"]
transcript = transcript if isinstance(transcript, list) else []
transcript = []
say_buf = []
def _flush_say():
@@ -461,12 +429,18 @@ async def _run_skill_test_job(
log.append({"type": "say", "text": "".join(say_buf)})
say_buf.clear()
messages = list(messages) if isinstance(messages, list) else _skill_test_messages(md, task)
messages = [
{"role": "system", "content":
"You are TESTING a skill. Below is a reusable skill (a procedure). Follow it "
"to complete the user's task for real, using your available tools, step by "
"step. If the skill is wrong, unclear, or references tools that don't exist, "
"do your best — the problems will be reviewed afterward.\n\n=== SKILL ===\n" + md},
{"role": "user", "content": task},
]
try:
async for chunk in stream_agent_loop(
url, model, messages, headers=headers,
temperature=0.3, max_tokens=0, max_rounds=8, owner=owner,
exact_approval=exact_approval,
):
if not chunk.startswith("data: ") or chunk.strip() == "data: [DONE]":
continue
@@ -484,25 +458,8 @@ async def _run_skill_test_job(
elif d.get("type") == "tool_output":
_flush_say()
out = str(d.get("output") or "")[:600]
tool_log = {"type": "tool_output", "output": out}
approval = d.get("ask_user")
if isinstance(approval, dict):
tool_log["ask_user"] = approval
log.append(tool_log)
log.append({"type": "tool_output", "output": out})
transcript.append(f"[output] {out}\n")
if (
isinstance(approval, dict)
and approval.get("kind") == "tool_approval"
and approval.get("approval_id")
):
# Manual skill tests have their own polling UI instead of a
# chat session. Pause the run and retain only server-side
# continuation state until the same owner approves/denies
# this exact sealed action.
job["status"] = "awaiting_approval"
job["approval"] = approval
job["_transcript"] = transcript
return
elif d.get("type") == "agent_step":
_flush_say()
log.append({"type": "agent_step", "round": d.get("round")})
@@ -514,9 +471,6 @@ async def _run_skill_test_job(
_flush_say()
log.append({"type": "error", "error": str(e)})
job.pop("approval", None)
job.pop("_transcript", None)
job.pop("_run", None)
log.append({"type": "evaluating"})
try:
job["verdict"] = await _eval_skill_run(md, task, "".join(transcript), url, model, headers)
@@ -740,8 +694,12 @@ async def _run_skill_test_once(md: str, task: str, url, model, headers, owner) -
import json as _json
from src.agent_loop import stream_agent_loop
transcript = []
approval_required = None
messages = _skill_test_messages(md, task)
messages = [
{"role": "system", "content":
"You are TESTING a skill. Follow this skill's procedure to complete the task "
"for real, using your tools, step by step.\n\n=== SKILL ===\n" + md},
{"role": "user", "content": task},
]
try:
# max_tokens explicitly set: passing 0 lets some upstreams (Ollama,
# OpenAI-compat) generate an empty completion, which manifested as
@@ -761,44 +719,11 @@ async def _run_skill_test_once(md: str, task: str, url, model, headers, owner) -
transcript.append(f"\n[tool {d.get('tool')}] {str(d.get('command') or d.get('args') or '')[:300]}\n")
elif d.get("type") == "tool_output":
transcript.append(f"[output] {str(d.get('output') or '')[:600]}\n")
approval = d.get("ask_user")
if (
isinstance(approval, dict)
and approval.get("kind") == "tool_approval"
):
approval_required = approval
break
elif d.get("type") == "agent_step":
transcript.append(f"\n--- round {d.get('round')} ---\n")
except Exception as e:
transcript.append(f"\n[run error] {e}\n")
text = "".join(transcript)
if approval_required is not None:
# Unattended audits have no authority to approve and no UI that could
# resume this record. Destructively deny it now instead of leaving a
# reusable opaque grant pending until TTL/cap eviction.
try:
from src.tool_approvals import tool_approval_store
tool_approval_store.consume(
approval_required.get("approval_id"),
decision="deny",
owner=owner,
session_id=None,
)
except Exception:
logger.debug("Could not retire unattended skill approval", exc_info=True)
return text, {
"verdict": "inconclusive",
"confidence": 1.0,
"summary": (
"This automated audit reached an exact action that requires "
"a human approval; no action was executed."
),
"issues": [
"Run this skill's manual test and review the sealed action."
],
"approval_required": True,
}
verdict = await _eval_skill_run(md, task, text, url, model, headers)
return text, verdict
@@ -938,26 +863,6 @@ async def _audit_one_skill(skills_manager, skill, url, model, headers,
transcript, verdict = await _run_skill_test_once(md, task, url, model, headers, owner)
v = verdict.get("verdict")
log(f"{name}: verdict = {v} ({verdict.get('summary', '')[:80]})")
if verdict.get("approval_required"):
# An unattended audit is not authority for an action influenced by the
# skill under test. Preserve the skill's current publication/confidence
# state and route the exact action to the manual test UI instead of
# letting a safety pause demote, rewrite, or auto-publish the skill.
skills_manager.set_audit(
name,
"inconclusive",
by_teacher=False,
worker_model=model,
owner=owner,
)
status = skill.get("status") or "draft"
log(f"{name}: {status} unchanged — exact action needs manual approval")
return {
"skill": name,
"result": "approval_required",
"verdict": verdict,
"status": status,
}
if v == "pass":
# Procedure works. If the reviewer still flagged metadata (tags/category/
# when_to_use/description), do ONE fixer pass to correct the frontmatter
@@ -1181,14 +1086,10 @@ async def run_scheduled_skill_audit(skills_manager: SkillsManager,
def setup_skills_routes(skills_manager: SkillsManager) -> APIRouter:
router = APIRouter(
prefix="/api/skills",
tags=["skills"],
dependencies=[Depends(require_interactive_request)],
)
router = APIRouter(prefix="/api/skills", tags=["skills"])
def _owner(request: Request) -> Optional[str]:
return require_interactive_request(request)
return get_current_user(request)
def _verify_owner(skill: dict, user: Optional[str]):
if user is None:
@@ -1530,19 +1431,6 @@ def setup_skills_routes(skills_manager: SkillsManager) -> APIRouter:
logger.warning(f"Skill-test model resolve failed: {_e}")
key = (user or "", name)
previous_job = _skill_test_jobs.get(key) or {}
previous_approval = previous_job.get("approval") or {}
if previous_approval.get("approval_id"):
try:
from src.tool_approvals import tool_approval_store
tool_approval_store.consume(
previous_approval["approval_id"],
decision="deny",
owner=user,
session_id=None,
)
except Exception:
logger.debug("Could not retire replaced skill approval", exc_info=True)
_skill_test_jobs[key] = {
"status": "running",
"task": task,
@@ -1551,138 +1439,10 @@ def setup_skills_routes(skills_manager: SkillsManager) -> APIRouter:
"started": _time.time(),
"log": [{"type": "skill_test_start", "task": task, "skill": name, "model": model}],
"verdict": None,
"_run": {
"md": md,
"url": url,
"model": model,
"headers": headers,
"owner": user,
},
}
_asyncio.create_task(_run_skill_test_job(key, name, md, task, url, model, headers, user, skills_manager))
return {"ok": True, "status": "running", "skill": name, "model": model}
@router.post("/{skill_id}/test-approval")
async def approve_skill_test_action(request: Request, skill_id: str):
"""Resume a manual skill test with one exact server-sealed action."""
import asyncio as _asyncio
from src.tool_approvals import tool_approval_store
user = _owner(request)
skills = skills_manager.load(owner=user)
match = next(
(s for s in skills if s.get("name") == skill_id or s.get("id") == skill_id),
None,
)
if not match:
raise HTTPException(404, "Skill not found")
_verify_owner(match, user)
name = match.get("name")
key = (user or "", name)
job = _skill_test_jobs.get(key)
if not job or job.get("status") != "awaiting_approval":
raise HTTPException(409, "This skill test is not awaiting an approval.")
body = await request.json()
if not isinstance(body, dict):
raise HTTPException(400, "Tool approval body must be a JSON object.")
approval_id = str(body.get("approval_id") or "")
decision = str(body.get("decision") or "").strip().lower()
expected = job.get("approval") or {}
if approval_id != str(expected.get("approval_id") or ""):
raise HTTPException(409, "This approval does not match the pending skill test action.")
if decision not in {"approve", "deny"}:
raise HTTPException(400, "Invalid tool approval decision.")
pending = tool_approval_store.peek(approval_id)
normalized_owner = str(user or "").strip().casefold()
if (
pending is None
or pending.owner != normalized_owner
or pending.session_id != ""
):
raise HTTPException(409, "This tool approval is invalid or expired.")
exact_approval = tool_approval_store.consume(
approval_id,
decision=decision,
owner=user,
session_id=None,
# The button here says "Allow once" and there is no chat to carry a
# scope into, so the gate must re-arm behind the sealed action.
allow_continuation=False,
)
if decision == "approve" and exact_approval is None:
raise HTTPException(409, "This tool approval could not be consumed.")
job.pop("approval", None)
if decision == "deny":
job.pop("_transcript", None)
job.pop("_run", None)
job["log"].append({
"type": "approval_denied",
"text": "Exact action denied; the skill test stopped without executing it.",
})
job["verdict"] = {
"verdict": "inconclusive",
"confidence": 1.0,
"summary": "The test stopped because its exact action was denied.",
"issues": [],
}
job["status"] = "done"
return {"ok": True, "status": "done", "decision": "deny"}
run = job.get("_run") or {}
transcript = job.pop("_transcript", [])
# stream_agent_loop owns its per-round message list internally. Rebuild
# continuation context from the original untrusted skill plus the
# accumulated transcript so repeated approvals do not lose earlier
# approved results, while keeping every transcript byte tainted.
messages = _skill_test_messages(
run.get("md", ""),
job.get("task", ""),
)
if transcript:
messages.append(untrusted_context_message(
"skill test transcript",
"".join(str(item) for item in transcript),
))
messages.extend([
{
"role": "assistant",
"content": str(expected.get("question") or "Allow this exact action once?"),
},
{
"role": "user",
"content": (
f"Approved the exact {exact_approval.pending.tool_name} "
"action shown above once."
),
},
])
job["status"] = "running"
job["log"].append({
"type": "approval_granted",
"text": (
f"Approved exact {exact_approval.pending.tool_name} action once; "
"resuming test."
),
})
_asyncio.create_task(_run_skill_test_job(
key,
name,
run.get("md", ""),
job.get("task", ""),
run.get("url"),
run.get("model"),
run.get("headers"),
run.get("owner"),
skills_manager,
messages=messages,
transcript=transcript,
exact_approval=exact_approval,
))
return {"ok": True, "status": "running", "decision": "approve"}
@router.get("/{skill_id}/test-status")
async def test_skill_status(request: Request, skill_id: str):
"""Current background-test state for a skill (status / log / verdict)."""
@@ -1699,7 +1459,6 @@ def setup_skills_routes(skills_manager: SkillsManager) -> APIRouter:
"model": job.get("model"),
"log": job.get("log", []),
"verdict": job.get("verdict"),
"approval": job.get("approval"),
}
@router.post("/audit-all")
-5
View File
@@ -1,5 +0,0 @@
"""Task route domain package (slice 2p, #4082/#4071).
Contains task_routes.py, migrated from the flat routes/ directory.
Backward-compat shim at routes/task_routes.py re-exports from here.
"""
File diff suppressed because it is too large Load Diff
+1177 -14
View File
File diff suppressed because it is too large Load Diff
+4 -28
View File
@@ -6,7 +6,7 @@ import asyncio
import shutil
import uuid
from pathlib import Path
from fastapi import APIRouter, Depends, Request, File, UploadFile, HTTPException, Form
from fastapi import APIRouter, Request, File, UploadFile, HTTPException, Form
from typing import List, Optional
import logging
from core.middleware import require_admin
@@ -21,12 +21,7 @@ from core.database import (
Note,
Session as DbSession,
)
from src.auth_helpers import (
effective_user,
is_bearer_principal,
require_chat_scope,
require_non_bearer_request,
)
from src.auth_helpers import effective_user
from src.attachment_refs import attachment_refs_from_metadata
from src.constants import GENERATED_IMAGES_DIR
from src.upload_handler import (
@@ -37,11 +32,7 @@ from src.upload_handler import (
logger = logging.getLogger(__name__)
router = APIRouter(
prefix="/api/upload",
tags=["upload"],
dependencies=[Depends(require_chat_scope)],
)
router = APIRouter(prefix="/api/upload", tags=["upload"])
UPLOAD_RESPONSE_HEADERS = {"X-Content-Type-Options": "nosniff"}
def _upload_ids_from_persisted_text(value: object) -> set[str]:
@@ -270,7 +261,6 @@ def setup_upload_routes(upload_handler):
session_id: Optional[str] = Form(None),
):
"""Upload files with enhanced security and organization."""
require_chat_scope(request)
if not isinstance(session_id, str):
session_id = None
if not files:
@@ -330,7 +320,6 @@ def setup_upload_routes(upload_handler):
@router.post("/cleanup")
async def manual_cleanup(request: Request):
"""Manually trigger cleanup of old uploads."""
require_chat_scope(request)
require_admin(request)
try:
cleaned_count = await asyncio.to_thread(
@@ -354,7 +343,6 @@ def setup_upload_routes(upload_handler):
@router.get("/stats")
async def upload_stats(request: Request):
"""Get statistics about uploaded files."""
require_chat_scope(request)
require_admin(request)
try:
return upload_handler.get_upload_stats()
@@ -367,7 +355,6 @@ def setup_upload_routes(upload_handler):
"""Serve an uploaded file by its ID. `?thumb=1` returns a small cached
JPEG thumbnail for images (used by chat attachment previews) so the
client isn't downloading the full-resolution photo just to show it tiny."""
require_chat_scope(request)
if not upload_handler.validate_upload_id(file_id):
raise HTTPException(400, "Invalid file ID")
import mimetypes as _mt
@@ -384,14 +371,7 @@ def setup_upload_routes(upload_handler):
auth_configured = bool(auth_mgr and auth_mgr.is_configured)
current_user = effective_user(request)
file_owner = info.get("owner") if info else None
if is_bearer_principal(request):
# A token owner is an owner-bound data principal, even when that
# owner is an administrator. Do not reuse the browser admin
# fallback for bearer downloads or an admin token can read another
# user's upload by ID.
if not current_user or file_owner != current_user:
raise HTTPException(404, "File not found")
elif auth_configured:
if auth_configured:
if not current_user:
raise HTTPException(403, "Access denied")
if file_owner != current_user and not auth_mgr.is_admin(current_user):
@@ -473,8 +453,6 @@ def setup_upload_routes(upload_handler):
"""Return the vision-model OCR/description for an uploaded image.
Cached under UPLOAD_DIR/.vision/{file_id}.txt first call computes,
subsequent loads are instant. Pass force=1 to recompute."""
require_chat_scope(request)
require_non_bearer_request(request)
if not upload_handler.validate_upload_id(file_id):
raise HTTPException(400, "Invalid file ID")
info = _load_upload_info(file_id)
@@ -519,8 +497,6 @@ def setup_upload_routes(upload_handler):
async def put_vision_text(request: Request, file_id: str):
"""Persist a user-edited vision/OCR text for an attachment. Stored in
the same cache file so the chat send picks it up as the override."""
require_chat_scope(request)
require_non_bearer_request(request)
if not upload_handler.validate_upload_id(file_id):
raise HTTPException(400, "Invalid file ID")
info = _load_upload_info(file_id)
+41 -157
View File
@@ -2,7 +2,6 @@
import uuid
import logging
import json
from typing import Optional
import httpx
@@ -10,15 +9,9 @@ from fastapi import APIRouter, HTTPException, Request, Form
from pydantic import BaseModel, Field
from core.database import SessionLocal, Webhook, ModelEndpoint
from src.auth_helpers import (
is_bearer_principal,
owner_filter,
request_capability,
require_chat_scope,
)
from src.auth_helpers import owner_filter
from src.url_security import validate_public_http_url
from src.webhook_manager import WebhookManager, validate_webhook_url, validate_events
from src.session_provenance import persist_session_endpoint_provenance
logger = logging.getLogger(__name__)
@@ -39,15 +32,14 @@ def _select_api_chat_fallback_endpoint(db, token_owner: Optional[str]):
legacy null-owner ("shared") rows. Owner-scoped: an unscoped .first() would
let a chat-scoped token fall back onto another user's private endpoint and
silently spend that owner's API key/quota. Prefer owner rows before shared
rows. Fails closed when token_owner is absent; the sync endpoint requires
an owner-scoped bearer before this helper is reached.
rows. Fails closed to null-owner rows only when token_owner is absent.
Does not validate base_url admin-configured local/LAN endpoints remain allowed.
"""
query = db.query(ModelEndpoint).filter(ModelEndpoint.is_enabled == True) # noqa: E712
if not token_owner:
return None
query = owner_filter(query, ModelEndpoint, token_owner)
return query.order_by(ModelEndpoint.owner.desc(), ModelEndpoint.created_at).first()
if token_owner:
query = owner_filter(query, ModelEndpoint, token_owner)
return query.order_by(ModelEndpoint.owner.desc(), ModelEndpoint.created_at).first()
return query.filter(ModelEndpoint.owner == None).order_by(ModelEndpoint.created_at).first() # noqa: E711
def _caller_owns_session(sess_owner, caller) -> bool:
@@ -69,89 +61,6 @@ def _caller_owns_session(sess_owner, caller) -> bool:
return sess_owner == caller
def _cached_endpoint_model_ids(endpoint) -> list[str]:
"""Return model IDs already stored for a configured endpoint.
The synchronous bearer integration may use a cached model or the provider's
``auto`` alias, but it must not turn an ordinary chat request into a remote
catalog probe. Malformed/legacy cache shapes are treated as empty.
"""
try:
from routes.model_routes import _effective_endpoint_kind, _picker_models_for_endpoint
base_url = getattr(endpoint, "base_url", "") or ""
kind = _effective_endpoint_kind(endpoint, base_url)
models, _ = _picker_models_for_endpoint(endpoint, base_url, kind)
return models
except Exception:
raw = getattr(endpoint, "cached_models", None)
pinned_raw = getattr(endpoint, "pinned_models", None)
hidden_raw = getattr(endpoint, "hidden_models", None)
if not raw and not pinned_raw:
return []
try:
value = json.loads(raw) if isinstance(raw, str) else raw
pinned = json.loads(pinned_raw) if isinstance(pinned_raw, str) else pinned_raw
hidden = json.loads(hidden_raw) if isinstance(hidden_raw, str) else hidden_raw
except (TypeError, ValueError):
return []
if isinstance(value, dict):
value = value.get("data") or value.get("models") or []
if isinstance(pinned, dict):
pinned = pinned.get("data") or pinned.get("models") or []
if isinstance(hidden, dict):
hidden = hidden.get("data") or hidden.get("models") or []
if not isinstance(value, list):
value = []
if not isinstance(pinned, list):
pinned = []
if not isinstance(hidden, list):
hidden = []
raw_ids = value + pinned
hidden_ids = {str(item).strip() for item in hidden if str(item).strip()}
ids = []
for item in raw_ids:
if isinstance(item, str) and item.strip():
model_id = item.strip()
elif isinstance(item, dict):
model_id = item.get("id") or item.get("name") or item.get("model")
if not isinstance(model_id, str) or not model_id.strip():
continue
model_id = model_id.strip()
else:
continue
if model_id not in hidden_ids and model_id not in ids:
ids.append(model_id)
return ids
def _validate_bearer_sync_model(endpoint, requested_model: str) -> str:
"""Validate a configured sync model without probing its provider."""
try:
from routes.model_routes import _validate_bearer_model_selection
return _validate_bearer_model_selection(endpoint, requested_model)
except ImportError:
# Keep the lightweight webhook test/import seam usable when optional
# route modules are deliberately stubbed. Production uses the
# central picker validator above; this fallback remains cache-only.
models = _cached_endpoint_model_ids(endpoint)
requested = str(requested_model or "").strip()
if requested and requested in models:
return requested
if not requested and models:
return models[0]
if (
requested
and not models
and not getattr(endpoint, "cached_models", None)
and not getattr(endpoint, "pinned_models", None)
and "localhost" in str(getattr(endpoint, "base_url", "")).lower()
):
return requested
raise HTTPException(400, "Model is not permitted for this endpoint")
def setup_webhook_routes(
webhook_manager: WebhookManager,
auth_manager,
@@ -327,16 +236,16 @@ def setup_webhook_routes(
@router.post("/v1/chat")
async def sync_chat(request: Request, body: SyncChatRequest):
if getattr(request.state, "api_token", False) is not True:
if not getattr(request.state, "api_token", False):
raise HTTPException(403, "This endpoint requires an API token")
token_owner = require_chat_scope(request)
capability = request_capability(request)
if not token_owner:
raise HTTPException(403, "API token has no owner")
scopes = set(getattr(request.state, "api_token_scopes", []) or [])
if "chat" not in scopes:
raise HTTPException(403, "API token is not scoped for chat")
token_owner = getattr(request.state, "api_token_owner", None)
from core.models import ChatMessage
from src.llm_core import llm_call_async
from src.endpoint_resolver import build_chat_url, build_headers, normalize_base
from src.endpoint_resolver import build_chat_url, build_headers, build_models_url, normalize_base
message = body.message.strip()
if not message:
@@ -366,12 +275,6 @@ def setup_webhook_routes(
_sess_owner = getattr(sess, "owner", None)
if not _caller_owns_session(_sess_owner, _tok_user):
raise HTTPException(404, "Session not found")
if is_bearer_principal(request):
from routes.chat_helpers import _validate_bearer_session_model
# Existing-session resume is an LLM boundary too; ownership
# alone must not authorize the persisted endpoint/model.
_validate_bearer_session_model(sess, owner=token_owner)
# --- Case 2: Direct API key + model (no pre-configured endpoint needed) ---
if not sess and body.api_key:
@@ -404,12 +307,6 @@ def setup_webhook_routes(
session_id=sid, name="API Chat", endpoint_url=endpoint_url,
model=model, owner=token_owner,
)
persist_session_endpoint_provenance(
session_manager,
sid,
sess,
endpoint_provenance="direct",
)
sess.headers = build_headers(api_key, base_url)
session_manager.save_sessions()
session_id = sid
@@ -429,27 +326,39 @@ def setup_webhook_routes(
base_url = normalize_base(ep.base_url)
endpoint_url = build_chat_url(base_url)
model = body.model or ""
model = body.model or "auto"
api_key = ep.api_key
if getattr(ep, "provider_auth_id", None):
try:
from src.endpoint_resolver import resolve_endpoint_runtime
runtime_kwargs = {}
if not capability.allow_live_probes:
runtime_kwargs["allow_live_probes"] = False
base_url, api_key = resolve_endpoint_runtime(
ep,
owner=token_owner,
**runtime_kwargs,
)
base_url, api_key = resolve_endpoint_runtime(ep, owner=token_owner)
endpoint_url = build_chat_url(base_url)
except Exception:
raise HTTPException(500, "Could not resolve endpoint credentials")
# This route is bearer-only. Explicit and empty selections both
# use the same server-owned, cache-only picker inventory; an empty
# inventory is an error rather than an implicit provider alias.
model = _validate_bearer_sync_model(ep, model)
if model == "auto":
try:
async with httpx.AsyncClient(timeout=5) as client:
models_url = build_models_url(base_url)
hdrs = build_headers(api_key, base_url)
if models_url:
resp = await client.get(models_url, headers=hdrs)
resp.raise_for_status()
data = resp.json()
items = data if isinstance(data, list) else (data.get("data") or [])
ids = [m.get("id") for m in items if isinstance(m, dict) and m.get("id")]
if not ids and isinstance(data, dict):
ids = [
m.get("name") or m.get("model")
for m in (data.get("models") or [])
if m.get("name") or m.get("model")
]
else:
import json as _json
ids = _json.loads(ep.cached_models or "[]")
model = ids[0] if ids else "auto"
except Exception:
raise HTTPException(500, "Could not discover models from endpoint")
if not session_manager:
raise HTTPException(500, "Session manager not available")
@@ -459,52 +368,27 @@ def setup_webhook_routes(
session_id=sid, name="API Chat", endpoint_url=endpoint_url,
model=model, owner=token_owner,
)
endpoint_id = getattr(ep, "id", None)
if endpoint_id:
persist_session_endpoint_provenance(
session_manager,
sid,
sess,
model_endpoint_id=endpoint_id,
endpoint_provenance="registered",
)
if api_key:
sess.headers = build_headers(api_key, base_url)
session_manager.save_sessions()
session_id = sid
# --- Send message and get response ---
if is_bearer_principal(request):
from routes.chat_helpers import _validate_bearer_session_model
# The fallback branch has just created the session, so it did not
# pass through the existing-session gate above. Recheck the
# durable endpoint identity immediately before the LLM boundary
# for every bearer path, including malformed endpoint rows.
_validate_bearer_session_model(sess, owner=token_owner)
sess.add_message(ChatMessage("user", message))
messages = [{"role": m.role, "content": m.content} for m in sess.history]
llm_kwargs = {}
if not capability.allow_live_probes:
llm_kwargs["allow_live_probes"] = False
reply = await llm_call_async(
sess.endpoint_url, sess.model, messages,
headers=sess.headers, timeout=120,
**llm_kwargs,
)
sess.add_message(ChatMessage("assistant", reply))
session_manager.save_sessions()
# /api/v1/chat remains a synchronous bearer integration: the response
# is returned normally, but the token must not fan that content out to
# an owner-configured asynchronous callback after authorization ends.
if not is_bearer_principal(request):
webhook_manager.fire_and_forget("chat.completed", {
"session_id": session_id, "model": sess.model,
"user_message": message[:2000], "response": reply[:2000],
})
webhook_manager.fire_and_forget("chat.completed", {
"session_id": session_id, "model": sess.model,
"user_message": message[:2000], "response": reply[:2000],
})
return {"response": reply, "session_id": session_id, "model": sess.model}
+1 -3
View File
@@ -2,7 +2,7 @@
import os
from fastapi import APIRouter, Request, HTTPException, Query
from src.auth_helpers import get_current_user, require_non_bearer_request
from src.auth_helpers import get_current_user
from src.tool_security import owner_is_admin_or_single_user
# Cap entries returned per directory (mirrors filesystem_tools._CODENAV_MAX_HITS).
@@ -24,7 +24,6 @@ def setup_workspace_routes():
NON_ADMIN_BLOCKED_TOOLS). A non-admin who can't use those tools must not
be able to map the host's directory tree either.
"""
require_non_bearer_request(request)
owner = get_current_user(request)
if not owner_is_admin_or_single_user(owner):
raise HTTPException(status_code=403, detail="Workspace browsing is admin-only")
@@ -76,7 +75,6 @@ def setup_workspace_routes():
instead of being stored client-side and silently dropped at chat time.
Admin-gated like /browse: it confirms path existence on the host.
"""
require_non_bearer_request(request)
owner = get_current_user(request)
if not owner_is_admin_or_single_user(owner):
raise HTTPException(status_code=403, detail="Workspace selection is admin-only")
+2 -2
View File
@@ -1,6 +1,6 @@
#!/usr/bin/env bash
# Encode a source screen-recording (.mkv) into web-optimized preview clips for
# the landing page: website/<name>.webm (VP9) + website/<name>.mp4 (H.264).
# the landing page: docs/<name>.webm (VP9) + docs/<name>.mp4 (H.264).
#
# ./encode_previews.sh <input> <name> [max_secs]
#
@@ -13,7 +13,7 @@ set -euo pipefail
IN="${1:?input file}"
NAME="${2:?output basename}"
MAX="${3:-30}"
OUT_DIR="$(cd "$(dirname "$0")/../website" && pwd)"
OUT_DIR="$(cd "$(dirname "$0")/../docs" && pwd)"
dur=$(ffprobe -v error -show_entries format=duration -of csv=p=0 "$IN" | cut -d. -f1)
dur=${dur:-0}
+2 -7
View File
@@ -327,12 +327,7 @@ def list_models():
@app.post("/v1/images/generations")
def generate(req: ImageRequest):
# The served model is the one this process was launched with. `req.model`
# is accepted for OpenAI wire compatibility and ignored, matching
# scripts/diffusion_server.py: honouring it would let a caller point the
# generator at any local directory or Hugging Face repo, and the HiDream
# branch runs a python script from inside that directory.
model = _args.model
model = req.model or _args.model
width, height = _size(req.size)
out_images = []
count = max(1, min(int(req.n or 1), 4))
@@ -398,7 +393,7 @@ async def edit_image(
size: str = Form("1024x1024"),
response_format: str = Form("b64_json"),
):
active_model = _args.model # pinned; see generate()
active_model = model or _args.model
if _is_lama_inpaint(active_model) or _is_ddcolor(active_model):
image_raw = await image.read()
mask_raw = await mask.read() if mask is not None else None
+3 -11
View File
@@ -2,7 +2,7 @@
"""odysseus-webhook — shell wrapper for scheduled-task webhook tokens.
Tasks in the scheduled-task system can carry a `webhook_token`. Any
HTTP POST to `/api/tasks/<task-id>/webhook/<token>` fires the task. This CLI lists,
HTTP POST to `/api/webhook/<token>` fires the task. This CLI lists,
rotates, and revokes those tokens.
odysseus-webhook list # tasks that have a token
@@ -21,7 +21,6 @@ quiet_logs()
import argparse, json, logging, os, secrets, sys
from pathlib import Path
from urllib.parse import quote
try:
from core.database import SessionLocal, ScheduledTask
@@ -54,14 +53,6 @@ def _summary(t: "ScheduledTask", reveal: bool = False) -> dict:
}
def _task_webhook_url(base: str, task_id: str, token: str) -> str:
"""Build the live task-route URL without leaking ids into path syntax."""
root = (base or "http://localhost:7000").rstrip("/")
task_part = quote(str(task_id), safe="")
token_part = quote(str(token), safe="")
return f"{root}/api/tasks/{task_part}/webhook/{token_part}"
def cmd_list(args):
db = SessionLocal()
try:
@@ -118,7 +109,8 @@ def cmd_url(args):
fail(f"no task with id {args.id!r}")
if not t.webhook_token:
fail(f"task {args.id!r} has no webhook token (rotate one first)")
url = _task_webhook_url(args.base, t.id, t.webhook_token)
base = (args.base or "http://localhost:7000").rstrip("/")
url = f"{base}/api/webhook/{t.webhook_token}"
emit({
"task_id": t.id,
"name": t.name,
+11 -41
View File
@@ -50,46 +50,16 @@ class DocsService:
List of DocChunk objects
"""
results = self.rag.search(query, k=top_k)
chunks = []
for result in results:
if not isinstance(result, dict):
continue
metadata = result.get("metadata")
if not isinstance(metadata, dict):
metadata = {}
text = result.get("document")
if text is None:
text = result.get("text")
if text is None:
text = result.get("content")
if text is None:
text = ""
source = result.get("source")
if source is None:
source = metadata.get("source")
if source is None:
source = "unknown"
score = result.get("similarity")
if score is None:
score = result.get("score")
if score is None:
score = 0.0
chunks.append(
DocChunk(
text=text,
source=source,
score=score,
metadata=metadata,
)
return [
DocChunk(
text=r.get("text", r.get("content", "")),
source=r.get("source", r.get("metadata", {}).get("source", "unknown")),
score=r.get("score", 0.0),
metadata=r.get("metadata"),
)
return chunks
for r in results
if isinstance(r, dict)
]
async def index(self, directory: str) -> IndexResult:
"""
@@ -103,8 +73,8 @@ class DocsService:
"""
result = self.rag.index_personal_documents(directory)
return IndexResult(
indexed=result.get("indexed_count", result.get("indexed", 0)),
failed=result.get("failed_count", result.get("failed", 0)),
indexed=result.get("indexed", 0),
failed=result.get("failed", 0),
errors=result.get("errors", []),
)
+331 -31
View File
@@ -2,18 +2,22 @@
import copy
import io
import ipaddress
import json
import os
import re
import logging
import socket
import ssl
from datetime import datetime, timedelta
from typing import List
from typing import Iterable, List, cast
from urllib.parse import urljoin, urlparse
import httpx
import httpcore
from bs4 import BeautifulSoup
from src.constants import WEB_FETCH_SOFT_MAX_BYTES, WEB_FETCH_HARD_MAX_BYTES, WEB_FETCH_USER_AGENT
from src import outbound_fetch as _outbound_fetch
from .analytics import RateLimitError, error_logger
from .cache import (
@@ -25,40 +29,336 @@ from .cache import (
logger = logging.getLogger(__name__)
def _is_private_address(addr):
return _outbound_fetch._is_private_address(addr)
_PRIVATE_NETWORKS = (
ipaddress.ip_network("0.0.0.0/8"),
ipaddress.ip_network("10.0.0.0/8"),
ipaddress.ip_network("127.0.0.0/8"),
ipaddress.ip_network("169.254.0.0/16"),
ipaddress.ip_network("172.16.0.0/12"),
ipaddress.ip_network("192.168.0.0/16"),
ipaddress.ip_network("::1/128"),
ipaddress.ip_network("fc00::/7"),
ipaddress.ip_network("fe80::/10"),
)
def _resolve_hostname_ips(hostname):
return _outbound_fetch._resolve_hostname_ips(hostname)
def _public_http_url(url):
return _outbound_fetch._public_http_url(url, resolver=_resolve_hostname_ips)
def _resolve_public_ips(url):
return _outbound_fetch._resolve_public_ips(url, resolver=_resolve_hostname_ips)
_PinnedBackend = _outbound_fetch._PinnedBackend
_PinnedTransport = _outbound_fetch._PinnedTransport
BodyTooLargeError = _outbound_fetch.BodyTooLargeError
_CappedFetch = _outbound_fetch._CappedFetch
def _get_public_url(url, headers, timeout, max_redirects=5, max_bytes=None):
return _outbound_fetch._get_public_url(
url,
headers=headers,
timeout=timeout,
max_redirects=max_redirects,
max_bytes=max_bytes,
resolve_public_ips=_resolve_public_ips,
transport_factory=_PinnedTransport,
def _is_private_address(addr: ipaddress._BaseAddress) -> bool:
if isinstance(addr, ipaddress.IPv6Address) and addr.ipv4_mapped is not None:
addr = addr.ipv4_mapped
return (
addr.is_private
or addr.is_loopback
or addr.is_link_local
or addr.is_reserved
or addr.is_multicast
or addr.is_unspecified
or any(addr in net for net in _PRIVATE_NETWORKS)
)
def _resolve_hostname_ips(hostname: str) -> list[ipaddress._BaseAddress]:
try:
infos = socket.getaddrinfo(hostname, None)
except Exception:
return []
out = []
for info in infos:
try:
out.append(ipaddress.ip_address(info[4][0]))
except Exception:
continue
return out
def _public_http_url(url: str) -> bool:
try:
parsed = urlparse(url)
if parsed.scheme not in ("http", "https"):
return False
host = (parsed.hostname or "").strip()
if not host:
return False
lower = host.lower()
if lower in ("localhost", "metadata", "metadata.google.internal"):
return False
if lower.endswith((".local", ".localhost", ".internal", ".lan", ".intranet")):
return False
try:
return not _is_private_address(ipaddress.ip_address(host))
except ValueError:
pass
addrs = _resolve_hostname_ips(host)
return bool(addrs) and not any(_is_private_address(a) for a in addrs)
except Exception:
return False
def _resolve_public_ips(url: str) -> list[ipaddress._BaseAddress]:
parsed = urlparse(url)
if parsed.scheme not in ("http", "https") or not parsed.hostname:
raise httpx.RequestError(f"Blocked non-public URL: {url}")
host = (parsed.hostname or "").strip().lower()
if host in ("localhost", "metadata", "metadata.google.internal"):
raise httpx.RequestError(f"Blocked non-public hostname: {host}")
try:
ip = ipaddress.ip_address(host)
if _is_private_address(ip):
raise httpx.RequestError(f"Blocked non-public IP literal: {host}")
return [ip]
except httpx.RequestError:
raise
except ValueError:
pass
addrs = _resolve_hostname_ips(host)
if not addrs or any(_is_private_address(a) for a in addrs):
raise httpx.RequestError(f"Blocked non-public URL: {url}")
return addrs
class _PinnedBackend(httpcore.NetworkBackend):
"""Network backend that connects to a pre-resolved IP.
httpcore derives the TLS SNI and the ``Host`` header from the URL's
origin, not from the host argument passed to ``connect_tcp``. So
routing the TCP connect to a resolved IP while leaving the URL
untouched keeps SNI / vhost behaviour correct and closes the
DNS-rebinding TOCTOU between the SSRF check and the connect.
"""
def __init__(self, ip: ipaddress._BaseAddress):
self._ip = str(ip)
self._real = httpcore.SyncBackend()
def connect_tcp(
self,
host: str,
port: int,
timeout: float | None = None,
local_address: str | None = None,
socket_options=None,
):
return self._real.connect_tcp(
self._ip, port, timeout, local_address, socket_options
)
def connect_unix_socket(self, path, timeout=None, socket_options=None):
return self._real.connect_unix_socket(path, timeout, socket_options)
def sleep(self, seconds: float) -> None:
return self._real.sleep(seconds)
# Map httpcore exception classes to their httpx equivalents. Built
# once at import time from the public exception classes; avoids any
# import of httpx's private transport machinery. httpcore's
# ``ConnectionNotAvailable`` is a pool-internal signal (the pool will
# close and retry on its own) — we never expect to see it surface to
# a transport caller, so it has no httpx counterpart here.
_HTTPCORE_TO_HTTPX_EXC = {
httpcore.ConnectError: httpx.ConnectError,
httpcore.ConnectTimeout: httpx.ConnectTimeout,
httpcore.LocalProtocolError: httpx.LocalProtocolError,
httpcore.NetworkError: httpx.NetworkError,
httpcore.PoolTimeout: httpx.PoolTimeout,
httpcore.ProtocolError: httpx.ProtocolError,
httpcore.ProxyError: httpx.ProxyError,
httpcore.ReadError: httpx.ReadError,
httpcore.ReadTimeout: httpx.ReadTimeout,
httpcore.RemoteProtocolError: httpx.RemoteProtocolError,
httpcore.TimeoutException: httpx.TimeoutException,
httpcore.UnsupportedProtocol: httpx.UnsupportedProtocol,
httpcore.WriteError: httpx.WriteError,
httpcore.WriteTimeout: httpx.WriteTimeout,
}
class _PinnedTransport(httpx.BaseTransport):
"""Transport that pins every TCP connect to a pre-resolved IP.
Uses only the public ``httpcore`` and ``httpx`` APIs no
subclassing of ``httpx.HTTPTransport``, no reads of private
``httpcore.ConnectionPool`` attributes, no imports from
``httpx private transport internals``. The URL is passed through unchanged so SNI
/ vhost work as if httpx had been given the hostname directly;
only the TCP destination is pinned, closing the DNS-rebinding
TOCTOU between the SSRF check and the connect.
"""
def __init__(self, ip: ipaddress._BaseAddress, *, http2: bool = False):
self._pool = httpcore.ConnectionPool(
ssl_context=ssl.create_default_context(),
http1=True,
http2=http2,
network_backend=_PinnedBackend(ip),
)
def __enter__(self):
self._pool.__enter__()
return self
def __exit__(self, exc_type=None, exc_value=None, traceback=None) -> None:
self._pool.__exit__(exc_type, exc_value, traceback)
def handle_request(self, request: httpx.Request) -> httpx.Response:
httpcore_req = httpcore.Request(
method=request.method,
url=httpcore.URL(
scheme=request.url.raw_scheme,
host=request.url.raw_host,
port=request.url.port,
target=request.url.raw_path,
),
headers=request.headers.raw,
content=request.stream,
extensions=request.extensions,
)
try:
httpcore_resp = self._pool.handle_request(httpcore_req)
# Eager materialisation matches the original
# ``response.text`` usage in fetch_webpage_content. The
# sync pool's stream is a plain Iterable[bytes] despite
# the httpcore type hint unioning the async variant.
content = b"".join(cast(Iterable[bytes], httpcore_resp.stream))
except Exception as exc:
mapped = _HTTPCORE_TO_HTTPX_EXC.get(type(exc))
if mapped is not None:
raise mapped(str(exc)) from exc
raise
return httpx.Response(
status_code=httpcore_resp.status,
headers=httpcore_resp.headers,
content=content,
extensions=httpcore_resp.extensions,
)
def close(self) -> None:
self._pool.close()
class BodyTooLargeError(Exception):
"""The server declared a body larger than the hard fetch ceiling."""
def __init__(self, url: str, declared_bytes: int):
self.url = url
self.declared_bytes = declared_bytes
super().__init__(
f"response body is {declared_bytes:,} bytes, over the "
f"{WEB_FETCH_HARD_MAX_BYTES:,}-byte hard cap"
)
class _CappedFetch:
"""Result of a size-capped streaming GET.
Carries just what fetch_webpage_content needs from an httpx.Response,
plus the cap bookkeeping: the (possibly truncated) body, whether the
cap cut it short, and the size the server declared via Content-Length
(wire bytes; None when absent).
"""
__slots__ = ("status_code", "headers", "content", "truncated",
"declared_bytes", "encoding", "url")
def __init__(self, status_code, headers, content, truncated,
declared_bytes, encoding, url):
self.status_code = status_code
self.headers = headers
self.content = content
self.truncated = truncated
self.declared_bytes = declared_bytes
self.encoding = encoding
self.url = url
@property
def text(self) -> str:
return self.content.decode(self.encoding or "utf-8", errors="replace")
def raise_for_status(self):
if self.status_code >= 400:
request = httpx.Request("GET", self.url)
raise httpx.HTTPStatusError(
f"HTTP {self.status_code} for {self.url}",
request=request,
response=httpx.Response(self.status_code, request=request),
)
def _get_public_url(url: str, headers: dict, timeout: int, max_redirects: int = 5,
max_bytes: int = None) -> "_CappedFetch":
"""Capped streaming GET with SSRF-guarded, DNS-pinned manual redirects.
Each hop is resolved once, validated as public, and then the actual TCP
connection is pinned to that resolved IP. The request URL is left unchanged
so Host and TLS SNI keep the original hostname.
"""
cap = min(max_bytes or WEB_FETCH_SOFT_MAX_BYTES, WEB_FETCH_HARD_MAX_BYTES)
current = url
for _ in range(max_redirects + 1):
ips = _resolve_public_ips(current)
# Force identity transfer-encoding. With gzip/deflate the wire bytes
# and Content-Length can be a small fraction of the decoded body, so a
# tiny compressed response could pass the hard-cap preflight and then
# expand past the ceiling in one decoded chunk before the streamed cap
# below can slice it.
req_headers = dict(headers or {})
req_headers["Accept-Encoding"] = "identity"
with httpx.Client(
headers=req_headers,
timeout=timeout,
follow_redirects=False,
transport=_PinnedTransport(ips[0]),
) as client:
with client.stream("GET", current) as response:
if response.status_code in (301, 302, 303, 307, 308):
location = response.headers.get("location")
if not location:
return _CappedFetch(response.status_code, response.headers, b"",
False, None, response.encoding, str(response.url))
current = urljoin(str(response.url), location)
continue
# A server can ignore the identity request and still return a
# compressed body; httpx.iter_bytes would then decode it, and a
# tiny gzip can balloon into one decoded chunk far past the cap.
# Refuse compressed Content-Encoding so the streamed cap stays
# a real memory bound.
enc = (response.headers.get("content-encoding") or "").strip().lower()
if enc and enc != "identity":
raise httpx.RequestError(
f"Refusing compressed response (Content-Encoding: {enc}) after "
"requesting identity: cannot bound decoded body size",
request=httpx.Request("GET", current),
)
declared = None
raw_len = response.headers.get("content-length")
if raw_len and raw_len.isdigit():
declared = int(raw_len)
if declared is not None and declared > WEB_FETCH_HARD_MAX_BYTES:
raise BodyTooLargeError(current, declared)
chunks = []
read = 0
truncated = False
for chunk in response.iter_bytes():
read += len(chunk)
if read > cap:
keep = cap - (read - len(chunk))
if keep > 0:
chunks.append(chunk[:keep])
truncated = True
break
chunks.append(chunk)
return _CappedFetch(response.status_code, response.headers,
b"".join(chunks), truncated, declared,
response.encoding, str(response.url))
raise httpx.RequestError("Too many redirects", request=httpx.Request("GET", current))
# PDF extraction (optional dependency)
try:
from pdfminer.high_level import extract_text as pdf_extract_text
-88
View File
@@ -1,88 +0,0 @@
# Specs DocumentMap
Last updated: dev@e71f8ce | 2026-08-25
This folder is the compact implementation-truth map for humans and coding agents working on Odysseus. Read this file first, then open only the subsystem specs that match the work.
Specs are living notes about current code shape and intended contracts. They are not product marketing, not PR planning, not templates, and not a replacement for source inspection or tests.
This `_readme.md` is the DocumentMap and control document. It is intentionally exempt from subsystem `Scope` and `Current Gaps` sections; keep it limited to the quality contract, working rules, subsystem map, and cross-cutting update triggers.
## Quality Contract
Each subsystem spec should stay compact and useful under context pressure:
- Start with `Last updated: dev@<short-sha> | YYYY-MM-DD`, using the
upstream `dev` commit the spec text was inspected against.
- Use a concrete `Scope` section that names real files, route surfaces, frontend modules, data stores, and integration points.
- Use domain-specific sections. Do not force every spec into the same headings when the subsystem needs `Streaming`, `Tool Results`, `Optional Dependencies`, `Current Gaps`, or another focused section.
- State ownership clearly: which file owns a mapping, which layer only forwards state, and which caller requests behavior without owning implementation.
- Include runtime behavior bullets for flows that matter.
- Include "Current call sites include" when behavior is spread across many files.
- Record transitional compatibility notes, especially `src/` versus `services/` duplication.
- Record degraded, optional, or platform behavior where it changes runtime expectations.
- Record policy/provenance where relevant: untrusted context, encrypted secrets, API token scopes, optional dependency/license implications, generated media, or user data.
- End with `Current Gaps` only when there is a real known gap, not as filler.
If code and specs disagree, treat code as ground truth. Update specs only when
the current task explicitly includes spec maintenance or the PR intentionally
includes specs; otherwise report the drift in the relevant issue, PR review, or
project documentation.
## Working Rules
- Start here before substantial work.
- Read the related subsystem spec before changing code in that area. For cross-cutting work, include the owning domain spec plus route/runtime, auth/security, persistence, frontend, tool/context, integration, and testing/devops specs as applicable.
- Treat specs as read-only context during ordinary project work, PR review, and code review. Do not edit specs unless the user explicitly asks for spec work or the current PR intentionally includes spec changes.
- During explicit spec-maintenance work, update the related spec when source inspection shows behavior, ownership, security boundaries, data shape, import paths, or implementation contracts have changed.
- During ordinary work, record source/spec drift in the relevant issue, PR review, or project documentation instead of mutating specs.
- Keep specs dense but readable. Prefer current facts and invariants over broad explanation.
- Every non-index `specs/*.md` file should appear exactly once in the Subsystem Map with a one-line description and no dead link.
- Specs contain implementation truth. Planning, research, branch notes, and decisions belong in tracked project docs. Drafts, audit reports, raw exports, and exploratory gap lists are not authoritative until promoted into tracked docs or specs.
- Use repo source and these specs as the authority for Odysseus architecture. Do not treat global skill registries or external agent metadata as repo ground truth.
## Subsystem Map
- [runtime.md](runtime.md): FastAPI startup, router registration, static serving, lifespan, app-wide middleware.
- [auth-security.md](auth-security.md): auth, privileges, API tokens, security headers, untrusted data, SSRF and admin boundaries.
- [persistence.md](persistence.md): SQLite models, startup migrations, encrypted columns, ownership columns, data directory rules.
- [chat.md](chat.md): chat routes, sessions, streaming, uploads-in-chat, compare handoff, research/chat mode dispatch.
- [compare.md](compare.md): model A/B comparison runs, voting/history, compare frontend panes, compare ownership.
- [llm-models.md](llm-models.md): LLM provider calls, endpoint discovery, model context length, fallbacks, model endpoints.
- [model-capability-canonical.md](model-capability-canonical.md): canonical provider/model capability shapes, evidence, payload resolution, and safe fallback.
- [model-quirks.md](model-quirks.md): model-specific behavior observations, evidence, and promotion gates.
- [model-providers/_readme.md](model-providers/_readme.md): provider-by-provider API/catalog shape index and compatibility status.
- [agent-tools.md](agent-tools.md): agent loop, tool schemas, tool execution, tool retrieval, tool security, MCP tool exposure.
- [context-building.md](context-building.md): URL/search/RAG/memory/skills/YouTube/email/tool-output context, untrusted wrapping, unavailable context, intent boundaries.
- [search.md](search.md): web search providers, ranking, cache/analytics, URL fetch/content extraction, `src.search`/`services.search` split.
- [documents-rag-uploads.md](documents-rag-uploads.md): uploads, documents, PDF/form handling, personal docs, RAG/vector stores.
- [memory-skills.md](memory-skills.md): memory storage, semantic memory, skill extraction/formatting, owner isolation.
- [research.md](research.md): deep research jobs, synthesis, sources, research library, research UI panel.
- [calendar-tasks-notes.md](calendar-tasks-notes.md): CalDAV calendars, scheduled tasks, reminders, assistant runs, notes/todos.
- [email-contacts.md](email-contacts.md): IMAP/SMTP email, email library, scheduled mail, contacts/CardDAV.
- [gallery-editor-media.md](gallery-editor-media.md): gallery, generated media, image editor drafts, signatures, emoji/font helpers.
- [cookbook-hwfit.md](cookbook-hwfit.md): model downloads, local/remote model serving, hardware detection, fit ranking.
- [speech.md](speech.md): STT and TTS services, routes, settings, optional dependencies.
- [frontend.md](frontend.md): static SPA, module loading, UI conventions, major JS areas, no-build frontend shape.
- [integrations.md](integrations.md): Codex/Claude scoped APIs, companion pairing, webhooks, external agent access.
- [shell-mcp.md](shell-mcp.md): shell execution, background jobs, MCP manager, built-in MCP servers.
- [settings-admin.md](settings-admin.md): settings, preferences, presets, backup/import/export, diagnostics, admin wipe.
- [testing-devops.md](testing-devops.md): pytest, JS tests, Docker, scripts, requirements, local dev expectations.
## Cross-Cutting Spec Update Triggers
Use these triggers only during explicit spec-maintenance work or a PR that
intentionally includes specs. For ordinary work and code review, use the same
list to choose which specs to read and where to report drift.
- New route file or route prefix: update [runtime.md](runtime.md) and the owning subsystem spec.
- New SQLAlchemy model, column migration, durable JSON/local store, data directory, backup/import domain, or non-SQL persistence behavior: update [persistence.md](persistence.md) and the owning subsystem spec.
- New tool, tool schema, agent prompt rule, or tool security behavior: update [agent-tools.md](agent-tools.md) and [context-building.md](context-building.md) if it adds model context.
- New MCP runtime/config/built-in behavior: update [shell-mcp.md](shell-mcp.md), [agent-tools.md](agent-tools.md), and [context-building.md](context-building.md) when MCP tool results enter model context.
- New external content source, tool result, MCP/app API result, or integration result shown to an LLM: update [context-building.md](context-building.md) and [auth-security.md](auth-security.md).
- New API-token scope, scoped external API, webhook, companion/pairing route, generic integration provider, or external-agent helper bundle: update [integrations.md](integrations.md), [auth-security.md](auth-security.md), and the owning subsystem spec.
- New secret store, decrypted-secret return path, settings backup/import/export behavior, diagnostics/log output, vault/tool secret flow, `.env*` policy change, or credential-bearing CLI output: update [auth-security.md](auth-security.md), [settings-admin.md](settings-admin.md), [testing-devops.md](testing-devops.md), and the owning subsystem spec.
- New optional dependency, degraded fallback, platform/Docker/native/launcher difference, GPU overlay behavior, or retired compatibility shim: update [testing-devops.md](testing-devops.md) and the owning subsystem spec; also update [runtime.md](runtime.md), [llm-models.md](llm-models.md), [shell-mcp.md](shell-mcp.md), [cookbook-hwfit.md](cookbook-hwfit.md), or [persistence.md](persistence.md) when that layer owns the behavior.
- New frontend module or modal/tool surface: update [frontend.md](frontend.md) and the owning subsystem spec.
- New static/PWA/service-worker/cache/CSP behavior: update [frontend.md](frontend.md), [runtime.md](runtime.md), and [auth-security.md](auth-security.md) when headers or trust boundaries change.
- New CLI script: update [testing-devops.md](testing-devops.md) and the owning subsystem spec.
-157
View File
@@ -1,157 +0,0 @@
# Agent Tools
Last updated: dev@e71f8ce | 2026-08-25
## Scope
This spec covers agent/tool behavior in:
- `src/agent_loop.py`;
- `src/llm_core.py`;
- `src/tool_schemas.py`;
- `src/tool_execution.py`;
- `src/tool_policy.py`;
- `src/tool_index.py`;
- `src/tool_parsing.py`;
- `src/tool_security.py`;
- `src/tool_capabilities.py`;
- `src/tool_approval_scopes.py`;
- `src/tool_approvals.py`;
- `src/attachment_refs.py` and shared upload lifecycle helpers in
`src/upload_handler.py` / `src/tool_utils.py`;
- `src/tool_implementations.py`;
- `src/tools/*.py`;
- `src/builtin_actions.py`;
- `src/ai_interaction.py`;
- `src/action_intents.py`;
- `src/goal_based_extractor.py`;
- `src/teacher_escalation.py`;
- `src/agent_tools/` modules and compatibility facade;
- `src/mcp_manager.py`;
- `src/builtin_mcp.py`;
- `src/bg_jobs.py` and `src/bg_monitor.py`;
- `routes/chat_routes.py`, `routes/chat_helpers.py`, `routes/model_routes.py`, `routes/skills_routes.py`, canonical `routes/mcp/mcp_routes.py` plus its shim, and `routes/workspace_routes.py`;
- `mcp_servers/*.py`;
- frontend stream/admin/settings files that display tool events, workspaces, and disabled tools;
- `tests/test_agent_loop.py`, `tests/test_tool_*`, and focused MCP/public-policy/schema tests.
## Agent Loop
`src.agent_loop` owns agent prompt assembly, request-local current date/time insertion, tool retrieval, prompted tool-block handling, native tool-call consumption after `llm_core` normalizes provider events, multi-round execution, tool result insertion, final metrics, and fallback responses. It requests context from documents, skills, tool retrieval, and messages; it should not own domain-specific business logic for every tool. Its prompt rules now bias structured/long-form writing toward living documents, route active compose/email drafts back into existing email documents, and prefer first-class `web_search`/`web_fetch` tools over shell/Python/curl for current web lookups when web tools are enabled.
`src.llm_core` owns provider payloads, native tool-schema emission, and provider stream parsing. `agent_loop` consumes normalized tool-call events and decides whether and how to execute them.
Agent mode enters through chat routes, including auto-escalation from intent helpers, detached `agent_runs` streaming, resume/stop behavior, and frontend tool-event rendering.
Guide-only/no-tools turns are runtime policy, not prompt advice. `src.tool_policy` detects strong latest-turn directives such as guide-only mode, no-tools mode, and explicit requests not to use tools; it builds a `ToolPolicy` that hides schemas, disables known native tools, disables MCP for that turn, skips tool retrieval, suppresses local/workspace context injection, blocks document streaming/teacher escalation, and gives `tool_execution` a final execution backstop.
Plan mode is a read-only investigation path inside the same loop. It adds a denylist for known mutating tools, filters write/unknown MCP tools, prepends plan-mode instructions, and uses the `update_plan` tool only after a plan is approved for execution. The backend path still exists for compatibility, but current browser chat forces incoming `plan_mode` off and the old plan-window UI module is gone.
Workspace mode is request-scoped. Admin chat can send a workspace directory selected through `static/js/workspace.js`; `agent_loop` injects that fact early in the prompt and `tool_execution` confines bash, python, read/write/edit-file, and code-navigation tools to that root. `routes.workspace_routes` owns admin-only browse/vet APIs, skips hidden/symlink directory traversal, caps listings, and rejects sensitive/root paths before a workspace reaches chat.
## Tool Registry
Tool registration is split:
- `src.agent_tools` is now a package/facade. `TOOL_HANDLERS` maps native tool names to handler functions across filesystem, subprocess, web, document, interaction, model-interaction, background-job, session, and admin modules, while `TOOL_TAGS` keeps compatibility metadata and the global MCP manager handle;
- `src.tools` owns domain do_* implementations for calendar, contacts, Cookbook, image, notes, research, search, system, and vault tools. `src.tool_implementations` is now a compatibility facade that re-exports those symbols and lazy-loads admin manage_* symbols to avoid circular imports;
- `src.agent_tools.admin_tools` owns admin manage_* tools for endpoints, MCP, webhooks, tokens, and settings, including command validation for `manage_mcp`;
- `src.tool_parsing._TOOL_NAME_MAP` owns aliases and prompted-block parsing;
- `src.tool_schemas.FUNCTION_TOOL_SCHEMAS` and `function_call_to_tool_block()` own native schema and native-call conversion;
- `src.tool_index.BUILTIN_TOOL_DESCRIPTIONS` owns retrieval text;
- `src.tool_execution.execute_tool_block()` owns dispatch and hard execution gates;
- `routes.model_routes.py` and frontend settings/admin surfaces expose global disabled-tool controls.
When adding, removing, or renaming a tool, update the registry chain, execution dispatch, retrieval text, prompt wording, disabled-tool UI, and tests together.
`src.tool_index.ALWAYS_AVAILABLE` is the retrieval catalog for high-frequency tools such as shell/python, web search/fetch, read/write/edit-file, code-nav, `manage_memory`, `ask_user`, `update_plan`, selected Cookbook serve controls, and `app_api`. Current prompt/schema assembly preserves only selected base tools unconditionally, then adds intent-, skill-, and retrieval-relevant tools so unrelated schemas do not flood small contexts.
## Tool Retrieval And Execution
`src.tool_index.ToolIndex` owns candidate retrieval using embeddings/keywords and cached index data. Security filtering is not its hard boundary: `agent_loop` hides unavailable schemas, and `tool_execution` blocks disabled, admin-only, and public-restricted calls before dispatch.
`src.tool_execution` owns built-in tool execution, MCP dispatch, path confinement, background markers, output truncation, internal HTTP loopback, owner/admin checks, policy-blocked execution results, and formatting tool results for the model/UI. File tools support exact edit diffs, full-file writes, read line ranges, and workspace confinement. Code-navigation tools (`grep`, `glob`, `ls`) prefer `rg`/structured filesystem traversal over ad hoc shell commands. Uploaded-file context uses stable `attachment_ref` manifests and owner-checked URIs; a compatibility local path is exposed only after upload-root and tool-root confinement. Shared truncation, upload-handler registration, and MCP manager compatibility helpers live in `src.tool_utils`.
Tool retrieval has domain-specific hooks beyond generic similarity: contact queries can surface `resolve_contact`/`manage_contact`; matched skills can add `manage_skills` and their required toolsets to the relevant tool set; explicit admin intents can include admin schemas so prompt text and native schema emission match.
Interaction/session/model helper tools are native first-class tools, not prompt-only conventions. `ask_user` and `update_plan` live in `src.agent_tools.interaction_tools`, model delegation/listing helpers live in `model_interaction_tools`, session creation/list/send/manage helpers live in `session_tools`, and `manage_bg_jobs` lives in `bg_job_tools`.
Prompted-tool parsing includes recovery paths for local/provider text leaks: bare JSON after a web-tool mention, OpenAI-style raw `{"function": ...}` payloads, StepFun/Gemma/DSML markup, Hermes/Qwen JSON bodies nested inside `tool_call` wrappers, and `<function_model><function_call>...</function_call><parameters>...</parameters></function_model>` wrappers from local MLX/Exo models. The Qwen bare end marker requires its pipe delimiter so ordinary text cannot terminate a tool block. Non-dict JSON arguments are rejected back to empty args instead of crashing the turn, common `tex` typos normalize to `text`, and delimiter scans are forward-only so unterminated tool markup cannot drive quadratic rescans. Executed raw tool JSON is stripped from assistant text afterward; this is still not a general-purpose JSON-command parser.
Current call sites include:
- agent mode tool calls from `src.agent_loop`;
- MCP route configuration and built-in MCP registration;
- background job monitoring and auto-continue;
- skill tests, teacher escalation, scheduled tasks, and background follow-up loops;
- UI-control and AI interaction helpers.
## Streaming And Continuations
Agent streaming emits normal content plus tool progress/output, document stream/update, ask-user choices, plan updates, budget, round exhaustion, loop-breaker, intent-nudge exhaustion, metrics, teacher escalation, research anchor, and finish/error events. Frontend chat stream code and detached replay depend on stable event names. If the stream generator closes while awaiting an in-flight tool, the loop cancels and awaits that tool task so subprocess-backed work is not left orphaned.
Long-running bash jobs can be detached with background markers. `src.bg_jobs` owns persistent job state/result files; `src.bg_monitor` owns auto-continuation when jobs finish. Detached chat runs are in-memory and do not survive server restart, while background job state is disk-backed.
Loop-breaker final-answer rounds, explicit repeated-tool/intent-nudge guard events, round-cap continuation signals, optional verifier retries, and teacher escalation are recovery behavior owned by `agent_loop` and `src.teacher_escalation`.
Approval replay injects the sealed first tool result before the resumed model round. If that replay round has neither assistant prose nor reasoning, `_append_tool_results()` omits the empty assistant spacer so Anthropic-compatible payloads do not contain a rejected non-final empty assistant message; reasoning-only carriers remain a documented compatibility edge.
## Security And Policy
- `src.tool_security` owns non-admin blocked-tool decisions.
- Non-admin users must not reach admin tools through agent mode, MCP, retrieval, or loopback calls.
- Agent owner is passed from chat route `get_current_user(request)`. In `AUTH_ENABLED=false` mode this is `None`, not the `""` value returned by route dependencies. `blocked_tools_for_owner()`, schema hiding, and `execute_tool_block()` all use that owner.
- Current dev tool security treats explicit `AUTH_ENABLED=false` as single-user even when an auth store exists, while auth-enabled pre-setup callers remain non-admin.
- Path-based tools must remain confined to allowed roots and reject sensitive paths. Sensitive-path checks are case-insensitive and apply to direct file tools and code-navigation tools; `grep`/`glob`/`ls` must not become existence or content oracles for `.env`, SSH/GPG material, `id_rsa`, and similar denylisted paths.
- Tool output is bounded/truncated where native execution owns the path, including displayed agent-tool output through the shared truncation helper. MCP output must be treated as untrusted; central MCP-output truncation before model re-entry remains a gap.
- Provider-emitted native tool calls are requests, not authorization. `tool_execution` and route-level policy remain the authority.
- `src.tool_capabilities` classifies each tool's effects and result integrity. Once external/workspace-untrusted content becomes model-visible, the request/session security context permits only explicitly low-impact tools without interruption and requires exact approval for high-impact, unknown, and arbitrary MCP calls.
- `src.tool_approvals` seals an opaque, expiring exact first action plus server-only selected tools and continuation query to owner, session, origin run, tool content, workspace, capability snapshot, and—when relevant—document id/version/content digest. Chat choices grant the resumed task or the same chat session; both consume the exact first action, task scope bypasses the gate only during that resumed run, and chat scope is reconstructed only from a resolved card bound to the exact session id. The browser never receives selected tools/query and submits only task/chat/deny. Non-chat callers retain single-action behavior; new normal turns and superseding actions retire unresolved approvals without clearing taint.
- Tool results that expose remote or stored untrusted content arm the gate even when their tool status is failed. Content-free failures and server-generated policy/approval placeholders do not. Native/provider tool messages and fenced results carry model-visible untrusted metadata/wrapping instead of relying on prompt wording alone.
- Attachment-bearing document, note, and calendar tools owner-reserve internal
upload references before durable writes and fail without mutation when the
referenced upload is unavailable.
- Guide-only/no-tools mode blocks tools before prompt assembly, before execution, and in chat preprocessing paths that would otherwise fetch context or start tool-backed research.
- Plan mode is policy, not prompt advice: mutating native tools are disabled through schema-derived detection plus a static backstop, and write/unknown MCP tools are hidden and runtime-blocked for that turn.
## Internal Loopback
`do_app_api()` is implemented in `src.tools.system` and re-exported by `src.tool_implementations`. It owns generic app API loopback, OpenAPI discovery, method/path blocklists, and fixed local target behavior. `_internal_headers()` adds the process-secret internal-tool token and optional `X-Odysseus-Owner`; `core.middleware.require_admin()` and auth middleware own the corresponding bypass and owner-stamping rules. Route-specific owner handling must still be audited.
## MCP
`src.mcp_manager` owns configured MCP server lifecycle, discovered tool state, qualified MCP names, OpenAI schema conversion, call routing, generation invalidation, and connect/disconnect status. It supports stdio, SSE, and Streamable HTTP transports; Streamable HTTP can publish a `needs_auth` state and uses `src.mcp_oauth` for OAuth/OIDC-style authorization, token refresh, and encrypted token storage. Arbitrary MCP tools classify fail-high for approvals. `src.builtin_mcp` owns built-in server registration and the native-vs-MCP split. `mcp_servers/` owns server-specific tools for email, image generation, memory, RAG, and optional browser tooling.
Native bash, python, file, web search, and web fetch tools continue through native fallback even when MCP is unavailable. Browser MCP is optional and can be skipped when cached Playwright/NPX packages are missing. Public users get no MCP schemas, and any `mcp__*` execution attempt must be blocked.
MCP prompt/schema rendering includes server-provided input schemas, but names, types, and parameter hint text are sanitized and length-capped before entering the prompt. Per-server disabled tools filter listings, prompt descriptions, and function schemas; execution-time disabled-tool enforcement remains a separate hardening item.
## Intent And Recovery Helpers
`src.action_intents` owns deterministic chat-to-agent promotion hints and returns a category/reason so route logs can explain auto-escalation decisions. Explicit web-search language is category `web`; it can promote the turn into agent mode and narrow tools toward web search/fetch, but route policy requires explicit web-search enablement and honors explicit denial. It must avoid promoting explanatory questions into agent mode. `src.builtin_actions` owns scheduler/background actions outside the normal live agent loop. `src.teacher_escalation` owns recovery/escalation and skill-creation flows. `src.goal_based_extractor` is research-adjacent and should stay cross-referenced from research behavior rather than treated as ordinary tool execution.
When an email reader is active, browser chat passes active email metadata and the agent loop injects it as protected, untrusted context so default reply/draft behavior targets the selected message. Active email compose documents are handled as existing email drafts rather than generic new-document requests.
## Degraded Behavior
- ToolIndex can degrade to keyword selection when embeddings, Chroma, index
warmup, or vector retrieval timeouts fail.
- Agent mode can degrade from native function schemas to prompted fenced-block parsing based on provider/tool-support heuristics. Local Ollama `/v1` and native `/api` endpoints default to text tools unless the endpoint explicitly advertises `supports_tools`; `gpt-oss` remains text-tool by default unless the endpoint opts in.
- MCP startup failure is non-critical; route/status surfaces expose per-server errors.
- `ODYSSEUS_DISABLE_MCP`, missing `mcp`, uncached browser MCP packages, and per-server disabled tools can remove tools without blocking the app.
- Global `builtin_browser` disable behavior may not currently match qualified `mcp__builtin_browser__*` tool names.
## Current Gaps
- Tool descriptions are duplicated across `FUNCTION_TOOL_SCHEMAS`, agent prompt sections, and `BUILTIN_TOOL_DESCRIPTIONS`.
- Agent prompts remain heavy for small local context windows.
- Some AI-control helpers are still globally wired from app startup rather than a narrower service layer.
- Tool registry consistency is manual across handler maps, tags, aliases, schemas, retrieval descriptions, execution dispatch, settings/model routes, and frontend toggles.
- MCP disabled-tool changes can stale-cache tool retrieval because disabled maps are not always an index generation input.
- External MCP output still needs a single central size cap before model re-entry; untrusted-result metadata and the post-external-context action gate now cover the prompt-injection/authorization boundary.
- Auth-disabled/no-login owner propagation is inconsistent between route dependencies and chat/agent execution, so tool-security and native tool storage behavior need dedicated regression coverage.
- Agent tests mostly cover helpers and targeted regressions, including round-cap
and disconnect cancellation paths, but not an end-to-end fake-LLM
`stream_agent_loop` path with retrieval, native schemas, prompted blocks,
disabled/admin hiding, MCP tools, plan/workspace state, user-time context, and
tool-result SSE.
+412
View File
@@ -0,0 +1,412 @@
# Architecture Runtime Inventory
> **Purpose**: Phase 0 planning baseline for codebase readability improvements (#4071).
> **Parent issue**: [#4082](https://github.com/odysseus-dev/odysseus/issues/4082)
> **Last updated**: dev@b58af42 | 2026-06-16
> **Status**: Draft — to be reviewed before follow-up slices open.
> **Snapshot basis**: Importer / file / import-line counts are refreshed to `dev@b58af42` (2026-06-16) and are recomputable via the commands in §3.4. **Line counts** in §2.1 / §2.2 are a snapshot from an earlier baseline and drift as `dev` moves — recompute any of them with `wc -l <file>`. This inventory tracks structure and risk, not live metrics.
This document maps the current runtime module structure, identifies high-risk boundaries, and recommends safe first refactor slices. It does **not** move files, change imports, or alter runtime behavior.
---
## 1. Current Structure Overview
### 1.1 Top-Level Layout
```
odysseus/
├── app.py # FastAPI app entrypoint (1,145 lines)
├── conf/ # Configuration (config.py, settings.py, settings_scrub.py)
├── src/ # 95 flat .py files + 2 subdirectories
│ ├── agent_tools/ # Tool helpers: document, filesystem, subprocess, web
│ └── search/ # Search subsystem
├── routes/ # 54 flat .py files — HTTP route handlers
├── core/ # 10 files — database models, auth, middleware, session
├── mcp_servers/ # 5 files — MCP server implementations
├── scripts/ # CLI tools and one-shot scripts
├── static/ # Frontend HTML/CSS/JS
├── tests/ # 583 test files (~54,800 lines)
└── services/ # (exists as needed)
```
### 1.2 Directory Flatness Metric
| Directory | Flat `.py` Files | Subdirectories | Concern |
|-----------|-----------------|----------------|---------|
| `src/` | **95** | 2 (`agent_tools/`, `search/`) | No domain grouping; 95 files in one directory |
| `routes/` | **54** | 0 | All route handlers in one flat directory |
| `core/` | 10 | 0 | Manageable, but `database.py` is oversized |
---
## 2. Largest Runtime Modules
### 2.1 Python Backend
| Rank | File | Lines | Classes | Functions | Risk |
|------|------|-------|---------|-----------|------|
| 1 | `src/tool_implementations.py` | **4,032** | 0 | ~48 | **HIGH** |
| 2 | `routes/email_routes.py` | **3,245** | — | — | **MEDIUM** |
| 3 | `routes/cookbook_routes.py` | **2,969** | — | — | **MEDIUM** |
| 4 | `src/agent_loop.py` | **2,961** | 0 | ~24 | **HIGH** |
| 5 | `src/task_scheduler.py` | **2,330** | — | 5 | MEDIUM |
| 6 | `routes/model_routes.py` | **2,266** | — | — | MEDIUM |
| 7 | `core/database.py` | **2,265** | 28 | ~59 helpers | **HIGH** |
| 8 | `src/builtin_actions.py` | **2,262** | 2 | ~24 | MEDIUM |
| 9 | `src/llm_core.py` | **2,164** | — | — | MEDIUM |
| 10 | `mcp_servers/email_server.py` | 2,197 | — | — | LOW (separate process) |
| 11 | `src/visual_report.py` | 1,918 | — | — | LOW |
| 12 | `routes/gallery_routes.py` | 1,896 | — | — | LOW |
| 13 | `src/ai_interaction.py` | 1,846 | — | — | MEDIUM |
| 14 | `routes/document_routes.py` | 1,717 | — | — | LOW |
| 15 | `routes/skills_routes.py` | 1,648 | — | — | LOW |
**Heuristic**: Files > 2,000 lines with 20+ public symbols and many importers are the highest-risk splits. Files 1,0002,000 lines are medium-risk if tightly coupled.
### 2.2 Frontend
| File | Lines | Concern |
|------|-------|---------|
| `static/style.css` | **36,653** | Entire app CSS in one file (tracked separately in #2617) |
| `static/js/document.js` | **9,776** | Single JS file for document functionality |
| `static/js/slashCommands.js` | 6,498 | |
| `static/js/settings.js` | 5,266 | |
| `static/js/emailLibrary.js` | 5,217 | |
| `static/js/notes.js` | 5,124 | |
| `static/js/chat.js` | 4,985 | |
| `static/app.js` | 4,090 | |
**Note**: Frontend modularization is tracked separately in #2617 (CSS) and is not the focus of this Phase 0 inventory. Frontend is listed here for completeness but follow-up slices should target Python backend boundaries first.
---
## 3. Import Dependency Graph
### 3.1 Who Depends on `core/database.py`
**102 files** import from `core.database` — this is the most depended-upon module:
- All route handlers (`routes/*.py`)
- Most `src/*.py` files
- `core/session_manager.py`, `core/auth.py`
- Multiple test files
**Implication**: Any split of `core/database.py` is the highest-risk refactor. It should be tackled **last**, never first.
### 3.2 Who Depends on `src/tool_implementations.py`
**17 files** import from `src.tool_implementations`:
- `src/agent_loop.py`, `src/builtin_actions.py`, `src/tool_index.py`
- `src/task_scheduler.py`, `src/tool_policy.py`
- Various tests
### 3.3 Who Depends on `src/agent_loop.py`
**22 files** import from `src.agent_loop`:
- `src/tool_policy.py`, `src/teacher_escalation.py`, `src/bg_monitor.py`
- `src/task_scheduler.py`
- Multiple test files
### 3.4 Cross-Layer Import Violations
**`src/` importing from `routes/`** (backwards dependency — domain logic depending on HTTP layer):
```
src/tool_implementations.py ──→ routes/calendar_routes.py
src/tool_implementations.py ──→ routes/cookbook_helpers.py
src/tool_implementations.py ──→ routes/email_helpers.py
src/tool_implementations.py ──→ routes/email_pollers.py
src/tool_implementations.py ──→ routes/email_routes.py
src/tool_implementations.py ──→ routes/model_routes.py
src/tool_implementations.py ──→ routes/note_routes.py
src/tool_implementations.py ──→ routes/prefs_routes.py
```
> These are **runtime imports** (inside function bodies, not at module top), which mitigates circular import risk but indicates fuzzy layer boundaries. Function-level inline imports from the HTTP layer into business logic are a code smell.
**Import counts (top-level)**:
| Direction | Count | Notes |
|-----------|-------|-------|
| `routes/``src/` | **374** | Expected: HTTP handlers call domain logic |
| `routes/``core/` | **126** | Expected: handlers access DB models |
| `src/``routes/` | **31** | **Unexpected**: domain logic reaching into HTTP layer (direct grep of import lines referencing `routes/`) |
| `src/``core/` | **106** | Acceptable but could be reduced with a data-access layer |
> **How the metrics in this document are computed** — recompute against current `dev` before treating any count as authoritative (the tree drifts; these numbers are a snapshot, not a live value):
> - `src/` flat `.py` files: `find src -maxdepth 1 -name '*.py' | wc -l`
> - `tests/` test files: `find tests -name 'test_*.py' | wc -l`
> - `core.database` importers: `grep -rlE '(from|import) +core\.database' --include='*.py' . | grep -v core/database.py | wc -l`
> - `src.agent_loop` importers: `grep -rlE '(from|import) +src\.agent_loop' --include='*.py' . | grep -v src/agent_loop.py | wc -l`
> - Cross-layer import lines: `grep -rhE '(from|import) +<pkg>' --include='*.py' <dir>/ | wc -l` (e.g. `(from|import) +routes` over `src/`)
---
## 4. Route Ownership Map
Routes can be grouped into logical feature domains. Current flat structure obscures these boundaries:
| Domain | Route Files | Total Lines | Review Complexity |
|--------|-------------|-------------|-------------------|
| **Email** | `email_routes.py`, `email_helpers.py`, `email_pollers.py` | 5,936 | HIGH — most complex domain |
| **Chat / Agent** | `chat_routes.py`, `chat_helpers.py`, `shell_routes.py`, `codex_routes.py`, `skills_routes.py` | 6,365 | HIGH — core interaction surface |
| **Cookbook** | `cookbook_routes.py`, `cookbook_helpers.py`, `cookbook_output.py` | 4,110 | MEDIUM |
| **Model / LLM** | `model_routes.py`, `assistant_routes.py`, `copilot_routes.py` | 2,764 | MEDIUM |
| **Calendar / Contacts** | `calendar_routes.py`, `contacts_routes.py` | 2,336 | MEDIUM |
| **Documents** | `document_routes.py`, `document_helpers.py` | 1,954 | LOW |
| **Auth** | `auth_routes.py`, `api_token_routes.py`, `device_flow.py` | 1,171 | LOW |
| **Tasks** | `task_routes.py` (standalone) | 1,157 | LOW |
| **Session** | `session_routes.py` (standalone) | 1,287 | LOW |
| **Gallery** | `gallery_routes.py`, `gallery_helpers.py` | 1,896 | LOW |
| **Memory** | `memory_routes.py` | — | LOW |
| **Research** | `research_routes.py` | — | LOW |
| **MCP** | `mcp_routes.py` | — | LOW |
| **Notes** | `note_routes.py` | — | LOW |
| **Other** | `prefs_routes.py`, `upload_routes.py`, `vault_routes.py`, `webhook_routes.py`, `workspace_routes.py`, `search_routes.py`, `history_routes.py`, `hwfit_routes.py`, `preset_routes.py`, `signature_routes.py`, `backup_routes.py`, `cleanup_routes.py`, `diagnostics_routes.py`, `embedding_routes.py`, `emoji_routes.py`, `font_routes.py`, `stt_routes.py`, `tts_routes.py`, `compare_routes.py`, `personal_routes.py`, `editor_draft_routes.py`, `admin_wipe_routes.py`, `chatgpt_subscription_routes.py` | 2,000+ | LOW individual, HIGH cumulative |
---
## 5. Tool Registry & Implementation Boundaries
### 5.1 Current Tool Architecture
| Component | File | Lines | Role |
|-----------|------|-------|------|
| Tool schemas | `src/tool_schemas.py` | 1,392 | JSON Schema tool definitions (Duck-TypedDict) |
| Tool index | `src/tool_index.py` | 542 | RAG-based tool retrieval from ChromaDB |
| Tool implementations | `src/tool_implementations.py` | 4,032 | 33 `do_*` functions — all tool execution logic |
| Tool security | `src/tool_security.py` | — | Owner-scoped tool blocking |
| Tool policy | `src/tool_policy.py` | — | Guide-only directive, plan-mode disabled tools |
| Tool utils | `src/tool_utils.py` | — | Shared tool helpers |
### 5.2 Tool Implementation Categories
The 33 `do_*` functions in `tool_implementations.py` fall into natural domain groups — the basis for slice 1's split in §6.2:
| Category | `do_*` functions | Count |
|----------|------------------|-------|
| **System / config** | `do_manage_skills`, `do_manage_tasks`, `do_manage_endpoints`, `do_manage_mcp`, `do_manage_webhooks`, `do_manage_tokens`, `do_manage_settings`, `do_api_call`, `do_app_api` | 9 |
| **Cookbook / model serving** | `do_download_model`, `do_serve_model`, `do_list_served_models`, `do_stop_served_model`, `do_tail_serve_output`, `do_list_downloads`, `do_cancel_download`, `do_search_hf_models`, `do_adopt_served_model`, `do_list_cookbook_servers`, `do_list_serve_presets`, `do_serve_preset`, `do_list_cached_models` | 13 |
| **Notes** | `do_manage_notes` | 1 |
| **Calendar** | `do_manage_calendar` | 1 |
| **Search** | `do_search_chats` | 1 |
| **Research** | `do_manage_research`, `do_trigger_research` | 2 |
| **Contacts** | `do_resolve_contact`, `do_manage_contact` | 2 |
| **Vault** | `do_vault_search`, `do_vault_get`, `do_vault_unlock` | 3 |
| **Image** | `do_edit_image` | 1 |
| | **Total** | **33** |
> Low-level tools (filesystem, subprocess, web fetch, document parsing) live in `src/agent_tools/`, **not** in `tool_implementations.py` — out of scope for this split.
---
## 6. Risk Assessment & Candidate Slice Ranking
> **Candidate proposals, not a committed plan.** The rankings, package shapes (e.g. `src/pkg/`, `src/domain/`, `src/infra/`, `src/api/`), split ordering, and route-grouping strategy below are **options for maintainer discussion**. Per #4082/#4071, slice ownership and order are settled by maintainers before any follow-up PR. §1–§3 above are the factual current-state inventory.
### 6.1 Risk Scale
| Level | Criteria |
|-------|----------|
| **LOW** | File has ≤3 importers AND ≤500 lines, OR is a pure refactor with clear boundaries |
| **MEDIUM** | File has 415 importers OR 5001,500 lines |
| **HIGH** | File has 16+ importers OR >2,000 lines, OR has cross-layer import violations |
### 6.2 Ranked Split Candidates
| Priority | Target | Risk | Rationale |
|----------|--------|------|-----------|
| **1** | `src/tool_implementations.py``src/tools/*.py` | **MEDIUM** | 4,032 lines → ~10 files by tool category. Already has natural boundaries. 17 importers, tracked in #3629. Use `__init__.py` shim to keep existing imports working. |
| **2** | `routes/` → domain subdirectories (one domain per PR) | **MEDIUM** | 54 flat files. Done **one domain at a time** (e.g. a standalone PR for the email domain, then chat, …), not a broad reorganization — route modules carry helper imports, registration assumptions, and test import paths. |
| **3** | `src/agent_loop.py``src/agent/loop.py` + submodules | **MEDIUM-HIGH** | 2,961 lines, 24 functions. Can extract prompt building, classification, verification, and runaway detection. Tracked in #3266. |
| **4** | `src/``src/pkg/`, `src/domain/`, `src/infra/`, `src/api/` | **MEDIUM** | Structural reorganization. Split flat `src/` into layered packages. Must come after routes and tools are stable. |
| **5** | `routes/email_*.py` consolidation | **LOW** | Already grouped by filename prefix. Low-risk cleanup within the email domain. |
| **6** | `core/database.py``src/infra/database/models/*.py` | **HIGH** | 28 classes, 102 importers. Highest-risk split. Must be **last** in any sequence. Requires careful import shim strategy. |
| **7** | Frontend CSS modularization | **MEDIUM** | 36,653 lines. Tracked in #2617. Separate timeline from backend work. |
| **8** | Frontend JS modularization | **MEDIUM** | 9,776 lines in `document.js`. Introduce ES modules at minimum. |
### 6.3 Candidate First 3 Behavior-Preserving Slices
**Slice 1: Split `tool_implementations.py`** (Lowest-risk high-impact)
- Create `src/tools/` package with one file per tool category
- Add `src/tools/__init__.py` re-exporting all symbols with current names
- Update 17 importers to use new paths (can be deferred via shim)
- Validation: `python -m pytest tests/ -x -q` + manual smoke test of tool execution
- Reference: #3629
**Slice 2: Group `routes/` by domain** (one domain per PR, not a broad sweep)
Route modules carry helper imports, router registration assumptions, and test import paths, so this must be done **one domain at a time** rather than as a single reorganization PR. Example sequence (each its own PR):
- PR 2a: move the **email** domain (`email_routes.py`, `email_helpers.py`, `email_pollers.py`) → `routes/email/` + shim
- PR 2b: move the **chat/agent** domain → `routes/chat/` + shim
- PR 2c: move the **cookbook** domain → `routes/cookbook/` + shim
- …and so on per domain from §4
Each PR: add `__init__.py` re-exporting old names, update `app.py` router imports, validation `python app.py` starts clean. **No behavior change** — pure file reorganization.
**Slice 3: Extract `agent_loop.py` submodules** (Improve reviewability)
- Move prompt assembly → `src/agent/prompt.py`
- Move request classification → `src/agent/classifier.py`
- Move sub-agent verification → `src/agent/verifier.py`
- Move runaway detection → `src/agent/runaway.py`
- Move context management → `src/agent/context.py`
- Keep `src/agent/loop.py` as the main orchestration module
- Validation: `python -m pytest tests/test_agent_loop.py tests/test_loop_breaker_runaway.py -v`
---
## 7. Safety Guardrails for Follow-Up Work
Per maintainer guidance in #4082 and #4071:
- [ ] **One domain/slice per PR** — never mix multiple reorganizations
- [ ] **No behavior changes** mixed with file moves — pure reorganization only
- [ ] **Keep compatibility shims**`__init__.py` re-exports for all existing import paths
- [ ] **Add or identify focused tests** before risky splits
- [ ] **Do not start with `core/database.py`** or broad route movement unless this inventory shows a safe boundary
- [ ] **Prefer small, reviewable slices** over large restructures
- [ ] **No packaging/runtime/tooling migration** mixed into file moves
- [ ] **No frontend framework migration** inside this stabilization lane
- [ ] **Validate with `python -m compileall`** — every PR must pass CI checks
- [ ] **Validate with `pytest`** — run the full test suite before opening each PR
---
## 8. Validation Commands
Each follow-up PR should be verifiable with these commands before submission:
```bash
# Syntax check — must pass with zero errors
python -m compileall src/ routes/ core/ conf/
# Full test suite — must match baseline pass rate
python -m pytest tests/ -x -q
# Import shim verification — existing import paths must still work
python -c "from src.tool_implementations import do_search_chats; print('OK')"
# App startup smoke test (if backend touched)
timeout 5 python app.py 2>&1 | head -5 || true
```
---
## 9. Open Questions
1. Is `#2538` (specs ground truth) the canonical behavior map baseline, and should this inventory be kept in sync with those specs once merged?
2. Should route grouping follow the domain map proposed here, or is there a different taxonomy preferred by maintainers?
3. For the `tool_implementations.py` split (#3629), is the tool categorization in §5.2 acceptable, or should it follow a different grouping?
4. Should compatibility shims (`__init__.py`) be temporary (removed in a follow-up wave) or permanent?
5. Should an ADR (Architecture Decision Record) document be started to track decisions made during this process?
---
## 10. Future Direction (NOT current state)
The following are **future refactor targets** (candidate directions **pending maintainer agreement**, not committed), recorded here so this inventory does not imply they exist today. None of them are present in the current `dev` tree:
- `main.py` — proposed rename of the `app.py` entrypoint. Today the app boots via `app.py`.
- `src/agent/` — proposed package to hold `agent_loop.py` submodules (prompt/classifier/verifier/runaway/context). Today `agent_loop.py` is a single flat file in `src/`.
- `src/infra/`, `src/domain/`, `src/pkg/`, `src/api/` — proposed layered reorganization of the flat `src/` directory (slice 4 in §6).
These become real only when the corresponding slices land.
---
## Appendix A: File Listing
### `src/` (95 files — 61 shown; run `ls src/*.py` for the full list)
```
agent_loop.py tool_implementations.py tool_schemas.py
tool_index.py tool_security.py tool_policy.py
tool_utils.py builtin_actions.py task_scheduler.py
llm_core.py model_context.py model_discovery.py
session_search.py context_budget.py context_compactor.py
ai_interaction.py action_intents.py agent_runs.py
app_helpers.py app_initializer.py config.py
database.py memory.py memory_provider.py
secret_storage.py prompt_security.py url_security.py
url_safety.py rate_limiter.py cleanup_service.py
readiness.py service_health.py exceptions.py
request_models.py assistant_log.py bg_monitor.py
builtin_mcp.py chat_helpers.py chroma_client.py
document_processor.py embedding_lanes.py deep_research.py
research_handler.py research_utils.py personal_docs.py
rag_manager.py rag_singleton.py topic_analyzer.py
visual_report.py youtube_handler.py pdf_forms.py
pdf_form_doc.py pdf_runtime.py caldav_writeback.py
email_thread_parser.py text_helpers.py user_time.py
teacher_escalation.py cookbook_serve_lifecycle.py
chatgpt_subscription.py mcp_manager.py
```
### `routes/` (54 files)
```
__init__.py _validators.py
auth_routes.py api_token_routes.py device_flow.py
chat_routes.py chat_helpers.py shell_routes.py
codex_routes.py skills_routes.py
email_routes.py email_helpers.py email_pollers.py
cookbook_routes.py cookbook_helpers.py cookbook_output.py
model_routes.py assistant_routes.py copilot_routes.py
calendar_routes.py contacts_routes.py
document_routes.py document_helpers.py
gallery_routes.py gallery_helpers.py
task_routes.py session_routes.py
note_routes.py memory_routes.py research_routes.py
mcp_routes.py search_routes.py history_routes.py
webhook_routes.py workspace_routes.py upload_routes.py
vault_routes.py prefs_routes.py preset_routes.py
signature_routes.py personal_routes.py hwfit_routes.py
backup_routes.py cleanup_routes.py diagnostics_routes.py
embedding_routes.py emoji_routes.py font_routes.py
stt_routes.py tts_routes.py compare_routes.py
editor_draft_routes.py chatgpt_subscription_routes.py admin_wipe_routes.py
```
### `core/` (10 files)
```
__init__.py constants.py database.py models.py
auth.py middleware.py session_manager.py exceptions.py
atomic_io.py platform_compat.py
```
---
## Appendix B: Key Import Relationships
```
core/database.py ←── 102 importers (routes/*, src/*, core/*, tests/*)
├── routes/auth_routes.py
├── routes/email_routes.py
├── src/builtin_actions.py
├── src/task_scheduler.py
├── src/tool_implementations.py (inline)
└── ...97 more
src/tool_implementations.py ←── 17 importers
├── src/agent_loop.py
├── src/builtin_actions.py
├── src/tool_index.py
├── src/task_scheduler.py
├── src/tool_policy.py
└── ...12 more (mostly tests)
src/agent_loop.py ←── 22 importers
├── src/tool_policy.py
├── src/teacher_escalation.py
├── src/bg_monitor.py
├── src/task_scheduler.py
└── 18 more (incl. tests)
```
-169
View File
@@ -1,169 +0,0 @@
# Auth And Security
Last updated: dev@e71f8ce | 2026-08-25
## Scope
This spec covers current security and trust-boundary behavior in:
- `core/auth.py`;
- `core/middleware.py`;
- `core/log_safety.py`;
- `core/database.py`;
- `app.py` auth middleware and token cache;
- `src/auth_helpers.py`;
- `src/owner_identity.py`;
- `src/tool_approval_scopes.py`, `src/tool_approvals.py`, and `src/tool_capabilities.py`;
- `src/tool_security.py`;
- `src/tool_execution.py`;
- `src/task_action_policy.py`;
- `src/prompt_security.py`;
- `src/url_safety.py` and `src/url_security.py`;
- `src/host_docker_access.py`;
- `src/attachment_refs.py` and upload lifecycle enforcement in
`src/upload_handler.py` / `routes/upload_routes.py`;
- `src/secret_storage.py`;
- `src/api_key_manager.py`;
- `src/integrations.py`;
- `src/webhook_manager.py`;
- `src/generated_images.py`;
- `scripts/diffusion_server.py`;
- `scripts/mlx_image_server.py`;
- `companion/routes.py` and `companion/pairing.py`;
- `routes/auth_routes.py`, `routes/api_token_routes.py`, and canonical `routes/vault/vault_routes.py` plus its top-level compatibility shim;
- admin-gated call sites in route files;
- `THREAT_MODEL.md` and `SECURITY.md`.
## Trust Boundary
Odysseus is a trusted-user private-network app. Admins intentionally have powerful local capabilities: shell, files, email, calendar, MCP, model serving, vault, settings, and API token management. The security model prevents unauthenticated access, non-admin escalation, prompt-injection through untrusted content, and accidental exposure of internal services.
`THREAT_MODEL.md` owns high-level security framing, but implementation claims here should be verified against current code when the threat model is stale. This spec records the implementation map that contributors should check before changing auth or untrusted-context flows. Security-header runtime details live in `runtime.md`.
## Auth Ownership
- `core.auth.AuthManager` owns users, password hashing, TOTP/backup codes, reserved usernames, privilege defaults, admin promote/demote state, and auth settings stored in `data/auth.json`. Auth config/setup mutations are lock-guarded, and session tokens are persisted separately in `data/sessions.json` behind their own lock.
- `app.py` owns request-time auth middleware, token-cache rebuild/invalidation, auth exemptions, API-token verification, and internal-tool identity stamping.
- `routes/auth_routes.py` owns HTTP endpoints for setup, signup/login/logout, 2FA, users, privileges, auth features, and integration settings.
- `core.middleware.require_admin()` owns the normal admin gate. Local wrappers must document and test any intentional divergence from that boundary.
- `src.auth_helpers.effective_user()` owns cookie/API-token owner attribution for selected route code. `require_user()` owns route-level degraded user resolution, `require_privilege()` owns privilege checks, and `owner_filter()` owns shared/null-owner query compatibility.
Reserved usernames include request-only sentinels `internal-tool`, `api`, `demo`, and `system`, plus the storage-only Default/Local owner `__odysseus_local__`. Loaded auth data drops reserved user records, and create/rename flows must reject real users with those names. `src.owner_identity` is the canonical owner vocabulary and `auth_disabled()` parser.
## Auth Runtime Flow
`AuthMiddleware` is the outer request gate because FastAPI middleware executes in reverse add order. It can return API `401` JSON or browser `/login` redirects before timeout/security-header middleware reaches the route.
Public/auth-exempt surfaces are limited to setup, signup/login/logout/status, feature/settings/integration preset reads, health/version/login, `/static/*`, and task webhook trigger paths. `routes/task/task_routes.py` owns validation of `POST /api/tasks/{task_id}/webhook/{token}` path credentials.
Login issues an `HttpOnly`, `SameSite=Lax` cookie with a seven-day max age when "remember" is enabled. `_secure_cookie()` (`routes/auth_routes.py:89`) decides the `Secure` attribute: an explicit `SECURE_COOKIES` of `true` or `false` is authoritative, and any other value, including unset and the present-but-empty value docker-compose injects, derives it from the request, marking the cookie `Secure` when the connection scheme or the first `X-Forwarded-Proto` hop is https. TOTP is checked before session issuance. Logout, password changes, user deletion, rename flows, expired sessions, and deleted-user sessions must keep revocation/migration behavior intact.
Deleting a user revokes that user's browser sessions and API-token rows, then the admin delete route invalidates the in-memory bearer-token cache so already-cached tokens stop authenticating.
Rename first changes the auth username, then migrates owner-bearing DB rows and disk-backed stores. Current rename coverage includes user preferences, active/disk research state, `memory.json`, upload metadata and owner-qualified upload index keys, skills frontmatter/usage state, cached browser sessions, and API-token cache invalidation. If owner migration fails after the auth rename, the route attempts to roll auth back to the old username instead of leaving a split identity.
Admin promotion/demotion is a live auth flag change through `AuthManager.set_admin()` and `PUT /api/auth/users/{username}/admin`. Demotion refuses to remove the last admin, permits self-demotion when another admin remains, restores the pre-admin privilege map when available, and does not revoke sessions or API tokens because later admin checks read the current `is_admin` flag.
## Owner Attribution
Cookie requests use the real username. Bearer-token requests are stamped as `request.state.current_user = "api"` plus `api_token_owner`, `api_token_scopes`, and token id. Routes that support API-token access must explicitly use `effective_user()` or route-local scope helpers instead of treating `"api"` as an owner.
Internal loopback calls may stamp `current_user = "internal-tool"` or a validated `X-Odysseus-Owner` username. Network/proxy validation for that bypass lives in `app.py`; `require_admin()` trusts the stamped sentinel or raw internal header and should be used behind equivalent middleware control.
Missing-owner values remain state-dependent at legacy call sites, but new storage-facing code has one normalization contract:
- Auth-enabled, configured auth with no `current_user` is unauthenticated and should fail closed at route dependencies.
- `AUTH_ENABLED=false` is an explicit local single-user/no-login mode. Existing route dependencies can still return `""`, and admin gates allow the local operator. `effective_storage_owner()` and `storage_owner_for_request()` normalize an absent owner to `__odysseus_local__` only in this mode.
- Chat/agent code that reads `get_current_user(request)` directly gets `None` when auth middleware is disabled, because no middleware stamps request state.
- SQL `NULL`/JSON missing owners remain legacy/shared compatibility data, not the same thing as a logged-out authenticated caller.
- `"api"` and `"internal-tool"` are request sentinels. They must not be persisted as normal storage owners unless a route explicitly defines that behavior.
- `__odysseus_local__` is a valid storage owner but never a login or request sentinel. Adoption is incremental: callers that do not use the storage-owner helper can still expose older `None`/empty/null compatibility behavior.
Authenticated `manage_tasks` mutations require an exact stored task-owner
match and reject both cross-owner and legacy null-owner rows. The `owner=None`
agent path keeps deliberate auth-disabled single-user compatibility, including
unscoped list/create/mutation behavior.
Owner-scoped route code should use `require_user()` or equivalent policy before querying per-owner data. Current note CRUD/reorder/reminder routes do this so an auth-enabled request that reaches the route without identity returns `401` instead of falling into single-user/null-owner compatibility behavior.
Scheduled task actions attribute differently again. `_execute_action` (`src/task_scheduler.py:1231`) invokes the action with `owner=task.owner` read from the stored `ScheduledTask` row, so no request and no resolved principal are in flight. These trigger paths converge there: schedule, event bus, manual run (`routes/task/task_routes.py:865`), the `manage_tasks` agent tool (`src/tools/system.py:469`), webhook triggers (`routes/task/task_routes.py:1045`), which are unauthenticated by design with the token as the only credential and execute under the stored `task.owner`, and success-chained tasks (`src/task_scheduler.py:1063-1074`), which additionally require the chained target to share `task.owner` and reject cycles. Trigger-side ownership checks use the `if user and task.owner != user` shape, so a falsy caller skips them. Action bodies that reach owner-scoped storage must treat `task.owner` as the authority; route-level `require_user()` never runs on this path.
## API Tokens And Scoped Integrations
`routes/api_token_routes.py` owns token CRUD and scope normalization. Partial updates preserve existing scopes unless new scopes are supplied, write scopes imply the matching read scopes where applicable, and Cookbook scopes are part of the normalized scope set. `app.py` caches active token prefix rows and verifies bearer tokens with bcrypt. API-token requests set `request.state.current_user = "api"` plus token owner/scopes.
Current call sites include Codex/Claude scoped APIs, `/api/v1/chat`, webhooks, selected session routes, companion pairing, and external integrations. `/api/codex/*` and `/api/v1/chat` enforce route-local scopes; companion and selected session routes use owner attribution. `companion/pairing.py` can mint chat-scoped tokens outside normal token CRUD.
Admin token CRUD is cookie/admin gated. Update/delete operations check token ownership, and cache rebuild ignores active tokens whose owner no longer maps to a known auth user. Scoped route code must use the token owner and declared scopes instead of falling back to cookie-user assumptions.
## Internal Tool Loopback
Agent tools call admin-gated HTTP routes through an in-process loopback. `core.middleware.INTERNAL_TOOL_TOKEN` owns the random per-process secret. `app.py` only accepts this bypass from direct loopback clients without proxy-forwarding headers.
`src.tool_security` owns non-admin tool blocking. Non-admin users must not reach admin tools through agent mode, MCP tools, or loopback calls.
`src.tool_security.owner_is_admin_or_single_user()` treats explicit `AUTH_ENABLED=false` as intentional single-user mode even when an auth store already exists, while keeping pre-setup auth-enabled callers non-admin.
Current admin gates include `require_admin()` call sites across admin wipe, backup, contacts, Cookbook, diagnostics, embeddings, MCP, model, personal docs, presets, skills, uploads, vault, webhook, and companion routes. Local wrappers also exist in auth routes, shell routes, and task action policy; changes to those wrappers need the same trust-boundary review as `require_admin()`. Scheduled task action policy treats `run_local`, `run_script`, `ssh_command`, and `cookbook_serve` as admin-only action tasks across create/update/manual-run/webhook/scheduler execution.
`tidy_research` can remove only empty or unparseable research JSON. Because a broken file has no trustworthy owner stamp, the action checks `owner_is_admin_or_single_user()` before enumerating files; regular users and the pre-setup window cannot run that global unattributable-file sweep.
## Untrusted Context Policy
`src.prompt_security` owns the model-facing untrusted data contract:
- `UNTRUSTED_CONTEXT_POLICY` states the policy in system prompt text.
- `untrusted_context_message(label, content)` wraps external content as user-role data with `metadata.trusted = False`, provenance metadata, and a default `tool_gate_untrusted` marker. Guard-like labels/content are escaped so source text cannot counterfeit the wrapper boundary.
Current untrusted surfaces include fetched URLs, web results, emails, memories, skills, notes, documents, active editor content, and tool output sourced from outside the server. Injecting those as trusted system instructions is a security bug.
`src.tool_capabilities` classifies native and MCP tools by effects and result integrity. After external/workspace-untrusted context becomes model-visible, `ToolRunSecurityContext` keeps a server-owned taint for the session turn: only explicitly low-impact tools can run immediately, while write, execute, network-egress, UI/external-side-effect, admin, destructive, unknown, and arbitrary MCP actions require exact approval. Failed tools can still arm the gate when their result carries remote or stored payload; content-free failures and server-generated blocked/approval placeholders do not.
`src.tool_approvals` owns opaque approvals sealed to the owner, session, origin run, exact first tool name/content, workspace, capability effects/result integrity, selected continuation tool set/query, and expiry. Document actions additionally seal document id, version, content digest, and workspace. Chat cards offer task scope, chat-session scope, or deny: both allow choices consume and execute the exact sealed first action after current-policy/freshness checks, task scope bypasses the gate only for the resumed task, and chat-session scope persists a resolved session-bound grant for later turns in that same chat. The browser submits only the opaque decision and cannot replace the sealed action, selected tools, query, composer text, or attachments. Non-chat callers retain single-action scope. A new ordinary turn or superseding action retires an unresolved approval without clearing taint.
## URL, Path, And Secret Policy
- `src/url_security.py` owns public HTTP(S) validation for integration/API-token supplied URLs. It should fail closed for private IP, loopback, invalid scheme, and unsafe redirect targets.
- `src/url_safety.py` owns local-first outbound URL safety for model endpoints and similar local services. Loopback/LAN can be allowed by default, and private-IP blocking is an explicit caller policy. Strict `block_private=True` also rejects RFC 6598 shared/CGNAT space (`100.64.0.0/10`) explicitly because Python does not classify that range as private.
- `core.log_safety.redact_url()` strips URL userinfo, query strings, and fragments before endpoint URLs enter logs. Model, chat/research endpoint, contact/CardDAV, and similar diagnostics should use this helper instead of logging raw admin-configured URLs.
- `src.webhook_manager` validates webhook URLs at create and delivery time,
rejects private/internal targets, disables redirects, and pins delivery to
the public IP set that passed validation immediately before the request.
- `src.integrations` owns admin-configured integration base URLs and secret
masking. `api_call` accepts only relative paths, rejects link-local/metadata destinations through `src.url_safety`, can additionally block RFC1918/loopback/private targets with `INTEGRATION_API_BLOCK_PRIVATE_IPS=true`, and pins requests to the IP set that passed SSRF validation while preserving the intended Host/TLS identity.
- `src.outbound_fetch` owns reusable public-URL classification, validates every redirect hop, rejects private/local resolved addresses, and pins the HTTP connection to the validated public IP while preserving original URL/SNI/Host semantics. `services.search.content` adapts that transport for extraction and caching.
- Path-based tools, upload/document/gallery/signature/generated-image routes, embedding cache paths, and research JSON helpers must stay confined to allowed roots and owner-scoped files. Native file/code-navigation tools also apply a case-insensitive sensitive-path denylist so `grep`, `glob`, `ls`, direct reads, and writes cannot reveal `.env`, SSH/GPG material, private-key filenames, or similar secret paths.
- Durable upload references are owner-reserved before chat/session, document,
note, or calendar writes. Cleanup scans every current durable reference
surface and fails closed on incomplete discovery or inconsistent upload-index
state rather than deleting a possibly live upload.
- File-backed SQLite startup restricts `app.db` and existing rollback/WAL/SHM
sidecars to `0600` on POSIX after resolving the real path from the parsed
engine URL. Windows, in-memory, and non-SQLite databases are excluded, and
failed POSIX restriction is logged as a secret-file warning.
- Secret-like DB columns use `EncryptedText` or `src.secret_storage`. Email passwords and Google OAuth mail tokens are encrypted manually in `EmailAccount` string columns; Google OAuth state is HMAC-signed and callback writes are owner-checked before token storage. `src.api_key_manager` keeps provider API keys encrypted in `data/api_keys.json`, writes by loading the raw encrypted dict so saving one provider does not rewrite other providers' keys as plaintext, and restricts local key-file permissions where the platform supports chmod. Vault state in `data/vault.json` is a chmod-restricted JSON secret store, not Fernet-encrypted DB storage. Do not log or return decrypted secrets except for intentional admin vault retrieval flows with audit/reason checks.
- `.env` files are secrets-only inputs and should not be read or printed during agent work.
`scripts/diffusion_server.py` is a local model-serving helper with its own web surface. It defaults CORS to deny, installs a trusted-host allowlist for loopback/bind addresses, and only extends Host/CORS through explicit CLI flags.
`scripts/mlx_image_server.py` serves exactly the model selected when the process starts. OpenAI-compatible request `model` fields are accepted but ignored for generation and edits, so an unauthenticated caller cannot select another local directory or Hugging Face repository and drive model-specific script/bridge execution.
Host Docker socket access is a high-trust admin/deployment choice, not a normal container capability. Default Docker Compose does not mount `/var/run/docker.sock`; `src.host_docker_access` only reports local Docker available inside a container when `ODYSSEUS_ENABLE_HOST_DOCKER=true` and the socket exists. Remote SSH Docker/Cookbook workflows remain the safer default.
## Degraded And Compatibility Behavior
- `AUTH_ENABLED=false` skips `AuthMiddleware` and `src.auth_helpers.require_user()` returns `""` from any host. This preserves local single-user/no-login operation; it is not permission for auth-enabled logged-out callers. Storage code that adopts `storage_owner_for_request()` receives the reserved Default/Local owner; direct `get_current_user()` readers still receive `None`. Owner-scoped routes that tolerate no-login mode should call the appropriate route or storage helper so auth-enabled anonymous requests fail closed.
- First-run setup mode redirects browser requests to `/login`, returns API `401 Setup required`, and keeps setup/status/login surfaces auth-exempt. Setup/signup/login are rate-limited; status is exempt but not rate-limited. Route helper fallbacks only tolerate unconfigured anonymous access from loopback.
- User privilege checks distinguish legacy empty `allowed_models=[]` from explicit no-model access through `allowed_models_restricted=True`.
- `LOCALHOST_BYPASS` in `app.py` only applies to direct loopback clients and excludes proxy/tunnel headers. Helper fallback code is weaker and should not be treated as the primary bypass boundary.
- Legacy migrations claim null-owner SQL/JSON data for the primary admin when possible, and startup repeats a null-owner sweep hourly. Remaining null-owner rows are surface-specific compatibility data that must be deliberately included, no-oped for single-user mode, or rejected for strict ownership gates.
- `.env` is loaded with `utf-8-sig`, so Windows BOM auth flags still parse.
## Current Gaps
- There is no shell/filesystem sandbox for admin tools.
- Token scopes remain coarse for some surfaces.
- `app.py` AuthMiddleware lacks direct regression coverage for bearer-token state/cache behavior, trusted-loopback proxy-header rejection, and internal-tool owner stamping.
- Codex/Claude scoped route enforcement still needs stronger regression coverage.
- `THREAT_MODEL.md` still has stale token-scope and `/api/v1/chat` SSRF gap text that should be reconciled with current route validation.
- The Default/Local owner contract is canonical but only incrementally adopted; route helper `""`, chat/agent `None`, SQL/JSON null-owner compatibility, and calendar fallback owner behavior still need domain-by-domain migration decisions.
-186
View File
@@ -1,186 +0,0 @@
# Calendar, Tasks, And Notes
Last updated: dev@e71f8ce | 2026-08-25
## Scope
This spec covers calendar, reminders, tasks, assistant runs, and notes in:
- app route wiring, auth exemptions, and scheduler startup in `app.py`;
- canonical database models in `core/database.py`, with `src/database.py` as a compatibility re-export;
- `routes/calendar_routes.py`, `src/caldav_sync.py`, and `src/caldav_writeback.py`;
- canonical `routes/task/task_routes.py`, compatibility shim `routes/task_routes.py`, `src/task_scheduler.py`, `src/task_endpoint.py`, `src/event_bus.py`, and `src/interactive_gate.py`;
- shared privileged task-action policy in `src/task_action_policy.py`;
- `routes/assistant_routes.py`;
- canonical `routes/note/note_routes.py`, compatibility shim
`routes/note_routes.py`, `src/builtin_actions.py`, and `src/action_intents.py`;
- agent/tool call sites in `src/tool_index.py` and `src/tool_implementations.py`;
- scoped Codex wrappers in `routes/codex_routes.py`;
- database models `CalendarCal`, `CalendarEvent`, `ScheduledTask`, `TaskRun`, `Note`, and `CrewMember`;
- direct DB CLIs `scripts/odysseus-calendar`, `scripts/odysseus-notes`, and `scripts/odysseus-tasks`;
- frontend modules `static/js/calendar.js`, `static/js/calendar/*`, `static/js/tasks.js`, `static/js/notes.js`, and `static/js/assistant.js`;
- tests covering calendar routes/utilities, CalDAV, recurrence, timezone handling, scheduler behavior, task webhooks, notes CLI/tool behavior, and task CLI behavior.
## Calendar
`routes/calendar_routes.py` owns `/api/calendar` behavior: config, multi-account CalDAV CRUD, connection test, sync, local calendar CRUD, event CRUD, recurrence expansion, ICS import/export, quick parse, and user timezone offset handling.
`src.caldav_sync` owns CalDAV fetch/sync. `src.caldav_writeback` owns pushing local changes back to remote calendars. Calendar routes request those behaviors; they do not own CalDAV protocol details.
Runtime behavior:
- local default calendars are created lazily per owner with stable UUID5 candidates. Default creation remains inside the caller's transaction so a failed event write cannot leave an orphaned calendar; SQLite serializes the absent-row check with `BEGIN IMMEDIATE`, other backends recover insert races inside a savepoint, and renamed-owner ID collisions advance through deterministic slots. List-only callers explicitly commit the lazy default.
- route-level no-login calendar access normalizes empty owner values to `ODYSSEUS_FALLBACK_OWNER` or `owner@localhost`, so route-created calendar rows do not use the empty string as their storage owner;
- CalDAV account config lives in per-user prefs as `caldav_accounts`, with the legacy `/api/calendar/config` route reading/upserting the first account;
- recurring rules are expanded server-side, including compound recurrence IDs;
- RRULE expansion is capped and marks truncated responses;
- event datetimes preserve UTC/local metadata through `CalendarEvent.is_utc` where supported;
- CalDAV pull uses a bounded sync window, scopes existing UID lookups to the synced calendar, stamps account ids and remote metadata on local calendars, maps Google principal URLs to event collections, preserves locally-created or writeback-pending events that are not yet remote-owned, and deletes stale in-window remote events only when remote object parsing did not fail;
- CalDAV writeback stores `remote_href`/`remote_etag`, clears `caldav_sync_pending` only after successful remote writes, and leaves create/update/delete pending markers for retry on failure;
- pull and writeback paths always close their `DAVClient`, including discovery,
database, and remote-write failure paths;
- sync direction can be pull, push, or both, and pending local writeback rows are included even before remote href metadata exists;
- ICS import is per-owner, capped, creates fresh local IDs in the target import calendar, and preserves zero-duration events as visible imported rows rather than dropping them as empty ranges;
- writeback is best-effort and local SQLite remains source of truth when remote writes fail.
Calendar credentials are encrypted at rest and are not returned to clients. CalDAV URL validation rejects unsafe schemes, credentials, fragments, localhost names, bad ports, unsafe IP literals, and hostnames resolving to disallowed addresses, with `ODYSSEUS_ALLOW_PRIVATE_CALDAV=1` as the explicit private-IP escape hatch. CalDAV sync/writeback clients disable redirects so credentials are not followed to another origin. The connection-test client keeps proxy/environment trust disabled but explicitly loads an operator `SSL_CERT_FILE` or `REQUESTS_CA_BUNDLE` when the file exists so private/self-signed deployments use the same CA trust intent as real sync.
## Tasks And Assistant Runs
`src.task_scheduler.TaskScheduler` owns scheduled task execution, next-run computation, strict single-slot execution, queued/running cleanup at startup, overdue next-run advancement, webhook-triggered tasks, notifications, run records, chained tasks, and event-triggered actions.
Cookbook serve scheduling crosses this domain. The Cookbook UI creates `cookbook_serve` scheduled tasks, can mirror them as Cookbook calendar events with `cookbook_event_uid`, and task deletion cleans up the linked event when present, falling back to exact-summary matching for legacy events without a stored UID. Cookbook command execution/lifecycle details stay in `cookbook-hwfit.md`.
`routes.task.task_routes` owns task CRUD, status, manual run/stop/cancel, pause/resume, owner-scoped run/activity history, metadata, onboarding defaults, cache clearing, parse endpoints, and webhook-token regeneration. `app.py` imports the canonical package path; `routes/task_routes.py` replaces its module entry with the canonical module for legacy import and monkeypatch compatibility. Chained-task `then_task_id` values are validated as same-owner relationships on create/update, and scheduler execution also rejects cross-owner or cyclic chains.
Task webhook paths are auth-exempt at the app middleware layer only for `/api/tasks/{task_id}/webhook/{token}`. The route still validates active task state plus task-specific webhook token before dispatch.
Task runtime behavior:
- task runs move through queued/running/success/error/skipped/aborted states;
- scheduler/background execution can wait for `src.interactive_gate` to report a quiet foreground window, and running background work can use browser heartbeat/chat-stream activity as a cancellation/defer signal where implemented;
- output targets include chat sessions, notifications, email, and MCP delivery paths;
- LLM and research tasks can carry a built-in `character_id` persona prompt that the scheduler prepends at execution time;
- task-created chat sessions can be foldered under `Tasks`, and startup migration backfills task/research folders for legacy sessions;
- event-bus triggers persist counters and `next_run` before scheduler handoff;
- the in-process scheduler is gated by `ODYSSEUS_INPROCESS_TASKS`, and multiple enabled app processes can double-run work.
- action tasks with `run_local`, `run_script`, `ssh_command`, or
`cookbook_serve` are admin-only. `routes.task_routes` enforces this on
create/update/manual run and hides those actions from `/meta/actions` for
non-admin owners; webhook and scheduler execution pause the task and clear
`next_run` if an admin-only action belongs to a non-admin owner.
- background LLM task execution uses the background workload path, and the
scheduler can abort/cancel active in-process task runs when foreground browser
activity appears.
- `tidy_research` scans all persisted research files because broken JSON has no trustworthy owner stamp, so it runs only for admins or the explicit auth-disabled single-user operator and refuses regular/pre-setup callers before enumeration.
`routes.assistant_routes.py` owns crew/assistant settings and run-status surfaces that use the scheduler. `TaskScheduler.ensure_assistant_defaults()` currently seeds the personal assistant crew member and pinned assistant session, but no longer auto-creates Morning/Midday/Evening check-in tasks. Existing crew-linked check-in tasks are still rendered and managed when present.
## Notes And Reminders
`routes.note.note_routes` owns notes/todos/reminders, and `app.py` imports that
canonical path. `routes.note_routes` replaces its module entry with the
canonical module for legacy import and monkeypatch compatibility. Notes are
SQLAlchemy `Note` rows and can include due dates, ordering, images, repeat
state, AI classification, source/session provenance, and agent session
linkage.
Notes CRUD/reorder/reminder routes resolve the acting owner through `require_user()`: auth-enabled anonymous requests fail closed before hitting owner-scoped queries, while documented no-login/single-user modes still resolve to the compatibility owner path.
Reminder policy:
- "remind me at 5pm" should become a todo/note with a due date;
- calendar event alarm/reminder UI writes reminder Notes;
- calendar events are for scheduled time blocks, meetings, appointments, or explicit calendar requests;
- creating a calendar event named "Reminder" does not create notification behavior.
Built-in reminder/persona prompt text is mirrored server-side for reminder synthesis and scheduled task execution; frontend persona selectors are UI over that server-owned id map, not the authority.
Reminder dispatch is Note-owned:
- `dispatch_reminder()` owns browser, email, ntfy, generic webhook, in-app notification, optional LLM reminder text, and dedupe behavior;
- the scheduler note scanner calls note-ping actions for backend due-note delivery with per-owner notification state, and calendar-event reminders are treated as Note-owned reminders rather than separate scheduler event pings;
- the notes frontend has a browser-tab fallback for visible sessions;
- calendar frontend reminder UI stores reminder records as Notes, not calendar-event notification jobs.
Email/ntfy failures degrade into channel result fields rather than blocking every reminder path. ntfy and generic webhook reminder URLs run through outbound URL safety checks, with `REMINDER_WEBHOOK_BLOCK_PRIVATE_IPS` controlling whether private/LAN targets are allowed. ntfy notification titles are converted to ASCII with replacement and capped at 200 characters before entering HTTP headers. Reminder dedupe uses owner-scoped cache files under `data/`.
## Agent, Codex, And CLI Surfaces
`do_manage_tasks`, `do_manage_notes`, and `do_manage_calendar` own agent-side writes. `do_manage_calendar` supports batch event creation plus list range aliases (`start`, `start_time`, `start_date`, `range_start`, `from`, `dtstart`, `since`, and matching end aliases), calendar name/short-id lookup, importance/tag aliases, and reminder offsets expressed as numbers, minute/hour words, or common abbreviations such as `min`/`mins`/`hr`/`hrs`. If a model supplies a loose `query`, `date_range`, or `range` without explicit start/end datetimes, `list_events` returns an error asking the caller to resolve the range and call again instead of guessing. Event classification reads `Memory.text` for personal context before LLM classification. `src.tool_index` encodes the reminder policy that notes/todos own reminders while calendar events own time blocks.
Agent native tool owner handling is not uniform today. `do_manage_tasks()` filters lists only when `owner` is truthy and creates tasks with the passed owner, so `owner=None` can create legacy/null-owner tasks. For authenticated/non-empty owners, edit/delete/pause/resume/run require an exact stored owner match and reject both cross-owner and null-owner rows; `owner=None` retains single-user compatibility. `do_manage_notes()` list/query behavior distinguishes `None` from `""`, with `None` acting as broader single-user compatibility while `""` filters to empty-owner rows in some paths. `do_manage_calendar()` query helpers filter only when owner is not `None`, while calendar creation routes through the calendar fallback owner for default calendars. These are compatibility behaviors, not a cross-user sharing model.
Note and calendar route/tool writers owner-reserve any canonical internal upload
references in content, checklist/color/image fields, descriptions, and
locations before their database writes. Missing or wrong-owner uploads fail the
write instead of creating a dangling durable reference; reservations serialize
with upload cleanup.
Chat forwards browser timezone offset and IANA timezone name so natural-language note/calendar tools can anchor dates to the user clock. A valid IANA zone wins over the fixed offset for current-time/DST reasoning; invalid or absent names fall back to the offset and then server-local/UTC compatibility behavior. Chat can auto-promote note/calendar/reminder intents to agent mode.
Codex todo/calendar wrappers enforce bearer-token owner and `todos:*` or `calendar:*` scopes, then delegate to note/calendar behavior as the token owner. Normal calendar/task/note routes are current-user/cookie routes and should not be treated as scoped bearer-token APIs unless they explicitly use token owner/scope policy.
Direct DB CLIs are local compatibility tools. They bypass HTTP route behavior, CalDAV writeback, and some owner/timezone parsing policy.
## Event Bus
`src.event_bus` owns event-triggered task counters and scheduler handoff. Current emitters include chat/session/document/memory/research/email/skill paths. Ownerless events resolve to a primary configured user instead of broadcasting to every owner.
The current event bus is not a calendar-event emitter despite the adjacent calendar/task/reminder domain.
## Timezone And Date Semantics
- calendar events store offset-aware input as UTC/naive fields plus `is_utc`;
- note `due_date` uses ISO-like strings interpreted through note/tool parsers;
- chat forwards browser UTC offset into `routes.calendar_routes` request-local state for natural-language date anchoring in calendar/note tool parsing;
- generic scheduled task clock times are stored as UTC values after local conversion;
- assistant check-ins can use an IANA timezone on `CrewMember`, with UTC fallback.
Dateutil fallbacks strip timezone-aware parser results back to the naive-UTC contract before recurrence/window comparisons. Calendar agent list tools accept current range aliases implemented by `src.tool_implementations`, and equal/same-day start/end ranges are normalized to a one-day window instead of silently returning no rows.
Natural-language parsers prefer time-first interpretations for short reminder/event phrases where the user supplies a clock time before a date phrase.
Calendar frontend week-start preference is browser-local (`cal-week-start`) with Monday/Sunday controls; it is not persisted as a server preference.
Natural-language date parsing and timezone behavior are compatibility-sensitive and need route/tool/frontend regression coverage when changed. Request-local timezone context is ephemeral and must not be persisted as user state. A valid browser IANA timezone is authoritative over a possibly stale or wrong-sign fixed offset because it carries daylight-saving rules.
## Degraded And Optional Behavior
- CalDAV sync no-ops with shaped errors when unconfigured, invalid, offline, or missing the optional `caldav` dependency.
- CalDAV writeback failures are non-fatal to local calendar writes and are mostly visible through logs.
- Missing or invalid `croniter` rejects cron schedules or yields no next run.
- Missing timezone support falls back to UTC or legacy behavior.
- ICS import depends on `icalendar`; missing dependency can fail before route-shaped error handling today.
- Notes reminders can still use local browser fallback when backend email/ntfy channels fail.
- App backup import/export does not currently include calendar events, scheduled tasks, task runs, or notes; calendar ICS import/export is separate and calendar-only.
## Security And Provenance
Calendar, task, note, and assistant routes are owner-scoped for normal users. Legacy null-owner behavior is compatibility-sensitive and should not silently grant authenticated owners broad mutation rights.
Because auth-disabled chat owners can arrive as `None`, tool-created rows may not use the same owner value as route-created rows. Multi-user or owner-model changes must audit both route and agent paths.
Task creation/update/manual run/webhook/scheduler execution blocks shell-like and Cookbook serve action types for non-admin users through `src.task_action_policy`, and tool security blocks privileged task/calendar tools for non-admin use. Assistant defaults reject synthetic owners such as `api` and `internal-tool`.
Note routes store caller-provided `source`, `session_id`, `image_url`, and agent-session provenance. Canonical internal upload references in persisted note/calendar fields are owner-reserved before writes, and upload-backed bytes remain protected when fetched through upload routes. Arbitrary non-upload image/provenance URLs are not otherwise normalized or validated by note storage.
## Testing Coverage
Existing coverage is strongest around CalDAV URL hardening/writeback, client cleanup and operator CA handling, bidirectional/pending CalDAV sync markers, CalDAV UID calendar scoping, calendar recurrence/timezone helpers, owner-scoped calendar basics, exact-owner task-tool mutations, scheduler restart/cancel/next-run behavior, webhook auth-exemption source shape, canonical/legacy note-module identity, note-route unauthenticated fail-closed behavior, note/calendar attachment reservations, notes CLI/tool due-date behavior, calendar reminder abbreviation parsing, task CLI preview, task persona fields, and same-owner chained task validation.
Route-level coverage is thinner for full calendar route behavior, task CRUD/security/run controls, live webhook token dispatch, notes owner CRUD/reminder delivery, assistant defaults/run status, event-bus triggers, Codex todo/calendar scopes, and frontend panel wiring.
## Current Gaps
- CardDAV still needs URL hardening parity with CalDAV; CalDAV now resolves hostnames during validation and revalidates writeback URLs.
- `do_manage_notes()` should match HTTP note-route owner behavior for legacy null-owner notes.
- Auth-disabled agent tools can produce or read broader owner scopes than route handlers because they receive `owner=None`; tasks, notes, and calendar need aligned policy/tests.
- Task webhook tests should keep exercising live route token behavior and
admin-only action blocking, not only middleware/source strings.
- Reminder delivery needs tests across frontend `/fire-reminder`, backend `dispatch_reminder()`, scheduler note pings, channel degradation, and dedupe.
- Codex todo/calendar scope and owner mapping needs dedicated regression coverage.
- Direct DB CLIs need either documented route-bypassing support status or shared helpers to avoid owner/timezone/writeback drift.
- `scripts/odysseus-webhook` builds the live `/api/tasks/{task_id}/webhook/{token}` path with percent-encoded path segments; its direct DB token rotation/revocation behavior remains a local compatibility surface.
- Assistant default documentation/code comments still mention check-ins that are no longer auto-seeded.
- App backup import/export does not cover the calendar/task/note rows described by this spec.
-154
View File
@@ -1,154 +0,0 @@
# Chat
Last updated: dev@e71f8ce | 2026-08-25
## Scope
This spec covers current chat behavior in:
- `routes/chat_routes.py` and `routes/chat_helpers.py`;
- `routes/session_routes.py` and canonical `routes/history/history_routes.py`,
with `routes/history_routes.py` as a compatibility shim;
- `src/chat_helpers.py`;
- `src/agent_runs.py`;
- `src/chat_handler.py` and `src/chat_processor.py`;
- `core/session_manager.py` and `core/models.py`;
- `src/attachment_refs.py` and `src/upload_handler.py` for durable attachment
references and write reservations;
- `src/context_budget.py`, `src/context_compactor.py`, and `src/topic_analyzer.py`;
- `src/foreground_model_routing.py`, `src/tool_approval_scopes.py`, `src/tool_approvals.py`, and `src/tool_capabilities.py`;
- `routes/workspace_routes.py` for workspace selection support;
- frontend modules `static/js/chat.js`, `static/js/chatStream.js`, `static/js/chatRenderer.js`, `static/js/sessions.js`, `static/js/search-chat.js`, `static/js/compare/stream.js`, `static/js/workspace.js`, `static/js/composerArrowUpRecall.js`, `static/js/streamingSegmenter.js`, `static/js/group.js`, and `static/js/notes.js`;
- integration points with uploads, documents, compare, research, agent tools, memory, RAG, search, and model endpoints.
## Session Ownership
`core.session_manager.SessionManager` owns session persistence and message writes. `routes/session_routes.py` owns session list/create/update/archive/delete/folder/importance behavior for the sidebar. `routes.history.history_routes` owns history/topic surfaces, with `routes/history_routes.py` kept as a compatibility shim.
`core.models.Session` and `ChatMessage` are pure data containers. They do not own persistence; `Session.add_message()` delegates to the configured session manager when present.
Startup session discovery selects non-archived sessions by the existence of persisted `ChatMessage` rows rather than trusting the denormalized `Session.message_count`. It computes authoritative counts only for the bounded discovery set, then keeps full message hydration lazy.
## Streaming
`routes/chat_routes.py` owns `/api/chat`, `/api/chat_stream`, detached stream resume/stop/status, injected context, chat-message search, and rewrite routes. Streaming is the main UI path.
`static/js/chat.js` owns send/abort/continue UI state, the main fetch/read loop, SSE parsing, rendering dispatch, workspace form wiring, and background/resumable stream tracking. `static/js/chatStream.js` owns UI-control event handling and stream/research notification helpers. `static/js/sessions.js` polls server stream status after refresh or session switch. `static/js/composerArrowUpRecall.js` owns prompt recall from the composer when the caret is at the top of an empty input.
Runtime behavior:
- the `/api/chat*` prefix is exempt from the global request hard timeout;
- browser chat sends `X-Tz-Offset` and an IANA timezone name; request-local helpers prefer a valid IANA zone for DST-aware current-time reasoning, then fall back to the fixed offset;
- browser chat can send a selected workspace path; route code only resolves it for admin/single-user flows, validates it as an existing directory, and forwards it so agent file/shell tools are confined by `src.tool_execution`;
- stream callbacks can outlive a deleted session, so persistence must fail closed instead of recreating orphan messages;
- message metadata carries timestamps, metrics, tool events, sources, hidden
thinking/reasoning text when providers expose it separately, context-trim
metrics, structured attachment references, and related UI state;
- metadata preserves requested and actual reply models and endpoints, per-round route transitions, and answering-route cost attribution; stable session ids remain available so prompt/sequence-memory and KV-cache paths can address the same conversation consistently;
- multimodal content can be a list of content blocks for the live provider call,
while persistence collapses raw media into readable text and stable
attachment-reference lines;
- agent streams forward explicit round-cap, tool-budget, repeated-tool-loop,
and intent-without-action guard events so the frontend can distinguish a
controlled stop from a stalled response.
`src.agent_runs` owns detached in-memory stream runs, replay buffers, replacement cancellation, resume subscribers, explicit stop, and terminal-buffer eviction. Closing the SSE connection does not necessarily stop generation. `static/js/chat.js` can live-resume a still-running detached stream through `/api/chat/resume/{session_id}`; rich responses reload from DB for canonical rendering. Detached runs are process-local and do not survive server restart.
Provider adapters live below chat in `src.llm_core`. Chat consumes normalized SSE output, fallback/error events, reasoning/tool deltas, and metrics. Foreground chat is strict to the selected route by default. Only the selected owner can opt in through `foreground_fallback_enabled` plus ordered `foreground_model_fallbacks`; the retired `default_model_fallbacks` key is ignored. Eligible pre-content availability failures can advance through at most ten owner-visible exact model candidates, while missing configuration/endpoints, provider/schema errors, clean empty completions, and post-content failures remain on the selected route and surface an error. Once a route produces substantive text/reasoning or a tool call it is pinned as the answering route.
Fallback candidates receive route-neutral context shaping. Only compaction performed for the answering route is persisted. Chat and agent metadata record requested/actual model and endpoint identity, round-by-round route transitions, and costs against the route that actually answered; the browser renders same-model endpoint changes as well as model changes.
## Context Preface
`routes.chat_helpers.build_chat_context()` owns the shared route pipeline: preset extraction, preprocessing, user-message persistence, incognito/no-memory/RAG/skills flags, prefetched compare search, YouTube transcript context, research-spinoff grounding, model normalization, and compaction.
`src.chat_processor.ChatProcessor.build_context_preface()` owns source preface construction. It can add memory, RAG, web search, URL page content, and skills index context before the model call.
Chat preface enhances the model's context. It must not rewrite the user message or force literal-vs-fetch interpretation before the model sees the request. See [context-building.md](context-building.md).
Chat-owned external context must enter the model through `untrusted_context_message()` unless a different treatment is explicitly documented. This includes memory, RAG, web search, URL fetches, prefetched search context, YouTube transcripts, research injection, and manual context injection.
## Modes And Handoffs
Chat can dispatch to normal LLM calls, agent mode, research mode, or compare-related flows. Session mode is stored on `sessions.mode`.
Legacy plan-mode backend plumbing still exists below chat, but `routes/chat_routes.py` currently forces browser/form `plan_mode` input off and the old visible plan window frontend module is not part of the current SPA. Treat plan-mode changes as compatibility work unless the UI contract is intentionally reintroduced.
Current call sites include:
- chat/research dispatch in `routes/chat_routes.py`;
- agent execution in `src/agent_loop.py`;
- deep research orchestration in `src/research_handler.py`;
- compare entry points in canonical `routes/compare/compare_routes.py` and frontend compare modules.
Agent-mode tool access is gated in layers. Chat route toggles and privileges
build a disabled-tool set; incognito and compare mode remove persistence-heavy
or UI-breaking tools; `src.action_intents.message_needs_tools()` provides
conservative regex auto-escalation hints; `src.agent_loop`,
`src.tool_security`, `src.tool_execution`, and internal loopback validation
remain server-side enforcement owners.
`allow_bash` and `allow_web_search` can be read from the JSON request body for browser chat posts that do not submit traditional form fields.
Web search tools are per-turn explicit opt-in. Either `allow_web_search=true`
or `use_web=true` can enable `web_search`/`web_fetch`, but an explicit
`allow_web_search=false` wins over `use_web=true` and keeps those tools
disabled. Explicit latest-turn web-search intent can still auto-escalate into
agent mode and narrows the available tool set toward `web_search`/`web_fetch`,
but it no longer re-enables web tools after an explicit denial or global
disable.
Guide-only/no-tools requests build an effective tool policy before preprocessing and agent dispatch. That policy suppresses tool-backed preprocessing/background extraction/research, disables schemas and MCP for the turn, and is still enforced by `src.tool_execution` if a model emits a tool call anyway.
When route context is trimmed without full compaction, chat emits a
`context_trimmed` SSE event and carries before/after message/token counts into
metrics. Provider reasoning/thinking deltas are streamed for live UI handling
but kept out of the visible saved assistant content and stored in metadata when
available.
## Attachments
`src.chat_handler.ChatHandler.preprocess_message()` owns owner-scoped upload-id resolution, attachment metadata, YouTube transcript/comment preprocessing, image/VL behavior, and enhanced text used by chat. `src.document_processor.build_user_content()` owns conversion of uploaded/chat-attached files into model-ready text or multimodal blocks. `src.attachment_refs` owns persisted text/reference normalization, and `SessionManager` owner-reserves attachment ids before appending or replacing durable message rows. `static/js/fileHandler.js` owns frontend pending-file state.
Attachment-only sends are valid. Missing or unauthorized ids are skipped during preprocessing, while a missing/wrong-owner durable reference aborts a message/history replacement before existing transcript rows are removed. Upload failures keep pending files for retry, unsupported media can degrade to text markers, optional Office/PDF/VL dependencies can emit extraction banners, Office attachments can create markdown documents when extracted server-side, and fillable-PDF auto-document failures fall back to normal PDF extraction. `chat_messages.content` and FTS do not retain provider data URLs; structured references stay in metadata for reloads. Chat does not own upload bytes or durable document storage; it requests document/upload behavior from those subsystems.
Frontend chat distinguishes normal resend from regenerate-from-here: normal resend appends a fresh user copy and carries upload IDs where available, while regeneration truncates from the selected point. AI-message delete prompts before removing the AI response plus preceding user turn. Desktop Enter submits; mobile Enter inserts a newline unless another platform-specific send control is used.
Native document tool outputs can open or refresh the document editor from
tool-result metadata, so the UI can recover if a later `doc_update` stream event
is missed. The chat renderer also hides raw/incomplete leaked tool JSON and
document fences from normal transcript text.
When untrusted external/workspace content has entered the agent context, high-impact tool calls pause as exact approval cards instead of executing. The browser can allow the rest of the interrupted task, allow this chat session, or deny; it submits only the opaque id/decision with an empty control-plane message and does not mutate the composer. The server restores the sealed first action plus private selected tools/query, revalidates policy and document freshness, consumes the first action, and resumes without persisting a synthetic user message. Task scope ends with that resumed run. Chat scope persists the resolved card and marks later context only for that exact session; forks do not inherit it. A normal message retires an unresolved card while preserving taint.
## Security And Provenance
`/api/chat` and `/api/chat_stream` verify session ownership before loading the session. Chat privilege gates enforce allowed models and daily message caps before LLM work. Active document injection, session auth/header recovery, endpoint repair, upload-id resolution and reservation, memory/RAG retrieval, and post-response work must stay owner-scoped.
The scoped API-token chat surface is `/api/v1/chat`. Browser chat routes can receive bearer-auth state from middleware, but route code must not assume `"api"` is a durable owner; API-token support requires explicit scope checks and token-owner attribution.
Incognito disables memory, skill, and chat-history tools and skips assistant DB persistence, but current user-message persistence and later cleanup are not a strict no-write guarantee. Treat incognito changes as security-sensitive until that contract is clarified.
## Search Boundary
`GET /api/search` in `routes/chat_routes.py` is chat-message search for the UI and slash commands. Web search routes are owned by canonical `routes/search/search_routes.py`; chat and agent web context call through `src.search`, compatibility shims, and search content fetchers. Do not confuse chat-history search with external web retrieval.
## Degraded And Compatibility Behavior
- Missing ChromaDB, embeddings, memory vectors, RAG managers, or skills indexes should remove injected context or fall back to keyword/text behavior without failing chat.
- Direct URL prefetch failures become compact untrusted context stating that the page was not read, with only transport-owned HTTP/size/rate-limit status where recognized; raw URLs, exception text, and response-controlled diagnostics are not echoed into logs or model context.
- Sessions hydrate legacy string headers and multimodal JSON-array content, export text/HTML/Markdown after flattening non-string blocks, can lazy-load from DB when cached state is empty, and preserve old history/index delete behavior where needed.
- Initial shell/session loading is non-blocking: the sidebar can render before a selected transcript is hydrated, and full transcript hydration is deferred until display or a model send requires it.
- Chat repairs empty selected models and orphaned endpoint references before provider calls when possible.
- Deleted-session stream writes fail closed.
- Docker/native endpoint differences are owned by runtime/model setup, but chat sessions depend on the saved endpoint URLs and headers.
- Copying a response from the UI copies the displayed answer text and omits hidden reasoning/thinking segments.
## Current Gaps
- Chat, agent, research, and compare orchestration still meet in a large route file.
- Context preface behavior is spread across `routes/chat_helpers.py`, `src/chat_processor.py`, route injections, and agent/tool paths.
- Detached stream lifecycle spans `routes/chat_routes.py`, `src/agent_runs.py`, `static/js/chat.js`, `static/js/sessions.js`, and non-chat callers.
- Some frontend stream state is still global/module-level in `static/js/chat.js` and needs careful session isolation when adding background or resumable flows.
- Chat lacks route-level SSE regression tests for `/api/chat_stream`, live resume/stop/status, mode handoff, persistence metadata, partial-save behavior, attachment/doc-update events, browser timezone offset/workspace handling, and literal URL context intent.
- Bearer-token behavior on browser chat routes and incognito persistence need explicit contract decisions and regression coverage.
-79
View File
@@ -1,79 +0,0 @@
# Compare
Last updated: dev@e71f8ce | 2026-08-25
## Scope
This spec covers model A/B comparison behavior in:
- canonical `routes/compare/compare_routes.py`, with `routes/compare_routes.py` as a compatibility shim;
- `routes/session_routes.py`;
- `routes/chat_routes.py` and `routes/chat_helpers.py`;
- `routes/model_routes.py`;
- canonical `routes/search/search_routes.py`, with `routes/search_routes.py` as a compatibility shim;
- `core/database.py` model `Comparison`;
- `src/llm_core.py` and `src/endpoint_resolver.py`;
- frontend modules under `static/js/compare/`;
- `static/js/chat.js`, `static/js/sessions.js`, `static/js/models.js`, and `static/js/slashCommands.js`;
- `tests/test_compare_*` and focused blind-compare redaction tests.
## Runtime Behavior
The active text compare UI creates ordinary `[CMP]` sessions through `/api/session`, then streams each pane through `/api/chat_stream` with `compare_mode=true`. Search compare is a separate branch: it can query `/api/search/query` directly and its synthesis sessions use ordinary chat streaming without `compare_mode=true`. `static/js/compare/index.js` owns compare orchestration, session creation, execution order, search-mode branching, and export actions. `static/js/compare/panes.js` owns pane add/remove/swap/reroll lifecycle. `static/js/compare/stream.js` owns pane streaming and event rendering.
`routes/compare/compare_routes.py` owns the `/api/compare` HTTP surface for alternate/legacy start/vote/history/delete behavior and the active `/api/compare/record` vote-summary endpoint. The top-level module is a compatibility alias. Legacy `/api/compare/start` uses neutral helper-session names and withholds model identities/mapping from the start response while blind mode is active. It does not own provider-specific payload behavior.
Current call sites include:
- `/api/session` compare session creation and cleanup in compare frontend modules;
- `/api/chat_stream` pane execution through chat routes and detached stream infrastructure, streamed directly into panes so upstream generation stops promptly when panes are stopped;
- `/api/models` and probe routes for model/endpoint selection;
- search-provider compare mode through `routes/search/search_routes.py`;
- `/api/compare/record` as a fire-and-forget backend vote summary, while active scoreboard state is localStorage-backed.
`Comparison` rows currently persist vote/history metadata: prompt, first model identifiers, winner, blind flag, optional N-model JSON in `blind_mapping`, vote timestamp, and owner. Response and metric columns exist in the schema but are not populated by the active compare UI flow. Compare history must be owner-scoped.
Frontend compare behavior is split by responsibility:
- `state.js` owns local compare state;
- `selector.js`, `models.js`, and `probe.js` own endpoint/model selection and probe UI;
- `panes.js` and `stream.js` own paired response rendering;
- `vote.js` and `scoreboard.js` own voting and history display.
Compare panes can receive `ask_user` or tool-approval controls from the shared chat stream. `static/js/compare/stream.js` routes those controls into the main chat renderer/control plane, pauses pane completion/autograding while a choice is pending, and can resume the pane after the user decision; compare orchestration keeps its busy state until those continuations settle.
Mobile compare layout collapses multi-pane grids to a single column so panes
remain readable on narrow screens while the desktop grid still uses the
selected column count.
## Ownership Boundaries
Compare owns paired evaluation flow and pane state. Chat routes own the actual stream execution path for compare panes. LLM provider code owns model-call mechanics. Session/model routes own endpoint-id resolution, owner-filtered endpoint/model visibility, header copying, and deleted-endpoint failures.
`compare_mode` in chat strips compare-breaking tools, disables document tools for `[CMP]` sessions, skips some research clarification, and suppresses memory, skill, and webhook side effects after pane responses.
Compare frontend code is part of the app DOM security surface. Current stream/search rendering sanitizes probe labels and tool labels, constrains search-result links to HTTP(S), uses safe generated-image display sources, and opens compare export/image popups with opener isolation.
## Policy Notes
- Current blind compare is UI/API masking until vote/reveal, not a full confidentiality boundary. `[CMP]` session names and session-list model fields are redacted for helper sessions, and legacy `/api/compare/start` withholds model identity/mapping while blind. Client-side selected model state and privileged/local inspection can still expose identity.
- Compare endpoint lists and secondary endpoint lookups use owner filtering so users see and resolve only shared or owned endpoints.
- Non-admin compare session creation must use registered owner-visible endpoints; compare must not allow arbitrary raw endpoint URLs to bypass session-route endpoint policy.
- Prefetched search, URL, RAG, and research context entering compare panes must use the untrusted-context wrapper.
- Compare panes use chat's foreground routing contract: selected routes are strict unless that owner explicitly enabled ordered foreground fallbacks. Verify each pane still reaches its intended route and that any opt-in route transition or error is visible.
## Degraded And Compatibility Behavior
- Missing/offline endpoints are surfaced by model/session routes; chat can clear orphaned endpoint references and recover empty models when possible.
- Compare streams inherit chat's opt-in, eligible-pre-output-only foreground fallback and provider-normalized SSE events, but compare frontend handling for errors and model/endpoint route transitions is thinner than chat's stream path.
- Shared legacy `ModelEndpoint.owner == NULL` rows remain visible through owner filters. Legacy `Comparison.owner == NULL` rows are not treated as shared for authenticated vote/delete/history flows.
- `/api/compare/start` and `/{comp_id}/vote` remain implemented but are not the active frontend path.
## Current Gaps
- Blind mode is not a confidentiality boundary; client/local state can still expose model identity before vote.
- `/api/compare/start` accepts raw endpoint URLs and can diverge from `/api/session` endpoint-owner/raw-endpoint policy.
- `src/agent_loop.py` advertises stale compare app API endpoints.
- Compare streaming and chat streaming are separate frontend paths but share model/provider infrastructure; regressions can happen when provider event shape changes.
- Compare frontend needs explicit fallback/error event handling parity with chat streaming.
- Compare tests cover endpoint owner helper behavior, blind compare redaction, ask-user/tool-approval routing, and portable JS helpers, but not full active `/api/session` pane creation, frontend pane lifecycle, or complete SSE fallback/error handling.

Some files were not shown because too many files have changed in this diff Show More