perf: batch page embeddings via /api/embed and upsert before pruning stale points

- embed_batch now issues one batched /api/embed request (the old loop
  made one /api/embeddings round-trip per chunk) with a per-text
  fallback that preserves None-for-failed semantics
- update_from_page embeds all chunks in that single call and stores
  them in one Qdrant batch upsert (upsert_points)
- reindex order reversed: upsert new points first, then prune stale ids
  (deterministic uuid5 ids make overwrite safe) so a mid-way failure no
  longer leaves the page with zero vectors
- VectorUpdateSummary gains status (success/partial/failed) and
  chunks_skipped; all-embeddings-failed keeps old vectors and reports
  failure instead of success=True

Measured on a 7-chunk page ingest (local server, llm_tester):
~375ms -> ~181ms median over 3 runs.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbFZyDvYksazX6nYQYZ67L
This commit is contained in:
2026-07-14 13:09:37 +02:00
co-authored by Claude Fable 5
parent c35d3c1fa9
commit 8348b4bf92
6 changed files with 246 additions and 38 deletions
+2
View File
@@ -67,6 +67,8 @@ class VectorUpdateSummary(BaseModel):
chunks_created: int = Field(default=0, description="New chunks created")
chunks_updated: int = Field(default=0, description="Existing chunks updated")
chunks_deleted: int = Field(default=0, description="Old chunks deleted")
chunks_skipped: int = Field(default=0, description="Chunks skipped (embedding failed)")
status: str = Field(default="success", description="'success', 'partial' (some chunks skipped), or 'failed'")
total_chunks: int = Field(default=0, description="Total chunks for this page")
embedding_dim: int = Field(default=768, description="Embedding dimensionality")
processing_time_ms: float = Field(..., description="Processing time in milliseconds")