perf: batch page embeddings via /api/embed and upsert before pruning stale points
- embed_batch now issues one batched /api/embed request (the old loop made one /api/embeddings round-trip per chunk) with a per-text fallback that preserves None-for-failed semantics - update_from_page embeds all chunks in that single call and stores them in one Qdrant batch upsert (upsert_points) - reindex order reversed: upsert new points first, then prune stale ids (deterministic uuid5 ids make overwrite safe) so a mid-way failure no longer leaves the page with zero vectors - VectorUpdateSummary gains status (success/partial/failed) and chunks_skipped; all-embeddings-failed keeps old vectors and reports failure instead of success=True Measured on a 7-chunk page ingest (local server, llm_tester): ~375ms -> ~181ms median over 3 runs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QbFZyDvYksazX6nYQYZ67L
This commit is contained in:
@@ -67,6 +67,8 @@ class VectorUpdateSummary(BaseModel):
|
||||
chunks_created: int = Field(default=0, description="New chunks created")
|
||||
chunks_updated: int = Field(default=0, description="Existing chunks updated")
|
||||
chunks_deleted: int = Field(default=0, description="Old chunks deleted")
|
||||
chunks_skipped: int = Field(default=0, description="Chunks skipped (embedding failed)")
|
||||
status: str = Field(default="success", description="'success', 'partial' (some chunks skipped), or 'failed'")
|
||||
total_chunks: int = Field(default=0, description="Total chunks for this page")
|
||||
embedding_dim: int = Field(default=768, description="Embedding dimensionality")
|
||||
processing_time_ms: float = Field(..., description="Processing time in milliseconds")
|
||||
|
||||
Reference in New Issue
Block a user