feat: add job cleanup loop, Scheduler task definitions, and registrar
- job_cleanup_loop (src/jobs/job_manager.py): hourly in-process pass over
JobManager.cleanup_expired_jobs, started at app startup and cancelled
at shutdown; Redis job payloads auto-expire but set memberships do not.
- docs/scheduler-tasks.md: the four production Scheduler task payloads
for the deploy checklist - nightly integrity check 04:30, weekly
quality report Sunday 03:00 (day_of_week=6, 0=Monday), daily Paperless
orphan-cleanup 05:00 hitting the existing
/maintenance/cleanup/paperless?user=jpmschweitzer&dry_run=false
endpoint, and disabling test_example_task - with exact HTTP bodies
(explicit user=jpmschweitzer, Authorization: Bearer ${LIBRARY_API_KEY}
placeholder).
- scripts/register_scheduler_tasks.py: reads SCHEDULER_URL from env,
DRY-RUN BY DEFAULT (prints the exact payloads, provably contacts
nothing), --execute gated and requiring LIBRARY_API_KEY to fill the
placeholder. NOT executed - definitions delivered for the deploy
checklist only.
9 new offline tests (loop passes/error-resilience/cancellation, payload
schedules, explicit production user, placeholder, dry-run default).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbFZyDvYksazX6nYQYZ67L
This commit is contained in:
@@ -0,0 +1,159 @@
|
||||
# Scheduler Task Definitions (Phase C deploy checklist)
|
||||
|
||||
Production task payloads for the homelab's database-driven **Scheduler**
|
||||
service. These are **definitions only** — nothing in this repo registers
|
||||
them automatically. Register them as part of the deploy checklist, either
|
||||
via the Scheduler UI/API or with the helper script:
|
||||
|
||||
```bash
|
||||
# Preview exactly what would be sent (default):
|
||||
SCHEDULER_URL=http://<scheduler-host>:8090 \
|
||||
.venv/bin/python scripts/register_scheduler_tasks.py
|
||||
|
||||
# Actually register/update the tasks (deploy checklist step):
|
||||
SCHEDULER_URL=http://<scheduler-host>:8090 \
|
||||
LIBRARY_API_KEY=<the library-desk API key> \
|
||||
.venv/bin/python scripts/register_scheduler_tasks.py --execute
|
||||
```
|
||||
|
||||
Conventions:
|
||||
|
||||
- All tasks call the **production** library-desk container
|
||||
(`http://library-desk:8089`) with the explicit production tenant
|
||||
`user=jpmschweitzer` (there is no default tenant — Phase B).
|
||||
- `${LIBRARY_API_KEY}` is a placeholder for the library-desk API key
|
||||
(`LIBRARY_API_KEY` in the container env). Never commit the real value.
|
||||
- Schedule fields use the Scheduler's convention: `-1` = every,
|
||||
`day_of_week`: `0` = Monday … `6` = Sunday.
|
||||
|
||||
---
|
||||
|
||||
## 1. Nightly integrity check — 04:30 daily
|
||||
|
||||
Read-only report: pages without vectors, orphaned vectors, unexpected
|
||||
Qdrant collections, Document nodes without wiki pages. Caches its result
|
||||
in Redis for the weekly quality report.
|
||||
|
||||
```json
|
||||
{
|
||||
"task_name": "library_integrity_check",
|
||||
"service": "library-desk",
|
||||
"executor": "rest_api_executor",
|
||||
"priority": 60,
|
||||
"description": "Nightly read-only integrity check for the library (vectors/graph/wiki/collections)",
|
||||
"enabled": true,
|
||||
"max_retries": 2,
|
||||
"timeout_seconds": 900,
|
||||
"minute": 30,
|
||||
"hour": 4,
|
||||
"day_of_month": -1,
|
||||
"month": -1,
|
||||
"day_of_week": -1,
|
||||
"config": {
|
||||
"method": "POST",
|
||||
"url": "http://library-desk:8089/maintenance/integrity-check",
|
||||
"headers": {
|
||||
"Content-Type": "application/json",
|
||||
"Authorization": "Bearer ${LIBRARY_API_KEY}"
|
||||
},
|
||||
"body": {
|
||||
"user": "jpmschweitzer"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## 2. Weekly quality report — Sunday 03:00
|
||||
|
||||
Runs the duplicate scan, flags stale/metadata-poor pages, folds in the
|
||||
latest integrity results, and writes the dated report page to
|
||||
`users/jpmschweitzer/system/quality-reports/YYYY-MM-DD`.
|
||||
|
||||
```json
|
||||
{
|
||||
"task_name": "library_quality_report",
|
||||
"service": "library-desk",
|
||||
"executor": "rest_api_executor",
|
||||
"priority": 60,
|
||||
"description": "Weekly library quality report (dedup, stale pages, missing metadata, integrity) written to the wiki",
|
||||
"enabled": true,
|
||||
"max_retries": 2,
|
||||
"timeout_seconds": 1800,
|
||||
"minute": 0,
|
||||
"hour": 3,
|
||||
"day_of_month": -1,
|
||||
"month": -1,
|
||||
"day_of_week": 6,
|
||||
"config": {
|
||||
"method": "POST",
|
||||
"url": "http://library-desk:8089/maintenance/quality-report",
|
||||
"headers": {
|
||||
"Content-Type": "application/json",
|
||||
"Authorization": "Bearer ${LIBRARY_API_KEY}"
|
||||
},
|
||||
"body": {
|
||||
"user": "jpmschweitzer",
|
||||
"stale_days": 30,
|
||||
"dedup_threshold": 0.9,
|
||||
"write_page": true
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## 3. Daily Paperless orphan cleanup — 05:00
|
||||
|
||||
Hits the **existing** cleanup endpoint (query parameters, empty body).
|
||||
`dry_run=false` deletes vectors/graph nodes for documents that were
|
||||
removed from Paperless-ngx.
|
||||
|
||||
```json
|
||||
{
|
||||
"task_name": "library_paperless_orphan_cleanup",
|
||||
"service": "library-desk",
|
||||
"executor": "rest_api_executor",
|
||||
"priority": 60,
|
||||
"description": "Daily cleanup of vectors/graph nodes for documents deleted from Paperless-ngx",
|
||||
"enabled": true,
|
||||
"max_retries": 2,
|
||||
"timeout_seconds": 900,
|
||||
"minute": 0,
|
||||
"hour": 5,
|
||||
"day_of_month": -1,
|
||||
"month": -1,
|
||||
"day_of_week": -1,
|
||||
"config": {
|
||||
"method": "POST",
|
||||
"url": "http://library-desk:8089/maintenance/cleanup/paperless?user=jpmschweitzer&dry_run=false",
|
||||
"headers": {
|
||||
"Authorization": "Bearer ${LIBRARY_API_KEY}"
|
||||
},
|
||||
"body": {}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## 4. Disable `test_example_task`
|
||||
|
||||
Not a new task: the leftover example task must be **disabled** (not
|
||||
deleted, so its history is preserved).
|
||||
|
||||
```
|
||||
PUT ${SCHEDULER_URL}/tasks/test_example_task
|
||||
Content-Type: application/json
|
||||
|
||||
{"enabled": false}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Related (already registered / in-process)
|
||||
|
||||
- `knowledge_consolidation` — every 30 minutes, POST
|
||||
`/consolidate/knowledge` (already registered; after the Phase C
|
||||
consolidation repair its runs log `searches_processed` and
|
||||
`duration_ms`, and searches are no longer consumed while the LLM is
|
||||
unavailable).
|
||||
- Redis job-set cleanup — runs **in-process** inside library-desk
|
||||
(hourly `job_cleanup_loop` started at app startup); no Scheduler task
|
||||
needed.
|
||||
Reference in New Issue
Block a user