feat(ui): client disk cache — FileAccess tiers beneath the LRU (D-255, T-1183) #204

Closed

jpmschweitzer wants to merge 0 commits from t1183-client-cache into main

pull from: t1183-client-cache

merge into: :main

:main

Conversation 2 Commits - Files Changed -

jpmschweitzer commented 2026-07-25 11:11:43 +02:00

Owner

Copy Link

Copy Source

Summary

The final D-255 chain ticket: the Tier 2/3 disk-backed cache store per D-255(d) and the round-2 three-tier spec. step_canvas_disk_cache.gd stores the WIRE payload pre-decode (store_var/get_var round-trips the PNG PackedByteArrays natively — Image decode never runs in this file), keyed by the same composite key as Tier 1, with a per-body index.json and malformed-index recovery (rebuild-or-discard, never crash).

Eviction — three independent mechanisms, exactly as ruled

• Rung-0 Global: retention floor, no sweep ever (also threaded into Tier 1 per the T-1182 handoff).
• Tier 2 geometry: byte-valid forever — evicted only by time-since-last-visit (14d starting tunable, on body-open) and LRU byte budget (256 MiB/body, 5-min coarse timer), two separate sweeps per the ruling.
• Tier 3 sim-state TTL: wired and tested; no production caller yet (no sim-state field exists on EncodedStepCanvas — the documented D-253 stub inheritance).

Hardening per D-255(d), both mandatory

• Per-body deep-rung cap: 512 Block+Chunk entries, enforced synchronously in put(), independent of the floor and byte budget. The ticket sanctions count-or-quota; count chosen as the direct D-226(d) information-content proxy.
• Schema/version tag on every entry: project.yaml version (the D-192 co-ship boundary makes the client version the wire-schema version); exact-inequality mismatch = miss + drop, NEVER decode — checked in both has() and get_canvas().

Verification

30 new disk-cache tests + 7 request-integration + 3 sweep-wiring (restart persistence incl. floor flag, sweep independence both directions, cap semantics, version-mismatch never-decode asserted at the call level, corrupt/truncated/dangling-file recovery). Full client suite 3,440/3,440, 0 orphans, through the gate; cold-parse clean. A live restart-persistence eyeball runs next — merge waits on it.

What changes for the player

Revisiting a body across restarts is instant at every previously-visited rung; abandoned bodies quietly reclaim fine-grained space after ~2 weeks; the whole-body view of every visited body stays forever. Update-safe by construction.

🤖 Generated with Claude Code

## Summary The final D-255 chain ticket: the Tier 2/3 disk-backed cache store per D-255(d) and the round-2 three-tier spec. step_canvas_disk_cache.gd stores the WIRE payload pre-decode (store_var/get_var round-trips the PNG PackedByteArrays natively — Image decode never runs in this file), keyed by the same composite key as Tier 1, with a per-body index.json and malformed-index recovery (rebuild-or-discard, never crash). ## Eviction — three independent mechanisms, exactly as ruled - Rung-0 Global: retention floor, no sweep ever (also threaded into Tier 1 per the T-1182 handoff). - Tier 2 geometry: byte-valid forever — evicted only by time-since-last-visit (14d starting tunable, on body-open) and LRU byte budget (256 MiB/body, 5-min coarse timer), two separate sweeps per the ruling. - Tier 3 sim-state TTL: wired and tested; no production caller yet (no sim-state field exists on EncodedStepCanvas — the documented D-253 stub inheritance). ## Hardening per D-255(d), both mandatory - Per-body deep-rung cap: 512 Block+Chunk entries, enforced synchronously in put(), independent of the floor and byte budget. The ticket sanctions count-or-quota; count chosen as the direct D-226(d) information-content proxy. - Schema/version tag on every entry: project.yaml version (the D-192 co-ship boundary makes the client version the wire-schema version); exact-inequality mismatch = miss + drop, NEVER decode — checked in both has() and get_canvas(). ## Verification 30 new disk-cache tests + 7 request-integration + 3 sweep-wiring (restart persistence incl. floor flag, sweep independence both directions, cap semantics, version-mismatch never-decode asserted at the call level, corrupt/truncated/dangling-file recovery). Full client suite 3,440/3,440, 0 orphans, through the gate; cold-parse clean. A live restart-persistence eyeball runs next — merge waits on it. ## What changes for the player Revisiting a body across restarts is instant at every previously-visited rung; abandoned bodies quietly reclaim fine-grained space after ~2 weeks; the whole-body view of every visited body stays forever. Update-safe by construction. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

jpmschweitzer added 2 commits 2026-07-25 11:11:43 +02:00

feat(ui): client disk cache — FileAccess tiers beneath the LRU (D-255, T-1183) ... 6b76e365bb

step_canvas_disk_cache.gd: the Tier 2/3 store per D-255(d) and the
round-2 three-tier spec. Payload is the WIRE, pre-decode — store_var/
get_var round-trips the PNG-encoded PackedByteArrays natively, and
Image.load_png_from_buffer never runs in this file. One composite key
shared verbatim with Tier 1 (hash filenames for filesystem safety);
per-body index.json with malformed-index recovery (rebuild-or-discard,
never crash).

Three independent eviction mechanisms, exactly as ruled: rung-0 Global
carries a retention floor no sweep touches (now also threaded into
Tier 1 per the T-1182 handoff); Tier 2 geometry is byte-valid forever
and evicts only by time-since-last-visit (14d starting tunable, on
body-open) and LRU byte budget (256 MiB/body, 5-min coarse timer) —
two separate sweeps; Tier 3 sim-state TTL is wired and tested but has
no production caller yet (no sim-state field exists on
EncodedStepCanvas — the D-253 stub inheritance, documented).

Hardening per D-255(d), both mandatory: per-body deep-rung cap
(512 Block+Chunk entries, enforced synchronously in put(), floor- and
budget-independent — the ticket sanctions count-or-quota; count chosen
as the direct D-226(d) information-content proxy) and a schema/version
tag on every entry (project.yaml version via the existing
loading_screen line-scan idiom — D-192 co-ship makes the client
version the wire-schema version; exact-inequality mismatch = miss +
drop, NEVER decode, checked in both has() and get_canvas()).

Integration: request_now() checks Tier 2 on a Tier-1 miss (synchronous
promote), Ready responses write through to both tiers, Pending never
writes; viewer runs the visit sweep on body-open + the background
sweep on a 5-min timer.

30 new disk-cache tests + 7 request-integration + 3 sweep-wiring tests
(restart persistence, sweep independence both directions, cap
semantics, version-mismatch never-decode, corrupt-index recovery).
Full client suite 3,440/3,440, 0 orphans; cold-parse clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

chore(meta): update changelog ... 6702e6c725

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

jpmschweitzer referenced this issue from a commit 2026-07-25 12:11:48 +02:00

fix(ui): harden step-canvas disk cache — PR #204 review round (T-1183)

jpmschweitzer added 1 commit 2026-07-25 12:11:48 +02:00

fix(ui): harden step-canvas disk cache — PR #204 review round (T-1183) ... 51fe35d00d

Atomic index writes (tmp+rename), orphan-payload reconciliation sweep
folded into the background sweep, payload shape guard mirroring the
protocol's own width/height discriminator, size_bytes via get_position()
instead of a full payload re-read, and a 64-bit SHA-256 payload filename
(String.hash()'s 31-bit space made a silent wrong-map filename collision
a ~1-in-16k event per cap-full body; migration self-heals via the orphan
sweep). Tier-3 class doc rewritten to name the real D-253 seam — the
glaciation/flooded_q wire fields exist but carry static values and no
staleness signal crosses the wire; T-1190 tracks threading a wire TTL
into put(sim_ttl_sec) when the driving clock lands — replacing the false
'no sim-state wire field exists' premise. Four new tests plus a
_stub_canvas fixture helper centralizing the canvas shape contract.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

jpmschweitzer commented 2026-07-25 12:12:17 +02:00

Author

Owner

Copy Link

Copy Source

Review: t1183-client-cache -> main (type: code)

Reviewers: Hoshe (code quality) + Tyre (architecture), per the code-branch matrix.
Both returned REQUEST_CHANGES; every finding was verified against the branch source and either fixed or retracted with evidence. Fix commit: 51fe35d00.

Hoshe (code quality): REQUEST_CHANGES

#	File	Issue	Resolution
1	step_canvas_disk_cache.gd:258 _save_index	Truncate-in-place index write — crash mid-write corrupts index.json; recovery discards all entries, orphaning payloads	Fixed — atomic tmp+DirAccess.rename_absolute swap; failure path warns and removes the tmp (51fe35d00)
2	step_canvas_disk_cache.gd:345 put()	Payload-before-index write order + no directory reconciliation — orphaned .dat files leak forever, invisible to the byte-budget sweep	Fixed — new run_orphan_sweep() (index-referenced-basename set vs one directory listing) folded into run_background_sweep(); write order kept and documented as deliberate (orphan-payload is the recoverable direction); test added (51fe35d00)
3	step_canvas_disk_cache.gd:298 get_canvas()	Corrupted payload "can crash or hang the client" on get_var()	Partially retracted — get_var() returns null on a failed decode and the existing not canvas is Dictionary check catches it, with _drop_entry() deleting both index row and file: full self-heal, no crash path; allocation is bounded by file length. Residual fixed: guard strengthened to require width/height (the protocol's own discriminator), contract documented, garbage-bytes test added (51fe35d00)
4	step_canvas_disk_cache.gd:380 put()	get_file_as_bytes().size() re-reads the whole just-written payload for its byte count	Fixed — file.get_position() on the still-open handle; accuracy test added (51fe35d00)

Tyre (architecture): REQUEST_CHANGES

#	File	Issue	Resolution
1	step_canvas_disk_cache.gd:43 Tier-3 doc + request.gd:161	Sim-state fields (glaciation/flooded_q) cached as never-stale geometry; doc claims "no sim-state WIRE field exists" citing round-1 — a live correctness gap	Partially retracted, doc fixed — the "live gap" scenario cannot occur today: the server hardwires flooded_q to 0 and ships glaciation as the static classification, byte-identical to an indefinitely-fresh derive with the TTL machinery explicitly wired ahead of D-253's driving clock (server step_canvas.rs module doc), and no staleness signal crosses the wire. TIER_GEOMETRY is therefore the correct classification today. Tyre's core complaint stands: the doc's premise was false and its citation superseded — rewritten to name the real seam, one-line note at the put() call site, and T-1190 filed to thread a wire TTL into the existing put(sim_ttl_sec) parameter when the D-253 clock lands (51fe35d00)
2	step_canvas_disk_cache.gd:380	Same double-read as Hoshe #4	Fixed (same change)

Lead addition (beyond reviewer findings)

_payload_filename() used String.hash() & 0x7FFFFFFF — a 31-bit space in which two live keys colliding on a filename silently serve the wrong map as a valid hit (~1-in-16k per cap-full body). Replaced with 16-hex SHA-256 (64-bit space, negligible); the scheme change self-heals — old rows miss and drop, old files are orphan-swept. Included in 51fe35d00.

Verification

• test_step_canvas_disk_cache 68/68 (64 existing — all kept green through the _stub_canvas fixture conversion — + 4 new), test_step_canvas_request 32/32, test_step_canvas_viewer 50/50; gdlint clean on both edited sources. Full suite runs at the push gate.
• Restart-persistence eyeball (the merge gate from the PR description): see the follow-up comment with evidence.

Verdict: CHANGES REQUESTED -> addressed in 51fe35d00; merge follows the eyeball evidence.

# Review: t1183-client-cache -> main (type: code) Reviewers: Hoshe (code quality) + Tyre (architecture), per the code-branch matrix. Both returned REQUEST_CHANGES; every finding was verified against the branch source and either fixed or retracted with evidence. Fix commit: `51fe35d00`. ### Hoshe (code quality): REQUEST_CHANGES | # | File | Issue | Resolution | |---|------|-------|------------| | 1 | step_canvas_disk_cache.gd:258 `_save_index` | Truncate-in-place index write — crash mid-write corrupts index.json; recovery discards all entries, orphaning payloads | **Fixed** — atomic tmp+`DirAccess.rename_absolute` swap; failure path warns and removes the tmp (`51fe35d00`) | | 2 | step_canvas_disk_cache.gd:345 `put()` | Payload-before-index write order + no directory reconciliation — orphaned `.dat` files leak forever, invisible to the byte-budget sweep | **Fixed** — new `run_orphan_sweep()` (index-referenced-basename set vs one directory listing) folded into `run_background_sweep()`; write order kept and documented as deliberate (orphan-payload is the recoverable direction); test added (`51fe35d00`) | | 3 | step_canvas_disk_cache.gd:298 `get_canvas()` | Corrupted payload "can crash or hang the client" on `get_var()` | **Partially retracted** — `get_var()` returns null on a failed decode and the existing `not canvas is Dictionary` check catches it, with `_drop_entry()` deleting both index row and file: full self-heal, no crash path; allocation is bounded by file length. Residual fixed: guard strengthened to require `width`/`height` (the protocol's own discriminator), contract documented, garbage-bytes test added (`51fe35d00`) | | 4 | step_canvas_disk_cache.gd:380 `put()` | `get_file_as_bytes().size()` re-reads the whole just-written payload for its byte count | **Fixed** — `file.get_position()` on the still-open handle; accuracy test added (`51fe35d00`) | ### Tyre (architecture): REQUEST_CHANGES | # | File | Issue | Resolution | |---|------|-------|------------| | 1 | step_canvas_disk_cache.gd:43 Tier-3 doc + request.gd:161 | Sim-state fields (`glaciation`/`flooded_q`) cached as never-stale geometry; doc claims "no sim-state WIRE field exists" citing round-1 — a live correctness gap | **Partially retracted, doc fixed** — the "live gap" scenario cannot occur today: the server hardwires `flooded_q` to 0 and ships `glaciation` as the static classification, byte-identical to an indefinitely-fresh derive with the TTL machinery explicitly wired ahead of D-253's driving clock (server step_canvas.rs module doc), and no staleness signal crosses the wire. TIER_GEOMETRY is therefore the *correct* classification today. Tyre's core complaint stands: the doc's premise was false and its citation superseded — rewritten to name the real seam, one-line note at the `put()` call site, and **T-1190** filed to thread a wire TTL into the existing `put(sim_ttl_sec)` parameter when the D-253 clock lands (`51fe35d00`) | | 2 | step_canvas_disk_cache.gd:380 | Same double-read as Hoshe #4 | **Fixed** (same change) | ### Lead addition (beyond reviewer findings) `_payload_filename()` used `String.hash() & 0x7FFFFFFF` — a 31-bit space in which two live keys colliding on a filename silently serve the wrong map as a valid hit (~1-in-16k per cap-full body). Replaced with 16-hex SHA-256 (64-bit space, negligible); the scheme change self-heals — old rows miss and drop, old files are orphan-swept. Included in `51fe35d00`. ### Verification - `test_step_canvas_disk_cache` 68/68 (64 existing — all kept green through the `_stub_canvas` fixture conversion — + 4 new), `test_step_canvas_request` 32/32, `test_step_canvas_viewer` 50/50; gdlint clean on both edited sources. Full suite runs at the push gate. - Restart-persistence eyeball (the merge gate from the PR description): see the follow-up comment with evidence. ### Verdict: CHANGES REQUESTED -> addressed in `51fe35d00`; merge follows the eyeball evidence.

jpmschweitzer commented 2026-07-25 12:12:21 +02:00

Author

Owner

Copy Link

Copy Source

Restart-persistence eyeball — PASS (the merge gate from the PR description)

Live protocol on the fixed tip 51fe35d00: cold client+server run visits Global/Region/District on GJ1c, clean quit, then a fresh client process + fresh server revisits the same rungs at the same centers against the untouched cache.

Evidence chain:

• Cold write: 3 disk entries (disk_entries 1→2→3), 16-hex sha256 payload filenames, schema_version "0.4.0" matching project.yaml exactly, Global retention_floor=true (Region/District false), 36,936,864 bytes total.
• Restart serve-from-disk: the fresh process reports disk_entries(GJ1c)=3 at its first Global capture — before any wire round-trip was possible — and writes zero new entries across all three rungs. Payload files byte-untouched (same mtimes); only last_read_at bumped per entry: the read-touch signature of genuine Tier-2 hits.
• Pixel identity: AE=0 (zero differing pixels) on all three rungs, run1 vs run2, via ImageMagick compare -metric AE and independently confirmed by identical SHA-256 per rung. Cross-process, cross-server, texel-exact.

Two anomalies surfaced and root-caused — neither a product bug: intermittent view-transform drift and a nav-stack pop were both traced (quantitatively: observed drift matched PAN_SPEED_PX_S × frame_time × held-frames to 0.02px) to real desktop input reaching the unfocused test window on this shared-desktop session. Neutralized in the harness with a verified input-suppression node (events intercepted AND zero resulting drift — positive proof, not absence-of-interference). One genuine low-priority product nit fell out: edge-scroll pans on mouse motion while the window is unfocused → T-1191. Harness techniques inventoried on T-1157.

Fidelity label: navigation was driven in-process via StepCanvasViewer's own methods (the same entry points its gdUnit suite exercises) rather than OS pointer injection, which this session's Wayland compositor blocks. Full production scene, real live server, real GPU rendering/capture — only the physical-pointer path was substituted, and that path is unchanged by this PR.

Gate status: pre-push suite green on the branch (3,448/3,448). Proceeding to merge.

# Restart-persistence eyeball — PASS (the merge gate from the PR description) Live protocol on the fixed tip `51fe35d00`: cold client+server run visits Global/Region/District on GJ1c, clean quit, then a **fresh client process + fresh server** revisits the same rungs at the same centers against the untouched cache. **Evidence chain:** - **Cold write:** 3 disk entries (`disk_entries` 1→2→3), 16-hex sha256 payload filenames, `schema_version "0.4.0"` matching project.yaml exactly, Global `retention_floor=true` (Region/District false), 36,936,864 bytes total. - **Restart serve-from-disk:** the fresh process reports `disk_entries(GJ1c)=3` at its *first* Global capture — before any wire round-trip was possible — and writes zero new entries across all three rungs. Payload files byte-untouched (same mtimes); only `last_read_at` bumped per entry: the read-touch signature of genuine Tier-2 hits. - **Pixel identity:** AE=0 (zero differing pixels) on all three rungs, run1 vs run2, via ImageMagick `compare -metric AE` and independently confirmed by identical SHA-256 per rung. Cross-process, cross-server, texel-exact. **Two anomalies surfaced and root-caused — neither a product bug:** intermittent view-transform drift and a nav-stack pop were both traced (quantitatively: observed drift matched `PAN_SPEED_PX_S × frame_time × held-frames` to 0.02px) to **real desktop input reaching the unfocused test window** on this shared-desktop session. Neutralized in the harness with a verified input-suppression node (events intercepted AND zero resulting drift — positive proof, not absence-of-interference). One genuine low-priority product nit fell out: edge-scroll pans on mouse motion while the window is unfocused → **T-1191**. Harness techniques inventoried on **T-1157**. **Fidelity label:** navigation was driven in-process via StepCanvasViewer's own methods (the same entry points its gdUnit suite exercises) rather than OS pointer injection, which this session's Wayland compositor blocks. Full production scene, real live server, real GPU rendering/capture — only the physical-pointer path was substituted, and that path is unchanged by this PR. Gate status: pre-push suite green on the branch (3,448/3,448). Proceeding to merge.

jpmschweitzer referenced this issue from a commit 2026-07-25 12:15:27 +02:00

chore(meta): pql changelog — T-1190 filed, T-1183 to review

jpmschweitzer closed this pull request 2026-07-25 12:17:42 +02:00

Pull request closed

This pull request cannot be reopened because the branch was deleted.

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

jpmschweitzer (Jeroen Schweitzer)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: jpmschweitzer/settled-reach#204