fix(client): cold-launch starvation — Pending never reached the retry path (T-1163) #193

2026-07-22T21:02:21+02:00

jpmschweitzer commented

2026-07-22 21:02:21 +02:00

Summary

T-1163 (critical): every cold first launch of the Atlas starved — a black map on entering any body. This PR is the complete fix chain from Jeroen's live report through five verification-driven diagnostic rounds to a passing cold acceptance run on the exact launch shape.

The root cause (after four disproven theories)

A stone-cold body answers the first window request with whole-response status: "Pending" (only the whole-body cache-hit branch sets Ready — layer_proxy.rs), and on_response()'s first check — status != "Ready" → return — swallowed it before the retry machinery could run. Retries frozen at 0 forever; both tile-mosaic and single-window first descents affected. Second connections worked by luck (the first request warms the whole-body cache), which also masked the bug in every warm-server verification drive ever run.

The four dead theories, each killed by verification rather than shipped: an unpaired redraw (lead's misread grep — the pairing existed), a ConnectionId(0) sentinel (Dudley's exhaustive audit exonerated the server and left a permanent bridge_tcp pin), the 10s retry ceiling (Stig's own replacement theory — killed by the live retries=0 numbers), and pending-attribution in the fan-out (close, but the responses never reached attribution at all).

The fixes

Status-first branching in on_response() per the atlas_generation_proxy reference shape: Pending → retry; Ready → existing null-window retry; NotFound/Error → immediate give-up (the principled policy, replacing the elapsed-retries ceiling).
Deterministic backoff + per-tile stagger: 0.5 s doubling to a 4 s cap, +0.1 s × tile index — six cold tiles retry as a spread trickle, strictly-increasing delays asserted, no lockstep pulses. MAX_RETRIES 20→30 (~110 s horizon under backoff).
Self-healing tile repaint (_process redraws viewer+overlay while tiles are pending) — closes the arrival-repaint class regardless of which signal edge drops.
Re-entry root cause: ImplantApp._on_screen_changed re-runs enter() on repeat same-screen pushes — same-body guard in RegionalScreen.enter() + immediate-free ImplantPanel.clear() (protects every implant app from same-frame stacking).
Legend re-fit (deferred reset_to_content_size() — manually-positioned Controls never shrink on their own; RichTextLabel fit_content reports degenerate minimums until laid out once — both found empirically after two plausible fixes were disproven by trace).
Visible "DERIVING TERRAIN…" state while zero tiles have arrived — a cold wait reads as loading, not broken.
make atlas moves to a release server (build-server-release + SR_SERVER_BIN override, SR_PORT-precedent shape): a cold debug server needs >10 s before a new body's first tile exists; release serves cold in well under a second.
Dudley's connection-zero regression module (bridge_tcp.rs, +497 lines): the first-ever connection id a fresh server assigns receives window responses — the server half's permanent pin, replicating the real main.rs bootstrap.

Test discipline

Wire-accurate regressions through the real fan-out (tile_set._on_atlas_layers_received, never tile.on_response() directly) with pendings matching the server's actual response construction (body_id only) — the exact shape whose absence let this ship. Single-window mirrors per the shared-class reach. Two first-draft tests that passed with the bug reverted were caught and strengthened before reporting; every fix revert-verified independently.

Cold acceptance (the merge gate, passed)

Fresh release server, first connection, Lendel entry: DERIVING TERRAIN at 2 s → terrain painting tile-by-tile at 5 s → all six tiles converged, 6 pending + 6 ready on the wire, legend exactly one legend tall. Captures in .cache/screenshots/cold-start-verify/.

Tickets: T-1163

## Summary T-1163 (critical): every cold first launch of the Atlas starved — a black map on entering any body. This PR is the complete fix chain from Jeroen's live report through five verification-driven diagnostic rounds to a passing cold acceptance run on the exact launch shape. ### The root cause (after four disproven theories) A stone-cold body answers the **first** window request with whole-response `status: "Pending"` (only the whole-body cache-hit branch sets Ready — `layer_proxy.rs`), and `on_response()`'s first check — `status != "Ready" → return` — swallowed it **before the retry machinery could run**. Retries frozen at 0 forever; both tile-mosaic and single-window first descents affected. Second connections worked by luck (the first request warms the whole-body cache), which also masked the bug in every warm-server verification drive ever run. The four dead theories, each killed by verification rather than shipped: an unpaired redraw (lead's misread grep — the pairing existed), a ConnectionId(0) sentinel (Dudley's exhaustive audit exonerated the server and left a permanent bridge_tcp pin), the 10s retry ceiling (Stig's own replacement theory — killed by the live retries=0 numbers), and pending-attribution in the fan-out (close, but the responses never reached attribution at all). ### The fixes - **Status-first branching** in `on_response()` per the `atlas_generation_proxy` reference shape: Pending → retry; Ready → existing null-window retry; NotFound/Error → **immediate give-up** (the principled policy, replacing the elapsed-retries ceiling). - **Deterministic backoff + per-tile stagger**: 0.5 s doubling to a 4 s cap, +0.1 s × tile index — six cold tiles retry as a spread trickle, strictly-increasing delays asserted, no lockstep pulses. MAX_RETRIES 20→30 (~110 s horizon under backoff). - **Self-healing tile repaint** (`_process` redraws viewer+overlay while tiles are pending) — closes the arrival-repaint class regardless of which signal edge drops. - **Re-entry root cause**: `ImplantApp._on_screen_changed` re-runs `enter()` on repeat same-screen pushes — same-body guard in `RegionalScreen.enter()` + **immediate-free `ImplantPanel.clear()`** (protects every implant app from same-frame stacking). - **Legend re-fit** (deferred `reset_to_content_size()` — manually-positioned Controls never shrink on their own; RichTextLabel fit_content reports degenerate minimums until laid out once — both found empirically after two plausible fixes were disproven by trace). - **Visible "DERIVING TERRAIN…" state** while zero tiles have arrived — a cold wait reads as loading, not broken. - **`make atlas` moves to a release server** (`build-server-release` + `SR_SERVER_BIN` override, SR_PORT-precedent shape): a cold debug server needs >10 s before a new body's first tile exists; release serves cold in well under a second. - **Dudley's connection-zero regression module** (`bridge_tcp.rs`, +497 lines): the first-ever connection id a fresh server assigns receives window responses — the server half's permanent pin, replicating the real `main.rs` bootstrap. ### Test discipline Wire-accurate regressions **through the real fan-out** (`tile_set._on_atlas_layers_received`, never `tile.on_response()` directly) with pendings matching the server's actual response construction (body_id only) — the exact shape whose absence let this ship. Single-window mirrors per the shared-class reach. Two first-draft tests that passed with the bug reverted were caught and strengthened before reporting; every fix revert-verified independently. ### Cold acceptance (the merge gate, passed) Fresh release server, **first connection**, Lendel entry: DERIVING TERRAIN at 2 s → terrain painting tile-by-tile at 5 s → all six tiles converged, 6 pending + 6 ready on the wire, legend exactly one legend tall. Captures in `.cache/screenshots/cold-start-verify/`. Tickets: T-1163

jpmschweitzer added 4 commits 2026-07-22 21:02:21 +02:00

fix(client): cold-start black screen — self-healing tile repaint, legend re-entry root cause, pending-tile wash d4526e51ff

Jeroen hit a black mosaic zooming into a body from a fresh make-atlas
spawn. Live diagnosis showed all six tiles held with colored textures
and no repaint; the original line-specific diagnosis (unpaired
queue_redraw in _on_tile_ready) turned out WRONG — the pairing already
existed (lead's truncated grep misread the function; Stig verified via
git log -p before acting). Rather than chase the exact dropped signal
edge, _process() now self-heals: both viewer and overlay redraw every
frame while the tile set has pending tiles (has_pending_tiles(), new) —
a strict superset that closes the black regardless of which edge drops,
pinned by a _draw()-counting real-subclass spy test.

Legend stacking root cause found by trace, not guess: ImplantApp.
_on_screen_changed() re-runs enter() unconditionally on repeat
same-screen pushes — each re-entry tore down and rebuilt all six tile
requests (the round-6 orphaning fingerprint via a new trigger) and
stacked another legend (~10 deep, full-height dark panel). Fixed both
ends: ImplantPanel.clear() frees immediately (same-frame re-entrant
refresh can never observe stale children — protects every implant app),
and RegionalScreen.enter() no-ops for the same body (different body
still re-enters fresh). The load-bearing regression asserts an ARRIVED
TILE'S DATA survives a repeat push — node identity would not catch the
teardown (the tile-set Node is a fixed field; only its internals reset).

Cold-start UX: pending tiles now draw the single-window path's
COLOR_BORDER_FADE wash instead of raw background — a deriving mosaic
reads as loading, not broken.

Suites green (zoom_ladder 48, viewer 74, tile_set 20, overlay 30,
overlays 46 + 2 new files), gdlint clean, revert-verified throughout.

fix(client): cold-start round 2 — DERIVING TERRAIN state, legend re-fit, release server for make atlas dd13760d62

Item 1: while zero tiles have arrived, the viewer draws a centered
screen-space 'DERIVING TERRAIN…' label (text_dim role, no new hue),
dropping the instant the first tile lands — a cold wait now reads as
loading, not broken. New has_any_tile_arrived() predicate (distinct
from has_pending_tiles(): both true mid-arrival, tested exactly there).

Item 2: legend re-fit root-caused empirically, two plausible fixes
disproven by trace before the real one: a manually-positioned Control's
size NEVER tracks a shrinking minimum in this parenting shape, and
RichTextLabel.fit_content reports degenerate minimums until laid out at
real width once — so reset must be DEFERRED and run after refill, not
inside clear(). ImplantPanel.reset_to_content_size() (call_deferred),
wired into both legend refresh()es. The load-bearing test compares
size.y to get_minimum_size().y — a size-to-size comparison passed
trivially with both numbers equally stuck (caught on first draft).

Item 3: make atlas now builds the RELEASE server and passes
SR_SERVER_BIN (a cold DEBUG server delivers zero tiles for >10s on a
new body — live-measured — vs 210ms warm; release serves cold in well
under a second). atlas_standalone._server_binary_path() honors the env
override per the SR_PORT two-tier precedent, debug path unchanged
when unset.

All fixes revert-verified; nine suites green collateral-checked;
gdlint clean.

test(simulation): connection-zero district_window delivery regression (bridge_tcp) ddc4483462

Dudley's audit of the suspected ConnectionId(0) starvation: exhaustive
trace of the delivery path (bridge/mod.rs insert/lookup/send paths,
atlas plugin serve/drain, gen_queue coalescing) found NO sentinel or
default-value collision — ConnectionId doesn't even derive Default, and
every id path is a plain monotonic allocation with linear lookup. The
real gap was coverage: every existing window test used ConnectionId(1).

New connection_zero_window_delivery module replicates main.rs's actual
bootstrap (blocking first-accept -> BridgeResource -> accept loop) and
pins: the first-ever connection receives a district window; the exact
six-tile entry shape delivers all six to connection 0; connection 1
keeps working. 5/5 repeated runs, single- and parallel-threaded.

The live 'starvation' itself was client-side (pending responses never
re-requested on the tile path — separate fix in flight); these tests
stay as the server half's permanent pin.

fix(client): cold-body Pending responses reach the retry path — the launch-shape starvation root cause 3891c83206

The true cause of the black first launch, server-confirmed after four
disproven theories: a stone-cold body answers the FIRST window request
with a whole-response status 'Pending' (only the whole-body cache-hit
branch sets Ready), and on_response()'s very first check — status !=
Ready -> return — swallowed it before the retry machinery could run.
retries stayed 0 forever; the DERIVING state never resolved. Second
connections worked by luck (the first request warms the whole-body
cache, so they read Ready and take the healthy path). Both tile fan-out
AND single-window first-descents were affected — one shared function,
one fix: branch on the outer status first (the atlas_generation_proxy
reference shape): Pending -> retry, Ready -> existing null-window retry,
NotFound/Error -> give up immediately (the principled give-up policy,
replacing the elapsed-retries ceiling).

Hardening in the same round: deterministic exponential backoff (0.5s
doubling, 4s cap) + per-tile stagger (0.1s * index — six tiles retry at
0.5/0.6/0.7/0.8/0.9/1.0s, strictly-increasing asserted, not jittered);
MAX_RETRIES 20->30 (~110s horizon under backoff).

Regressions are wire-accurate by construction: whole-response Pending
(body_id only, no center/n/granularity — verified against the server's
own response construction) delivered through the REAL fan-out
(tile_set._on_atlas_layers_received, never tile.on_response directly),
mirrored at single-window level. Two first-draft tests that passed with
the bug reverted were caught and strengthened before reporting; every
fix and hardening piece revert-verified independently (7+1 failing
tests without them).

jpmschweitzer commented

2026-07-22 21:20:22 +02:00

Review: ladder-cold-start → main (type: code)

Branch tip reviewed: 8bb09abad (fix round included). Reviewers: Tyre (architecture), Hoshe (QA).

Tyre (Architecture): APPROVE

The status-first branching in on_response() is the architecturally correct reading of the T-1124 §4 async serving contract, and the fix chain's diagnostic discipline (five theories, each killed by live evidence before shipping) is exemplary. Three capture items, all executed rather than parked:

#	Item	Disposition
1	The Pending / Ready+null-window two-legal-wire-shapes-for-one-logical-state fact is a protocol invariant, not an implementation accident — record it in governance	Done — D-226 pending-shape protocol note under the T-1124 §4 amendment area (commit `4c34327b4`): both shapes mean "still deriving, re-poll"; only NotFound/Error are terminal; any server refactor of the asymmetry must migrate every consumer in the same change
2	Tiled orbital terminal-recovery gap: after ~110 s of genuine unbroken Pending, the DERIVING state has no player-reachable retry path (pan no-op, zoom-out floored, re-entry guarded)	Ticketed — T-1164 (client, medium)
3	The cold-launch bug class was only findable by a human launching the real stack cold — the missing test tier is a scripted cold-launch integration test	Ticketed — T-1166 under T-1037 (fresh release server, connection-zero descent, all-tiles-arrive deadline, non-black frame assertion); plus T-1165 under T-1037 for root-causing the dropped repaint edge the `_process` self-heal currently compensates for

Hoshe (QA): APPROVE (after one fix round)

Initial verdict REQUEST_CHANGES with two findings; both fixed and independently re-verified at tip (not taken on report):

#	Finding	Fix
1	Missing regression: a whole-response `Pending` for a different body must not touch a request's retry state — the explicitly-called-out edge case had no test	`8bb09abad` — `test_pending_for_a_different_body_does_not_touch_retry_state`. Revert-verified twice independently (coordinator and reviewer each reordered the guards and saw exactly this one test fail by name); final own-body Ready assertion proves state genuinely untouched
2	Duplicated 5-line stagger comment block in `atlas_window_tile_set.gd` (copy-paste artifact)	`4c34327b4` — collapsed to one copy, `grep -c` = 1

Hoshe's verification highlights: empirically confirmed the flagship retry regressions fail against the pre-fix code (live revert, not commit-message self-report); verified the layer_proxy.rs response-construction claims by direct source inspection; audited every ImplantPanel.clear() caller for same-frame hazards; verified Dudley's bridge_tcp.rs connection-zero module replicates the real main.rs blocking-first-accept bootstrap line-for-line; backoff math and six-tile stagger collision-freedom verified analytically. Noted for awareness (not blocking): the client discards an attached district_window when the outer status is Pending (independently-evicting caches) — an efficiency gap, converges via retry.

Full regression sweep at tip: client 3678/3678, server suite green, cargo fmt/gdlint clean, pql decisions validate clean.

Cold acceptance (the merge gate for this PR): PASSED

Fresh release server, first-ever connection, Lendel entry: DERIVING TERRAIN at 2 s → terrain painting tile-by-tile at 5 s → all six tiles converged (6 pending + 6 ready observed on the wire), legend exactly one legend tall. Captures: .cache/screenshots/cold-start-verify/.

Verdict: APPROVED

Tickets: T-1163 (fix). Filed from review: T-1164, T-1165, T-1166.

## Review: ladder-cold-start → main (type: code) Branch tip reviewed: `8bb09abad` (fix round included). Reviewers: Tyre (architecture), Hoshe (QA). ### Tyre (Architecture): APPROVE The status-first branching in `on_response()` is the architecturally correct reading of the T-1124 §4 async serving contract, and the fix chain's diagnostic discipline (five theories, each killed by live evidence before shipping) is exemplary. Three capture items, all executed rather than parked: | # | Item | Disposition | |---|------|-------------| | 1 | The Pending / Ready+null-window two-legal-wire-shapes-for-one-logical-state fact is a protocol invariant, not an implementation accident — record it in governance | **Done** — D-226 pending-shape protocol note under the T-1124 §4 amendment area (commit `4c34327b4`): both shapes mean "still deriving, re-poll"; only NotFound/Error are terminal; any server refactor of the asymmetry must migrate every consumer in the same change | | 2 | Tiled orbital terminal-recovery gap: after ~110 s of genuine unbroken Pending, the DERIVING state has no player-reachable retry path (pan no-op, zoom-out floored, re-entry guarded) | **Ticketed** — T-1164 (client, medium) | | 3 | The cold-launch bug class was only findable by a human launching the real stack cold — the missing test tier is a scripted cold-launch integration test | **Ticketed** — T-1166 under T-1037 (fresh release server, connection-zero descent, all-tiles-arrive deadline, non-black frame assertion); plus T-1165 under T-1037 for root-causing the dropped repaint edge the `_process` self-heal currently compensates for | ### Hoshe (QA): APPROVE (after one fix round) Initial verdict REQUEST_CHANGES with two findings; both fixed and independently re-verified at tip (not taken on report): | # | Finding | Fix | |---|---------|-----| | 1 | Missing regression: a whole-response `Pending` for a *different* body must not touch a request's retry state — the explicitly-called-out edge case had no test | `8bb09abad` — `test_pending_for_a_different_body_does_not_touch_retry_state`. Revert-verified twice independently (coordinator and reviewer each reordered the guards and saw exactly this one test fail by name); final own-body Ready assertion proves state genuinely untouched | | 2 | Duplicated 5-line stagger comment block in `atlas_window_tile_set.gd` (copy-paste artifact) | `4c34327b4` — collapsed to one copy, `grep -c` = 1 | Hoshe's verification highlights: empirically confirmed the flagship retry regressions fail against the pre-fix code (live revert, not commit-message self-report); verified the `layer_proxy.rs` response-construction claims by direct source inspection; audited every `ImplantPanel.clear()` caller for same-frame hazards; verified Dudley's `bridge_tcp.rs` connection-zero module replicates the real `main.rs` blocking-first-accept bootstrap line-for-line; backoff math and six-tile stagger collision-freedom verified analytically. Noted for awareness (not blocking): the client discards an attached `district_window` when the outer status is `Pending` (independently-evicting caches) — an efficiency gap, converges via retry. Full regression sweep at tip: client 3678/3678, server suite green, `cargo fmt`/gdlint clean, `pql decisions validate` clean. ### Cold acceptance (the merge gate for this PR): PASSED Fresh release server, first-ever connection, Lendel entry: DERIVING TERRAIN at 2 s → terrain painting tile-by-tile at 5 s → all six tiles converged (6 pending + 6 ready observed on the wire), legend exactly one legend tall. Captures: `.cache/screenshots/cold-start-verify/`. ### Verdict: APPROVED Tickets: T-1163 (fix). Filed from review: T-1164, T-1165, T-1166.

jpmschweitzer referenced this issue from a commit

2026-07-22 21:22:03 +02:00

docs(ui): PR #193 review capture — D-226 pending-shape protocol note + stagger comment dedupe

jpmschweitzer referenced this issue from a commit

2026-07-22 21:22:03 +02:00

test(ui): wrong-body Pending must not touch retry state (PR #193 Hoshe finding 1)

jpmschweitzer added 2 commits 2026-07-22 21:22:03 +02:00

docs(ui): PR #193 review capture — D-226 pending-shape protocol note + stagger comment dedupe 4c34327b41

Tyre's APPROVE items (T-1163): record the two-legal-wire-shapes-for-one-
logical-state protocol invariant (whole-response Pending AND Ready+null
district_window both mean 're-poll'; only NotFound/Error are terminal) as
a D-226 note under the T-1124 §4 amendment area, so a future server
refactor of the asymmetry must migrate every consumer in the same change.
Drop the duplicated 5-line stagger comment in atlas_window_tile_set.gd.

Follow-up tickets filed on main: T-1164 (tiled terminal-recovery),
T-1165 (queue_redraw edge root-cause), T-1166 (cold-launch test tier).

Tickets: T-1163

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

test(ui): wrong-body Pending must not touch retry state (PR #193 Hoshe finding 1) 8bb09abad0

Pins on_response()'s guard ordering: body_id check BEFORE the status
branch, so another body's cold Pending can never burn one of our 30
retries or reschedule our timer (multi-body browsing / shared-broadcast
tile fan-out). Revert-verified: reordering the guards makes this test
fail; final own-body Ready assertion proves the request is genuinely
untouched, not just un-retried. 44/44 in the file.

Tickets: T-1163

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

jpmschweitzer closed this pull request

2026-07-22 21:29:49 +02:00

jpmschweitzer referenced this issue from a commit

2026-07-22 21:35:36 +02:00

chore(meta): close T-1163 — PR #193 merged (cold-launch starvation fix); review captures T-1164/T-1165/T-1166 filed

jpmschweitzer referenced this pull request

2026-07-22 23:12:48 +02:00

feat(simulation): nature-first octaves + per-rung filter policy (T-1162, T-1161) #194