fix(client): cold-body Pending responses reach the retry path — the launch-shape starvation root cause
The true cause of the black first launch, server-confirmed after four disproven theories: a stone-cold body answers the FIRST window request with a whole-response status 'Pending' (only the whole-body cache-hit branch sets Ready), and on_response()'s very first check — status != Ready -> return — swallowed it before the retry machinery could run. retries stayed 0 forever; the DERIVING state never resolved. Second connections worked by luck (the first request warms the whole-body cache, so they read Ready and take the healthy path). Both tile fan-out AND single-window first-descents were affected — one shared function, one fix: branch on the outer status first (the atlas_generation_proxy reference shape): Pending -> retry, Ready -> existing null-window retry, NotFound/Error -> give up immediately (the principled give-up policy, replacing the elapsed-retries ceiling). Hardening in the same round: deterministic exponential backoff (0.5s doubling, 4s cap) + per-tile stagger (0.1s * index — six tiles retry at 0.5/0.6/0.7/0.8/0.9/1.0s, strictly-increasing asserted, not jittered); MAX_RETRIES 20->30 (~110s horizon under backoff). Regressions are wire-accurate by construction: whole-response Pending (body_id only, no center/n/granularity — verified against the server's own response construction) delivered through the REAL fan-out (tile_set._on_atlas_layers_received, never tile.on_response directly), mirrored at single-window level. Two first-draft tests that passed with the bug reverted were caught and strengthened before reporting; every fix and hardening piece revert-verified independently (7+1 failing tests without them).
This commit is contained in:
@@ -57,6 +57,27 @@ static func _mock_response(body_id: String, window: Variant) -> Dictionary:
|
||||
return {"body_id": body_id, "status": "Ready", "district_window": window}
|
||||
|
||||
|
||||
## PR #192 cold-start round 3: the WIRE-ACCURATE shape of a cold body's
|
||||
## first-ever response (server/src/atlas/layer_proxy.rs's
|
||||
## get_or_generate()/serve_district_window(), whole-body cache MISS branch —
|
||||
## `AtlasLayerResponse { status: Pending, district_window: None, ... }`,
|
||||
## confirmed directly against that source). `body_id` is the only
|
||||
## identifying field.
|
||||
static func _pending_response(body_id: String) -> Dictionary:
|
||||
return {"body_id": body_id, "status": "Pending", "district_window": null}
|
||||
|
||||
|
||||
static func _not_found_response(body_id: String) -> Dictionary:
|
||||
return {"body_id": body_id, "status": "NotFound", "district_window": null}
|
||||
|
||||
|
||||
## Matches atlas_map_protocol.gd's `_decode_status_field()` — the decoded
|
||||
## `AtlasLayerStatus::Error(String)` variant's status STRING is always just
|
||||
## "Error" (the message rides in a separate `error` field).
|
||||
static func _error_response(body_id: String, message: String = "boom") -> Dictionary:
|
||||
return {"body_id": body_id, "status": "Error", "error": message, "district_window": null}
|
||||
|
||||
|
||||
func _make_request() -> Variant:
|
||||
var owner_stub := RefCounted.new()
|
||||
var req = auto_free(AtlasWindowRequest.new(owner_stub))
|
||||
@@ -454,3 +475,143 @@ func test_clamp_window_n_mirror_v2_region_per_axis_cap_alone_satisfies_wire_cap_
|
||||
func test_cell_grid_side_region_mirror_minimum_is_one() -> void:
|
||||
assert_int(AtlasWindowRequest._cell_grid_side_region_mirror(1)).is_equal(1)
|
||||
assert_int(AtlasWindowRequest._cell_grid_side_region_mirror(50)).is_equal(1)
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# PR #192 cold-start round 3 — the single-window half of the launch-shape
|
||||
# gap: the SAME status-gate bug the tile fan-out has (test_atlas_window_tile_set.gd's
|
||||
# own regressions) applies equally here, since on_response() is the shared
|
||||
# class both paths use. A first descent onto a cold body with NO tiles
|
||||
# (District/Quarter rung, or a small Region body) hits the identical
|
||||
# whole-body-cache-miss -> status:"Pending" wire shape.
|
||||
# =============================================================================
|
||||
|
||||
|
||||
## The exact bug, single-window shape: a whole-response Pending on a cold
|
||||
## first request must increment the retry counter and schedule a re-poll —
|
||||
## not be silently dropped by the OLD `status != "Ready" -> return` gate.
|
||||
func test_cold_request_whole_response_pending_increments_retry_count() -> void:
|
||||
var req = _make_request()
|
||||
req.request_now("GJ380c", Vector2i(2, 2), 2)
|
||||
assert_bool(req.is_pending()).is_true()
|
||||
|
||||
req.on_response(_pending_response("GJ380c"))
|
||||
|
||||
assert_bool(req.is_pending()).override_failure_message(
|
||||
"a whole-response Pending must leave the request still pending, not"
|
||||
+ " silently give up"
|
||||
).is_true()
|
||||
assert_int(req._retries).override_failure_message(
|
||||
"a whole-response Pending must increment the retry counter — the"
|
||||
+ " exact bug: the OLD status-gate dropped this before ever reaching"
|
||||
+ " the retry-scheduling code, leaving retries at 0 forever"
|
||||
).is_equal(1)
|
||||
|
||||
|
||||
## Full convergence: a whole-response Pending, then a real Ready for the
|
||||
## RE-REQUEST, must be accepted — proving the retry loop's own re-request
|
||||
## actually gets picked up, not just that the counter increments. The
|
||||
## mid-test retries==1 assertion is what makes this genuinely load-bearing:
|
||||
## without it, a Ready delivered ANY time after a Pending (retried or not)
|
||||
## trivially passes this test's final assertion, since accepting a fresh
|
||||
## Ready response was never the broken behavior — only the retry itself was.
|
||||
func test_cold_request_converges_after_pending_then_real_ready() -> void:
|
||||
var req = _make_request()
|
||||
req.request_now("GJ380c", Vector2i(2, 2), 2)
|
||||
|
||||
req.on_response(_pending_response("GJ380c"))
|
||||
assert_bool(req.is_pending()).is_true()
|
||||
assert_int(req._retries).override_failure_message(
|
||||
"sanity: the retry must have actually been scheduled before this test"
|
||||
+ " waits for it to fire — otherwise the final assertion below would"
|
||||
+ " pass even if the retry never happened at all"
|
||||
).is_equal(1)
|
||||
|
||||
await get_tree().create_timer(0.6).timeout # past the first retry delay
|
||||
|
||||
var window: Dictionary = _mock_window(Vector2i(2, 2), 2)
|
||||
req.on_response(_mock_response("GJ380c", window))
|
||||
assert_bool(req.is_pending()).override_failure_message(
|
||||
"a real Ready response after the pending/retry cycle must be accepted"
|
||||
).is_false()
|
||||
|
||||
|
||||
## NotFound must give up immediately, not retry.
|
||||
func test_cold_request_not_found_gives_up_immediately_without_retry() -> void:
|
||||
var req = _make_request()
|
||||
req.request_now("GJ380c", Vector2i(2, 2), 2)
|
||||
|
||||
req.on_response(_not_found_response("GJ380c"))
|
||||
|
||||
assert_bool(req.is_pending()).override_failure_message(
|
||||
"NotFound must give up immediately, not stay pending waiting for a retry"
|
||||
).is_false()
|
||||
assert_int(req._retries).is_equal(0)
|
||||
|
||||
|
||||
## Error must give up immediately too.
|
||||
func test_cold_request_error_gives_up_immediately_without_retry() -> void:
|
||||
var req = _make_request()
|
||||
req.request_now("GJ380c", Vector2i(2, 2), 2)
|
||||
|
||||
req.on_response(_error_response("GJ380c"))
|
||||
|
||||
assert_bool(req.is_pending()).override_failure_message(
|
||||
"Error must give up immediately, not stay pending waiting for a retry"
|
||||
).is_false()
|
||||
assert_int(req._retries).is_equal(0)
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# _retry_delay_for() — deterministic exponential backoff + per-tile stagger
|
||||
# (PR #192 cold-start round 3 hardening: 6 tiles retrying in perfect
|
||||
# lockstep on a whole-response Pending is a real request-pulse risk even
|
||||
# though it isn't what caused the starvation bug above).
|
||||
# =============================================================================
|
||||
|
||||
|
||||
func test_retry_delay_for_first_retry_is_the_initial_delay() -> void:
|
||||
assert_float(AtlasWindowRequest._retry_delay_for(1, 0)).is_equal_approx(
|
||||
AtlasWindowRequest.INITIAL_RETRY_DELAY, 0.0001
|
||||
)
|
||||
|
||||
|
||||
func test_retry_delay_for_doubles_each_retry_until_the_cap() -> void:
|
||||
assert_float(AtlasWindowRequest._retry_delay_for(1, 0)).is_equal_approx(0.5, 0.0001)
|
||||
assert_float(AtlasWindowRequest._retry_delay_for(2, 0)).is_equal_approx(1.0, 0.0001)
|
||||
assert_float(AtlasWindowRequest._retry_delay_for(3, 0)).is_equal_approx(2.0, 0.0001)
|
||||
assert_float(AtlasWindowRequest._retry_delay_for(4, 0)).is_equal_approx(4.0, 0.0001)
|
||||
# Retry 5 would double past MAX_RETRY_DELAY (8.0) — must clamp, not keep growing.
|
||||
assert_float(AtlasWindowRequest._retry_delay_for(5, 0)).is_equal_approx(
|
||||
AtlasWindowRequest.MAX_RETRY_DELAY, 0.0001
|
||||
)
|
||||
assert_float(AtlasWindowRequest._retry_delay_for(20, 0)).is_equal_approx(
|
||||
AtlasWindowRequest.MAX_RETRY_DELAY, 0.0001
|
||||
)
|
||||
|
||||
|
||||
## The deterministic stagger: tile index i's delay is offset by
|
||||
## STAGGER_STEP*i on top of the same backoff schedule — directly assertable,
|
||||
## not a randomized jitter a test would have to tolerance-check.
|
||||
func test_retry_delay_for_staggers_deterministically_by_tile_index() -> void:
|
||||
var base: float = AtlasWindowRequest._retry_delay_for(1, 0)
|
||||
for i in range(6):
|
||||
var expected: float = base + AtlasWindowRequest.STAGGER_STEP * float(i)
|
||||
assert_float(AtlasWindowRequest._retry_delay_for(1, i)).override_failure_message(
|
||||
"tile index %d's first-retry delay must be exactly base + STAGGER_STEP*%d" % [i, i]
|
||||
).is_equal_approx(expected, 0.0001)
|
||||
|
||||
|
||||
## Six tiles that all went pending in the same frame must NOT all retry at
|
||||
## the exact same instant — the anti-storm property this hardening exists
|
||||
## for, pinned directly: every tile's delay for the SAME retry count must be
|
||||
## strictly increasing with its stagger index.
|
||||
func test_retry_delay_for_six_tiles_never_collide_on_the_same_retry() -> void:
|
||||
var delays: Array = []
|
||||
for i in range(6):
|
||||
delays.append(AtlasWindowRequest._retry_delay_for(1, i))
|
||||
for i in range(1, delays.size()):
|
||||
assert_float(delays[i]).override_failure_message(
|
||||
"tile %d's delay must be strictly greater than tile %d's — a storm"
|
||||
+ " pulse means two tiles retrying at the same instant" % [i, i - 1]
|
||||
).is_greater(delays[i - 1])
|
||||
|
||||
@@ -30,6 +30,29 @@ static func _mock_response(body_id: String, window: Variant) -> Dictionary:
|
||||
return {"body_id": body_id, "status": "Ready", "district_window": window}
|
||||
|
||||
|
||||
## PR #192 cold-start round 3: the WIRE-ACCURATE shape of a cold body's
|
||||
## first-ever response — server/src/atlas/layer_proxy.rs's
|
||||
## get_or_generate()/serve_district_window() on a whole-body cache MISS
|
||||
## builds `AtlasLayerResponse { status: Pending, district_window: None, ... }`
|
||||
## (confirmed directly against that source). `body_id` is the ONLY
|
||||
## identifying field — no center/n/granularity anywhere, matching the real
|
||||
## wire's total lack of per-request attribution on this specific shape.
|
||||
static func _pending_response(body_id: String) -> Dictionary:
|
||||
return {"body_id": body_id, "status": "Pending", "district_window": null}
|
||||
|
||||
|
||||
static func _not_found_response(body_id: String) -> Dictionary:
|
||||
return {"body_id": body_id, "status": "NotFound", "district_window": null}
|
||||
|
||||
|
||||
## Matches atlas_map_protocol.gd's `_decode_status_field()` — the decoded
|
||||
## `AtlasLayerStatus::Error(String)` variant's status STRING is always just
|
||||
## "Error" (the message rides in a separate `error` field, not appended to
|
||||
## the status string), confirmed against that decoder directly.
|
||||
static func _error_response(body_id: String, message: String = "boom") -> Dictionary:
|
||||
return {"body_id": body_id, "status": "Error", "error": message, "district_window": null}
|
||||
|
||||
|
||||
func _make_tile_set() -> Variant:
|
||||
var owner_stub := RefCounted.new()
|
||||
var ts = auto_free(AtlasWindowTileSet.new(owner_stub))
|
||||
@@ -52,6 +75,23 @@ func test_enter_produces_the_expected_tile_count_for_lendel() -> void:
|
||||
assert_bool(ts.is_multi_tile()).is_true()
|
||||
|
||||
|
||||
## PR #192 cold-start round 3 hardening: each tile's own AtlasWindowRequest
|
||||
## must get a DISTINCT, index-matching `_stagger_index` — the anti-storm
|
||||
## property (deterministic retry-delay stagger, see
|
||||
## AtlasWindowRequest._retry_delay_for()'s own doc) depends entirely on this
|
||||
## wiring; without it every tile silently staggers at index 0 and retries
|
||||
## in lockstep again, exactly the storm risk this hardening exists to close.
|
||||
func test_enter_wires_a_distinct_stagger_index_per_tile() -> void:
|
||||
var ts = _make_tile_set()
|
||||
ts.enter("GJ380c", 6238.4)
|
||||
for i in range(ts._tiles.size()):
|
||||
var req = ts._tiles[i]["request"]
|
||||
assert_int(req._stagger_index).override_failure_message(
|
||||
"tile %d's request must be wired with _stagger_index=%d, matching"
|
||||
+ " its own position in the tile set" % [i, i]
|
||||
).is_equal(i)
|
||||
|
||||
|
||||
## A tiny (non-tiling) body produces exactly ONE tile — the degenerate case
|
||||
## compute_tile_grid() itself already covers; this confirms the ORCHESTRATION
|
||||
## (not just the grid math) handles it without crashing or requesting zero
|
||||
@@ -142,6 +182,123 @@ func test_all_tiles_arriving_flips_fully_arrived() -> void:
|
||||
).is_true()
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# PR #192 cold-start round 3 — THE launch-shape regression: a cold body's
|
||||
# FIRST-EVER response is a wire-accurate whole-response Pending (status
|
||||
# "Pending", district_window null, body_id only — no center/n/granularity
|
||||
# anywhere, confirmed against server/src/atlas/layer_proxy.rs directly).
|
||||
# Driven through the REAL fan-out (SimBridge.atlas_layers_received.emit(),
|
||||
# reaching every tile via tile_set._on_atlas_layers_received), not a direct
|
||||
# tile.on_response() call — the coordinator's own regression-discipline ask,
|
||||
# since a direct per-tile call is implicit attribution a real wire fan-out
|
||||
# doesn't have. Before the status-gate fix, on_response()'s FIRST line
|
||||
# (`status != "Ready" -> return`) dropped this response for every tile
|
||||
# before ever reaching the retry-scheduling code — retries stayed at 0
|
||||
# forever, matching the coordinator's own live cold-server capture exactly.
|
||||
# =============================================================================
|
||||
|
||||
|
||||
## The exact bug: a whole-response Pending on cold entry must increment
|
||||
## every tile's retry counter and schedule a re-poll — not be silently
|
||||
## dropped. Fails hard against the pre-fix code (retries stay 0 forever).
|
||||
func test_cold_entry_whole_response_pending_increments_every_tiles_retry_count() -> void:
|
||||
var ts = _make_tile_set()
|
||||
ts.enter("GJ380c", 6238.4)
|
||||
|
||||
SimBridge.atlas_layers_received.emit(_pending_response("GJ380c"))
|
||||
|
||||
for tile_dict: Dictionary in ts._tiles:
|
||||
var req = tile_dict["request"]
|
||||
assert_bool(req.is_pending()).override_failure_message(
|
||||
"every tile must still be pending after a whole-response Pending"
|
||||
).is_true()
|
||||
assert_int(req._retries).override_failure_message(
|
||||
"a whole-response Pending must increment the retry counter — the"
|
||||
+ " exact bug: the OLD status-gate silently dropped this before"
|
||||
+ " ever reaching the retry-scheduling code, leaving retries at 0 forever"
|
||||
).is_equal(1)
|
||||
|
||||
|
||||
## The full convergence: repeated whole-response Pendings (simulating a slow
|
||||
## cold AnalyzeBody), THEN real per-tile Ready responses for the tiles' own
|
||||
## RE-REQUESTS — every tile must eventually fill. No re-request means this
|
||||
## hangs (waiting past the retry delay for a re-poll that never happens) or
|
||||
## fails (has_pending_tiles() never flips false) — exactly the launch-shape
|
||||
## gap the coordinator named: "every unit test delivered Ready immediately."
|
||||
func test_cold_entry_converges_after_repeated_pending_then_real_readies() -> void:
|
||||
var ts = _make_tile_set()
|
||||
ts.enter("GJ380c", 6238.4)
|
||||
var tiles: Array = ts.get_tiles()
|
||||
|
||||
# Two consecutive whole-response Pendings — a slow cold derive, not a
|
||||
# single flip. Real waits so the scheduled retry timers can actually fire.
|
||||
SimBridge.atlas_layers_received.emit(_pending_response("GJ380c"))
|
||||
await get_tree().create_timer(0.7).timeout # past the first backoff delay
|
||||
SimBridge.atlas_layers_received.emit(_pending_response("GJ380c"))
|
||||
await get_tree().create_timer(0.8).timeout # past the second (staggered) delay
|
||||
|
||||
assert_bool(ts.has_pending_tiles()).override_failure_message(
|
||||
"sanity: still pending after 2 cold cycles"
|
||||
).is_true()
|
||||
for tile_dict: Dictionary in ts._tiles:
|
||||
var req = tile_dict["request"]
|
||||
assert_int(req._retries).override_failure_message(
|
||||
"sanity: each tile's retry counter must have actually incremented"
|
||||
+ " twice — otherwise the final assertion below would pass even if"
|
||||
+ " the retries never happened at all (a fresh Ready was never the"
|
||||
+ " broken behavior, only the retry itself was)"
|
||||
).is_equal(2)
|
||||
|
||||
for tile: Dictionary in tiles:
|
||||
var window: Dictionary = _mock_window(
|
||||
tile["center"], AtlasWindowRequest.SERVER_DISTRICT_WINDOW_MAX_N_REGION
|
||||
)
|
||||
SimBridge.atlas_layers_received.emit(_mock_response("GJ380c", window))
|
||||
|
||||
assert_bool(ts.has_pending_tiles()).override_failure_message(
|
||||
"after the retries fire and real Readies land, every tile must have adopted its window"
|
||||
).is_false()
|
||||
assert_bool(ts.is_fully_arrived()).is_true()
|
||||
|
||||
|
||||
## NotFound must give up IMMEDIATELY, not retry — a body that doesn't exist
|
||||
## will never resolve by waiting (the coordinator's own "give up on error
|
||||
## responses, not elapsed patience" framing — this replaces the old
|
||||
## elapsed-retries-only give-up policy with a correctness-based one for the
|
||||
## cases where retrying is provably pointless).
|
||||
func test_cold_entry_not_found_gives_up_immediately_without_retry() -> void:
|
||||
var ts = _make_tile_set()
|
||||
ts.enter("GJ380c", 6238.4)
|
||||
|
||||
SimBridge.atlas_layers_received.emit(_not_found_response("GJ380c"))
|
||||
|
||||
for tile_dict: Dictionary in ts._tiles:
|
||||
var req = tile_dict["request"]
|
||||
assert_bool(req.is_pending()).override_failure_message(
|
||||
"NotFound must give up immediately, not stay pending waiting for a retry"
|
||||
).is_false()
|
||||
assert_int(req._retries).override_failure_message(
|
||||
"NotFound must never increment the retry counter — retrying a"
|
||||
+ " nonexistent body is provably pointless"
|
||||
).is_equal(0)
|
||||
|
||||
|
||||
## Error must give up immediately too, same reasoning as NotFound — a
|
||||
## resolve/IO failure won't resolve itself by polling.
|
||||
func test_cold_entry_error_gives_up_immediately_without_retry() -> void:
|
||||
var ts = _make_tile_set()
|
||||
ts.enter("GJ380c", 6238.4)
|
||||
|
||||
SimBridge.atlas_layers_received.emit(_error_response("GJ380c"))
|
||||
|
||||
for tile_dict: Dictionary in ts._tiles:
|
||||
var req = tile_dict["request"]
|
||||
assert_bool(req.is_pending()).override_failure_message(
|
||||
"Error must give up immediately, not stay pending waiting for a retry"
|
||||
).is_false()
|
||||
assert_int(req._retries).is_equal(0)
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# has_pending_tiles() / has_any_tile_arrived() — PR #192 cold-start dossier.
|
||||
# Distinct predicates (both can be true at once, mid-arrival): the viewer's
|
||||
|
||||
Reference in New Issue
Block a user