test(config): reject captures where the renderer drew nothing

run-visual verified only that the captured PNG was non-empty AS A FILE. A
blank screen is a perfectly valid ~19 KB PNG, so it passed — and once a blank
capture had been recorded as a golden, every later blank capture matched it at
0.0% and the scenario PASSED. atlas_GJ338Bd_Block and atlas_GJ445c-m1_Chunk
sat green against blank goldens while the suite's other 30 scenarios failed.

That is the worst kind of test result: indistinguishable from success, and
load-bearing for exactly the work it fails to cover. e024cfb3f recorded this
same failure once already ("the Atlas Global goldens have been measuring
nothing"); it recurred because nothing checked the property, only the file.

tooling/visual-blank-check measures the share of the frame taken by its single
most common colour. On this project's real captures the classes are far apart:

  Global (real world map)     38.7% modal
  Region (flat colour wash)    7.2% modal   <- dither; least uniform of all
  District                    45.4% modal
  Block / Chunk / Quarter     92.9-94.6% modal   <- nothing drawn

Nothing falls between 45% and 93%, so the 0.85 default sits in open space
rather than being tuned against a boundary case. Deliberately NOT an aesthetic
judgement: the Region wash is a real product gap (T-1213) and scores 7.2%,
comfortably "content". The question is only whether a world reached the
screen.

Wired into both paths, and the update path is the one that matters — refusing
to RECORD a blank golden is what stops the trap being re-armed. Ad-hoc
--screenshot only warns, since capturing a rung that renders nothing is a
legitimate thing to want to do; that is how the empty deep rungs were found.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-08-07 09:12:12 +02:00
co-authored by Claude Opus 5
parent aeab41555a
commit 6147529fe8
2 changed files with 131 additions and 0 deletions
+105
View File
@@ -0,0 +1,105 @@
#!/usr/bin/env python3
"""Reject a screen capture in which the renderer drew nothing.
WHY THIS EXISTS (2026-08-06). `tests/run-visual` verified only that the
captured PNG was non-empty *as a file*. A capture of a blank screen is a
perfectly valid ~19 KB PNG, so it passed — and then, because the golden had
been recorded from an equally blank capture, `visual-diff` reported 0.0%
difference and the scenario PASSED. Two Atlas scenarios
(`atlas_GJ338Bd_Block`, `atlas_GJ445c-m1_Chunk`) sat green for months against
blank goldens. This is the same failure `e024cfb3f` recorded once already
("the Atlas Global goldens have been measuring nothing"); it kept recurring
because nothing checked the property, only the file.
A green comparison between two blank images is the worst kind of test result:
it is indistinguishable from success and it is load-bearing for exactly the
work it fails to cover.
THE METRIC. Fraction of the frame occupied by the single most common colour.
Measured on this project's real captures:
Global (real world map) 38.7% modal
Region (flat colour wash) 7.2% modal <- dither; least uniform of all
District 45.4% modal
Block / Chunk / Quarter 92.9-94.6% modal <- nothing drawn
Nothing falls between 45% and 93%, so the default 0.85 threshold sits in open
space rather than being tuned to a boundary case.
Note what this deliberately does NOT do: it does not judge whether a render is
GOOD. The Region wash is a real product gap (T-1213), and it scores 7.2% —
comfortably "content". The question here is only "did the renderer put a world
on the screen", never "is the world any good". Keep it that way; a check that
creeps toward aesthetics will start failing legitimate frames and get muted.
Usage:
tooling/visual-blank-check <PNG> [--max-modal 0.85] [--quiet]
Exit: 0 = has content, 1 = blank, 2 = error (unreadable/missing).
"""
import sys
from pathlib import Path
DEFAULT_MAX_MODAL = 0.85
def modal_fraction(path: Path) -> tuple[float, int]:
"""Return (modal colour's share of the frame, distinct colour count)."""
try:
from PIL import Image
except ImportError:
print("visual-blank-check: Pillow not available", file=sys.stderr)
raise SystemExit(2)
with Image.open(path) as im:
rgb = im.convert("RGB")
total = rgb.size[0] * rgb.size[1]
if total == 0:
return 1.0, 0
# maxcolors high enough that a real frame is never bucketed away;
# getcolors returns None if it overflows, which itself means the frame
# is richly coloured and therefore certainly not blank.
colors = rgb.getcolors(maxcolors=1 << 24)
if colors is None:
return 0.0, 1 << 24
top = max(colors)[0]
return top / total, len(colors)
def main(argv: list[str]) -> int:
args = [a for a in argv if not a.startswith("--")]
if not args:
print(__doc__, file=sys.stderr)
return 2
max_modal = DEFAULT_MAX_MODAL
for i, a in enumerate(argv):
if a == "--max-modal" and i + 1 < len(argv):
max_modal = float(argv[i + 1])
quiet = "--quiet" in argv
path = Path(args[0])
if not path.is_file():
print(f"visual-blank-check: no such file: {path}", file=sys.stderr)
return 2
modal, distinct = modal_fraction(path)
if modal > max_modal:
print(
f"BLANK: {path.name} is {modal:.1%} a single colour "
f"({distinct} distinct) — the renderer drew nothing but chrome. "
f"A golden recorded from this would pass against any other blank "
f"capture forever.",
file=sys.stderr,
)
return 1
if not quiet:
print(f"content: {path.name} modal={modal:.1%} distinct={distinct}")
return 0
if __name__ == "__main__":
raise SystemExit(main(sys.argv[1:]))