test(config): reject captures where the renderer drew nothing
run-visual verified only that the captured PNG was non-empty AS A FILE. A
blank screen is a perfectly valid ~19 KB PNG, so it passed — and once a blank
capture had been recorded as a golden, every later blank capture matched it at
0.0% and the scenario PASSED. atlas_GJ338Bd_Block and atlas_GJ445c-m1_Chunk
sat green against blank goldens while the suite's other 30 scenarios failed.
That is the worst kind of test result: indistinguishable from success, and
load-bearing for exactly the work it fails to cover. e024cfb3f recorded this
same failure once already ("the Atlas Global goldens have been measuring
nothing"); it recurred because nothing checked the property, only the file.
tooling/visual-blank-check measures the share of the frame taken by its single
most common colour. On this project's real captures the classes are far apart:
Global (real world map) 38.7% modal
Region (flat colour wash) 7.2% modal <- dither; least uniform of all
District 45.4% modal
Block / Chunk / Quarter 92.9-94.6% modal <- nothing drawn
Nothing falls between 45% and 93%, so the 0.85 default sits in open space
rather than being tuned against a boundary case. Deliberately NOT an aesthetic
judgement: the Region wash is a real product gap (T-1213) and scores 7.2%,
comfortably "content". The question is only whether a world reached the
screen.
Wired into both paths, and the update path is the one that matters — refusing
to RECORD a blank golden is what stops the trap being re-armed. Ad-hoc
--screenshot only warns, since capturing a rung that renders nothing is a
legitimate thing to want to do; that is how the empty deep rungs were found.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -210,6 +210,11 @@ if [[ "$MODE" == "screenshot" ]]; then
|
||||
PNG="$CACHE_DIR/$TARGET.png"
|
||||
if [[ -f "$PNG" ]]; then
|
||||
echo "Screenshot: $PNG ($(stat -c%s "$PNG" 2>/dev/null || stat -f%z "$PNG") bytes)"
|
||||
# Advisory here rather than fatal — an ad-hoc capture of a rung that
|
||||
# genuinely renders nothing is a legitimate thing to want to look at
|
||||
# (that is how the empty deep rungs were found). But say so out loud,
|
||||
# because file size alone reads as success.
|
||||
"$ROOT/tooling/visual-blank-check" "$PNG" --quiet || true
|
||||
else
|
||||
echo "Error: capture failed — $PNG not found" >&2
|
||||
exit 1
|
||||
@@ -298,6 +303,27 @@ for scenario in "${SCENARIOS[@]}"; do
|
||||
continue
|
||||
fi
|
||||
|
||||
# Verify the renderer actually drew something.
|
||||
#
|
||||
# "Non-empty file" was the only content check until 2026-08-06, and a blank
|
||||
# screen is a perfectly valid ~19 KB PNG. Worse, once a blank capture was
|
||||
# recorded as a golden, every later blank capture matched it at 0.0% and
|
||||
# the scenario PASSED — atlas_GJ338Bd_Block and atlas_GJ445c-m1_Chunk were
|
||||
# green against blank goldens. A test that cannot fail is worse than no
|
||||
# test, because it is counted as coverage.
|
||||
#
|
||||
# Checked in BOTH modes, and the update mode matters most: refusing to
|
||||
# RECORD a blank golden is what stops the trap being re-armed.
|
||||
set +e
|
||||
BLANK_OUT=$("$ROOT/tooling/visual-blank-check" "$CAPTURED" 2>&1)
|
||||
BLANK_RC=$?
|
||||
set -e
|
||||
if [[ $BLANK_RC -ne 0 ]]; then
|
||||
echo " FAIL: $BLANK_OUT" >&2
|
||||
FAILED=$((FAILED + 1))
|
||||
continue
|
||||
fi
|
||||
|
||||
if [[ "$MODE" == "update" ]]; then
|
||||
cp "$CAPTURED" "$GOLDEN_DIR/$scenario.png"
|
||||
echo " Updated golden: $GOLDEN_DIR/$scenario.png"
|
||||
|
||||
Reference in New Issue
Block a user