fix(config): run-godot reported a suite that never ran as a pass

Found by walking into it. test_step_canvas_annotation_layer.gd had a parse
error from an earlier edit in this session, so gdUnit4 could not load it and
ran the other suites instead. The harness printed 3610 passed / 0 failed and
exit 0. Fifty tests had not run for hours and nothing said so — the full
suite reports 3660 with the file repaired, and that difference was invisible.

Two states are now hard harness failures rather than test results:

  load_error — a suite failed to LOAD. Any pass count excludes it, so a green
  number is a lie. The hint names the offending file.

  no_tests  — zero tests executed. A run that executes nothing can never be
  a pass; previously a mistyped --filter printed "Tests passed".

Both add a "harness_error" field to the summary JSON and exit 2. The exit
code cannot inherit gdUnit4's, which returns 0 in both states — that is
precisely why they were invisible.

Verified by injecting each failure rather than by reasoning about it. The
load_error guard was checked in the case that actually matters: one broken
file among many, where total stays large and failed stays zero. That run now
reports 3610/0 WITH harness_error and exits 2, where before it was
indistinguishable from success.

Also repairs the file itself: a missed set_frame() argument (the parse error),
and a cell-placement test still asserting pre-inversion spacing. Rewritten to
assert the invariant that survives the extent inversion, the viewport aspect
ratio and panning — half the SHORT axis is half a rung cell — instead of a
literal. Two things it deliberately does not assert, both of which the
previous version got wrong: "the corner is half a district away" holds only
on a square canvas, and the canvas is one district WIDE without sitting ON a
district. It is a free-floating window centred wherever the player panned;
zoom is stepped, pan is continuous. A rung names a scale, not a cell you are
inside. A second test pins that with a deliberately unaligned world centre,
so a future change that snaps the canvas to the rung lattice — making pan
step instead of slide — fails here.

Pair session with Jeroen, 2026-07-27.

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
2026-07-27 00:12:26 +02:00
co-authored by Claude
parent d58397c59f
commit b2ef73256a
3 changed files with 112 additions and 8 deletions
+46
View File
@@ -106,12 +106,34 @@ if [[ "$TOTAL" -eq 0 ]]; then
fi
fi
# --- Harness-integrity guards (2026-07-26) -------------------------------
# A test file that fails to PARSE never runs, and gdUnit4 reports whatever
# DID run as a clean pass — so a broken suite reads as success. Both halves
# of that bit on the same day:
# * a filtered run printed {"total":0,...} followed by "Tests passed";
# * a full run printed 3610 passed / 0 failed while silently dropping an
# entire suite whose parse error had gone unnoticed for hours.
# Neither is a test RESULT, so neither may be reported as one. A run that
# executed zero tests is never a pass, and a run that could not load a suite
# is a harness failure regardless of how many other suites went green.
LOAD_ERROR_COUNT=$(grep -c 'Failed to load script' "$LOG_FILE" 2>/dev/null || true)
LOAD_ERROR_LIST=$(grep -oE 'Failed to load script "[^"]+"' "$LOG_FILE" 2>/dev/null | sort -u || true)
HARNESS_ERROR=""
if [[ "${LOAD_ERROR_COUNT:-0}" -gt 0 ]]; then
HARNESS_ERROR="load_error"
elif [[ "$TOTAL" -eq 0 && "$TIMED_OUT" != "true" ]]; then
HARNESS_ERROR="no_tests"
fi
LOG_LINES=$(wc -l < "$LOG_FILE" 2>/dev/null || echo 0)
# Single-line JSON summary on stdout — machine-parseable, small.
if [[ "$TIMED_OUT" == "true" ]]; then
printf '{"suite":"godot","total":%d,"passed":%d,"failed":%d,"duration_ms":%d,"timeout":true,"timeout_sec":%d,"log":"%s"}\n' \
"${TOTAL:-0}" "${PASSED:-0}" "${FAILED:-0}" "$DURATION_MS" "$TIMEOUT_SEC" "$LOG_FILE"
elif [[ -n "$HARNESS_ERROR" ]]; then
printf '{"suite":"godot","total":%d,"passed":%d,"failed":%d,"duration_ms":%d,"harness_error":"%s","log":"%s"}\n' \
"${TOTAL:-0}" "${PASSED:-0}" "${FAILED:-0}" "$DURATION_MS" "$HARNESS_ERROR" "$LOG_FILE"
else
printf '{"suite":"godot","total":%d,"passed":%d,"failed":%d,"duration_ms":%d,"log":"%s"}\n' \
"${TOTAL:-0}" "${PASSED:-0}" "${FAILED:-0}" "$DURATION_MS" "$LOG_FILE"
@@ -129,6 +151,23 @@ TEST_TIMEOUT: tests/run-godot exceeded ${TIMEOUT_SEC}s wall-clock cap (killed=${
grep -E 'STARTED|PASSED|FAILED' ${LOG_FILE} | tail -20
The last STARTED without a matching PASSED/FAILED is the hang site.
EOF
elif [[ "$HARNESS_ERROR" == "load_error" ]]; then
cat >&2 <<EOF
HARNESS_ERROR: a test suite failed to LOAD — this is not a test result.
${LOAD_ERROR_LIST}
Those suites did not run at all. Any pass count above EXCLUDES them, so a
green number here would be a lie. Usually a parse error (arity change, a
renamed symbol, a stale call signature) in the listed file. Find it with:
grep -n 'Parse Error' ${LOG_FILE} | head
log=${LOG_FILE} (${LOG_LINES} lines)
EOF
elif [[ "$HARNESS_ERROR" == "no_tests" ]]; then
cat >&2 <<EOF
HARNESS_ERROR: zero tests executed — this is not a pass.
filter=${FILTER:-<none>} target=${TEST_TARGET} log=${LOG_FILE} (${LOG_LINES} lines)
Either the filter matched no suite (check the name), or the target failed to
load. A run that executes nothing can never be reported as success.
EOF
elif [[ "${FAILED:-0}" -gt 0 ]]; then
cat >&2 <<EOF
Tests finished with failures. log=${LOG_FILE} (${LOG_LINES} lines)
@@ -141,4 +180,11 @@ else
echo "Tests passed. log=${LOG_FILE} (${LOG_LINES} lines)" >&2
fi
# A harness failure must never inherit gdUnit4's exit code — it exits 0 both
# when a suite fails to load and when the filter matches nothing, which is the
# whole reason these two states were invisible.
if [[ -n "$HARNESS_ERROR" && "$EXIT_CODE" -eq 0 ]]; then
exit 2
fi
exit $EXIT_CODE