fix(config): run-godot reported a suite that never ran as a pass
Found by walking into it. test_step_canvas_annotation_layer.gd had a parse error from an earlier edit in this session, so gdUnit4 could not load it and ran the other suites instead. The harness printed 3610 passed / 0 failed and exit 0. Fifty tests had not run for hours and nothing said so — the full suite reports 3660 with the file repaired, and that difference was invisible. Two states are now hard harness failures rather than test results: load_error — a suite failed to LOAD. Any pass count excludes it, so a green number is a lie. The hint names the offending file. no_tests — zero tests executed. A run that executes nothing can never be a pass; previously a mistyped --filter printed "Tests passed". Both add a "harness_error" field to the summary JSON and exit 2. The exit code cannot inherit gdUnit4's, which returns 0 in both states — that is precisely why they were invisible. Verified by injecting each failure rather than by reasoning about it. The load_error guard was checked in the case that actually matters: one broken file among many, where total stays large and failed stays zero. That run now reports 3610/0 WITH harness_error and exits 2, where before it was indistinguishable from success. Also repairs the file itself: a missed set_frame() argument (the parse error), and a cell-placement test still asserting pre-inversion spacing. Rewritten to assert the invariant that survives the extent inversion, the viewport aspect ratio and panning — half the SHORT axis is half a rung cell — instead of a literal. Two things it deliberately does not assert, both of which the previous version got wrong: "the corner is half a district away" holds only on a square canvas, and the canvas is one district WIDE without sitting ON a district. It is a free-floating window centred wherever the player panned; zoom is stepped, pan is continuous. A rung names a scale, not a cell you are inside. A second test pins that with a deliberately unaligned world centre, so a future change that snaps the canvas to the rung lattice — making pan step instead of slide — fails here. Pair session with Jeroen, 2026-07-27. Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -106,12 +106,34 @@ if [[ "$TOTAL" -eq 0 ]]; then
|
||||
fi
|
||||
fi
|
||||
|
||||
# --- Harness-integrity guards (2026-07-26) -------------------------------
|
||||
# A test file that fails to PARSE never runs, and gdUnit4 reports whatever
|
||||
# DID run as a clean pass — so a broken suite reads as success. Both halves
|
||||
# of that bit on the same day:
|
||||
# * a filtered run printed {"total":0,...} followed by "Tests passed";
|
||||
# * a full run printed 3610 passed / 0 failed while silently dropping an
|
||||
# entire suite whose parse error had gone unnoticed for hours.
|
||||
# Neither is a test RESULT, so neither may be reported as one. A run that
|
||||
# executed zero tests is never a pass, and a run that could not load a suite
|
||||
# is a harness failure regardless of how many other suites went green.
|
||||
LOAD_ERROR_COUNT=$(grep -c 'Failed to load script' "$LOG_FILE" 2>/dev/null || true)
|
||||
LOAD_ERROR_LIST=$(grep -oE 'Failed to load script "[^"]+"' "$LOG_FILE" 2>/dev/null | sort -u || true)
|
||||
HARNESS_ERROR=""
|
||||
if [[ "${LOAD_ERROR_COUNT:-0}" -gt 0 ]]; then
|
||||
HARNESS_ERROR="load_error"
|
||||
elif [[ "$TOTAL" -eq 0 && "$TIMED_OUT" != "true" ]]; then
|
||||
HARNESS_ERROR="no_tests"
|
||||
fi
|
||||
|
||||
LOG_LINES=$(wc -l < "$LOG_FILE" 2>/dev/null || echo 0)
|
||||
|
||||
# Single-line JSON summary on stdout — machine-parseable, small.
|
||||
if [[ "$TIMED_OUT" == "true" ]]; then
|
||||
printf '{"suite":"godot","total":%d,"passed":%d,"failed":%d,"duration_ms":%d,"timeout":true,"timeout_sec":%d,"log":"%s"}\n' \
|
||||
"${TOTAL:-0}" "${PASSED:-0}" "${FAILED:-0}" "$DURATION_MS" "$TIMEOUT_SEC" "$LOG_FILE"
|
||||
elif [[ -n "$HARNESS_ERROR" ]]; then
|
||||
printf '{"suite":"godot","total":%d,"passed":%d,"failed":%d,"duration_ms":%d,"harness_error":"%s","log":"%s"}\n' \
|
||||
"${TOTAL:-0}" "${PASSED:-0}" "${FAILED:-0}" "$DURATION_MS" "$HARNESS_ERROR" "$LOG_FILE"
|
||||
else
|
||||
printf '{"suite":"godot","total":%d,"passed":%d,"failed":%d,"duration_ms":%d,"log":"%s"}\n' \
|
||||
"${TOTAL:-0}" "${PASSED:-0}" "${FAILED:-0}" "$DURATION_MS" "$LOG_FILE"
|
||||
@@ -129,6 +151,23 @@ TEST_TIMEOUT: tests/run-godot exceeded ${TIMEOUT_SEC}s wall-clock cap (killed=${
|
||||
grep -E 'STARTED|PASSED|FAILED' ${LOG_FILE} | tail -20
|
||||
The last STARTED without a matching PASSED/FAILED is the hang site.
|
||||
EOF
|
||||
elif [[ "$HARNESS_ERROR" == "load_error" ]]; then
|
||||
cat >&2 <<EOF
|
||||
HARNESS_ERROR: a test suite failed to LOAD — this is not a test result.
|
||||
${LOAD_ERROR_LIST}
|
||||
Those suites did not run at all. Any pass count above EXCLUDES them, so a
|
||||
green number here would be a lie. Usually a parse error (arity change, a
|
||||
renamed symbol, a stale call signature) in the listed file. Find it with:
|
||||
grep -n 'Parse Error' ${LOG_FILE} | head
|
||||
log=${LOG_FILE} (${LOG_LINES} lines)
|
||||
EOF
|
||||
elif [[ "$HARNESS_ERROR" == "no_tests" ]]; then
|
||||
cat >&2 <<EOF
|
||||
HARNESS_ERROR: zero tests executed — this is not a pass.
|
||||
filter=${FILTER:-<none>} target=${TEST_TARGET} log=${LOG_FILE} (${LOG_LINES} lines)
|
||||
Either the filter matched no suite (check the name), or the target failed to
|
||||
load. A run that executes nothing can never be reported as success.
|
||||
EOF
|
||||
elif [[ "${FAILED:-0}" -gt 0 ]]; then
|
||||
cat >&2 <<EOF
|
||||
Tests finished with failures. log=${LOG_FILE} (${LOG_LINES} lines)
|
||||
@@ -141,4 +180,11 @@ else
|
||||
echo "Tests passed. log=${LOG_FILE} (${LOG_LINES} lines)" >&2
|
||||
fi
|
||||
|
||||
# A harness failure must never inherit gdUnit4's exit code — it exits 0 both
|
||||
# when a suite fails to load and when the filter matches nothing, which is the
|
||||
# whole reason these two states were invisible.
|
||||
if [[ -n "$HARNESS_ERROR" && "$EXIT_CODE" -eq 0 ]]; then
|
||||
exit 2
|
||||
fi
|
||||
|
||||
exit $EXIT_CODE
|
||||
|
||||
Reference in New Issue
Block a user