ci: run the pytest suite as four parallel shards

The suite is 12.1k tests in a single CI job — nearly six minutes of pytest
that every push and every PR waits on in one block, on top of a setup step
that already installs npm, Playwright, FFmpeg and bubblewrap. Split it into
four sections the matrix runs in parallel: 352s becomes a 98s longest pole
locally.

Shards partition by test *file*, not by the area_* taxonomy markers. Those
markers do not partition the suite - a file can carry a hand-applied area_*
mark on top of the one conftest derives from its filename, so a marker-based
split would run those tests in more than one section. Assignment is a total
function of the file path instead, so every file lands in exactly one shard
and the four together run every test exactly once.

Sharding deselects rather than narrowing collection, so every test module is
still imported, in the same order, in every shard. The import-time stubbing
in conftest and the session-scoped static server behave identically whether
the suite runs whole or in sections - this suite has known collection-order
coupling and splitting by path would have walked into it.

Balance uses the existing `slow` marker as its weight signal rather than a
committed duration table that would go stale unnoticed. Files pack
heaviest-first into the lightest shard, which is deterministic for a given
file set, so every parallel job computes the same plan from the same commit.

Verified: the four shards together reproduce the full run exactly - 12141
tests selected across the four, and the same 59 failures, 78 skips and 2
xfails, by node ID and not merely by count.
This commit is contained in:
Léo
2026-10-02 18:42:10 +02:00
parent 08c2dc881e
commit 2bda788311
5 changed files with 485 additions and 6 deletions
+33
View File
@@ -93,6 +93,39 @@ fast lane; the test stays runnable directly, e.g.:
./venv/bin/python -m pytest -m slow
```
## Parallel shards (`--shard N/M`)
CI no longer runs the whole suite as one workload. The `python-tests` job is a
four-way matrix, and each job runs one section:
```bash
./venv/bin/python -m pytest -q --shard 1/4
```
`tests/_shards.py` owns the partition and `tests/conftest.py` applies it. The
unit of a shard is a **test file**, so tests that share module state stay
together, and assignment is a total function of the file path - every file
lands in exactly one shard, and the four shards together run every test exactly
once. The partition is deliberately *not* built on the `area_*` markers: those
do not partition the suite, because a file may carry a hand-applied `area_*`
mark on top of the one derived from its filename.
Sharding deselects; it does not narrow collection. Every test module is still
imported, in the same order, in every shard, so the import-time stubbing in
`conftest.py` behaves identically whether the suite runs whole or in sections.
Only the deselected tests' call phase is skipped.
Balance comes from the `slow` marker: a `slow` item is weighted far above an
ordinary one, and files are packed heaviest-first into the lightest shard. The
plan depends only on the collected file set, so every parallel job computes the
same one from the same commit. As more tests earn a `slow` mark from duration
evidence, the sections even out further - no duration table to keep current.
`--shard 1/1` is a no-op, and a selector that is malformed or out of range ends
the run with a usage error rather than quietly testing a subset. If you change
the shard count, change `DEFAULT_SHARD_COUNT` and the `ci.yml` matrix together;
`tests/test_shards.py` fails when they drift apart.
## Order-sensitivity reporting (report-only)
`tests/run_order_report.py` runs pytest with the collected test items shuffled