mirror of
https://github.com/pewdiepie-archdaemon/odysseus.git
synced 2026-10-29 17:22:25 +01:00
docs(tests): record the known full-suite failures
specs/testing-devops.md lists "no canonical full-suite known-failing/flaky ledger" as a gap. Without one a first local run is uninterpretable: you cannot tell a regression from a platform artifact, so you either chase a non-bug or ignore a real one. Six failures on macOS against lab@c499c01b, each with its cause and a verdict rather than a blanket "environmental": - three compare an unresolved /tmp path against a resolved /private/tmp one. Those are test bugs and the file says so. - one asserts ffmpeg exit 0 for a .webp still, which is a build option Homebrew does not always carry. Needs a skip or a PNG fallback. - one opens real sockets and needs a fast connection refusal. Environmental. - one Playwright colour-contrast test had been written off as a flake. It is not: three consecutive runs failed identically at ~31s. Recorded as unexplained and possibly a real defect, because calling it noise is what stopped anyone looking. Also documents the prerequisites, since most surprise failures are a missing npm ci rather than anything here, and the CHROMADB_PORT precaution: the client reaches Chroma over HTTP regardless of the data directory, so a test run can attach to a store holding real data.
This commit is contained in:
@@ -33,6 +33,8 @@ the sub-area. The `area_*` names are registered in `pyproject.toml`; the dynamic
|
||||
`sub_*` names are registered before collection by `pytest_configure` in
|
||||
`tests/conftest.py`, so unknown-mark warnings still flag genuine typos.
|
||||
|
||||
The full suite does not come back clean on every machine. [KNOWN_FAILURES.md](KNOWN_FAILURES.md) lists which failures are expected, which are test bugs worth fixing, and the prerequisites a clean run needs; anything not on that list is a regression until shown otherwise.
|
||||
|
||||
For common focused runs, use `tests/run_focus.py`. It validates area and
|
||||
sub-area names, accepts sub-areas with or without the `sub_` prefix, and passes
|
||||
extra pytest arguments after `--`:
|
||||
|
||||
Reference in New Issue
Block a user