feat(tooling): two-pass clip discriminator for garment QA (T-1089)

Second garment-only render pass per view at the identical paused animation
time; the analyzer intersects so body-key pixels split into exposed_skin
(no garment behind — informational: collars, sleeveless arms) vs
clip_through (garment behind — gating). Highlights differ: lime exposed,
red clip. Peasant re-run: 72 captures, 56 clip-through flags — real
collar micro-clips under crouch/walk plus suspected 1px boundary
artifacts; gate threshold + garment-mask dilation are the tuning knobs,
to be calibrated against the first real modern garments.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
2026-07-06 14:51:04 +02:00
co-authored by Claude Fable 5
parent eaca6c8c44
commit 74fa16260d
3 changed files with 274 additions and 121 deletions
+24 -12
View File
@@ -1,26 +1,38 @@
# garment-qa — chromakey garment-clipping QA (T-1089)
Automatically detects clothing clip-through. The capture scene paints the body
segments a garment *claims to cover* (coverage.json `hides`) flat **unshaded magenta**,
leaves the garment and uncovered segments (head/hands/etc.) normal, then cycles
animation clips × sampled frames × 4 camera yaws and saves a PNG per view. Any magenta
the camera sees = body poking through cloth. The analyzer counts connected magenta
pixels per capture and flags clips.
Automatically detects clothing clip-through with a **two-pass depth-proximity**
chromakey. Per view (clips × sampled frames × 4 camera yaws), the capture scene renders
the SAME frozen animation frame twice: **pass A** paints the body segments a garment
*claims to cover* (coverage.json `hides`) flat **unshaded magenta**, garment + head/hands
normal; **pass B** is identical except the garment is flat **cyan** and every garment
vertex is nudged `clip_epsilon_m` (default 3 cm) **toward the camera**. A body-key pixel
that is magenta in A but cyan in B means the ε-shifted cloth now covers it — the body sat
within ε *in front of* the cloth = poking through. This separates a true clip from a limb
merely crossing in front of the torso, an open collar, or a bare arm over background
(all of which stay magenta because the cloth behind them is > ε away).
**Run:** `tooling/garment-qa/run-garment-qa [config.json]` (defaults to
`configs/peasant.json`). It launches Godot (`~/bin/godot4`, `opengl3`, `xvfb-run` when
headless) on the capture scene, then runs the analyzer.
**Config** (path passed to the scene via `GARMENT_QA_CONFIG`): `garments`
(`{item_id,slot}` catalogue items, or `{glb,slot,covers[]}` for a raw WIP garment),
(`{item_id,slot}` catalogue items, or `{glb,covers[]}` for a raw WIP garment),
`body_types`, `clips`, `frames_per_clip`, `yaws`, `head_id/hair_id/eyebrow_id/skin_tone`,
`out_dir`.
**Read the report:** `<out_dir>/report.json`per-capture `key_pixels`,
`largest_component`, `fail`, plus a `by_group` (body__clip) roll-up; the same summary
prints to stdout. A capture FAILS when its largest connected magenta blob ≥ `--min-pixels`
(default 8, tolerating AA edges). Failing frames are copied to `<out_dir>/failures/` with
clip pixels recoloured lime and each blob boxed red.
**Read the report:** `<out_dir>/report.json`the same summary prints to stdout. Two
metrics per capture:
- **`clip_through_pixels`** / `largest_clip_component` — key pixels the ε-shifted cloth
covers (skin ≤ ε in front of cloth). **This is the real defect and it gates:** a capture
`fail`s when its largest connected clip blob ≥ `--min-pixels` (default 8, tolerating AA
edges). `clip_through_failures` + the `by_group` `clipfail`/`worstClip` columns roll it up.
- **`exposed_skin_pixels`** — key pixels the shift did NOT cover (skin well in front of any
cloth: collar/sleeve/ankle gaps, a limb crossing the torso). Informational; does not gate.
Tune sensitivity with `clip_epsilon_m` in the config (smaller = only tighter pokes count)
and `--min-pixels` on the analyzer. Failing/flagged frames are copied to
`<out_dir>/failures/*_HL.png` with **exposed_skin lime**, **clip_through filled red**, and
each clip blob boxed.
**Note:** the Godot scene lives at `client/tools/garment_qa/chromakey_scene.gd` (not here)
because Godot `res://` paths cannot leave the client project root.
+115 -54
View File
@@ -1,16 +1,24 @@
#!/usr/bin/env python3
"""Chromakey garment-clipping analyzer (T-1089).
"""Chromakey garment-clipping analyzer (T-1089, two-pass).
Counts key-colour (pure magenta) pixels in each capture PNG produced by
client/tools/garment_qa/chromakey_scene.gd. Any solid patch of key pixels showing
through a garment is a body-clip-through: the QA scene painted the covered body
segments flat magenta, so magenta the camera can see = body poking through cloth.
The capture scene writes two PNGs per view at the identical (paused) animation frame:
<stem>.png pass A — covered body segments flat magenta, garment normal.
<stem>__shift.png pass B — same, but the garment is flat CYAN and nudged
CLIP_EPSILON metres toward the camera (a view-space depth bias).
A capture FAILS when its largest connected key-pixel component is >= --min-pixels
(default 8; a small tolerance for anti-aliased edges). For each failing capture a
highlighted copy is written to <out_dir>/failures/ with the clip pixels recoloured
lime and each component boxed in red. Results land in <out_dir>/report.json plus a
one-screen summary on stdout.
A body-key pixel that is magenta in A but CYAN in B means the epsilon-shifted garment
now covers it — the body sat within epsilon in front of the cloth. Intersecting the two
separates depth-proximate clip-through from mere overlap:
clip_through — magenta in A AND cyan in B: skin <= epsilon in front of cloth = poking
through. The true defect. Largest connected blob >= --min-pixels
(default 8, tolerating AA edges) FAILS the capture.
exposed_skin — magenta in A AND still magenta in B: skin well in front of any cloth
(a limb crossing the torso, an open collar, a bare arm over background).
Informational only; does not gate.
For each capture a highlighted copy lands in <out_dir>/failures/ with exposed_skin
recoloured lime and clip_through filled red (+ red box per clip blob). Results land in
<out_dir>/report.json plus a one-screen summary on stdout.
Reads the same JSON config as the capture scene (via --config) to locate out_dir,
or takes --dir directly.
@@ -26,25 +34,51 @@ from pathlib import Path
from PIL import Image, ImageChops, ImageDraw
# Key-colour gate. Pure magenta is (255, 0, 255); the blue channel is the decisive
# discriminator — skin/garment texture is never simultaneously high-red, low-green
# AND high-blue, so this never fires on legitimate body or cloth pixels.
R_MIN = 200
G_MAX = 60
B_MIN = 200
# Key-colour gate (pass A). Pure magenta is (255, 0, 255); the blue channel is the
# decisive discriminator — skin/garment texture is never simultaneously high-red,
# low-green AND high-blue, so this never fires on legitimate body or cloth pixels.
KEY_R_MIN = 200
KEY_G_MAX = 60
KEY_B_MIN = 200
# Shifted-garment gate (pass B). Flat cyan is (0, 255, 255); low red + high green/blue
# isolates the shifted cloth from magenta body-key, skin, and the dark background.
CYAN_R_MAX = 60
CYAN_G_MIN = 190
CYAN_B_MIN = 190
def _threshold(band: Image.Image, lo: int | None, hi: int | None) -> Image.Image:
def f(v: int) -> int:
if lo is not None and v < lo:
return 0
if hi is not None and v > hi:
return 0
return 255
return band.point(f)
def build_key_mask(img: Image.Image) -> Image.Image:
"""Return an "L" mask, 255 where the pixel is key-colour, else 0."""
""""L" mask, 255 where the pass-A pixel is key-colour (magenta), else 0."""
r, g, b = img.convert("RGB").split()
r_ok = r.point(lambda v: 255 if v >= R_MIN else 0)
g_ok = g.point(lambda v: 255 if v <= G_MAX else 0)
b_ok = b.point(lambda v: 255 if v >= B_MIN else 0)
r_ok = _threshold(r, KEY_R_MIN, None)
g_ok = _threshold(g, None, KEY_G_MAX)
b_ok = _threshold(b, KEY_B_MIN, None)
return ImageChops.multiply(ImageChops.multiply(r_ok, g_ok), b_ok)
def build_cyan_mask(img: Image.Image) -> Image.Image:
""""L" mask, 255 where the pass-B pixel is the shifted flat-cyan garment, else 0."""
r, g, b = img.convert("RGB").split()
r_ok = _threshold(r, None, CYAN_R_MAX)
g_ok = _threshold(g, CYAN_G_MIN, None)
b_ok = _threshold(b, CYAN_B_MIN, None)
return ImageChops.multiply(ImageChops.multiply(r_ok, g_ok), b_ok)
def connected_components(mask: Image.Image) -> list[dict]:
"""8-connected components of the key-pixel mask.
"""8-connected components of a 0/255 mask.
Only the mask bounding box is scanned, so cost tracks the (sparse) clip area,
not the whole frame. Returns one dict per component: size + pixel bbox.
@@ -83,7 +117,6 @@ def connected_components(mask: Image.Image) -> list[dict]:
components.append(
{
"size": size,
# bbox back in full-image coordinates
"bbox": [x0 + min_x, y0 + min_y, x0 + max_x + 1, y0 + max_y + 1],
}
)
@@ -91,39 +124,59 @@ def connected_components(mask: Image.Image) -> list[dict]:
return components
def write_highlight(img: Image.Image, mask: Image.Image, comps: list[dict], dest: Path) -> None:
"""Recolour key pixels lime and box each component in red."""
def write_highlight(
img: Image.Image, exposed: Image.Image, clip: Image.Image, comps: list[dict], dest: Path
) -> None:
"""Recolour exposed_skin lime, fill clip_through red, box each clip blob."""
out = img.convert("RGB")
out.paste((0, 255, 0), mask=mask)
out.paste((0, 255, 0), mask=exposed)
out.paste((255, 0, 0), mask=clip)
draw = ImageDraw.Draw(out)
for comp in comps:
draw.rectangle(comp["bbox"], outline=(255, 0, 0), width=2)
draw.rectangle(comp["bbox"], outline=(255, 80, 80), width=2)
dest.parent.mkdir(parents=True, exist_ok=True)
out.save(dest)
def analyze_dir(out_dir: Path, min_pixels: int) -> dict:
captures = sorted(p for p in out_dir.glob("*.png") if p.is_file())
captures = sorted(
p for p in out_dir.glob("*.png") if p.is_file() and not p.stem.endswith("__shift")
)
fail_dir = out_dir / "failures"
results: list[dict] = []
for path in captures:
shift_path = path.with_name(path.stem + "__shift.png")
img = Image.open(path)
mask = build_key_mask(img)
total = mask.histogram()[255]
comps = connected_components(mask) if total else []
key = build_key_mask(img)
if shift_path.exists():
cyan = build_cyan_mask(Image.open(shift_path))
clip = ImageChops.multiply(key, cyan)
exposed = ImageChops.multiply(key, ImageChops.invert(cyan))
shift_missing = False
else:
# No shift pass — cannot classify; treat all key as exposed, clip empty.
clip = Image.new("L", img.size, 0)
exposed = key
shift_missing = True
comps = connected_components(clip)
largest = comps[0]["size"] if comps else 0
failed = largest >= min_pixels
if failed:
write_highlight(img, mask, comps, fail_dir / (path.stem + "_HL.png"))
if failed or exposed.getbbox() is not None:
write_highlight(img, exposed, clip, comps, fail_dir / (path.stem + "_HL.png"))
results.append(
{
"file": path.name,
"key_pixels": total,
"components": len(comps),
"largest_component": largest,
"component_sizes": [c["size"] for c in comps[:10]],
"key_pixels": key.histogram()[255],
"exposed_skin_pixels": exposed.histogram()[255],
"clip_through_pixels": clip.histogram()[255],
"clip_components": len(comps),
"largest_clip_component": largest,
"clip_component_sizes": [c["size"] for c in comps[:10]],
"fail": failed,
"shift_pass_missing": shift_missing,
}
)
@@ -137,33 +190,41 @@ def summarize(out_dir: Path, min_pixels: int, results: list[dict]) -> dict:
# filename: <body>__<clip>__f<n>__yaw<deg>.png
parts = r["file"].split("__")
group = "__".join(parts[:2]) if len(parts) >= 2 else r["file"]
entry = by_group.setdefault(group, {"captures": 0, "failures": 0, "worst": 0})
entry = by_group.setdefault(
group, {"captures": 0, "clip_failures": 0, "worst_clip": 0, "worst_exposed": 0}
)
entry["captures"] += 1
entry["failures"] += 1 if r["fail"] else 0
entry["worst"] = max(entry["worst"], r["largest_component"])
entry["clip_failures"] += 1 if r["fail"] else 0
entry["worst_clip"] = max(entry["worst_clip"], r["largest_clip_component"])
entry["worst_exposed"] = max(entry["worst_exposed"], r["exposed_skin_pixels"])
return {
"out_dir": str(out_dir),
"min_component_pixels": min_pixels,
"key_gate": {"r_min": R_MIN, "g_max": G_MAX, "b_min": B_MIN},
"key_gate": {"r_min": KEY_R_MIN, "g_max": KEY_G_MAX, "b_min": KEY_B_MIN},
"shift_gate": {"r_max": CYAN_R_MAX, "g_min": CYAN_G_MIN, "b_min": CYAN_B_MIN},
"total_captures": len(results),
"failures": len(failures),
"clip_through_failures": len(failures),
"by_group": by_group,
"captures": results,
}
def print_summary(report: dict) -> None:
print("=" * 64)
print("garment-qa chromakey analysis")
print(f" out_dir : {report['out_dir']}")
print(f" min clip pixels : {report['min_component_pixels']}")
print(f" captures : {report['total_captures']}")
print(f" FAILING captures : {report['failures']}")
print("-" * 64)
print(f" {'group (body__clip)':<28}{'caps':>6}{'fails':>7}{'worst':>7}")
print("=" * 74)
print("garment-qa chromakey analysis (two-pass: clip_through gates, exposed_skin info)")
print(f" out_dir : {report['out_dir']}")
print(f" min clip blob pixels : {report['min_component_pixels']}")
print(f" captures : {report['total_captures']}")
print(f" CLIP-THROUGH failures: {report['clip_through_failures']}")
print("-" * 74)
hdr = f" {'group (body__clip)':<26}{'caps':>6}{'clipfail':>10}{'worstClip':>11}{'worstExp':>10}"
print(hdr)
for group, g in sorted(report["by_group"].items()):
print(f" {group:<28}{g['captures']:>6}{g['failures']:>7}{g['worst']:>7}")
print("=" * 64)
print(
f" {group:<26}{g['captures']:>6}{g['clip_failures']:>10}"
f"{g['worst_clip']:>11}{g['worst_exposed']:>10}"
)
print("=" * 74)
def resolve_out_dir(args: argparse.Namespace) -> Path:
@@ -186,7 +247,7 @@ def main() -> int:
"--min-pixels",
type=int,
default=8,
help="largest connected key-pixel component that counts as a clip (default 8)",
help="largest connected clip-through blob that counts as a failure (default 8)",
)
parser.add_argument("--report", help="report JSON path (default: <out_dir>/report.json)")
args = parser.parse_args()
@@ -200,8 +261,8 @@ def main() -> int:
report_path.write_text(json.dumps(report, indent=2))
print_summary(report)
print(f" report : {report_path}")
if report["failures"]:
print(f" highlighted failing frames : {out_dir / 'failures'}")
if report["clip_through_failures"]:
print(f" highlighted frames : {out_dir / 'failures'}")
return 0