feat(tooling): gemma_naming.py batch naming pipeline for atlas (#833)
New end-to-end pipeline that walks every markers.json in the reach and
fills empty `name` fields using the Gemma 2 voice pipeline via
`sr-voice serve --stdio`. Per D-191 §4: the same Gemma 2 pipeline the
client uses for NPC voicing also produces the atlas content, which is
dual-purposed as a quality test of the LLM plumbing.
Pipeline per body (hop-ordered, core-first):
1. Load markers.json; identify feature records whose `name` is
blank (null or ""). Hand-authored names are never overwritten;
the 6 template bodies and any partial authoring stay put.
2. Look up body context (planet_class, settlement_pattern,
cultural_corridor, population, economic_role) from systems.db.
3. Build a short corridor-aware few-shot prompt per feature type.
Prompts carry 3 concrete `Style: X. Answer: Y` examples so
Gemma 2 2B completes a pattern instead of generating to an
open-ended instruction — this is the single biggest lever
against placeholder echoes on a small model.
4. Stream the prompt into a long-lived sr-voice subprocess, read
the JSONL response, post-process (strip markdown, label
prefixes, brackets, reject 5+ word outputs and placeholder
tokens), check the earth-name blocklist, check per-(corridor,
feature_type) + per-body dedup, check the per-stem cap, retry
up to 3 times with a bumped seed.
5. On persistent failure, fall back to a deterministic palette
generator so every feature ends up with a name.
6. Write markers.json atomically and refresh atlas_* DB rows via
sync_markers_to_db. Commit the DB per body so a crash loses
at most one body of state.
7. Restart the sr-voice subprocess every `--refresh` requests
(default: 200) to prevent KV-cache context bleed.
Core design decisions:
- Determinism: per-(world_seed, body_id, feature_local_id, attempt)
seed so the full run is reproducible.
- Ordering: bodies are processed in ascending `hop_distance_from_gateway`
so core bodies get first pick at every unique Gemma output and
outer sectors fall into the palette fallback when they lose the
dedup race.
- Dedup scope: (cultural_corridor, feature_type) across the run,
PLUS a per-body cross-type set so the same name can't be a river
AND an ocean AND a mountain on the same world. Hand-authored names
are seeded into both sets on load so templates win priority.
- Stem cap: each non-generic root token (e.g. 'Arcturus', 'Meridian')
may appear at most `--stem-cap` times across the full run (default
20), preventing single-word runaway. Fallback names bypass the cap.
- Earth blocklist: 181 curated entries covering major Earth cities,
mountains, rivers, oceans, historical/colonial spellings, and
Greek/Roman mythology that reads too literally. Prefixed variants
('Nouveau Paris', 'New Tokyo') explicitly allowed per the product
intent that Earth-echo names are fine but must not dominate.
Leading 'The ' is stripped before comparison so 'The Great Divide'
also matches.
Operational features:
- `--shard N/M` slices the body list into M partitions for parallel
runs. Two terminals × `--shard 0/2` + `--shard 1/2` fits the
~2.5 GB/instance VRAM footprint twice under the 50% cap on a
16 GB AMD GPU and roughly halves wall time.
- `--log PATH` writes a timestamped tee of every status line to a
file. Default: `.tmp/gemma_naming.shard{N}of{M}.log` when a
non-trivial shard is in use.
- SQLite `PRAGMA journal_mode=WAL` + `busy_timeout=15000` so two
concurrent shards serialize writes without lock errors.
- Per-body progress lines report `body K/N`, `sys K/N`, and
`hop=H` so the user can watch core sectors finish first.
- Each body logs the new names it produced per feature type so the
user can eyeball quality as the run progresses.
- Checkpoint summary every 25 bodies: cumulative names, rate,
ETA — gives the log regular scroll points.
- `--mock` uses `server/sr-voice/mock-stdio.sh` for dry-fire
pipeline validation without a model load (tested end-to-end).
Supporting files:
- `tooling/planet-gen/earth_blocklist.txt` — 181 curated entries.
- `tooling/db/backfill_cultural_corridor.py` — one-off migration
that fills the `cultural_corridor` column on both `star_systems`
and `bodies` from the `geographic_sector` values. Before this
pass, 99.4% of rows (3221/3240) had a NULL cultural_corridor
despite `wiki_sync.py` being aware of the column — the wiki
index.md files only carry the sector header, which was never
propagated to the DB column. Idempotent, safe to re-run after
any wiki_sync rebuild, explicit transaction wrapper with
rollback on failure.
Full batch runtime estimate: ~20 hours single-shard / ~10 hours
double-shard on this hardware. Smoke tests across five hardened
iterations (v1–v5) on GJ71b/c/d/d-1/e confirm the pipeline produces
clean, varied, culturally-coherent names with zero post-processing
residue.
This commit is contained in:
Binary file not shown.
Executable
+162
@@ -0,0 +1,162 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
Backfill star_systems.cultural_corridor and bodies.cultural_corridor.
|
||||||
|
|
||||||
|
The schema has a `cultural_corridor` column on both tables, but
|
||||||
|
`wiki_sync.py` never populated it from the wiki index.md files — the
|
||||||
|
cultural/geographic identity of each system lives in
|
||||||
|
`star_systems.geographic_sector` instead (values: core, north_reach,
|
||||||
|
south_reach, east_reach, west_reach, deep_frontier). These two fields
|
||||||
|
refer to the same concept: which arc of the reach the system belongs
|
||||||
|
to. Leaving `cultural_corridor` NULL on 99%+ of rows defeats every
|
||||||
|
downstream consumer that actually wants to filter by corridor
|
||||||
|
(gemma_naming.py, future atlas UI queries, narrative tools).
|
||||||
|
|
||||||
|
This script treats `geographic_sector` as the source of truth and
|
||||||
|
copies it into `cultural_corridor`:
|
||||||
|
|
||||||
|
star_systems.cultural_corridor := star_systems.geographic_sector
|
||||||
|
WHERE cultural_corridor IS NULL
|
||||||
|
|
||||||
|
bodies.cultural_corridor := parent star_systems.cultural_corridor
|
||||||
|
WHERE bodies.cultural_corridor IS NULL
|
||||||
|
|
||||||
|
It is safe to re-run — idempotent, NULL-only updates, explicit
|
||||||
|
transaction wrapper so a crash never leaves a half-populated state.
|
||||||
|
Run it after any `wiki_sync.py` pass that creates fresh systems.db
|
||||||
|
rows.
|
||||||
|
|
||||||
|
Usage:
|
||||||
|
tooling/db/backfill_cultural_corridor.py
|
||||||
|
tooling/db/backfill_cultural_corridor.py --db path/to/systems.db
|
||||||
|
tooling/db/backfill_cultural_corridor.py --dry-run
|
||||||
|
|
||||||
|
Decisions: D-191 (atlas pipeline — downstream consumer)
|
||||||
|
"""
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import sqlite3
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
SCRIPT_DIR = Path(__file__).resolve().parent
|
||||||
|
REPO_ROOT = (SCRIPT_DIR / ".." / "..").resolve()
|
||||||
|
DB_PATH = REPO_ROOT / "server" / "data" / "systems.db"
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
parser = argparse.ArgumentParser(
|
||||||
|
description="Backfill cultural_corridor on star_systems and bodies"
|
||||||
|
)
|
||||||
|
parser.add_argument("--db", default=str(DB_PATH), help="Path to systems.db")
|
||||||
|
parser.add_argument(
|
||||||
|
"--dry-run",
|
||||||
|
action="store_true",
|
||||||
|
help="Report counts without writing",
|
||||||
|
)
|
||||||
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
db_path = Path(args.db)
|
||||||
|
if not db_path.exists():
|
||||||
|
print(f"error: {db_path} not found", file=sys.stderr)
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
conn = sqlite3.connect(str(db_path))
|
||||||
|
conn.execute("PRAGMA foreign_keys=ON")
|
||||||
|
|
||||||
|
# Counts before.
|
||||||
|
before_systems_null = conn.execute(
|
||||||
|
"SELECT COUNT(*) FROM star_systems WHERE cultural_corridor IS NULL"
|
||||||
|
).fetchone()[0]
|
||||||
|
before_bodies_null = conn.execute(
|
||||||
|
"SELECT COUNT(*) FROM bodies WHERE cultural_corridor IS NULL"
|
||||||
|
).fetchone()[0]
|
||||||
|
|
||||||
|
print(f"\n cultural_corridor backfill")
|
||||||
|
print(f" DB: {db_path}")
|
||||||
|
if args.dry_run:
|
||||||
|
print(f" Mode: DRY RUN")
|
||||||
|
print()
|
||||||
|
print(f" Before:")
|
||||||
|
print(f" star_systems.cultural_corridor NULL: {before_systems_null}")
|
||||||
|
print(f" bodies.cultural_corridor NULL: {before_bodies_null}")
|
||||||
|
|
||||||
|
conn.execute("BEGIN")
|
||||||
|
try:
|
||||||
|
# 1. Star systems — copy geographic_sector into cultural_corridor
|
||||||
|
# where the latter is still NULL. If geographic_sector is also
|
||||||
|
# NULL, leave cultural_corridor NULL — there is nothing to
|
||||||
|
# copy and a bogus placeholder is worse than honest NULL.
|
||||||
|
sys_rows_updated = conn.execute(
|
||||||
|
"""
|
||||||
|
UPDATE star_systems
|
||||||
|
SET cultural_corridor = geographic_sector
|
||||||
|
WHERE cultural_corridor IS NULL
|
||||||
|
AND geographic_sector IS NOT NULL
|
||||||
|
"""
|
||||||
|
).rowcount
|
||||||
|
|
||||||
|
# 2. Bodies — inherit from the parent star_systems row.
|
||||||
|
body_rows_updated = conn.execute(
|
||||||
|
"""
|
||||||
|
UPDATE bodies
|
||||||
|
SET cultural_corridor = (
|
||||||
|
SELECT s.cultural_corridor
|
||||||
|
FROM star_systems s
|
||||||
|
WHERE s.system_id = bodies.system_id
|
||||||
|
)
|
||||||
|
WHERE cultural_corridor IS NULL
|
||||||
|
AND EXISTS (
|
||||||
|
SELECT 1
|
||||||
|
FROM star_systems s
|
||||||
|
WHERE s.system_id = bodies.system_id
|
||||||
|
AND s.cultural_corridor IS NOT NULL
|
||||||
|
)
|
||||||
|
"""
|
||||||
|
).rowcount
|
||||||
|
|
||||||
|
if args.dry_run:
|
||||||
|
conn.rollback()
|
||||||
|
print()
|
||||||
|
print(f" Would update:")
|
||||||
|
print(f" star_systems: {sys_rows_updated}")
|
||||||
|
print(f" bodies: {body_rows_updated}")
|
||||||
|
print(f"\n Dry run — no changes written.")
|
||||||
|
else:
|
||||||
|
conn.commit()
|
||||||
|
print()
|
||||||
|
print(f" Updated:")
|
||||||
|
print(f" star_systems: {sys_rows_updated}")
|
||||||
|
print(f" bodies: {body_rows_updated}")
|
||||||
|
|
||||||
|
# Counts after.
|
||||||
|
after_systems_null = conn.execute(
|
||||||
|
"SELECT COUNT(*) FROM star_systems WHERE cultural_corridor IS NULL"
|
||||||
|
).fetchone()[0]
|
||||||
|
after_bodies_null = conn.execute(
|
||||||
|
"SELECT COUNT(*) FROM bodies WHERE cultural_corridor IS NULL"
|
||||||
|
).fetchone()[0]
|
||||||
|
print()
|
||||||
|
print(f" After:")
|
||||||
|
print(f" star_systems.cultural_corridor NULL: {after_systems_null}")
|
||||||
|
print(f" bodies.cultural_corridor NULL: {after_bodies_null}")
|
||||||
|
|
||||||
|
# Show the distribution so the outcome is visible.
|
||||||
|
print()
|
||||||
|
print(f" star_systems.cultural_corridor distribution:")
|
||||||
|
for corridor, count in conn.execute(
|
||||||
|
"SELECT cultural_corridor, COUNT(*) FROM star_systems "
|
||||||
|
"GROUP BY cultural_corridor ORDER BY COUNT(*) DESC"
|
||||||
|
).fetchall():
|
||||||
|
print(f" {corridor!r}: {count}")
|
||||||
|
except BaseException:
|
||||||
|
conn.rollback()
|
||||||
|
conn.close()
|
||||||
|
raise
|
||||||
|
|
||||||
|
conn.close()
|
||||||
|
print()
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,216 @@
|
|||||||
|
# Earth-name blocklist for gemma_naming.py (#833, D-191 §4).
|
||||||
|
#
|
||||||
|
# Per memory feedback_earth_echo_names: Earth-sounding names are OK and
|
||||||
|
# expected — the setting is a blended reality. This file is a small,
|
||||||
|
# curated list of Earth *majors* whose raw names should not appear as
|
||||||
|
# settled-reach feature names because they are too on-the-nose. Prefixed
|
||||||
|
# variants like "Nouveau Paris", "Neu Berlin", or "New Tokyo" are allowed
|
||||||
|
# (they pass because the exact-match check does not strip prefixes).
|
||||||
|
#
|
||||||
|
# Match rule: case-insensitive whole-name equality between a generated
|
||||||
|
# feature name and any entry below. One entry per line, `#` comments are
|
||||||
|
# allowed, blank lines ignored.
|
||||||
|
#
|
||||||
|
# Expand this list if the generator starts producing raw-Earth-major
|
||||||
|
# names in frequent runs. Err on the side of short — we do NOT want to
|
||||||
|
# block every vaguely Earthy token.
|
||||||
|
|
||||||
|
# Capitals & megacities
|
||||||
|
Paris
|
||||||
|
London
|
||||||
|
Tokyo
|
||||||
|
Beijing
|
||||||
|
Moscow
|
||||||
|
Berlin
|
||||||
|
Madrid
|
||||||
|
Rome
|
||||||
|
Cairo
|
||||||
|
Mumbai
|
||||||
|
Delhi
|
||||||
|
Jakarta
|
||||||
|
Seoul
|
||||||
|
Bangkok
|
||||||
|
Istanbul
|
||||||
|
Lagos
|
||||||
|
Nairobi
|
||||||
|
Johannesburg
|
||||||
|
Sydney
|
||||||
|
Washington
|
||||||
|
Ottawa
|
||||||
|
Mexico City
|
||||||
|
Rio de Janeiro
|
||||||
|
Rio
|
||||||
|
Buenos Aires
|
||||||
|
Lima
|
||||||
|
Bogota
|
||||||
|
Dhaka
|
||||||
|
Karachi
|
||||||
|
Lahore
|
||||||
|
Tehran
|
||||||
|
Baghdad
|
||||||
|
Riyadh
|
||||||
|
Manila
|
||||||
|
Hanoi
|
||||||
|
Kuala Lumpur
|
||||||
|
Singapore
|
||||||
|
Vienna
|
||||||
|
Prague
|
||||||
|
Warsaw
|
||||||
|
Amsterdam
|
||||||
|
Brussels
|
||||||
|
Copenhagen
|
||||||
|
Stockholm
|
||||||
|
Oslo
|
||||||
|
Helsinki
|
||||||
|
Reykjavik
|
||||||
|
Lisbon
|
||||||
|
Athens
|
||||||
|
Budapest
|
||||||
|
Dublin
|
||||||
|
Edinburgh
|
||||||
|
Glasgow
|
||||||
|
|
||||||
|
# Iconic cities / regional majors
|
||||||
|
New York
|
||||||
|
Los Angeles
|
||||||
|
Chicago
|
||||||
|
Houston
|
||||||
|
Miami
|
||||||
|
Boston
|
||||||
|
Seattle
|
||||||
|
Toronto
|
||||||
|
Vancouver
|
||||||
|
Montreal
|
||||||
|
Barcelona
|
||||||
|
Munich
|
||||||
|
Hamburg
|
||||||
|
Frankfurt
|
||||||
|
Milan
|
||||||
|
Venice
|
||||||
|
Florence
|
||||||
|
Naples
|
||||||
|
Marseille
|
||||||
|
Lyon
|
||||||
|
Geneva
|
||||||
|
Zurich
|
||||||
|
Shanghai
|
||||||
|
Hong Kong
|
||||||
|
Taipei
|
||||||
|
Osaka
|
||||||
|
Kyoto
|
||||||
|
Yokohama
|
||||||
|
Busan
|
||||||
|
Ho Chi Minh
|
||||||
|
Kolkata
|
||||||
|
Chennai
|
||||||
|
Bangalore
|
||||||
|
Hyderabad
|
||||||
|
Addis Ababa
|
||||||
|
Casablanca
|
||||||
|
Accra
|
||||||
|
Kinshasa
|
||||||
|
Cape Town
|
||||||
|
Tel Aviv
|
||||||
|
Dubai
|
||||||
|
Doha
|
||||||
|
Damascus
|
||||||
|
Beirut
|
||||||
|
Jerusalem
|
||||||
|
Auckland
|
||||||
|
Wellington
|
||||||
|
Melbourne
|
||||||
|
Brisbane
|
||||||
|
Perth
|
||||||
|
|
||||||
|
# Mountain / natural majors that read as Earth
|
||||||
|
Everest
|
||||||
|
Kilimanjaro
|
||||||
|
Fuji
|
||||||
|
Matterhorn
|
||||||
|
Olympus
|
||||||
|
Etna
|
||||||
|
Vesuvius
|
||||||
|
Denali
|
||||||
|
Aconcagua
|
||||||
|
Ararat
|
||||||
|
K2
|
||||||
|
Annapurna
|
||||||
|
Elbrus
|
||||||
|
Kosciuszko
|
||||||
|
|
||||||
|
# Rivers
|
||||||
|
Amazon
|
||||||
|
Nile
|
||||||
|
Danube
|
||||||
|
Mississippi
|
||||||
|
Mekong
|
||||||
|
Ganges
|
||||||
|
Yangtze
|
||||||
|
Rhine
|
||||||
|
Volga
|
||||||
|
Thames
|
||||||
|
Seine
|
||||||
|
Tigris
|
||||||
|
Euphrates
|
||||||
|
Congo
|
||||||
|
Niger
|
||||||
|
Zambezi
|
||||||
|
|
||||||
|
# Oceans / seas
|
||||||
|
Atlantic
|
||||||
|
Pacific
|
||||||
|
Indian
|
||||||
|
Arctic
|
||||||
|
Mediterranean
|
||||||
|
Baltic
|
||||||
|
Caspian
|
||||||
|
Aegean
|
||||||
|
Adriatic
|
||||||
|
Caribbean
|
||||||
|
|
||||||
|
# Historical / colonial / alternate spellings of Earth cities — Gemma
|
||||||
|
# knows these from training data and emits them as if they were
|
||||||
|
# neutral names.
|
||||||
|
Calcutta
|
||||||
|
Bombay
|
||||||
|
Madras
|
||||||
|
Bangalore
|
||||||
|
Poona
|
||||||
|
Benares
|
||||||
|
Peking
|
||||||
|
Canton
|
||||||
|
Nanking
|
||||||
|
Chungking
|
||||||
|
Saigon
|
||||||
|
Rangoon
|
||||||
|
Ceylon
|
||||||
|
Batavia
|
||||||
|
Angora
|
||||||
|
Smyrna
|
||||||
|
Constantinople
|
||||||
|
Formosa
|
||||||
|
Tasmania
|
||||||
|
Rhodesia
|
||||||
|
Persia
|
||||||
|
Mesopotamia
|
||||||
|
|
||||||
|
# Earth natural-phenomenon names Gemma likes to reach for
|
||||||
|
Aurora Borealis
|
||||||
|
Aurora Australis
|
||||||
|
Great Divide
|
||||||
|
Great Lakes
|
||||||
|
Great Basin
|
||||||
|
Grand Canyon
|
||||||
|
Death Valley
|
||||||
|
Sahara
|
||||||
|
Gobi
|
||||||
|
Patagonia
|
||||||
|
Serengeti
|
||||||
|
Outback
|
||||||
|
Tundra
|
||||||
|
Siberia
|
||||||
|
Amazon Basin
|
||||||
|
|
||||||
|
# Greek/Roman mythology that reads too literally as Earth classical
|
||||||
|
Olympus Mons
|
||||||
|
Mount Olympus
|
||||||
Executable
+1467
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user