diff --git a/docs/architecture/atlas-naming-pipeline.md b/docs/architecture/atlas-naming-pipeline.md new file mode 100644 index 000000000..bd8fcb2c9 --- /dev/null +++ b/docs/architecture/atlas-naming-pipeline.md @@ -0,0 +1,488 @@ +# Atlas Naming Pipeline + +Reference document for the feature-naming pipeline that populates +`markers.json` files and the `atlas_*` tables in `systems.db`. + +Related: ticket #833, D-191 §4. + +--- + +## 1. Overview + +The atlas naming pipeline generates human-readable names for every +unnamed geographic feature across ~2,394 planetary bodies in the +Settled Reach. Features covered: cities (capital and secondary), +rivers, oceans/seas/lakes, mountain ranges, and points of interest +(transit, institutional, cultural). + +Inputs: +- `wiki/star-systems/*/bodies/*/markers.json` — one file per body, + containing arrays of feature records with optional `name` fields +- `server/data/systems.db` — `star_systems`, `bodies`, `atlas_*` tables +- `wiki/star-systems/*/index.md` and `gttr.md` — cultural context for + register selection +- `tooling/planet-gen/earth_blocklist.txt` — exact-match rejection list + for well-known Earth place names + +Outputs: +- `markers.json` files updated in-place with the new `name` fields +- `atlas_cities`, `atlas_rivers`, `atlas_mountain_ranges`, `atlas_oceans`, + `atlas_pois` rows in `systems.db` synced immediately after each body + +Hand-authored names in `markers.json` are never overwritten. Re-running the +pipeline is safe: any body whose features all have non-empty `name` fields is +skipped entirely. + +Wall time on an RX 9070 (ROCm) is roughly 4-6 hours for a full run. + +--- + +## 2. Architecture + +### The inference binary: sr-voice-tooling + +The pipeline does NOT use the game's Gemma 2 runtime (`sr-voice`). It uses a +separate binary, `sr-voice-tooling`, built against Gemma 4 (GGUF) and compiled +for the ROCm / HIP stack. The binary lives outside all git worktrees at: + +``` +~/Projects/settled-reach/binaries/sr-voice-tooling +~/Projects/settled-reach/models/gemma-4.gguf +``` + +The split exists because the game's runtime (Gemma 2, 2B parameters, 1024-token +context) is optimised for low-latency in-game NPC dialogue, while the tooling +binary trades latency for richer outputs and a larger context window. + +### stdio JSONL protocol + +`VoiceSubprocess` in `gemma_naming.py` manages the binary as a long-lived +subprocess. Communication is newline-delimited JSON on stdin/stdout: + +Request (one JSON object per line): +```json +{"prompt": "...", "seed": 12345678} +``` + +Response: +```json +{"text": "..."} +``` + +or on error: +```json +{"error": "..."} +``` + +The subprocess is restarted every `--refresh` requests (default: 1000) to +prevent KV-cache context bleed accumulating across bodies. Seeds are derived +deterministically from `(body_id, feature_type, world_seed)` so the same +prompt always produces the same output — re-running for a partially-completed +body reproduces the same names, which helps diagnose quality issues without +introducing fresh variation. + +### Running inside distrobox + +The ROCm HIP kernels embedded in `sr-voice-tooling` are compiled for gfx1201 +(AMD Navi 48 / RX 9070). Because `libhipblas.so.2` and related ROCm libraries +are only installed in the `reach-build` distrobox container, `run-atlas-naming.sh` +and the test scripts invoke the binary via `distrobox enter reach-build --`. +The shell wrapper verifies the gfx1201 target is present before starting. + +--- + +## 3. Pipeline Stages + +For each body, `process_body()` executes the following stages in order. + +### Stage 1: Register selection (per system, wiki-grounded) + +At the start of each new system, `select_register()` asks Gemma to choose +the best cultural register from the corridor's `CORRIDOR_SUBSTYLES` list, +based on the system's `wiki/star-systems/{slug}/index.md` and `gttr.md`. + +`load_wiki_context()` reads both files. `_extract_cultural_lines()` scans +`index.md` for lines containing cultural keywords (heritage, founding, settler, +diaspora, specific language names, etc.) and returns up to 8 of them. The first +substantive paragraph from `gttr.md` (skipping the title line) is used as the +context hook. + +The prompt is few-shot: two worked examples from different corridors, then the +target system's context followed by the numbered option list. Gemma replies +with a single digit. The pipeline retries up to `max_attempts=3` times with +rotated seeds. On failure it falls back to `palette_for()`, a hash-based +deterministic selection: `hash(system_id) % len(substyles)`. + +The selected register applies to all bodies in that system. All bodies in the +same system therefore share a consistent cultural voice. + +### Stage 2: Mood injection (per body) + +`mood_for_body(body_id, world_seed)` picks one entry from `MOOD_POOL` via +`SHA-256(f"mood|{body_id}|{world_seed}")`. The 13-word pool is: + +``` +ambition, family, wealth, community, industry, pride, fleeting, +hope, fear, isolation, devotion, defiance, loss +``` + +The chosen mood is injected into each batch prompt as a clause: +`"The settlers here had a sense of {mood}."` This is the primary mechanism for +vocabulary divergence across bodies in the same system — same cultural register, +different emotional colouring. The mood is deterministic per body so re-runs +produce identical results. + +### Stage 3: Batch naming (2x oversampled) + +`name_features_batch()` (in `naming_core.py`) groups blank features by type and +requests `count * 2` names in a single prompt call. The oversample factor means +the ranking step (Stage 4) has candidates to work with even if the model repeats +some names. + +`build_batch_prompt()` assembles the prompt with: +- A preamble emphasising mundane, practical settler names over epic or classical + ones +- Optional system name, body name, and `system_hook` (the gttr.md + characterisation) +- The mood clause (if a mood was injected) +- The "already used" list (trimmed to fit within `ctx_size` tokens) +- Two fixed few-shot examples showing comma-separated batch format + (Scottish Highland and Dutch colonial) +- The tail: `"Style: {inflection}. {count*2} names:"` + +The model pattern-completes a comma-separated list. `parse_batch_response()` +takes the first line only (the model often continues with explanation), splits +on commas, strips quotes, filters via `is_valid_name()`, and deduplicates +preserving order. + +For single-name prompts (the legacy `_build_prompt()` / `name_feature()` path), +the prompt uses a rotating pool of few-shot style+answer pairs. The pool is +picked deterministically by `hash(body_id | local_id | attempt)` so retries +rotate to a different example set rather than re-querying with the same prompt +and a different seed. + +### Stage 4: Levenshtein distinctiveness ranking + +`select_distinct(candidates, count, taken)` in `naming_core.py` greedily +selects the N most distinct names from the oversampled candidate list. + +The distinctiveness metric is `word_avg_distance()`: for each word in the +shorter name, find the closest word (by Levenshtein edit distance) in the +longer name, then average across words. This means shared structural words +(Serra, The, Glen, Ridge) lower a candidate's preference score but do not +block it outright — the unique words in the name pull the average up. + +Selection is greedy: pick the candidate with the highest minimum distance to +all already-selected names plus all `taken` names, add it to the selected set, +repeat. No hard threshold — all candidates except exact string matches to +`taken` are eligible. + +### Stage 5: Cross-body dedup + +The `corpus` dict in `main()` maps `(system_id, feature_type)` to the set of +all names already assigned within that system. Before each batch call, `taken` +is built from `corpus[(system_id, ft)]` plus `body_used` (all names already +assigned to the current body across all feature types). This enforces two +constraints: + +- No body has the same name for two different features (e.g. a river and a + mountain range). +- No two bodies in the same system share the same name for the same feature + type (e.g. two rivers in GJ 144 named "Cooper's Creek"). + +Cross-corridor collisions (the same name appearing in two different systems +from different corridors) are allowed and expected. + +`taken` is pruned when it exceeds the context budget: the most recently added +entries are dropped first. This means a very large system (many bodies, many +features) may eventually lose dedup coverage for its earliest names at the +tail of the run. + +### Stage 6: Adjacent-register refill + +If the batch call + ranking still yields fewer than `count` names (after +dedup against `taken`), `name_features_batch()` fires a refill call using the +next sub-style in the corridor's `CORRIDOR_SUBSTYLES` list, wrapping around. +The refill asks for `shortfall * 3` candidates. Refill results go through the +same `select_distinct()` ranking against `taken + selected`. This ensures +the feature list is always fully populated even when the primary register's +vocabulary is exhausted. + +--- + +## 4. Cultural Registers + +### CORRIDOR_SUBSTYLES + +`CORRIDOR_SUBSTYLES` in `gemma_naming.py` is a `dict[str, list[dict[str, str]]]` +mapping corridor name to an ordered list of sub-style dicts. Each sub-style has +two keys: `inflection` (the style label injected into the prompt) and `examples` +(a comma-separated string of illustrative names, shown during `select_register()` +option display but NOT injected into batch prompts — only the inflection label +reaches the model during naming). + +Active corridors and their sub-styles: + +**core** (6 sub-styles) +- English countryside, rural, agricultural settlers +- British colonial settlement era +- American frontier, practical, geographic +- American municipal, administrative, cosmopolitan +- Classical references, institutional, civic +- Australian and New Zealand settler + +**north_reach** (5 sub-styles) +- English rural, village and parish names +- Scottish Highland and Lowland place-names +- Australian outback, station and property names +- Irish rural and coastal settlement +- South African English settler + +**south_reach** (5 sub-styles) +- Portuguese colonial era, Iberian +- Brazilian interior, frontier settlement +- East African Swahili coastal +- Cape Verdean and West African +- Angolan and Mozambican settlement + +**east_reach** (5 sub-styles) +- Korean place-name tradition +- Japanese rural and coastal settlement +- Taiwanese and Hakka settler +- Filipino settler community +- Mixed East Asian diaspora, cosmopolitan + +**west_reach** (5 sub-styles) +- German settlement, orderly and compound names +- Dutch colonial, low-country +- Nordic and Scandinavian +- Polish and Czech settler +- Baltic and Finnish settler + +**deep_frontier** (3 sub-styles) +- frontier founder-name era, surname-first, any Earth culture +- frontier descriptive, geographic features named by surveyors +- frontier outpost, functional and military + +Legacy aliases (`sol-gateway-axis`, `inner_corridor`, `inner_orbit`) map to +`core`. `frontier` maps to `deep_frontier`. The pipeline reads the corridor +from `COALESCE(cultural_corridor, geographic_sector, 'core')` in +`star_systems` — `geographic_sector` is the live field; `cultural_corridor` +was a legacy column never populated beyond the sol-gateway-axis. + +### Sub-style rotation + +All bodies within a system use the same register (chosen by `select_register()` +or `palette_for()`). Across systems within a corridor the hash-based fallback +rotates through the sub-style list: `hash(system_id) % len(substyles)`. This +ensures neighbouring systems don't all sound identical even when `select_register()` +fails. + +### Inflection as dominant bias, not hard lock + +The `CORRIDOR_SUBSTYLES` comment is explicit: the inflection is a dominant bias, +not a hard cultural lock. A British surveyor on an east_reach moon still names +a river after their aunt in Dorset. The few-shot examples in the batch prompt +show cross-cultural names to reinforce this — the examples are Scottish Highland +and Dutch colonial regardless of the target corridor. + +### SECTOR_PRIORITY + +`SECTOR_PRIORITY` sets the processing order: `core=0`, `north_reach=1`, +`south_reach=2`, `east_reach=3`, `west_reach=4`, `deep_frontier=5`. Core +bodies are named first so they win the dedup race; frontier bodies fall into +the adjacent-register refill path when names collide. + +--- + +## 5. Body Processing Order + +`discover_bodies()` discovers all `markers.json` files via +`WIKI_SYSTEMS.glob("*/bodies/*/markers.json")` and sorts them by a 5-tuple +sort key loaded from `load_body_hop_order()`: + +``` +(hop_distance, system_id, is_inhabited_desc, population_rank, body_id) +``` + +- **Hop distance** from Gateway (GJ 71), ascending. Core systems process first. +- **System grouping**: all bodies in a system are processed consecutively + (same `system_id` in the sort key). +- **Inhabited first**: within a system, inhabited bodies sort before + uninhabited ones. +- **Population rank**: within inhabited bodies, higher-population bodies sort + earlier (so the main inhabited world of a system wins the dedup race against + minor colonies). +- Bodies whose `body_id` is absent from `hop_order` (markers.json orphans + not in the DB) sort to the end with a sentinel hop of 99. + +--- + +## 6. Key Files and Their Roles + +| File | Role | +|------|------| +| `tooling/planet-gen/gemma_naming.py` | Main pipeline: `VoiceSubprocess`, `CORRIDOR_SUBSTYLES`, `palette_for()`, `select_register()`, `_build_prompt()` (single-name path), `_PROMPT_CONFIG`, `process_body()`, `main()`. | +| `tooling/planet-gen/naming_core.py` | Shared algorithms: `levenshtein()`, `word_avg_distance()`, `select_distinct()`, `build_batch_prompt()`, `parse_batch_response()`, `name_features_batch()`, `mood_for_body()`, `MOOD_POOL`, `is_valid_name()`. | +| `tooling/planet-gen/run-atlas-naming.sh` | Shell wrapper for overnight runs. Validates the binary (`gfx1201` kernel check), model, and distrobox container before exec-ing into `gemma_naming.py`. Hardcodes paths — no arguments accepted. | +| `tooling/planet-gen/qa_naming.py` | QA report: reads `systems.db` and `markers.json` files, runs the full check suite, prints a summary with issue rates. | +| `tooling/planet-gen/test_batch_naming.py` | Integration test harness. Runs 4 simulated systems (20 bodies) against real Gemma 4 via `sr-voice-tooling`, accumulates the taken list across bodies, prints candidates/selected/refill for visual inspection. | +| `tooling/planet-gen/test_register_selection.py` | Tests `select_register()` against 8 real systems spanning all corridors, prints what Gemma picks vs. the hash-based fallback. | +| `tooling/planet-gen/earth_blocklist.txt` | Lowercase exact-match list of Earth major place names. `is_blocked()` also strips a leading "The " before comparing. | +| `~/Projects/settled-reach/binaries/sr-voice-tooling` | The Gemma 4 inference binary. Lives outside the repo. Built with `CMAKE_HIP_ARCHITECTURES=gfx1201` inside the `reach-build` distrobox. | +| `~/Projects/settled-reach/models/gemma-4.gguf` | The model weights. Shared across worktrees. | +| `server/data/systems.db` | SQLite database. `star_systems.geographic_sector` is the corridor source of truth. `atlas_*` tables are synced per body immediately after `markers.json` is written. | + +`generate_atlas.py` provides `ensure_atlas_schema()` and `sync_markers_to_db()`, +both imported by `gemma_naming.py`. This makes `generate_atlas.py` the single +authoritative path for `atlas_*` row writes. + +--- + +## 7. Known Limitations + +### Classical register sameyness + +The "Classical references, institutional, civic" sub-style in the `core` +corridor (Concordia, Aurelius, Prefecture, Forum) has a narrow vocabulary. +Gemma 4 has roughly 15 usable stems per register. On a system with many +bodies, classical register bodies at the tail of the run exhaust the fresh +stem pool and the adjacent-register refill kicks in. The refill names are +structurally correct but may not feel classical. This is by design: the +frontier fallback is correct behaviour, not a failure mode. + +### Few-shot example bleed + +The two fixed batch few-shot examples (Scottish Highland, Dutch colonial) are +cross-corridor by design, but on short prompts (low `ctx_size`) they can +dominate the model's completion distribution. The result is names that have +a vaguely British or Dutch character even for east_reach or south_reach +bodies. Increasing `ctx_size` reduces this; at 1024 tokens it is detectable +but not severe. + +### Narrow register exhaustion on large systems + +The `taken` list is trimmed from the tail when it exceeds the token budget. +For systems with 30+ bodies (e.g. very large multi-moon systems), early +names may drop off the `taken` list before all bodies are processed, allowing +duplicates to slip through. `qa_naming.py`'s `check_exact_dupes_within_system()` +catches these; manual re-runs with `--body` are the fix. + +### Stem repetition ("Ridge Ridge Ridge") + +`check_stem_repetition()` detects this: 4 or more features on the same body +sharing the same first word. It is caused by the model having a strong +associative path from the inflection to one compound-word opener (e.g. +"Ridge" for frontier descriptive). The Levenshtein ranking partially mitigates +it — two names with identical first words have a low `word_avg_distance` and +the second one loses the greedy selection — but it does not eliminate it when +the entire candidate pool shares the stem. + +### Single-name vs. batch path + +`gemma_naming.py` contains two prompt-building paths: the legacy +`_build_prompt()` / `name_feature()` single-name path and the newer +`build_batch_prompt()` / `name_features_batch()` batch path. The batch path +is what `process_body()` calls via `_batch_fill()`. The single-name path +and its `_PROMPT_CONFIG` pools are still present and used by `_build_prompt()`, +which is called by `name_feature()` — that function may be invoked on retry +paths or by older call sites. The two paths use different prompt structures +and produce different output distributions. + +--- + +## 8. QA Process + +### Running + +```bash +python3 tooling/planet-gen/qa_naming.py +python3 tooling/planet-gen/qa_naming.py --verbose +``` + +Requires `server/data/systems.db` to be present with populated `atlas_*` tables. + +### What the checks look for + +| Check | Function | Trigger | +|-------|----------|---------| +| Prompt fragment leaks | `check_prompt_fragments()` | Name contains substrings like `"style:"`, `"generate"`, `"already used"`, `"must be"`, etc. | +| Exact dupes within system | `check_exact_dupes_within_system()` | Same name, same feature type, two different bodies in the same system. | +| Exact dupes within body | `check_exact_dupes_within_body()` | Same name, different feature types, same body (e.g. a river and a mountain range both called "Thornbury"). | +| Stem repetition | `check_stem_repetition()` | 4 or more features of the same type on a body share the same first word. | +| Body name echo | `check_body_name_echo()` | A word from the body's proper name appears as a word in a feature name. | +| Register bleed | `check_register_bleed()` | Keyword heuristics flag obviously wrong cultural associations (e.g. "fjord" in south_reach, "sakura" in west_reach). | +| Feature type mismatch | `check_feature_type_mismatch()` | Street-vocabulary words on a mountain; building-vocabulary words on a river or ocean. | +| Short names | `check_short_long_names()` | Name is 3 characters or fewer. | +| Long names | `check_short_long_names()` | Name is more than 40 characters. | +| Numbers in names | `check_numbers_in_names()` | Name contains a digit. | +| Coverage gaps | `coverage_gaps()` | Bodies with one or more unnamed features remaining in `atlas_*` tables. Inhabited bodies listed separately. | + +The summary block at the end prints a total issue rate (`issues / total_named * +100`). A clean run at the end of a full batch should sit below 1%. Prompt +fragment leaks above zero indicate a context-budget or parsing failure and +should be investigated before ship. + +--- + +## 9. How to Extend + +### Adding a new cultural register + +1. Add a new dict entry to the relevant list in `CORRIDOR_SUBSTYLES`: + ```python + {"inflection": "...", "examples": "..."} + ``` +2. The pipeline picks it up automatically. `palette_for()` uses + `len(substyles)` for the modulo, so the new sub-style becomes available + on the next hash rotation. `select_register()` presents it as a numbered + option to Gemma. +3. Run `test_register_selection.py` to verify Gemma picks the new register + for a system whose wiki content matches it. + +### Tuning temperature and sampling + +Temperature is a parameter of the `sr-voice-tooling` binary, not of the +Python scripts. Pass flags to the binary via the `--sr-voice` argument or +edit `run-atlas-naming.sh`. The scripts have no Python-side temperature knob. + +### Adding new moods + +Extend `MOOD_POOL` in `naming_core.py`. The deterministic hash spreads the +new mood across bodies automatically on the next full run. Existing bodies +whose `markers.json` already has names will be skipped, so only un-named +bodies pick up the new mood entries. + +### Running for a single system or body + +Single body: +```bash +python3 tooling/planet-gen/gemma_naming.py --body GJ380c +``` + +Single body with verbose output showing per-feature retry detail: +```bash +python3 tooling/planet-gen/gemma_naming.py --body GJ380c --verbose +``` + +Smoke test (first 5 bodies in hop order): +```bash +python3 tooling/planet-gen/gemma_naming.py --limit 5 --verbose +``` + +Without a real model (mock stdio for pipeline testing): +```bash +python3 tooling/planet-gen/gemma_naming.py --mock --limit 5 +``` + +Dump prompts to JSONL without calling the model (for A/B comparison against +a different backend): +```bash +python3 tooling/planet-gen/gemma_naming.py \ + --dump-prompts /tmp/prompts.jsonl \ + --limit 10 +``` + +All invocations use `--refresh 1000` by default (subprocess restart every +1000 requests). Lower it to `--refresh 100` if you observe quality degradation +across a long single-system run. diff --git a/server/data/systems-schema.sql b/server/data/systems-schema.sql index 796b7933c..fa58aac7c 100644 --- a/server/data/systems-schema.sql +++ b/server/data/systems-schema.sql @@ -38,6 +38,15 @@ CREATE TABLE IF NOT EXISTS star_systems ( cultural_corridor TEXT, generation_priority TEXT, + -- Short narrative hook extracted from the wiki's gttr.md file — the + -- first characterisation paragraph ("where the rules live", "forty + -- years old and still in the draft", etc). Populated by + -- tooling/db/populate_gttr_hook.py from wiki/star-systems//gttr.md. + -- Used as compact cultural context in the Gemma 2 naming pipeline + -- (#833) so per-system feel is grounded in the canonical identity + -- rather than generic corridor labels. + gttr_hook TEXT, + -- Economics (D-172) currency_zone TEXT DEFAULT 'TRACTUS_PRIMARY', -- TRACTUS_PRIMARY | MARK_PRIMARY | MIXED diff --git a/server/data/systems.db b/server/data/systems.db index 34ce951c9..e339af242 100644 Binary files a/server/data/systems.db and b/server/data/systems.db differ diff --git a/server/sr-voice/.gitignore b/server/sr-voice/.gitignore new file mode 100644 index 000000000..5ac889a78 --- /dev/null +++ b/server/sr-voice/.gitignore @@ -0,0 +1,4 @@ +# Built binaries — platform-specific, rebuilt via distrobox + cargo. +# Per #850, release binaries will ship as CI artifacts, not in the repo. +/bin/ +/target/ diff --git a/server/sr-voice/src/inference.rs b/server/sr-voice/src/inference.rs index ee1729575..e95ccdcbd 100644 --- a/server/sr-voice/src/inference.rs +++ b/server/sr-voice/src/inference.rs @@ -39,11 +39,19 @@ pub struct InferenceEngine { impl InferenceEngine { /// Load a GGUF model from disk. + /// + /// Offloads all layers to the GPU via ROCm. The binary is built with + /// llama-cpp-rs + ROCm support (see Makefile `build-sr-voice` target), + /// but `LlamaModelParams::default()` sets `n_gpu_layers = 0`, which + /// runs the entire model on CPU at ~10× lower throughput. Setting + /// `n_gpu_layers` to a large sentinel value (999) asks llama.cpp to + /// offload every layer the model has; it clamps to the real count. + /// For Gemma 2 2B (27 layers) this fully GPU-offloads the model. pub fn load(config: &InferenceConfig) -> Result { let backend = LlamaBackend::init().map_err(|e| VoiceError::ModelLoadFailed(e.to_string()))?; - let model_params = LlamaModelParams::default(); + let model_params = LlamaModelParams::default().with_n_gpu_layers(999); let model = LlamaModel::load_from_file( &backend, Path::new(&config.model_path), diff --git a/tooling/db/backfill_cultural_corridor.py b/tooling/db/backfill_cultural_corridor.py new file mode 100755 index 000000000..5f7cfc686 --- /dev/null +++ b/tooling/db/backfill_cultural_corridor.py @@ -0,0 +1,162 @@ +#!/usr/bin/env python3 +""" +Backfill star_systems.cultural_corridor and bodies.cultural_corridor. + +The schema has a `cultural_corridor` column on both tables, but +`wiki_sync.py` never populated it from the wiki index.md files — the +cultural/geographic identity of each system lives in +`star_systems.geographic_sector` instead (values: core, north_reach, +south_reach, east_reach, west_reach, deep_frontier). These two fields +refer to the same concept: which arc of the reach the system belongs +to. Leaving `cultural_corridor` NULL on 99%+ of rows defeats every +downstream consumer that actually wants to filter by corridor +(gemma_naming.py, future atlas UI queries, narrative tools). + +This script treats `geographic_sector` as the source of truth and +copies it into `cultural_corridor`: + + star_systems.cultural_corridor := star_systems.geographic_sector + WHERE cultural_corridor IS NULL + + bodies.cultural_corridor := parent star_systems.cultural_corridor + WHERE bodies.cultural_corridor IS NULL + +It is safe to re-run — idempotent, NULL-only updates, explicit +transaction wrapper so a crash never leaves a half-populated state. +Run it after any `wiki_sync.py` pass that creates fresh systems.db +rows. + +Usage: + tooling/db/backfill_cultural_corridor.py + tooling/db/backfill_cultural_corridor.py --db path/to/systems.db + tooling/db/backfill_cultural_corridor.py --dry-run + +Decisions: D-191 (atlas pipeline — downstream consumer) +""" + +import argparse +import sqlite3 +import sys +from pathlib import Path + +SCRIPT_DIR = Path(__file__).resolve().parent +REPO_ROOT = (SCRIPT_DIR / ".." / "..").resolve() +DB_PATH = REPO_ROOT / "server" / "data" / "systems.db" + + +def main(): + parser = argparse.ArgumentParser( + description="Backfill cultural_corridor on star_systems and bodies" + ) + parser.add_argument("--db", default=str(DB_PATH), help="Path to systems.db") + parser.add_argument( + "--dry-run", + action="store_true", + help="Report counts without writing", + ) + args = parser.parse_args() + + db_path = Path(args.db) + if not db_path.exists(): + print(f"error: {db_path} not found", file=sys.stderr) + sys.exit(1) + + conn = sqlite3.connect(str(db_path)) + conn.execute("PRAGMA foreign_keys=ON") + + # Counts before. + before_systems_null = conn.execute( + "SELECT COUNT(*) FROM star_systems WHERE cultural_corridor IS NULL" + ).fetchone()[0] + before_bodies_null = conn.execute( + "SELECT COUNT(*) FROM bodies WHERE cultural_corridor IS NULL" + ).fetchone()[0] + + print(f"\n cultural_corridor backfill") + print(f" DB: {db_path}") + if args.dry_run: + print(f" Mode: DRY RUN") + print() + print(f" Before:") + print(f" star_systems.cultural_corridor NULL: {before_systems_null}") + print(f" bodies.cultural_corridor NULL: {before_bodies_null}") + + conn.execute("BEGIN") + try: + # 1. Star systems — copy geographic_sector into cultural_corridor + # where the latter is still NULL. If geographic_sector is also + # NULL, leave cultural_corridor NULL — there is nothing to + # copy and a bogus placeholder is worse than honest NULL. + sys_rows_updated = conn.execute( + """ + UPDATE star_systems + SET cultural_corridor = geographic_sector + WHERE cultural_corridor IS NULL + AND geographic_sector IS NOT NULL + """ + ).rowcount + + # 2. Bodies — inherit from the parent star_systems row. + body_rows_updated = conn.execute( + """ + UPDATE bodies + SET cultural_corridor = ( + SELECT s.cultural_corridor + FROM star_systems s + WHERE s.system_id = bodies.system_id + ) + WHERE cultural_corridor IS NULL + AND EXISTS ( + SELECT 1 + FROM star_systems s + WHERE s.system_id = bodies.system_id + AND s.cultural_corridor IS NOT NULL + ) + """ + ).rowcount + + if args.dry_run: + conn.rollback() + print() + print(f" Would update:") + print(f" star_systems: {sys_rows_updated}") + print(f" bodies: {body_rows_updated}") + print(f"\n Dry run — no changes written.") + else: + conn.commit() + print() + print(f" Updated:") + print(f" star_systems: {sys_rows_updated}") + print(f" bodies: {body_rows_updated}") + + # Counts after. + after_systems_null = conn.execute( + "SELECT COUNT(*) FROM star_systems WHERE cultural_corridor IS NULL" + ).fetchone()[0] + after_bodies_null = conn.execute( + "SELECT COUNT(*) FROM bodies WHERE cultural_corridor IS NULL" + ).fetchone()[0] + print() + print(f" After:") + print(f" star_systems.cultural_corridor NULL: {after_systems_null}") + print(f" bodies.cultural_corridor NULL: {after_bodies_null}") + + # Show the distribution so the outcome is visible. + print() + print(f" star_systems.cultural_corridor distribution:") + for corridor, count in conn.execute( + "SELECT cultural_corridor, COUNT(*) FROM star_systems " + "GROUP BY cultural_corridor ORDER BY COUNT(*) DESC" + ).fetchall(): + print(f" {corridor!r}: {count}") + except BaseException: + conn.rollback() + conn.close() + raise + + conn.close() + print() + + +if __name__ == "__main__": + main() diff --git a/tooling/db/populate_gttr_hook.py b/tooling/db/populate_gttr_hook.py new file mode 100755 index 000000000..c5992de68 --- /dev/null +++ b/tooling/db/populate_gttr_hook.py @@ -0,0 +1,215 @@ +#!/usr/bin/env python3 +""" +Extract a short narrative hook from every wiki/star-systems//gttr.md +and store it on `star_systems.gttr_hook`. + +The GTTR ("Drifter's Guide to the Reach") files follow a consistent +format: an H1 title, then a paragraph that opens with `**NAME**` followed +by a 1-3 sentence characterisation. That characterisation is the hook — +per D-191 §4 and the #833 naming pipeline it is the single strongest +cultural signal available for each system, capturing things like: + + - Gateway: "the most connected system in the Reach, the site of its + oldest research institution, and the location of a gate that goes + to Earth and that nobody uses" + - Ran: "a very nice place to live, provided you are the sort of + person whose grandparents owned the land" + - ACB: "where the Lattice Commission lives, which means it is where + the rules live" + - Posto Avançado: "forward post — the place beyond the established + line" + +This script parses each gttr.md, regex-extracts the first `**NAME**` +paragraph, normalises whitespace, and truncates at a soft word cap so +the hook stays cheap to inject into prompts. Empty or unmatched files +leave the column NULL. + +Idempotent, safe to re-run after any wiki update. Explicit transaction +wrapper with rollback on exception. + +Usage: + tooling/db/populate_gttr_hook.py + tooling/db/populate_gttr_hook.py --max-words 45 + tooling/db/populate_gttr_hook.py --dry-run + tooling/db/populate_gttr_hook.py --system "GJ 71" +""" + +import argparse +import re +import sqlite3 +import sys +from pathlib import Path + +SCRIPT_DIR = Path(__file__).resolve().parent +REPO_ROOT = (SCRIPT_DIR / ".." / "..").resolve() +DB_PATH = REPO_ROOT / "server" / "data" / "systems.db" +WIKI_SYSTEMS = REPO_ROOT / "wiki" / "star-systems" + +# Matches the first paragraph that opens with `**NAME**` at the start +# of a line. Captures everything up to a blank line or the next H1/H2. +# Non-greedy on the content. +# Match a line that opens with `**NAME**` (any non-asterisk chars, since +# names contain accents from many alphabets — Á, Ž, Ç, Ñ, etc) followed +# by the paragraph body up to a blank line or the next markdown heading. +_HOOK_RE = re.compile( + r"^\*\*([^\n*]+?)\*\*(.+?)(?:\n\n|\n#)", + re.DOTALL | re.MULTILINE, +) + + +def _system_slug_to_system_id(slug: str) -> str: + """Convert 'GJ-244A' → 'GJ 244A'. Mirrors populate_terrain_reference.py.""" + if slug.startswith("GJ-"): + return "GJ " + slug[3:] + return slug + + +def extract_hook(gttr_path: Path, max_words: int) -> str | None: + """Return a compact one-paragraph hook for this gttr.md, or None if + the file is missing or the opening paragraph can't be located.""" + if not gttr_path.exists(): + return None + text = gttr_path.read_text() + + match = _HOOK_RE.search(text) + if not match: + return None + + name_token = match.group(1).strip() + body = match.group(2).strip() + + # Reconstruct "NAME is …" without the markdown asterisks. Add a + # single space between the name and the body since `.strip()` above + # removed the leading space from the captured body. + hook = f"{name_token} {body}" + + # Collapse whitespace so multi-line paragraphs become one clean line. + hook = re.sub(r"\s+", " ", hook).strip() + + # Strip a leading orphan "is" that comes from `**NAME**` + " is …": + # the regex captures the word "is" on its own because the opener is + # typically `**GATEWAY** (known as Tau Ceti...) is the most connected…`. + # Normal reading already works — this is just hygiene. + hook = re.sub(r"\s*\(\s*\)\s*", " ", hook) + + # Hard cap at max_words. Truncate at the last word boundary before + # the cap and append an ellipsis so the reader knows it continues. + words = hook.split() + if len(words) > max_words: + hook = " ".join(words[:max_words]).rstrip(",;:") + "…" + + return hook + + +def main(): + parser = argparse.ArgumentParser( + description="Populate star_systems.gttr_hook from wiki gttr.md files" + ) + parser.add_argument("--db", default=str(DB_PATH), help="Path to systems.db") + parser.add_argument( + "--system", + help="Process only this system_id (e.g. 'GJ 71')", + ) + parser.add_argument( + "--max-words", + type=int, + default=45, + help="Soft cap on hook length (default: 45 words)", + ) + parser.add_argument( + "--dry-run", + action="store_true", + help="Extract and print but do not write to the DB", + ) + parser.add_argument( + "--verbose", + action="store_true", + help="Print every extracted hook", + ) + args = parser.parse_args() + + db_path = Path(args.db) + if not db_path.exists(): + print(f"error: {db_path} not found", file=sys.stderr) + sys.exit(1) + + conn = sqlite3.connect(str(db_path), timeout=30.0) + conn.execute("PRAGMA journal_mode=WAL") + conn.execute("PRAGMA busy_timeout=15000") + conn.execute("PRAGMA foreign_keys=ON") + + # Ensure the column exists — idempotent ADD COLUMN for old DBs. + try: + conn.execute("ALTER TABLE star_systems ADD COLUMN gttr_hook TEXT") + except sqlite3.OperationalError: + pass # column already exists + + print(f"\n populate_gttr_hook.py") + print(f" DB: {db_path}") + print(f" max words: {args.max_words}") + if args.dry_run: + print(f" Mode: DRY RUN") + print() + + rows = conn.execute( + "SELECT system_id, proper_name FROM star_systems ORDER BY system_id" + ).fetchall() + if args.system: + rows = [r for r in rows if r[0] == args.system] + + updated = 0 + missing = 0 + unmatched = 0 + + conn.execute("BEGIN") + try: + for system_id, proper_name in rows: + slug = system_id.replace(" ", "-", 1) + gttr_path = WIKI_SYSTEMS / slug / "gttr.md" + + hook = extract_hook(gttr_path, args.max_words) + if hook is None: + if not gttr_path.exists(): + missing += 1 + if args.verbose: + print(f" MISSING {system_id:10s} {gttr_path}") + else: + unmatched += 1 + if args.verbose: + print(f" UNMATCHED {system_id:10s} {gttr_path}") + continue + + if args.verbose: + display = f"{system_id} ({proper_name})" if proper_name else system_id + print(f" {display}") + print(f" → {hook}") + + if not args.dry_run: + conn.execute( + "UPDATE star_systems SET gttr_hook = ? WHERE system_id = ?", + (hook, system_id), + ) + updated += 1 + + if args.dry_run: + conn.rollback() + else: + conn.commit() + except BaseException: + conn.rollback() + conn.close() + raise + + conn.close() + + print(f"\n Done:") + print(f" updated: {updated}") + print(f" missing: {missing}") + print(f" unmatched: {unmatched}") + if args.dry_run: + print(f"\n Dry run — no DB writes.") + print() + + +if __name__ == "__main__": + main() diff --git a/tooling/planet-gen/earth_blocklist.txt b/tooling/planet-gen/earth_blocklist.txt new file mode 100644 index 000000000..a1943dec0 --- /dev/null +++ b/tooling/planet-gen/earth_blocklist.txt @@ -0,0 +1,267 @@ +# Earth-name blocklist for gemma_naming.py (#833, D-191 §4). +# +# Per memory feedback_earth_echo_names: Earth-sounding names are OK and +# expected — the setting is a blended reality. This file is a small, +# curated list of Earth *majors* whose raw names should not appear as +# settled-reach feature names because they are too on-the-nose. Prefixed +# variants like "Nouveau Paris", "Neu Berlin", or "New Tokyo" are allowed +# (they pass because the exact-match check does not strip prefixes). +# +# Match rule: case-insensitive whole-name equality between a generated +# feature name and any entry below. One entry per line, `#` comments are +# allowed, blank lines ignored. +# +# Expand this list if the generator starts producing raw-Earth-major +# names in frequent runs. Err on the side of short — we do NOT want to +# block every vaguely Earthy token. + +# Capitals & megacities +Paris +London +Tokyo +Beijing +Moscow +Berlin +Madrid +Rome +Cairo +Mumbai +Delhi +Jakarta +Seoul +Bangkok +Istanbul +Lagos +Nairobi +Johannesburg +Sydney +Washington +Ottawa +Mexico City +Rio de Janeiro +Rio +Buenos Aires +Lima +Bogota +Dhaka +Karachi +Lahore +Tehran +Baghdad +Riyadh +Manila +Hanoi +Kuala Lumpur +Singapore +Vienna +Prague +Warsaw +Amsterdam +Brussels +Copenhagen +Stockholm +Oslo +Helsinki +Reykjavik +Lisbon +Athens +Budapest +Dublin +Edinburgh +Glasgow + +# Iconic cities / regional majors +New York +Los Angeles +Chicago +Houston +Miami +Boston +Seattle +Toronto +Vancouver +Montreal +Barcelona +Munich +Hamburg +Frankfurt +Milan +Venice +Florence +Naples +Marseille +Lyon +Geneva +Zurich +Shanghai +Hong Kong +Taipei +Osaka +Kyoto +Yokohama +Busan +Ho Chi Minh +Kolkata +Chennai +Bangalore +Hyderabad +Addis Ababa +Casablanca +Accra +Kinshasa +Cape Town +Tel Aviv +Dubai +Doha +Damascus +Beirut +Jerusalem +Auckland +Wellington +Melbourne +Brisbane +Perth + +# Mountain / natural majors that read as Earth +Everest +Kilimanjaro +Fuji +Matterhorn +Olympus +Etna +Vesuvius +Denali +Aconcagua +Ararat +K2 +Annapurna +Elbrus +Kosciuszko + +# Rivers +Amazon +Nile +Danube +Mississippi +Mekong +Ganges +Yangtze +Rhine +Volga +Thames +Seine +Tigris +Euphrates +Congo +Niger +Zambezi + +# Oceans / seas +Atlantic +Pacific +Indian +Arctic +Mediterranean +Baltic +Caspian +Aegean +Adriatic +Caribbean + +# Historical / colonial / alternate spellings of Earth cities — Gemma +# knows these from training data and emits them as if they were +# neutral names. +Calcutta +Bombay +Madras +Bangalore +Poona +Benares +Peking +Canton +Nanking +Chungking +Saigon +Rangoon +Ceylon +Batavia +Angora +Smyrna +Constantinople +Formosa +Tasmania +Rhodesia +Persia +Mesopotamia + +# Earth natural-phenomenon names Gemma likes to reach for +Aurora Borealis +Aurora Australis +Great Divide +Great Lakes +Great Basin +Grand Canyon +Death Valley +Sahara +Gobi +Patagonia +Serengeti +Outback +Tundra +Siberia +Amazon Basin + +# Greek/Roman mythology that reads too literally as Earth classical +Olympus Mons +Mount Olympus + +# European rivers the model keeps reaching for +Rhine +Weser +Elbe +Oder +Vistula +Loire +Rhône +Douro +Tagus +Ebro +Po +Arno +Tiber +Sava +Drava +Vlatava +Vltava +Dnieper +Don +Volga +Dniester + +# Nordic / Eastern European cities +Reykjavik +Oslo +Bergen +Tromsø +Gothenburg +Gdansk +Krakow +Warsaw +Prague +Brno +Bratislava +Budapest +Debrecen +Bucharest +Sofia +Belgrade +Zagreb +Ljubljana +Tallinn +Riga +Vilnius +Kiev +Kyiv +Minsk +Odessa +Lviv diff --git a/tooling/planet-gen/fix_fewshot_bleed.py b/tooling/planet-gen/fix_fewshot_bleed.py new file mode 100644 index 000000000..36ea080bf --- /dev/null +++ b/tooling/planet-gen/fix_fewshot_bleed.py @@ -0,0 +1,206 @@ +#!/usr/bin/env python3 +"""Replace few-shot example names that bled into the output. + +The batch naming prompt uses Scottish Highland and Dutch colonial +examples. The Scottish ones (Glen Moray, Dunvegan Ridge, Torridon, +Cairn Brae, The Kelpie's Spine) leaked into 270 features. This script +replaces them with unique names from a combined Scottish/Welsh/Irish +pool, ensuring no collisions with the existing corpus. +""" + +import json +import sqlite3 +import sys +from collections import defaultdict +from pathlib import Path + +TOOLING_DIR = Path(__file__).resolve().parent +REPO_ROOT = (TOOLING_DIR / ".." / "..").resolve() +DB_PATH = REPO_ROOT / "server" / "data" / "systems.db" +WIKI_SYSTEMS = REPO_ROOT / "wiki" / "star-systems" + +sys.path.insert(0, str(TOOLING_DIR)) +from generate_atlas import sync_markers_to_db + +# The few-shot names to replace +FEWSHOT_NAMES = { + "glen moray", "dunvegan ridge", "torridon", "cairn brae", "the kelpie's spine", + "kloosterbeek", "nieuw rijn", "hoogland run", "van diemen's creek", +} + +# Scottish / Welsh / Irish replacement pool — 300+ names to cover 270 replacements +# with room for Levenshtein filtering. Mix of geographic feature styles. +REPLACEMENT_POOL = [ + # Scottish + "Glenfinnan", "Dalwhinnie Pass", "Cairngorm", "Loch Maree", + "Kinlochleven", "Strathspey", "Brae Morar", "Skye Reach", + "Ardnamurchan", "Kintail", "Glen Affric", "Lochaber", + "Killiecrankie", "Rannoch Moor", "Glen Coe", "Strathnaver", + "Applecross", "Torrisdale", "Durness", "Assynt", + "Coigach", "Inverpolly", "Sandwood", "Cape Wrath", + "Sutherland", "Helmsdale", "Brora", "Golspie", + "Cromarty", "Dornoch", "Nairn", "Forres", + "Culbin", "Findhorn", "Spey Bay", "Buckie", + "Banff", "Fraserburgh", "Peterhead", "Cruden Bay", + "Slains", "Ythan", "Bennachie", "Morven", + "Lochnagar", "Braemar", "Balmoral", "Crathie", + "Ballater", "Dinnet", "Tarland", "Lumphanan", + "Corgarff", "Tomintoul", "Glenlivet", "Dufftown", + "Craigellachie", "Aberlour", "Knockando", "Archiestown", + "Rothes", "Elgin", "Lossiemouth", "Burghead", + "Kinloss", "Alves", "Pluscarden", "Dallas", + # Welsh + "Cwm Idwal", "Beddgelert", "Crib Goch", "Tryfan", + "Ogwen", "Llyn Padarn", "Dolgellau", "Harlech", + "Rhinog", "Cader Idris", "Barmouth", "Aberdovey", + "Tywyn", "Machynlleth", "Pumlumon", "Hafren", + "Elan Valley", "Claerwen", "Llandrindod", "Brecon", + "Pen y Fan", "Corn Du", "Crickhowell", "Llangorse", + "Talgarth", "Hay Bluff", "Mynydd Troed", "Mynydd Llangorse", + "Skirrid", "Blorenge", "Llanfoist", "Govilon", + "Gilwern", "Llangattock", "Crug Hywel", "Cwm Clydach", + "Pontneddfechan", "Ystradfellte", "Sgwd yr Eira", "Henrhyd", + "Carreg Cennen", "Dinefwr", "Llandeilo", "Dryslwyn", + "Tywi Valley", "Carmarthen", "Kidwelly", "Pembrey", + "Gower", "Rhossili", "Oxwich", "Port Eynon", + "Pennard", "Langland", "Caswell", "Mumbles", + "Merthyr Mawr", "Ogmore", "Dunraven", "Llantwit", + "Monknash", "Nash Point", "Aberthaw", "Fonmon", + # Irish + "Glendalough", "Lugnaquilla", "Glen Imaal", "Wicklow Gap", + "Sally Gap", "Kippure", "Djuce", "Maulin", + "Djouce", "Great Sugar Loaf", "Bray Head", "Killiney", + "Dalkey", "Howth", "Lambay", "Ireland's Eye", + "Malahide", "Portmarnock", "Donabate", "Skerries", + "Balbriggan", "Gormanston", "Bettystown", "Laytown", + "Slane", "Newgrange", "Dowth", "Knowth", + "Tara", "Trim", "Navan", "Kells", + "Loughcrew", "Oldcastle", "Castlepollard", "Fore", + "Delvin", "Mullingar", "Kilbeggan", "Tullamore", + "Clara", "Ferbane", "Banagher", "Shannonbridge", + "Clonmacnoise", "Ballinasloe", "Aughrim", "Loughrea", + "Portumna", "Mountshannon", "Killaloe", "Ballina", + "Nenagh", "Roscrea", "Templemore", "Thurles", + "Cashel", "Cahir", "Clonmel", "Carrick-on-Suir", + "Piltown", "Mooncoin", "Waterford", "Tramore", + "Bunmahon", "Ardmore", "Youghal", "Midleton", + "Cobh", "Crosshaven", "Kinsale", "Clonakilty", + "Skibbereen", "Bantry", "Glengarriff", "Kenmare", + "Sneem", "Caherdaniel", "Waterville", "Cahersiveen", + "Valentia", "Portmagee", "Skellig", "Dingle", + "Brandon", "Castlegregory", "Fenit", "Tralee", + "Listowel", "Ballybunion", "Tarbert", "Glin", + "Foynes", "Askeaton", "Adare", "Patrickswell", + # More Scottish/Gaelic to fill + "Stornoway", "Tarbert", "Scalpay", "Eriskay", + "Barra", "Vatersay", "Minguilay", "Pabbay", + "Berneray", "Monach Isles", "Balranald", "Lochmaddy", + "Benbecula", "Grimsay", "Ronay", "Wiay", + "Canna", "Rum", "Eigg", "Muck", + "Ardnish", "Arisaig", "Morar", "Mallaig", + "Knoydart", "Barrisdale", "Arnisdale", "Glenelg", + "Sandaig", "Brochs of Borve", "Callanish", "Garenin", + "Carloway", "Arnol", "Barvas", "Tolsta", + "Ness", "Europie", "Swainbost", "Skigersta", + # Additional Welsh/Irish + "Aberystwyth", "Llanberis", "Betws-y-Coed", "Conwy", + "Caernarfon", "Pwllheli", "Abersoch", "Nefyn", + "Llanbedrog", "Criccieth", "Porthmadog", "Portmeirion", + "Trawsfynydd", "Ffestiniog", "Blaenau", "Llyn Tegid", + "Corwen", "Llangollen", "Chirk", "Oswestry", +] + + +def load_global_names(conn): + """Load all existing names globally for uniqueness checking.""" + names = set() + for table in ['atlas_cities', 'atlas_rivers', 'atlas_mountain_ranges', + 'atlas_oceans', 'atlas_pois']: + rows = conn.execute( + f"SELECT lower(name) FROM {table} WHERE name IS NOT NULL AND name != ''" + ).fetchall() + names.update(r[0] for r in rows) + return names + + +def main(): + conn = sqlite3.connect(str(DB_PATH), timeout=30.0) + conn.execute("PRAGMA journal_mode=WAL") + conn.execute("PRAGMA busy_timeout=15000") + + global_names = load_global_names(conn) + print(f"Loaded {len(global_names)} existing names") + + # Build available replacements (not already in corpus) + available = [n for n in REPLACEMENT_POOL if n.lower() not in global_names] + print(f"Available replacements: {len(available)} (from pool of {len(REPLACEMENT_POOL)})") + + # Find all features that need replacement + replacements_needed = [] + for markers_path in sorted(WIKI_SYSTEMS.glob("*/bodies/*/markers.json")): + body_id = markers_path.parent.name + m = json.loads(markers_path.read_text()) + for section in ("cities", "rivers", "oceans", "mountain_ranges", "pois"): + for feat in m.get(section, []): + name = feat.get("name", "") + if name and name.lower() in FEWSHOT_NAMES: + replacements_needed.append((markers_path, body_id, section, feat)) + + print(f"Features to replace: {len(replacements_needed)}") + + if len(available) < len(replacements_needed): + print(f"WARNING: only {len(available)} replacements for {len(replacements_needed)} features") + print(" some features will keep their few-shot names") + + # Group by file, reload, replace, write + replacement_idx = 0 + used_per_body = defaultdict(set) + changed_bodies = [] + + # Group by file + by_file = defaultdict(list) + for markers_path, body_id, section, feat in replacements_needed: + by_file[markers_path].append((body_id, section, feat["id"] if "id" in feat else None)) + + for markers_path, entries in by_file.items(): + body_id = markers_path.parent.name + m = json.loads(markers_path.read_text()) + changed = False + + for _, section, feat_id in entries: + for feat in m.get(section, []): + name = feat.get("name") or "" + if not name or name.lower() not in FEWSHOT_NAMES: + continue + + assigned = None + for attempt in range(len(available)): + candidate = available[(replacement_idx + attempt) % len(available)] + if candidate.lower() not in used_per_body[body_id]: + assigned = candidate + replacement_idx = (replacement_idx + attempt + 1) % len(available) + break + + if assigned: + feat["name"] = assigned + used_per_body[body_id].add(assigned.lower()) + changed = True + + if changed: + markers_path.write_text(json.dumps(m, indent=2) + "\n") + changed_bodies.append(body_id) + # Sync to DB if body exists + try: + sync_markers_to_db(conn, body_id, m) + except Exception: + pass # orphan body + + conn.commit() + conn.close() + + print(f"\nReplaced few-shot names on {len(changed_bodies)} bodies") + print("Done.") + + +if __name__ == "__main__": + main() diff --git a/tooling/planet-gen/gemma_naming.py b/tooling/planet-gen/gemma_naming.py new file mode 100755 index 000000000..494d59369 --- /dev/null +++ b/tooling/planet-gen/gemma_naming.py @@ -0,0 +1,2343 @@ +#!/usr/bin/env python3 +""" +gemma_naming.py — Batch-name every empty name field in the reach's +markers.json files using the Gemma 4 E2B tooling pipeline (#833, D-191 §4). + +Pipeline per body: + 1. Load markers.json; identify feature records whose `name` is empty + or null. Hand-authored names are never overwritten. + 2. Build a short corridor-aware prompt per feature. + 3. Stream the prompts into `sr-voice serve --stdio` (long-lived + subprocess, restarted every --refresh requests to prevent KV-cache + context bleed). + 4. Post-process each response: strip quotes, trim whitespace, reject + blocklisted Earth majors, retry with a bumped seed on collision or + on blocklist hit (up to 3 attempts), fall back to a deterministic + palette-driven name on persistent failure. + 5. Dedup within (cultural_corridor, feature_type) so two bodies in the + same corridor never ship the same river name; cross-corridor + collisions are allowed (two "Aldren"s on opposite arcs is fine). + 6. Write markers.json back (only if any field changed). + 7. Sync every touched body's atlas_* rows in systems.db so + `atlas_cities.name`, `atlas_rivers.name`, etc. pick up the new + strings without needing a follow-up generate_atlas.py pass. + +Usage: + tooling/planet-gen/gemma_naming.py # full batch, real model + tooling/planet-gen/gemma_naming.py --body GJ380c # single body + tooling/planet-gen/gemma_naming.py --limit 5 --verbose # smoke test + tooling/planet-gen/gemma_naming.py --mock # mock-stdio.sh (no model) + tooling/planet-gen/gemma_naming.py \\ + --sr-voice ~/Projects/settled-reach/binaries/sr-voice-tooling \\ + --model ~/Projects/settled-reach/models/gemma-4.gguf + +Exit codes: + 0 pipeline completed (possibly with skipped bodies) + 1 fatal error (subprocess crash, missing binary, missing schema) +""" + +import argparse +import datetime +import hashlib +import json +import os +import re +import signal +import subprocess +import sys +import time +from pathlib import Path + +TOOLING_DIR = Path(__file__).resolve().parent +REPO_ROOT = (TOOLING_DIR / ".." / "..").resolve() + +# Reuse the atlas DB sync logic from generate_atlas.py so there is one +# authoritative path for atlas_* row updates. +sys.path.insert(0, str(TOOLING_DIR)) +from generate_atlas import ( # noqa: E402 + GRID_H, + GRID_W, + ensure_atlas_schema, + sync_markers_to_db, +) + +import sqlite3 # noqa: E402 +from naming_core import ( # noqa: E402 + name_features_batch, + mood_for_body, +) + +DB_PATH = REPO_ROOT / "server" / "data" / "systems.db" +WIKI_SYSTEMS = REPO_ROOT / "wiki" / "star-systems" +BLOCKLIST_PATH = TOOLING_DIR / "earth_blocklist.txt" + +# NOTE: The following are vestigial from the Gemma 2 single-name pipeline. +# The live path uses _batch_fill() → name_features_batch() from naming_core. +# TODO(#833): remove in a cleanup pass. Full dead-code island (~750 lines): +# - _CAPTURE_FILE, --dump-prompts argparse (here + main()) +# - _build_prompt() and its few-shot example pools (~lines 500-929) +# - post_process(), is_placeholder(), _PLACEHOLDER_TOKENS, _LABEL_PREFIX, +# _MD_BOLD, _MD_UNDER (~lines 1040-1100) +# - is_blocked(), load_blocklist() (~lines 1105-1140) +# - fallback_name(), _FALLBACK_STEMS, _FALLBACK_SUFFIXES (~lines 1145-1205) +# - name_feature() with its retry loop and _is_duplicate (~lines 1530-1655) +_CAPTURE_FILE = None # vestigial — see note above + +# Default binary + model paths. The sr-voice binary is platform-specific +# (GPU backend baked in per-build) and lives OUTSIDE any git worktree so +# it survives sprint-worktree cleanup: +# +# ~/Projects/settled-reach/binaries/sr-voice-rocm (AMD / ROCm) +# ~/Projects/settled-reach/binaries/sr-voice-cuda (NVIDIA, future) +# ~/Projects/settled-reach/binaries/sr-voice-vulkan (cross-vendor, future) +# ~/Projects/settled-reach/binaries/sr-voice-cpu (fallback) +# +# See #850 for the multi-backend release-binary ticket. The container +# used to build these is created via the distrobox recipe documented in +# the sr-voice README. +# +# The Gemma 2 model weights live under main/server/models and are shared +# across worktrees (too large to duplicate). +HOME_PROJECTS = Path.home() / "Projects" / "settled-reach" +BINARIES_DIR = HOME_PROJECTS / "binaries" +MODELS_DIR = HOME_PROJECTS / "models" +MAIN_WORKDIR = Path("/var/mnt/data/projects/settled-reach/main") + + +def _find_sr_voice() -> Path: + """Resolve the default sr-voice binary path. + + Preference order: + 1. $HOME/Projects/settled-reach/binaries/sr-voice-tooling — Gemma 4 + tooling binary, preferred for content generation. + 2. $HOME/Projects/settled-reach/binaries/sr-voice-rocm — Gemma 2 + ROCm binary, fallback. + 3. main workdir's target/release/sr-voice — legacy. + """ + tooling_bin = BINARIES_DIR / "sr-voice-tooling" + if tooling_bin.exists(): + return tooling_bin + rocm_bin = BINARIES_DIR / "sr-voice-rocm" + if rocm_bin.exists(): + return rocm_bin + return MAIN_WORKDIR / "server" / "sr-voice" / "target" / "release" / "sr-voice" + + +def _find_default_model() -> Path: + """Resolve the default model path. Prefers Gemma 4 over Gemma 2.""" + gemma4 = MODELS_DIR / "gemma-4.gguf" + if gemma4.exists(): + return gemma4 + return MAIN_WORKDIR / "server" / "models" / "gemma2.gguf" + + +DEFAULT_SR_VOICE = _find_sr_voice() +DEFAULT_MODEL = _find_default_model() +MOCK_STDIO = REPO_ROOT / "server" / "sr-voice" / "mock-stdio.sh" + + +# --------------------------------------------------------------------------- +# Tee logger — stdout + log file in one call +# --------------------------------------------------------------------------- + +class Logger: + """Write lines to stdout AND an optional log file. + + Every message gets a prefix of the form `[HH:MM:SS +00h03m]`: + - HH:MM:SS is wall-clock local time, + - +NNhMMm is the elapsed time since the Logger was constructed. + The elapsed offset tells the user at a glance how long the run has + been going without scrolling back to the banner line. Flushes after + every line so a kill -9 loses at most one entry. + """ + + def __init__(self, log_path: Path | None): + self.log_path = log_path + self.started_at = time.monotonic() + self.fh = None + if log_path is not None: + log_path.parent.mkdir(parents=True, exist_ok=True) + # Truncate on open so each run starts fresh — the user can + # rename an old log before kicking off the next run. + self.fh = log_path.open("w", buffering=1) # line buffered + + def _elapsed(self) -> str: + secs = int(time.monotonic() - self.started_at) + return f"+{secs // 3600:02d}h{(secs % 3600) // 60:02d}m" + + def _prefix(self) -> str: + clock = datetime.datetime.now().strftime("%H:%M:%S") + return f"[{clock} {self._elapsed()}]" + + def __call__(self, msg: str = "") -> None: + line = f"{self._prefix()} {msg}" if msg else "" + print(line, flush=True) + if self.fh is not None: + self.fh.write(line + "\n") + self.fh.flush() + + def raw(self, msg: str = "") -> None: + """Print without the timestamp prefix (for banner lines).""" + print(msg, flush=True) + if self.fh is not None: + self.fh.write(msg + "\n") + self.fh.flush() + + def close(self) -> None: + if self.fh is not None: + self.fh.close() + self.fh = None + + +# --------------------------------------------------------------------------- +# Corridor palettes (D-191 §4, glossary.md §Corridors, decisions/economics.md D-175) +# --------------------------------------------------------------------------- + +# Each palette is the cultural inflection the prompt asks Gemma to +# produce names in. Palette keys match the values of +# `star_systems.geographic_sector` directly — that column is the real +# source of corridor identity in systems.db (cultural_corridor is a +# legacy field that was never populated beyond sol-gateway-axis). +CORRIDOR_SUBSTYLES: dict[str, list[dict[str, str]]] = { + # Each corridor has a list of sub-style inflections. The pipeline + # picks one per body via hash(body_id) so neighbouring bodies on the + # same planet get different registers, and Gemma's narrow per-register + # vocabulary (~15 stems) stays fresh across hundreds of bodies. + # + # IMPORTANT: the inflection is a DOMINANT bias, not a hard lock. + # A British surveyor on an east_reach moon still names a river after + # their aunt in Dorset. Each sub-style explicitly names its register + # AND invites diaspora variety. + "core": [ + {"inflection": "English countryside, rural, agricultural settlers", + "examples": "Thornbury, Bramblewood, Millbrook, Ashford, Weston"}, + {"inflection": "British colonial settlement era", + "examples": "New Bristol, Port Augusta, Kingstown, Admiralty, Georgetown"}, + {"inflection": "American frontier, practical, geographic", + "examples": "Dusty Creek, Twin Oaks, Cedar Flat, Hawk's Hollow, Red Bluff"}, + {"inflection": "American municipal, administrative, cosmopolitan", + "examples": "Prospect Heights, Liberty, Union, Meridian, Commonwealth"}, + {"inflection": "Classical references, institutional, civic", + "examples": "Concordia, Aurelius, Prefecture, Senate Landing, Forum"}, + {"inflection": "Australian and New Zealand settler", + "examples": "Redfern, Glenelg, Wollongong, Kaikoura, Hawke's Bay"}, + ], + "north_reach": [ + {"inflection": "English rural, village and parish names", + "examples": "Wolcott, Mildern, Ashbourne, Briarfell, Tarndale"}, + {"inflection": "Scottish Highland and Lowland place-names", + "examples": "Glenmoray, Dunfermline, Kinross, Brae, Dalwhinnie"}, + {"inflection": "Australian outback, station and property names", + "examples": "Redfern, Birdsville, Tennant, Woomera, Coober"}, + {"inflection": "Irish rural and coastal settlement", + "examples": "Ballymore, Kilrush, Dunmore, Tralee, Skellig"}, + {"inflection": "South African English settler", + "examples": "Grahamstown, Oudtshoorn, Stellenbosch, Graaff, Beaufort"}, + ], + "south_reach": [ + {"inflection": "Portuguese colonial era, Iberian", + "examples": "Monteforte, Serra, Tavira, Oliveira, Porto Novo"}, + {"inflection": "Brazilian interior, frontier settlement", + "examples": "Ribeirão, Campo Largo, Várzea, Ilhabela, Pinheiro"}, + {"inflection": "East African Swahili coastal", + "examples": "Inhambane, Kilimi, Ngola, Manhica, Quelimane"}, + {"inflection": "Cape Verdean and West African", + "examples": "Cabo, Moçambo, Ribeira, Mindelo, Tarrafal"}, + {"inflection": "Angolan and Mozambican settlement", + "examples": "Huambo, Lobito, Nampula, Lichinga, Benguela"}, + ], + "east_reach": [ + {"inflection": "Korean place-name tradition", + "examples": "Hanyang, Seorak, Baektu, Saeyeon, Taegong"}, + {"inflection": "Japanese rural and coastal settlement", + "examples": "Takamine, Ginoza, Tsukuri, Aomori, Fukagawa"}, + {"inflection": "Taiwanese and Hakka settler", + "examples": "Jiufen, Beigang, Hsinchu, Meinong, Tainan"}, + {"inflection": "Filipino settler community", + "examples": "Batangas, Legazpi, Tuguegarao, Zambales, Tarlac"}, + {"inflection": "Mixed East Asian diaspora, cosmopolitan", + "examples": "Naruhan, Morimine, Kōzan, Midori, Kawasaki"}, + ], + "west_reach": [ + {"inflection": "German settlement, orderly and compound names", + "examples": "Altdorf, Drachenberg, Feldberg, Krakenberg, Lüneborg"}, + {"inflection": "Dutch colonial, low-country", + "examples": "Kloosterdam, Hoogland, Oudewater, Nieuwpoort, Voorhout"}, + {"inflection": "Nordic and Scandinavian", + "examples": "Sørholm, Torsfell, Bergfjord, Lindeborg, Nordhölm"}, + {"inflection": "Polish and Czech settler", + "examples": "Krakowice, Bystrica, Wieliczka, Tarnów, Ostrava"}, + {"inflection": "Baltic and Finnish settler", + "examples": "Järvenpää, Tallinna, Pärnu, Turku, Rakvere"}, + ], + "deep_frontier": [ + {"inflection": "frontier founder-name era, surname-first, any Earth culture", + "examples": "Okafor Reach, Stenner Cross, Weller Hold, Pruitt Basin"}, + {"inflection": "frontier descriptive, geographic features named by surveyors", + "examples": "Red Mesa, Dry Fork, Iron Flat, Long Ridge, Dust Basin"}, + {"inflection": "frontier outpost, functional and military", + "examples": "Forward Post, Relay Station, Survey Camp, Waypoint, Anchor"}, + ], +} + +# Legacy aliases +CORRIDOR_SUBSTYLES["sol-gateway-axis"] = CORRIDOR_SUBSTYLES["core"] +CORRIDOR_SUBSTYLES["inner_corridor"] = CORRIDOR_SUBSTYLES["core"] +CORRIDOR_SUBSTYLES["inner_orbit"] = CORRIDOR_SUBSTYLES["core"] +CORRIDOR_SUBSTYLES["frontier"] = CORRIDOR_SUBSTYLES["deep_frontier"] + +DEFAULT_SUBSTYLES = CORRIDOR_SUBSTYLES["core"] + +# Processing order for the main run — core first so those bodies win +# the dedup race and the outer sectors fall into the palette fallback +# path when names collide. +SECTOR_PRIORITY: dict[str, int] = { + "core": 0, + "north_reach": 1, + "south_reach": 2, + "east_reach": 3, + "west_reach": 4, + "deep_frontier": 5, +} + + +def palette_for(corridor: str | None, system_id: str = "") -> dict[str, str]: + """Pick a sub-style for this system within its corridor. + + All bodies in the same system get the same sub-style (consistent + cultural register per star system). Different systems rotate through + the sub-style list via hash(system_id). + + This is the FALLBACK path — the preferred path is select_register() + which asks Gemma to pick the register based on wiki/GTTR content. + """ + substyles = CORRIDOR_SUBSTYLES.get(corridor or "core", DEFAULT_SUBSTYLES) + idx = int(hashlib.sha256(system_id.encode()).hexdigest()[:8], 16) % len(substyles) + return substyles[idx] + + +def _system_slug(system_id: str) -> str: + """Convert system_id ('GJ 411') to wiki directory slug ('GJ-411').""" + if system_id.startswith("GJ "): + return "GJ-" + system_id[3:] + return system_id + + +def load_wiki_context(system_id: str) -> tuple[str | None, str | None]: + """Read index.md and gttr.md for a system from wiki/star-systems/. + + Returns (index_text, gttr_text). Either or both may be None if the + file doesn't exist. + """ + slug = _system_slug(system_id) + sys_dir = WIKI_SYSTEMS / slug + index_path = sys_dir / "index.md" + gttr_path = sys_dir / "gttr.md" + index_text = index_path.read_text() if index_path.exists() else None + gttr_text = gttr_path.read_text() if gttr_path.exists() else None + return index_text, gttr_text + + +def _extract_cultural_lines(wiki_text: str, max_lines: int = 8) -> str: + """Pull the most culturally relevant lines from a wiki index.md. + + Scans for lines mentioning heritage, founding identity, language, + cultural texture, or corridor affiliation. Falls back to the first + prose paragraphs if no keyword hits. Keeps the excerpt short enough + for Gemma 2 2B's 1024-token context. + """ + keywords = ( + "cultural", "heritage", "founding", "settler", "surname", + "language", "tradition", "diaspora", "population carried", + "portuguese", "iberian", "japanese", "korean", "chinese", + "filipino", "german", "dutch", "nordic", "scandinavian", + "polish", "czech", "finnish", "baltic", "swahili", "african", + "angolan", "cape verde", "irish", "scottish", "australian", + "british", "brazilian", "mozambic", "norwegian", "frisian", + "afrikaans", "lusophone", "corridor", + ) + hits: list[str] = [] + prose: list[str] = [] + for line in wiki_text.splitlines(): + stripped = line.strip() + if not stripped or stripped.startswith("#") or stripped.startswith("|") or stripped.startswith("---") or stripped.startswith("