settled-reach/docs/architecture/atlas-naming-pipeline.md

# Atlas Naming Pipeline

Reference document for the feature-naming pipeline that populates
`markers.json` files and the `atlas_*` tables in `systems.db`.

Related: ticket #833, D-191 §4.

---

## 1. Overview

The atlas naming pipeline generates human-readable names for every
unnamed geographic feature across ~2,394 planetary bodies in the
Settled Reach. Features covered: cities (capital and secondary),
rivers, oceans/seas/lakes, mountain ranges, and points of interest
(transit, institutional, cultural).

Inputs:
- `wiki/star-systems/*/bodies/*/markers.json` — one file per body,
  containing arrays of feature records with optional `name` fields
- `server/data/systems.db` — `star_systems`, `bodies`, `atlas_*` tables
- `wiki/star-systems/*/index.md` and `gttr.md` — cultural context for
  register selection
- `tooling/planet-gen/earth_blocklist.txt` — exact-match rejection list
  for well-known Earth place names

Outputs:
- `markers.json` files updated in-place with the new `name` fields
- `atlas_cities`, `atlas_rivers`, `atlas_mountain_ranges`, `atlas_oceans`,
  `atlas_pois` rows in `systems.db` synced immediately after each body

Hand-authored names in `markers.json` are never overwritten. Re-running the
pipeline is safe: any body whose features all have non-empty `name` fields is
skipped entirely.

Wall time on an RX 9070 (ROCm) is roughly 4-6 hours for a full run.

---

## 2. Architecture

### The inference binary: sr-voice-tooling

The pipeline does NOT use the game's Gemma 2 runtime (`sr-voice`). It uses a
separate binary, `sr-voice-tooling`, built against Gemma 4 (GGUF) and compiled
for the ROCm / HIP stack. The binary lives outside all git worktrees at:

```
~/Projects/settled-reach/binaries/sr-voice-tooling
~/Projects/settled-reach/models/gemma-4.gguf
```

The split exists because the game's runtime (Gemma 2, 2B parameters, 1024-token
context) is optimised for low-latency in-game NPC dialogue, while the tooling
binary trades latency for richer outputs and a larger context window.

### stdio JSONL protocol

`VoiceSubprocess` in `gemma_naming.py` manages the binary as a long-lived
subprocess. Communication is newline-delimited JSON on stdin/stdout:

Request (one JSON object per line):
```json
{"prompt": "...", "seed": 12345678}
```

Response:
```json
{"text": "..."}
```

or on error:
```json
{"error": "..."}
```

The subprocess is restarted every `--refresh` requests (default: 1000) to
prevent KV-cache context bleed accumulating across bodies. Seeds are derived
deterministically from `(body_id, feature_type, world_seed)` so the same
prompt always produces the same output — re-running for a partially-completed
body reproduces the same names, which helps diagnose quality issues without
introducing fresh variation.

### Running inside distrobox

The ROCm HIP kernels embedded in `sr-voice-tooling` are compiled for gfx1201
(AMD Navi 48 / RX 9070). Because `libhipblas.so.2` and related ROCm libraries
are only installed in the `reach-build` distrobox container, `run-atlas-naming.sh`
and the test scripts invoke the binary via `distrobox enter reach-build --`.
The shell wrapper verifies the gfx1201 target is present before starting.

---

## 3. Pipeline Stages

For each body, `process_body()` executes the following stages in order.

### Stage 1: Register selection (per system, wiki-grounded)

At the start of each new system, `select_register()` asks Gemma to choose
the best cultural register from the corridor's `CORRIDOR_SUBSTYLES` list,
based on the system's `wiki/star-systems/{slug}/index.md` and `gttr.md`.

`load_wiki_context()` reads both files. `_extract_cultural_lines()` scans
`index.md` for lines containing cultural keywords (heritage, founding, settler,
diaspora, specific language names, etc.) and returns up to 8 of them. The first
substantive paragraph from `gttr.md` (skipping the title line) is used as the
context hook.

The prompt is few-shot: two worked examples from different corridors, then the
target system's context followed by the numbered option list. Gemma replies
with a single digit. The pipeline retries up to `max_attempts=3` times with
rotated seeds. On failure it falls back to `palette_for()`, a hash-based
deterministic selection: `hash(system_id) % len(substyles)`.

The selected register applies to all bodies in that system. All bodies in the
same system therefore share a consistent cultural voice.

### Stage 2: Mood injection (per body)

`mood_for_body(body_id, world_seed)` picks one entry from `MOOD_POOL` via
`SHA-256(f"mood|{body_id}|{world_seed}")`. The 13-word pool is:

```
ambition, family, wealth, community, industry, pride, fleeting,
hope, fear, isolation, devotion, defiance, loss
```

The chosen mood is injected into each batch prompt as a clause:
`"The settlers here had a sense of {mood}."` This is the primary mechanism for
vocabulary divergence across bodies in the same system — same cultural register,
different emotional colouring. The mood is deterministic per body so re-runs
produce identical results.

### Stage 3: Batch naming (2x oversampled)

`name_features_batch()` (in `naming_core.py`) groups blank features by type and
requests `count * 2` names in a single prompt call. The oversample factor means
the ranking step (Stage 4) has candidates to work with even if the model repeats
some names.

`build_batch_prompt()` assembles the prompt with:
- A preamble emphasising mundane, practical settler names over epic or classical
  ones
- Optional system name, body name, and `system_hook` (the gttr.md
  characterisation)
- The mood clause (if a mood was injected)
- The "already used" list (trimmed to fit within `ctx_size` tokens)
- Two fixed few-shot examples showing comma-separated batch format
  (Scottish Highland and Dutch colonial)
- The tail: `"Style: {inflection}.   {count*2} names:"`

The model pattern-completes a comma-separated list. `parse_batch_response()`
takes the first line only (the model often continues with explanation), splits
on commas, strips quotes, filters via `is_valid_name()`, and deduplicates
preserving order.

For single-name prompts (the legacy `_build_prompt()` / `name_feature()` path),
the prompt uses a rotating pool of few-shot style+answer pairs. The pool is
picked deterministically by `hash(body_id | local_id | attempt)` so retries
rotate to a different example set rather than re-querying with the same prompt
and a different seed.

### Stage 4: Levenshtein distinctiveness ranking

`select_distinct(candidates, count, taken)` in `naming_core.py` greedily
selects the N most distinct names from the oversampled candidate list.

The distinctiveness metric is `word_avg_distance()`: for each word in the
shorter name, find the closest word (by Levenshtein edit distance) in the
longer name, then average across words. This means shared structural words
(Serra, The, Glen, Ridge) lower a candidate's preference score but do not
block it outright — the unique words in the name pull the average up.

Selection is greedy: pick the candidate with the highest minimum distance to
all already-selected names plus all `taken` names, add it to the selected set,
repeat. No hard threshold — all candidates except exact string matches to
`taken` are eligible.

### Stage 5: Cross-body dedup

The `corpus` dict in `main()` maps `(system_id, feature_type)` to the set of
all names already assigned within that system. Before each batch call, `taken`
is built from `corpus[(system_id, ft)]` plus `body_used` (all names already
assigned to the current body across all feature types). This enforces two
constraints:

- No body has the same name for two different features (e.g. a river and a
  mountain range).
- No two bodies in the same system share the same name for the same feature
  type (e.g. two rivers in GJ 144 named "Cooper's Creek").

Cross-corridor collisions (the same name appearing in two different systems
from different corridors) are allowed and expected.

`taken` is pruned when it exceeds the context budget: the most recently added
entries are dropped first. This means a very large system (many bodies, many
features) may eventually lose dedup coverage for its earliest names at the
tail of the run.

### Stage 6: Adjacent-register refill

If the batch call + ranking still yields fewer than `count` names (after
dedup against `taken`), `name_features_batch()` fires a refill call using the
next sub-style in the corridor's `CORRIDOR_SUBSTYLES` list, wrapping around.
The refill asks for `shortfall * 3` candidates. Refill results go through the
same `select_distinct()` ranking against `taken + selected`. This ensures
the feature list is always fully populated even when the primary register's
vocabulary is exhausted.

---

## 4. Cultural Registers

### CORRIDOR_SUBSTYLES

`CORRIDOR_SUBSTYLES` in `gemma_naming.py` is a `dict[str, list[dict[str, str]]]`
mapping corridor name to an ordered list of sub-style dicts. Each sub-style has
two keys: `inflection` (the style label injected into the prompt) and `examples`
(a comma-separated string of illustrative names, shown during `select_register()`
option display but NOT injected into batch prompts — only the inflection label
reaches the model during naming).

Active corridors and their sub-styles:

**core** (6 sub-styles)
- English countryside, rural, agricultural settlers
- British colonial settlement era
- American frontier, practical, geographic
- American municipal, administrative, cosmopolitan
- Classical references, institutional, civic
- Australian and New Zealand settler

**north_reach** (5 sub-styles)
- English rural, village and parish names
- Scottish Highland and Lowland place-names
- Australian outback, station and property names
- Irish rural and coastal settlement
- South African English settler

**south_reach** (5 sub-styles)
- Portuguese colonial era, Iberian
- Brazilian interior, frontier settlement
- East African Swahili coastal
- Cape Verdean and West African
- Angolan and Mozambican settlement

**east_reach** (5 sub-styles)
- Korean place-name tradition
- Japanese rural and coastal settlement
- Taiwanese and Hakka settler
- Filipino settler community
- Mixed East Asian diaspora, cosmopolitan

**west_reach** (5 sub-styles)
- German settlement, orderly and compound names
- Dutch colonial, low-country
- Nordic and Scandinavian
- Polish and Czech settler
- Baltic and Finnish settler

**deep_frontier** (3 sub-styles)
- frontier founder-name era, surname-first, any Earth culture
- frontier descriptive, geographic features named by surveyors
- frontier outpost, functional and military

Legacy aliases (`sol-gateway-axis`, `inner_corridor`, `inner_orbit`) map to
`core`. `frontier` maps to `deep_frontier`. The pipeline reads the corridor
from `COALESCE(cultural_corridor, geographic_sector, 'core')` in
`star_systems` — `geographic_sector` is the live field; `cultural_corridor`
was a legacy column never populated beyond the sol-gateway-axis.

### Sub-style rotation

All bodies within a system use the same register (chosen by `select_register()`
or `palette_for()`). Across systems within a corridor the hash-based fallback
rotates through the sub-style list: `hash(system_id) % len(substyles)`. This
ensures neighbouring systems don't all sound identical even when `select_register()`
fails.

### Inflection as dominant bias, not hard lock

The `CORRIDOR_SUBSTYLES` comment is explicit: the inflection is a dominant bias,
not a hard cultural lock. A British surveyor on an east_reach moon still names
a river after their aunt in Dorset. The few-shot examples in the batch prompt
show cross-cultural names to reinforce this — the examples are Scottish Highland
and Dutch colonial regardless of the target corridor.

### SECTOR_PRIORITY

`SECTOR_PRIORITY` sets the processing order: `core=0`, `north_reach=1`,
`south_reach=2`, `east_reach=3`, `west_reach=4`, `deep_frontier=5`. Core
bodies are named first so they win the dedup race; frontier bodies fall into
the adjacent-register refill path when names collide.

---

## 5. Body Processing Order

`discover_bodies()` discovers all `markers.json` files via
`WIKI_SYSTEMS.glob("*/bodies/*/markers.json")` and sorts them by a 5-tuple
sort key loaded from `load_body_hop_order()`:

```
(hop_distance, system_id, is_inhabited_desc, population_rank, body_id)
```

- **Hop distance** from Gateway (GJ 71), ascending. Core systems process first.
- **System grouping**: all bodies in a system are processed consecutively
  (same `system_id` in the sort key).
- **Inhabited first**: within a system, inhabited bodies sort before
  uninhabited ones.
- **Population rank**: within inhabited bodies, higher-population bodies sort
  earlier (so the main inhabited world of a system wins the dedup race against
  minor colonies).
- Bodies whose `body_id` is absent from `hop_order` (markers.json orphans
  not in the DB) sort to the end with a sentinel hop of 99.

---

## 6. Key Files and Their Roles

| File | Role |
|------|------|
| `tooling/planet-gen/gemma_naming.py` | Main pipeline: `VoiceSubprocess`, `CORRIDOR_SUBSTYLES`, `palette_for()`, `select_register()`, `_build_prompt()` (single-name path), `_PROMPT_CONFIG`, `process_body()`, `main()`. |
| `tooling/planet-gen/naming_core.py` | Shared algorithms: `levenshtein()`, `word_avg_distance()`, `select_distinct()`, `build_batch_prompt()`, `parse_batch_response()`, `name_features_batch()`, `mood_for_body()`, `MOOD_POOL`, `is_valid_name()`. |
| `tooling/planet-gen/run-atlas-naming.sh` | Shell wrapper for overnight runs. Validates the binary (`gfx1201` kernel check), model, and distrobox container before exec-ing into `gemma_naming.py`. Hardcodes paths — no arguments accepted. |
| `tooling/planet-gen/qa_naming.py` | QA report: reads `systems.db` and `markers.json` files, runs the full check suite, prints a summary with issue rates. |
| `tooling/planet-gen/test_batch_naming.py` | Integration test harness. Runs 4 simulated systems (20 bodies) against real Gemma 4 via `sr-voice-tooling`, accumulates the taken list across bodies, prints candidates/selected/refill for visual inspection. |
| `tooling/planet-gen/test_register_selection.py` | Tests `select_register()` against 8 real systems spanning all corridors, prints what Gemma picks vs. the hash-based fallback. |
| `tooling/planet-gen/earth_blocklist.txt` | Lowercase exact-match list of Earth major place names. `is_blocked()` also strips a leading "The " before comparing. |
| `~/Projects/settled-reach/binaries/sr-voice-tooling` | The Gemma 4 inference binary. Lives outside the repo. Built with `CMAKE_HIP_ARCHITECTURES=gfx1201` inside the `reach-build` distrobox. |
| `~/Projects/settled-reach/models/gemma-4.gguf` | The model weights. Shared across worktrees. |
| `server/data/systems.db` | SQLite database. `star_systems.geographic_sector` is the corridor source of truth. `atlas_*` tables are synced per body immediately after `markers.json` is written. |

`generate_atlas.py` provides `ensure_atlas_schema()` and `sync_markers_to_db()`,
both imported by `gemma_naming.py`. This makes `generate_atlas.py` the single
authoritative path for `atlas_*` row writes.

---

## 7. Known Limitations

### Classical register sameyness

The "Classical references, institutional, civic" sub-style in the `core`
corridor (Concordia, Aurelius, Prefecture, Forum) has a narrow vocabulary.
Gemma 4 has roughly 15 usable stems per register. On a system with many
bodies, classical register bodies at the tail of the run exhaust the fresh
stem pool and the adjacent-register refill kicks in. The refill names are
structurally correct but may not feel classical. This is by design: the
frontier fallback is correct behaviour, not a failure mode.

### Few-shot example bleed

The two fixed batch few-shot examples (Scottish Highland, Dutch colonial) are
cross-corridor by design, but on short prompts (low `ctx_size`) they can
dominate the model's completion distribution. The result is names that have
a vaguely British or Dutch character even for east_reach or south_reach
bodies. Increasing `ctx_size` reduces this; at 1024 tokens it is detectable
but not severe.

### Narrow register exhaustion on large systems

The `taken` list is trimmed from the tail when it exceeds the token budget.
For systems with 30+ bodies (e.g. very large multi-moon systems), early
names may drop off the `taken` list before all bodies are processed, allowing
duplicates to slip through. `qa_naming.py`'s `check_exact_dupes_within_system()`
catches these; manual re-runs with `--body` are the fix.

### Stem repetition ("Ridge Ridge Ridge")

`check_stem_repetition()` detects this: 4 or more features on the same body
sharing the same first word. It is caused by the model having a strong
associative path from the inflection to one compound-word opener (e.g.
"Ridge" for frontier descriptive). The Levenshtein ranking partially mitigates
it — two names with identical first words have a low `word_avg_distance` and
the second one loses the greedy selection — but it does not eliminate it when
the entire candidate pool shares the stem.

### Single-name vs. batch path

`gemma_naming.py` contains two prompt-building paths: the legacy
`_build_prompt()` / `name_feature()` single-name path and the newer
`build_batch_prompt()` / `name_features_batch()` batch path. The batch path
is what `process_body()` calls via `_batch_fill()`. The single-name path
and its `_PROMPT_CONFIG` pools are still present and used by `_build_prompt()`,
which is called by `name_feature()` — that function may be invoked on retry
paths or by older call sites. The two paths use different prompt structures
and produce different output distributions.

---

## 8. QA Process

### Running

```bash
python3 tooling/planet-gen/qa_naming.py
python3 tooling/planet-gen/qa_naming.py --verbose
```

Requires `server/data/systems.db` to be present with populated `atlas_*` tables.

### What the checks look for

| Check | Function | Trigger |
|-------|----------|---------|
| Prompt fragment leaks | `check_prompt_fragments()` | Name contains substrings like `"style:"`, `"generate"`, `"already used"`, `"must be"`, etc. |
| Exact dupes within system | `check_exact_dupes_within_system()` | Same name, same feature type, two different bodies in the same system. |
| Exact dupes within body | `check_exact_dupes_within_body()` | Same name, different feature types, same body (e.g. a river and a mountain range both called "Thornbury"). |
| Stem repetition | `check_stem_repetition()` | 4 or more features of the same type on a body share the same first word. |
| Body name echo | `check_body_name_echo()` | A word from the body's proper name appears as a word in a feature name. |
| Register bleed | `check_register_bleed()` | Keyword heuristics flag obviously wrong cultural associations (e.g. "fjord" in south_reach, "sakura" in west_reach). |
| Feature type mismatch | `check_feature_type_mismatch()` | Street-vocabulary words on a mountain; building-vocabulary words on a river or ocean. |
| Short names | `check_short_long_names()` | Name is 3 characters or fewer. |
| Long names | `check_short_long_names()` | Name is more than 40 characters. |
| Numbers in names | `check_numbers_in_names()` | Name contains a digit. |
| Coverage gaps | `coverage_gaps()` | Bodies with one or more unnamed features remaining in `atlas_*` tables. Inhabited bodies listed separately. |

The summary block at the end prints a total issue rate (`issues / total_named *
100`). A clean run at the end of a full batch should sit below 1%. Prompt
fragment leaks above zero indicate a context-budget or parsing failure and
should be investigated before ship.

---

## 9. How to Extend

### Adding a new cultural register

1. Add a new dict entry to the relevant list in `CORRIDOR_SUBSTYLES`:
   ```python
   {"inflection": "...", "examples": "..."}
   ```
2. The pipeline picks it up automatically. `palette_for()` uses
   `len(substyles)` for the modulo, so the new sub-style becomes available
   on the next hash rotation. `select_register()` presents it as a numbered
   option to Gemma.
3. Run `test_register_selection.py` to verify Gemma picks the new register
   for a system whose wiki content matches it.

### Tuning temperature and sampling

Temperature is a parameter of the `sr-voice-tooling` binary, not of the
Python scripts. Pass flags to the binary via the `--sr-voice` argument or
edit `run-atlas-naming.sh`. The scripts have no Python-side temperature knob.

### Adding new moods

Extend `MOOD_POOL` in `naming_core.py`. The deterministic hash spreads the
new mood across bodies automatically on the next full run. Existing bodies
whose `markers.json` already has names will be skipped, so only un-named
bodies pick up the new mood entries.

### Running for a single system or body

Single body:
```bash
python3 tooling/planet-gen/gemma_naming.py --body GJ380c
```

Single body with verbose output showing per-feature retry detail:
```bash
python3 tooling/planet-gen/gemma_naming.py --body GJ380c --verbose
```

Smoke test (first 5 bodies in hop order):
```bash
python3 tooling/planet-gen/gemma_naming.py --limit 5 --verbose
```

Without a real model (mock stdio for pipeline testing):
```bash
python3 tooling/planet-gen/gemma_naming.py --mock --limit 5
```

Dump prompts to JSONL without calling the model (for A/B comparison against
a different backend):
```bash
python3 tooling/planet-gen/gemma_naming.py \
    --dump-prompts /tmp/prompts.jsonl \
    --limit 10
```

All invocations use `--refresh 1000` by default (subprocess restart every
1000 requests). Lower it to `--refresh 100` if you observe quality degradation
across a long single-system run.