settled-reach/spikes/heightmap-pipeline/PIPELINE.md

Heightmap Pipeline — Spike Documentation

Ticket: #778 Author: Araminta Date: 2026-04-05 Status: Spike complete — awaiting review before batch (#794, Sprint 33) Output: heightmaps/GJ144d_kallast.png (4096×2048px)

---

What This Spike Validates

This spike validates the full annotated heightmap pipeline from wiki data through to a deliverable PNG. Every stage ran successfully on Kallast (GJ144d, Ran system):

• pyplatec tectonic simulation → elevation grid
• Erosion pass → softer ridges, valley hints
• Dynamic sea level → correct 40% land coverage from wiki spec
• Terrain classification → 13 biome classes
• D8 flow accumulation → river network
• Settlement placement snapped to appropriate terrain class
• Road network connecting all cities
• Annotated render with title/legend in Settled Reach visual grammar

The pipeline is confirmed viable for batch production (#794).

---

Planet: Kallast (GJ144d)

Selected because it showcases all annotation types:

Property	Value	Source
Planet ID	GJ144d	systems.db
System	Ran (GJ 144)	systems.db
Biome	temperate	systems.db
Hydrosphere	ocean	systems.db
Land coverage	40%	wiki: "amber continental shelves"
Population	2,000,000,000	systems.db
Settlement wave	1 (580y)	systems.db
Settlement pattern	urban_concentrated	systems.db
Industrial	Agricultural_Syndic	systems.db
Terrain character	Extensive temperate plains, amber-toned grassland	wiki narrative

Kallast was chosen over higher-population worlds (Haodu, etc.) because the wiki narrative explicitly describes the terrain features we need to annotate: "amber continental shelves broken by irrigation channels wide enough to see from low orbit." That text is a direct visual brief. The pipeline output should feel consistent with it.

---

Pipeline Architecture

Input: Planet profile (wiki + systems.db)
         ↓ planet_type, land_fraction, settlement data

Stage 1: Tectonic simulation (pyplatec)
         platec.create(seed, W, H, sea_level=land_fraction, …)
         platec.step() × 200  [200 steps for mature, well-eroded world]
         platec.get_heightmap() → float list → reshape → normalize [0, 1]
         Runtime: ~1s at 512×256 (scales linearly with grid × steps)

Stage 2: Erosion (scipy gaussian_filter)
         Slope-weighted smoothing: steep cells erode more
         4 passes on mature world (reduce to 2 for young volcanic)
         Runtime: <0.5s at 512×256

Stage 3: Dynamic sea level
         sea_level = np.percentile(terrain, (1 - land_fraction) * 100)
         CRITICAL: pyplatec output is heavily right-skewed (most cells at low
         elevation). A fixed sea_level fraction (e.g. 0.40) does NOT produce
         40% land — you get ~0.2% land. Always compute from actual distribution.

Stage 4: Terrain classification (13 classes)
         Thresholds as fractions of the land elevation range [sea_level, max]
         so classification scales correctly across different pyplatec outputs.
         Classes: ocean_deep → ocean_mid → ocean_shallow → coast → lowland →
                  plains → grassland → hills → forest → highland → mountain →
                  peak → snow

Stage 5: D8 flow accumulation → river network
         Sort land cells by elevation descending
         Each cell drains to steepest downslope neighbour (8-directional)
         Flow threshold: 30 (calibrated for 512×256 grid with 40% land)
         Note: threshold scales with grid size and terrain relief — calibrate
         per planet type. Very flat worlds (like Kallast) need lower threshold.

Stage 6: Settlement placement
         For each city from wiki data: snap to nearest plains/grain_belt cell
         within expanding search radius (20 → 40 → 60 → 80 cells)
         Preference order: plains (class 5) > grain_belt (6) > lowland (4) >
                           coast (3) > hills (7)

Stage 7: Road network
         Tier-1 and tier-2 cities connected by major roads (all-pairs from capital)
         Tier-3 nodes connected to nearest tier-1/2 by minor roads
         Rendered as polylines on the annotated layer

Stage 8: Geographic render (PIL)
         1. Base terrain color layer (RGB from class colors)
         2. Elevation shading on land (ambient occlusion proxy)
         3. Dilate river mask at source grid resolution (2 iterations, preserves topology)
         4. Scale up terrain to output resolution (4096×2048) via LANCZOS
         5. Paint rivers AFTER upscale via NEAREST-neighbor upscaled mask
            CRITICAL: painting before LANCZOS blurs rivers into invisibility.
            Post-upscale NEAREST gives each source cell a 4×4px block — clearly legible.
         6. Lat/lon grid lines (every 30°), scaled width
         7. Title panel + legend — natural geographic features only
            (ocean, coast, plains, grassland, mountain, river)
            Text/panel sizes scale with UI_SCALE = OUTPUT_W / 1024
         NOTE: settlements, roads, freight elevators are NOT rendered here.
               They live in the JSON sidecar and are overlaid by the atlas app.

---

Configuration per Planet Type

For the batch run (#794), per-planet config differs in:

Parameter	Kallast	Young volcanic	Ice world	Desert	Ocean world
plate_count	10	4	7	6	8
sim_steps	200	100	150	150	180
erosion_passes	4	1	3 (glacial)	2 (aeolian)	3
land_fraction	0.40	0.55	0.30	0.60	0.15
river_threshold	60	320	80	60	200

The land_fraction comes directly from the wiki's hydrosphere field:

• ocean → 0.30–0.45
• liquid_water → 0.40–0.60
• ice → 0.20–0.35
• none → 0.90–0.99

---

Two-Layer Model

Heightmaps are geographic only. Human data lives in JSON sidecars.

kallast_heightmap.png          ← geographic render: terrain, rivers, biomes, grid
kallast_heightmap_settlements.json  ← human layer: city names + grid coordinates

The PNG renders: terrain classification colors, elevation shading, dilated river network, lat/lon grid, title panel.

The PNG does NOT render: settlements, roads, freight elevators, city labels, irrigation channels, or any human-activity markers. Those exist in the JSON sidecar and are overlaid separately by the atlas app (Phase 3) when the map is interactive.

Rationale: A geographic heightmap is a stable base layer. The human overlay changes as the simulation runs (cities grow, shrink, change character). Keeping them separate means the PNG can be regenerated from terrain data without recomputing settlement placement, and vice versa.

Output Files

File	Size	Description
kallast_heightmap.png	4096×2048px	Geographic world map (deliverable)
kallast_terrain.npy	~2MB	Raw normalised elevation grid (numpy float32, 1024×512)
kallast_heightmap_settlements.json	<1KB	City positions for atlas DB import (human layer sidecar)

For batch production, the .npy and .json files are inputs to the Phase 3 atlas pipeline — they pre-seed the city layer rather than requiring re-computation.

---

Known Issues / Calibration Notes for Batch

1. River painting order is critical. Painting river pixels into the source-resolution array before LANCZOS upscaling blurs them into invisibility. Always dilate at source resolution, then upscale with NEAREST neighbor and paint AFTER. Enforced in render().

2. Flat worlds produce sparse rivers. Kallast has low terrain relief. Threshold=60 at 1024×512 gives 113 pre-dilation cells (1164 post). Scale threshold with grid area: threshold_1024 ≈ threshold_512 * 4. For this flat world, halve the baseline to get denser coverage.

3. City placement uses wiki narrative coordinates, not astrophysical simulation. Relative positions (e.g. "Kallast Prime at 45% longitude, 48% latitude") are editorial decisions. The snap algorithm finds nearest suitable terrain class within search radius. This is intentional — settlement locations should reflect the world's narrative.

4. Agricultural layer. The wiki describes irrigation channels wide enough to see from orbit. These are human infrastructure — they belong in the JSON sidecar, not the geographic heightmap. Phase 3 atlas work should render irrigation channels as a separate overlay from hydrology + settlement data.

---

Batch Run Estimate (#794)

Grid size: 1024×512. Output: 4096×2048.

Phase	Step	Time per planet	301 planets
Tectonic (200 steps, 1024×512)	~3.5s	1054s
Erosion (4 passes)	~1.0s	301s
Hydrology	~0.5s	151s
Placement + roads	~0.5s	151s
Render + export	~0.8s	241s
Total		~6.3s/planet	~32 minutes

Full batch of 301 systems runs in ~32 minutes single-threaded. Parallelisable across all CPU cores (no shared state) — realistically ~8 minutes on 4 cores.

Note: if batch time is a concern, sim_steps=100 halves tectonic time with acceptable terrain quality for most planet types. Only mature worlds (Kallast, old ocean worlds) benefit meaningfully from 200 steps.

---

Running the Spike

# Standard (200 tectonic steps, ~1.5s)
python3 spikes/heightmap-pipeline/generate_kallast.py

# Fast mode (50 steps — good for testing annotation, poor terrain)
python3 spikes/heightmap-pipeline/generate_kallast.py --fast

# Different seed (changes continent layout)
python3 spikes/heightmap-pipeline/generate_kallast.py --seed 42

# Custom output path
python3 spikes/heightmap-pipeline/generate_kallast.py --output heightmaps/GJ144d_kallast_v2.png

Dependencies: pyplatec, scipy, numpy, Pillow (all installable via pip)

---

Open Questions for Review

Before starting batch (#794), Jeroen should confirm:

1. Visual style. Does the terrain color palette work? The grassland amber (#a59b4b) reads as temperate plains — is this the right mood for Kallast?
2. Annotation density. 12 settlements for a 2B-population world — too sparse? too many? For the batch, settlement count would be derived from wiki city data (if any) or a formula from population + settlement_pattern.
3. Output resolution. 1024×512 adequate for wiki use? Or do we need 2048×1024 for the implant atlas app (Phase 3)?
4. River threshold calibration. The flat terrain of Kallast needed threshold=30. Should we auto-calibrate per planet by targeting N river-mouth cells, rather than a fixed threshold?