diff --git a/.claude/settings.json b/.claude/settings.json index b1d7d9b97..8cacc72e8 100644 --- a/.claude/settings.json +++ b/.claude/settings.json @@ -36,6 +36,10 @@ "Bash(db/connectors/sqlite-init)", "Bash(db/connectors/decisions-sync)", + "Bash(db/connectors/audio-generate *)", + "Bash(db/connectors/audio-health)", + "Bash(db/connectors/audio-post *)", + "Bash(make *)", "Bash(make)", diff --git a/.claude/skills/gen-audio/SKILL.md b/.claude/skills/gen-audio/SKILL.md new file mode 100644 index 000000000..5f8d19d3e --- /dev/null +++ b/.claude/skills/gen-audio/SKILL.md @@ -0,0 +1,187 @@ +--- +name: gen-audio +description: > + Generate audio assets for The Settled Reach using the Stable Audio Open API + (self-hosted Gradio app at tower-of-joy:11500). Use when generating any game + audio: ambient loops, SFX, UI sounds, monologue chimes, footsteps, or any + sound asset from docs/assets/audio/. Also use when the user asks about audio + generation, sound design pipeline, or audio asset iteration. Triggers on: + "generate audio", "make sounds", "create ambient", "audio pipeline", + "generate sfx", "stable audio", "gen audio", "sound design". +--- + +# Audio Generation — The Settled Reach + +Generate sonically consistent audio assets using the Stable Audio Open API via +wrapper scripts at `db/connectors/audio-*`. + +Asset descriptions, filenames, bus routing, and design intent are documented in +`docs/assets/audio/`. This skill provides the prompt system, generation +workflow, and quality validation. + +## API Access + +**Never call the API directly.** Use the wrapper scripts: + +```bash +# Check API health +db/connectors/audio-health + +# Generate audio +db/connectors/audio-generate "prompt text" \ + --duration 10 \ + --steps 100 \ + --cfg 7 \ + --output path/to/output.wav \ + --timeout 600 +``` + +### Parameters + +| Parameter | Default | Range | Notes | +|-----------|---------|-------|-------| +| `--duration` | 10 | 0-47s | Max 47s per generation. For longer loops, generate 45s with crossfade overlap. | +| `--steps` | 100 | 10-200 | More steps = better quality, slower. Use 50 for quick previews, 100-150 for final. | +| `--cfg` | 7 | 1-15 | Classifier-free guidance. Higher = more prompt-adherent but less natural. 5-9 is the sweet spot. | +| `--output` | auto | — | Output file path. Auto-names from prompt if omitted. | +| `--timeout` | 600 | — | Max wait in seconds. Generation can take 2-5 minutes on 11GB VRAM. | + +### Critical Constraints + +- **NEVER parallelize requests.** The server has 11GB VRAM and runs one + generation at a time. Always wait for a generation to complete before + starting the next. Sequential only. +- **Generation takes 2-5 minutes** per clip depending on duration and steps. + Be patient. The timeout default (600s) is generous. +- **Max 47 seconds** per generation. For 60-90s ambient loops, generate 45s + clips and crossfade-stitch in post-processing. +- **Output is WAV at 44.1kHz stereo.** Convert to .ogg for Godot import: + `ffmpeg -i input.wav -c:a libvorbis -q:a 6 output.ogg` + +## Prompt Assembly + +Every generation uses three parts: + +``` +[SONIC FAMILY PREFIX] + [CATEGORY TEMPLATE] + [ASSET DESCRIPTION from docs/assets/audio/] +``` + +Never call the API with just the asset description. Always prepend the sonic +family prefix and matching category template. + +- **Sonic palette and families:** Read `references/sonic-palette.md` +- **Category templates:** Read `references/category-templates.md` and match by + asset type (ambient, sfx, ui) +- **Asset description:** Look up the specific asset in `docs/assets/audio/{category}.md` + +## Single Asset Workflow + +1. Find the asset in `docs/assets/audio/{ambient,sfx,ui}.md` — note filename, + duration, bus, method, and design intent. +2. Read `references/sonic-palette.md` for the sonic family prefix. +3. Read `references/category-templates.md` for the matching template. +4. Assemble the full prompt. +5. Run `db/connectors/audio-health` to verify the API is up. +6. Run `db/connectors/audio-generate` with the assembled prompt. **One request + at a time. Wait for completion.** +7. Listen to the output (or describe it based on file size/duration). +8. If acceptable, convert to .ogg and place in `client/assets/audio/`. +9. Update the asset status in `docs/assets/audio/{category}.md`. + +## Iteration Workflow + +For each asset, generate 4-6 candidates: + +1. **Generate candidates** — vary the prompt slightly (add/remove descriptors, + adjust CFG between 5-9). Run each generation sequentially — never in + parallel. +2. **Solo test** — does each candidate sound right alone? +3. **Stack test** — play the candidate alongside other layers. Does it mask or + clash? +4. **Fatigue test** (loops only) — can you listen for 5+ minutes without a + jarring repeat? +5. **Close-your-eyes test** — does it create a mental image or sensation? +6. Select the best candidate, trim, normalize, convert. + +## Post-Processing + +After selecting the best generation: + +```bash +# Trim silence from start/end +ffmpeg -i input.wav -af "silenceremove=start_periods=1:start_silence=0.1:start_threshold=-50dB,areverse,silenceremove=start_periods=1:start_silence=0.1:start_threshold=-50dB,areverse" trimmed.wav + +# LUFS normalize to -16 LUFS (broadcast standard, good for game audio) +ffmpeg -i trimmed.wav -af loudnorm=I=-16:LRA=11:TP=-1 normalized.wav + +# Convert to .ogg for Godot +ffmpeg -i normalized.wav -c:a libvorbis -q:a 6 output.ogg + +# For loops: verify loop point +ffplay -loop 0 output.ogg +``` + +For ambient loops, create crossfade overlap: +```bash +# Create a 45s loop with 3s crossfade overlap +# (manual: export 48s, crossfade first 3s with last 3s in Audacity) +``` + +## Manual Synthesis (Insert-Tech Sounds) + +For sounds under 200ms (cursor hover, weapon aim), Stable Audio Open cannot +produce meaningful output. Use manual synthesis instead: + +```python +# Example: 50ms cursor hover tick +import numpy as np +import wave + +sr = 44100 +duration = 0.05 # 50ms +t = np.linspace(0, duration, int(sr * duration), endpoint=False) +freq = 3200 # Hz +signal = np.sin(2 * np.pi * freq * t) +envelope = np.exp(-t * 80) # exponential decay +audio = (signal * envelope * 32767).astype(np.int16) + +with wave.open("cursor_hover.wav", "w") as f: + f.setnchannels(1) + f.setsampwidth(2) + f.setframerate(sr) + f.writeframes(audio.tobytes()) +``` + +## Quality Checklist + +After generating, verify: +- Sound matches the sonic family (insert-tech = synthetic/precise, organic = warm/natural) +- Frequency range doesn't mask other layers (check docs/assets/audio/palette.md) +- Duration matches spec +- No unwanted artifacts (clicks, pops, digital noise at start/end) +- Loop point is clean (ambient loops only) +- Volume sits well relative to other assets (LUFS normalized) +- Passes the close-your-eyes test + +## File Placement + +Generated assets go to `client/assets/audio/` with exact filenames from the +asset docs: + +``` +client/assets/audio/ + amb_station_base.ogg # Ambient bus + amb_workplace_layer.ogg # Ambient bus + amb_bar_layer.ogg # Ambient bus + amb_corridor_layer.ogg # Ambient bus + sfx_footstep_metal.ogg # Player Actions bus + sfx_footstep_metal_run.ogg # Player Actions bus + cursor_hover.ogg # UI Sounds bus + implant_open.ogg # UI Sounds bus + fog_recognition.ogg # UI Sounds bus + weapon_aim.ogg # UI Sounds bus + sfx_monologue_chime.ogg # UI Sounds bus + sfx_monologue_chime_urgent.ogg # UI Sounds bus +``` + +AudioManager discovers these by directory scan — filenames must match exactly. diff --git a/.claude/skills/gen-audio/references/category-templates.md b/.claude/skills/gen-audio/references/category-templates.md new file mode 100644 index 000000000..4dafd7e3b --- /dev/null +++ b/.claude/skills/gen-audio/references/category-templates.md @@ -0,0 +1,67 @@ +# Category Templates — Audio Generation + +Match the asset's category to the right template. Append after the sonic family +prefix and before the specific asset description. + +## Ambient Loops + +**Template:** +> Continuous ambient soundscape loop, seamless looping audio, no distinct +> beginning or end, steady background atmosphere, [DURATION]s duration + +**Notes:** +- Generate at 45s (the SAO sweet spot below the 47s ceiling) +- Add 3-5s of overlap material for crossfade looping in post +- These play continuously — they must be boring enough to fade into the + background but rich enough to reward attention +- Station baseline prefix is required (see sonic-palette.md) + +**Steps:** 100-150 (higher quality for loops — artifacts are more noticeable) +**CFG:** 5-7 (lower guidance = more natural variation, less mechanical) + +## Sound Effects (SFX) + +**Template:** +> Single isolated sound effect, clean recording, [DURATION] duration, +> clear attack and natural decay, no background noise, no reverb tail +> beyond natural + +**Notes:** +- For footsteps: specify surface material, pace, weight +- For environmental: specify the physical mechanism (door hinge, cargo latch) +- Keep short and punchy — these are event-driven, not continuous + +**Steps:** 80-100 (shorter sounds need fewer steps) +**CFG:** 7-9 (more guidance = more precise sound matching) + +## UI Sounds + +**Template:** +> Interface feedback sound, digital UI element, [DURATION] duration, +> immediate attack, clean decay, isolated sound with no background + +**Notes:** +- Most UI sounds are under 200ms — use manual synthesis, not SAO +- For SAO-generated UI sounds (>200ms like fog_recognition, chimes): + generate at 3-5s duration, then trim to the best portion +- Insert-tech prefix is required for lattice/interface sounds +- Organic prefix for fog_recognition (cognitive, not technological) + +**Steps:** 80-100 +**CFG:** 7-9 (precision matters for UI sounds) + +## Chimes / Notification Sounds + +**Template:** +> Single musical tone, crystalline quality, [DURATION] duration, gentle +> attack, resonant sustain, natural fade out, no accompaniment, no rhythm + +**Notes:** +- Generate at 3-5s to give SAO enough temporal context +- Trim to the best 0.5-1.0s section +- The attack transient is where the character lives — select for that +- Normal vs urgent variants: same base prompt, vary "gentle/soft" vs + "brighter/sharper/more present" + +**Steps:** 100 (tonal quality matters) +**CFG:** 6-8 (some freedom for natural harmonic content) diff --git a/.claude/skills/gen-audio/references/sonic-palette.md b/.claude/skills/gen-audio/references/sonic-palette.md new file mode 100644 index 000000000..d9d2751d8 --- /dev/null +++ b/.claude/skills/gen-audio/references/sonic-palette.md @@ -0,0 +1,45 @@ +# Sonic Palette — Prompt Prefixes + +Two sonic families. Every generated sound uses one of these as a prompt prefix. + +## Insert-Tech (Synthetic) + +For UI sounds, neural lattice interface, augmented cognition. + +**Prompt prefix:** +> Clean digital audio, synthetic electronic tone, precise and clinical sound +> design, no reverb, no room ambience, studio-dry recording, futuristic +> interface sound + +**Character:** Mathematical precision. The sound of well-designed technology. +No organic texture, no room reflections. Exists "inside the head," not in +physical space. + +**Frequency range:** 800Hz-4kHz primary. Sharp attack, controlled decay. + +**Used for:** cursor_hover, implant_open, weapon_aim, sfx_monologue_chime, +sfx_monologue_chime_urgent + +## Organic (Biological/Environmental) + +For environmental sounds, human cognition, physical world. + +**Prompt prefix:** +> Warm organic audio, natural sound recording, slight room ambience, +> lived-in industrial space, realistic acoustic properties, authentic +> environmental sound + +**Character:** Human warmth. Natural attack and decay. Reverberant — exists +in physical space. The sound of a real place with real materials. + +**Frequency range:** 60Hz-3kHz primary. Soft or natural attack, room tail. + +**Used for:** fog_recognition, all amb_* ambient loops, all sfx_footstep_* + +## Station Baseline + +For ambient loops specifically, add the station context: + +> Interior of a large industrial space station, metal and composite +> construction, background mechanical hum from power systems, air circulation +> audible, no music, no prominent voices unless specified diff --git a/.claude/skills/gen-image/SKILL.md b/.claude/skills/gen-image/SKILL.md index 8aaaa2c9f..d91004e4e 100644 --- a/.claude/skills/gen-image/SKILL.md +++ b/.claude/skills/gen-image/SKILL.md @@ -1,5 +1,5 @@ --- -name: asset-gen +name: gen-image description: > Generate themed visual assets for the Lords of Ash CK3 total conversion mod using the generate_image MCP tool (Nano Banana / Gemini 2.5 Flash Image).