The listener opened one asyncpg connection, called add_listener, and set
running = True. Nothing watched that connection afterwards. When it dropped, the
subscription was gone for good while running still reported True, so the service
stayed healthy in every way anything could observe and silently stopped indexing
page edits. Recovery needed a manual container restart.
That happened on 2026-08-08 when postgres-shared was redeployed. The sibling
settings_client survived the same event because it uses asyncpg.create_pool,
which replaces dead connections; a bare LISTEN connection has no such recovery.
A supervisor task now waits on asyncpg's termination callback and reconnects
with bounded exponential backoff, 1s doubling to a 60s cap. It retries forever
rather than giving up after N attempts: a database under maintenance does come
back, and a listener that stopped trying would reproduce exactly the silent
deafness this exists to prevent. The termination listener is re-registered on
every new connection because asyncpg clears its listener list as soon as it
fires them, so a one-time registration survives exactly one drop.
running is now derived from the connection rather than assigned, and stop() sets
a flag the termination callback and supervisor both check so a deliberate
shutdown cannot race into a reconnect.
NOTIFY is fire-and-forget, so events emitted during an outage are lost and
cannot be replayed. The reconnect logs the gap and names
POST /maintenance/integrity-check rather than reporting a clean recovery.
Reconciling automatically is left out on purpose: deriving the tenant for a
changed page is subtle here, and getting it wrong writes into the wrong user's
namespace.
Verified against the real database by terminating the listener's backend with
pg_terminate_backend. Old code: running=True with is_closed()=True, dead
forever. New code: reconnects on its own onto a new server pid. The same probe
was run against both implementations so the check is known to discriminate.
One existing test mocked the connection with a bare AsyncMock, which models
asyncpg's synchronous is_closed() as a coroutine — always truthy, so the
connection read as closed once running started deriving from it. Corrected.
Co-Authored-By: Claude <noreply@anthropic.com>
jpmschweitzerandClaude
4f09a12171