Long-Running Chain Listeners That Actually Stay Up

A blockchain event listener looks trivial in the docs. You call contract.on("Transfer", handler) and events arrive. Then you run it in production for a week and discover the interesting failure mode: it stops receiving events and never tells you.

Websocket providers drop connections. Public RPC nodes rate-limit, lag behind the chain head, or return stale data. Reorgs rewrite blocks you already processed. None of these throw an exception in your handler.

Treat the subscription as untrusted

The first shift is to stop trusting the push stream as your source of truth. A subscription is a latency optimisation; the chain is the source of truth.

That means tracking the last block you fully processed, in durable storage, and continuously reconciling it against the current head. If the gap between your cursor and the head exceeds a threshold, you are stalled — regardless of whether the socket claims to be open. That check is your real liveness signal, and it is the one worth paging on.

Backfill and live ingestion should share a code path

The temptation is to write a fast live handler and a separate catch-up script. Then they drift, and the backfill applies slightly different logic than the listener did.

A better shape: one function that ingests a block range. Live ingestion calls it with [cursor+1, head] on every tick. Recovery calls it with whatever range you missed. Manual backfill calls it with an arbitrary window. Same parsing, same idempotency guards, same writes. The listener becomes a trigger, not a pipeline.

Respect the node, not just the rate limit

RPC-safe execution is more than a semaphore. Chunk your getLogs ranges so a query never asks for more than the provider will return, and shrink the chunk adaptively when you get a range-too-large error rather than failing the run. Cache block timestamps you have already fetched. Batch calls where the provider supports it.

And keep at least two providers configured. Not for throughput — for the afternoon when your primary starts returning results from twelve blocks ago.

Confirmations are a product decision

Reorgs mean a block you processed can be replaced. You have two options and they are both fine, as long as you pick deliberately.

Wait n confirmations before treating an event as final, and accept the latency. Or process optimistically for responsiveness and mark records provisional, with a rollback path when the block hash at that height no longer matches what you recorded. The second is more work but it is what real-time UX demands.

What you cannot do is ignore the question and hope the chain never reorgs. It will, usually during the week you ship.

The boring part that matters most

Structured logs on every ingestion cycle: cursor position, head, gap, events written, duration. One metric for cursor lag. One alert when lag exceeds your threshold for more than a few minutes.

A listener that quietly stops is worse than one that crashes, because a crash restarts. Make silence impossible to miss.