Hi, Im checking on the heartbeat flow, The below configuration is not in use, as per comment it seems. Am I missing anything? Please guide me.
/** * How long before a supervisor can go without heartbeating before nimbus considers it dead and stops assigning new work to it. */ @isInteger @isPositiveNumber public static final String NIMBUS_SUPERVISOR_TIMEOUT_SECS = "nimbus.supervisor.timeout.secs"; On Sat, Jul 25, 2026 at 1:45 AM Gianluca Graziadei < [email protected]> wrote: > Hi, > > I dug through the JIRA history and your reconciliation holds, though with > two key refinements: the bottleneck in STORM-2693 was actually per-round > read-and-recompute overhead on Nimbus rather than Zookeeper write pressure, > which is why the fix prioritized caching and supervisor reporting over a > faster store. SupervisorInfo stayed in ZK not for liveness monitoring, but > because it is shared cluster-state metadata that any elected Nimbus leader > needs to access for scheduling (content of the serialized > https://github.com/apache/storm/blob/master/storm-client/src/jvm/org/apache/storm/generated/SupervisorInfo.java > ) > > Best, > > Gianluca > > Il giorno ven 24 lug 2026 alle ore 12:25 Karthick < > [email protected]> ha scritto: > >> Thanks, that's a clear and helpful breakdown. The push-for-liveness / >> pull-for-stats split makes sense: keeping a continuous in-memory pulse per >> worker in Nimbus (HeartbeatCache) while serving heavy stats on demand >> from ZK keeps Nimbus lightweight, and the Supervisor Thrift relay keeps >> those frequent liveness writes off ZK. >> >> One thing I left out of my original question: the *supervisor liveness* >> heartbeat — which is also liveness, but it still goes to ZooKeeper >> (ephemeral znode at /supervisors/<id>, via SupervisorHeartbeat), >> consumed by Nimbus under nimbus.supervisor.timeout.secs. So "liveness" >> isn't uniformly on the Thrift path. >> >> My working reconciliation is that the deciding factor is *volume and >> semantics, not liveness-vs-stats*: >> >> - Worker/executor liveness is high-volume (potentially thousands of >> heartbeats per cluster, every ~1s) → moved off ZK onto the Supervisor >> Thrift relay to avoid write pressure. >> - Supervisor liveness is low-volume (one per node, every ~5s) → cheap >> on ZK, and the *ephemeral* znode gives automatic crash detection when >> the session dies, which the Thrift path wouldn't provide for free. >> >> Does that match the design intent — i.e. supervisor heartbeats stayed on >> ZK deliberately because per-node volume is low and ephemeral-node semantics >> are valuable, whereas per-worker heartbeats were the actual ZK scaling >> problem? >> >> Also noted on 3.0.0's ZK read/serialization improvements — thanks for the >> pointer, will look into it. >> >> >> On Thu, Jul 23, 2026 at 4:53 PM Karthick <[email protected]> >> wrote: >> >>> Hi all, >>> >>> I'm studying the Storm 2.0 heartbeat/liveness paths and want to confirm >>> my understanding of a design decision. >>> >>> As I read the 2.0 code, there are two distinct worker-originated >>> heartbeats: >>> >>> 1. *Liveness* — the worker writes an LSWorkerHeartbeat to local disk >>> (Worker.doHeartBeat); the supervisor reads those files and relays a >>> batch to the leader Nimbus over *Thrift* (ReportWorkerHeartbeats → >>> Nimbus.sendSupervisorWorkerHeartbeats → HeartbeatCache), governed by >>> nimbus.task.timeout.secs. >>> 2. *Stats* — Worker.doExecutorHeartbeats writes a heartbeat object ( >>> time-secs + uptime + executor stats) to *ZooKeeper*, which the >>> UI/metrics consume (and which HeartbeatCache.updateFromZkHeartbeat >>> can still use for liveness on the ZK strategy). >>> >>> My understanding is that liveness was moved off ZooKeeper (the 1.x >>> model, and later Pacemaker) because high-volume per-worker heartbeat writes >>> made ZK a scaling bottleneck, and since heartbeats are ephemeral they don't >>> need ZK's persistence/consistency — so the supervisor-relay-over-Thrift >>> model removes those writes from ZK entirely. >>> >>> A few questions: >>> >>> - Is that the correct/primary motivation for the Thrift >>> supervisor-relay path, or were there other drivers (connection count, >>> watch >>> load, Nimbus HA, recovery on leader change)? >>> - Why do executor *stats* still go through ZooKeeper rather than >>> riding the same Thrift path — is it purely that stats are lower-frequency >>> and UI-oriented, or is there a stronger reason? >>> - Is there a JIRA / design doc that captures this transition (beyond >>> docs/Pacemaker.md) that I could read? >>> >>> Thanks for any pointers — trying to make sure I document this accurately. >>> >>
