Hi! Thanks for spotting it. It seems that config was never used from the beginning. Storm 3.0.0 dropped Clojure in favour of Java, but the port kept the logic as much as possible. There used to be a comment in a Clojure source fle:
- ;; TODO: this is broken. need to maintain a map since last time - ;; supervisor hearbeats like is done for tasks - ;; maybe it's ok to trust ephemeral nodes here? - ;;[[id info]] - ;; (when (< (time-delta (:time-secs info)) - ;; (conf NIMBUS-SUPERVISOR-TIMEOUT-SECS)) - ;; [[id info]] - ;; ) https://github.com/apache/storm/commit/40e092dda0e7affe89c00552bc138a251f095dc0 The actual mechanism still relies on the ephemeral nodes (and always had, apparently) - ZooKeeper session timeout (storm.zookeeper.session.timeout, default 20000 ms). This is negotiated with ZK and drives when ZK deletes the ephemeral node after the supervisor stops responding. - Nimbus scheduler tick (nimbus.monitor.freq.secs) — how often Nimbus re-reads ZK and reassigns. supervisor.heartbeat.frequency.secs is misleading because even if it would not fire, Zookeeper client uses pings to keep the session alive, according to what I can gather (this property keeps the Supervisor data read by Nimbus fresh) @Richard Zowalla @Gianluca Graziadei @Julien Nioche would like to hear on thoughts on this. 1- We remove the dead configs/code/documentation and just properly document the actual mechanism 2- We implement a different Supervisor liveness mechanism based on that property that has been never used Either way, this is a bug. @Karthick do you want to open an issue for this? Thank you On Tue, 4 Aug 2026 at 05:29, Karthick <[email protected]> wrote: > > Hi, > Im checking on the heartbeat flow, The below configuration is not in use, as > per comment it seems. Am I missing anything? Please guide me. > > /** > * How long before a supervisor can go without heartbeating before nimbus > considers it dead and stops assigning new work to it. > */ > @isInteger > @isPositiveNumber > public static final String NIMBUS_SUPERVISOR_TIMEOUT_SECS = > "nimbus.supervisor.timeout.secs"; > > > On Sat, Jul 25, 2026 at 1:45 AM Gianluca Graziadei > <[email protected]> wrote: >> >> Hi, >> >> I dug through the JIRA history and your reconciliation holds, though with >> two key refinements: the bottleneck in STORM-2693 was actually per-round >> read-and-recompute overhead on Nimbus rather than Zookeeper write pressure, >> which is why the fix prioritized caching and supervisor reporting over a >> faster store. SupervisorInfo stayed in ZK not for liveness monitoring, but >> because it is shared cluster-state metadata that any elected Nimbus leader >> needs to access for scheduling (content of the serialized >> https://github.com/apache/storm/blob/master/storm-client/src/jvm/org/apache/storm/generated/SupervisorInfo.java) >> >> Best, >> >> Gianluca >> >> >> Il giorno ven 24 lug 2026 alle ore 12:25 Karthick >> <[email protected]> ha scritto: >>> >>> Thanks, that's a clear and helpful breakdown. The push-for-liveness / >>> pull-for-stats split makes sense: keeping a continuous in-memory pulse per >>> worker in Nimbus (HeartbeatCache) while serving heavy stats on demand from >>> ZK keeps Nimbus lightweight, and the Supervisor Thrift relay keeps those >>> frequent liveness writes off ZK. >>> >>> One thing I left out of my original question: the supervisor liveness >>> heartbeat — which is also liveness, but it still goes to ZooKeeper >>> (ephemeral znode at /supervisors/<id>, via SupervisorHeartbeat), consumed >>> by Nimbus under nimbus.supervisor.timeout.secs. So "liveness" isn't >>> uniformly on the Thrift path. >>> >>> My working reconciliation is that the deciding factor is volume and >>> semantics, not liveness-vs-stats: >>> >>> Worker/executor liveness is high-volume (potentially thousands of >>> heartbeats per cluster, every ~1s) → moved off ZK onto the Supervisor >>> Thrift relay to avoid write pressure. >>> Supervisor liveness is low-volume (one per node, every ~5s) → cheap on ZK, >>> and the ephemeral znode gives automatic crash detection when the session >>> dies, which the Thrift path wouldn't provide for free. >>> >>> Does that match the design intent — i.e. supervisor heartbeats stayed on ZK >>> deliberately because per-node volume is low and ephemeral-node semantics >>> are valuable, whereas per-worker heartbeats were the actual ZK scaling >>> problem? >>> >>> Also noted on 3.0.0's ZK read/serialization improvements — thanks for the >>> pointer, will look into it. >>> >>> >>> >>> On Thu, Jul 23, 2026 at 4:53 PM Karthick <[email protected]> wrote: >>>> >>>> Hi all, >>>> >>>> I'm studying the Storm 2.0 heartbeat/liveness paths and want to confirm my >>>> understanding of a design decision. >>>> >>>> As I read the 2.0 code, there are two distinct worker-originated >>>> heartbeats: >>>> >>>> Liveness — the worker writes an LSWorkerHeartbeat to local disk >>>> (Worker.doHeartBeat); the supervisor reads those files and relays a batch >>>> to the leader Nimbus over Thrift (ReportWorkerHeartbeats → >>>> Nimbus.sendSupervisorWorkerHeartbeats → HeartbeatCache), governed by >>>> nimbus.task.timeout.secs. >>>> Stats — Worker.doExecutorHeartbeats writes a heartbeat object (time-secs + >>>> uptime + executor stats) to ZooKeeper, which the UI/metrics consume (and >>>> which HeartbeatCache.updateFromZkHeartbeat can still use for liveness on >>>> the ZK strategy). >>>> >>>> My understanding is that liveness was moved off ZooKeeper (the 1.x model, >>>> and later Pacemaker) because high-volume per-worker heartbeat writes made >>>> ZK a scaling bottleneck, and since heartbeats are ephemeral they don't >>>> need ZK's persistence/consistency — so the supervisor-relay-over-Thrift >>>> model removes those writes from ZK entirely. >>>> >>>> A few questions: >>>> >>>> Is that the correct/primary motivation for the Thrift supervisor-relay >>>> path, or were there other drivers (connection count, watch load, Nimbus >>>> HA, recovery on leader change)? >>>> Why do executor stats still go through ZooKeeper rather than riding the >>>> same Thrift path — is it purely that stats are lower-frequency and >>>> UI-oriented, or is there a stronger reason? >>>> Is there a JIRA / design doc that captures this transition (beyond >>>> docs/Pacemaker.md) that I could read? >>>> >>>> Thanks for any pointers — trying to make sure I document this accurately.
