Hi Zhe, Thanks for the careful reading — your two-case split is the intended semantics, and the current wording is too easy to misread. We will make that boundary explicit in the FIP and the migration runbook.
There is no dual-write path. MetadataBackend.from() requires exactly one of zookeeper.address or coordinator.seed.nodes; setting both is rejected at startup. Migration is also one-way (zk-to-raft-export / import / live-migrate). There is no Raft→ZK reverse mapper and none is planned. The rollback boundary is therefore whether Raft has become the serving source of truth, not whether the ZK tree still exists on disk. 1. Rollback before Raft serves metadata writes — supported This covers the whole pre-cutover window: Raft quorum bootstrap, baseline import, live-migrate re-scan, and the write-fence + final scan, before coordinators are restarted with coordinator.seed.nodes. ZK remains the source of truth. Raft is only a seeded replica. Rollback is: stop the Raft coordinators restore server.yaml with zookeeper.address (and without seed nodes) restart from the still-current ZK metadata If a write-fence was already applied for the final scan, un-fence ZK so it can accept writes again. The imported Raft state can be discarded. 2. Rollback after Raft has served metadata writes — unsupported Cutover is the config switch: coordinators (and TabletServers) start with coordinator.seed.nodes and omit zookeeper.address. From that point Raft is the source of truth. Any later metadata mutation lives only on Raft, including: DDL (create/drop database/table, schema/partition changes) ACL / cluster-config updates sequence-id block allocation (ALLOCATE_ID_RANGE for table / partition / writer ids) TabletServer registration and liveness (Raft RegisterTabletServer / heartbeat leases, not ZK ephemerals) Pointing those nodes back at the old ZK tree would silently drop those post-cutover writes. That is especially unsafe for writer ids: RaftSequenceIDCounter pre-reserves blocks on Raft, so a ZK rollback can reuse ids that were already handed out. We will not provide a reverse migration. “Keep ZK until soak” means only: keep the frozen pre-cutover ZK snapshot (and the final export) as a last-resort restore of that snapshot, accepting loss of every Raft-era metadata change. It is not a lossless rollback after Raft has served writes. After soak, decommission ZK. We will update migrate-zk-to-raft.md (and the FIP/design §6) with this 1 vs 2 distinction so “rollback while ZK data is still intact” cannot be read as safe after Raft is the source of truth. Best regards, Forward On 2026/08/24 14:35:07 Zhe Wang wrote: > Hi Forward, > > Thanks for the detailed FIP. The overall migration direction makes sense to > me. > > One area I would like to understand better is the rollback boundary during > ZK-to-Raft migration. > > The FIP says rollback is possible while the ZK data is still intact by > stopping Raft coordinators, restoring `zookeeper.address`, and restarting. > That seems clear before the Raft cluster starts serving new metadata writes. > > Could we clarify what the intended rollback semantics are after cutover? > > For example, if the cluster has already switched to the Raft metadata > backend and accepted new metadata changes, such as creating/dropping > tables, changing ACLs/configs, allocating sequence ID blocks, or updating > TabletServer registrations, then switching back to the old ZK tree would > appear to lose those post-cutover metadata changes unless there is a > reverse migration path from Raft back to ZK. > > Would it make sense for the FIP/docs to explicitly distinguish these cases? > > 1. Rollback before serving writes on Raft: > stop Raft coordinators, restore `zookeeper.address`, and restart from > the still-current ZK metadata. > > 2. Rollback after Raft has served metadata writes: > either unsupported, or requires a defined reverse migration / restore > procedure. > > I think documenting this boundary would make the migration runbook safer > for operators, especially since “rollback while ZK data is still intact” > could otherwise be interpreted as safe even after the Raft backend has > become the source of truth. > > Best regards, > Zhe Wang > > > Forward Xu <[email protected]> 于2026年8月22日周六 16:50写道： > > > Hi Fei, > > > > Thanks for the careful read, and for offering to help with review and > > testing. > > I agree this change touches coordination, storage, deployment, and > > migration, > > so the first Apache cut should be conservative. > > > > The prototype branch currently implements more than I now think we should > > merge as one release. Your two points are the right correction. > > > > --------------------------------------------------------------------- > > 1. Merge / release shape: Phases 1–4 first, Phase 5 later > > --------------------------------------------------------------------- > > > > Agreed. > > > > On `feature/zookeeper-removal-raft`, Phases 1–5 are all marked done > > *in-repo*. > > That was a development convenience so the branch could be the future end > > state (Raft default, ZK deprecated, migration docs on the website). It was > > not meant to be "land Phases 1–5 in the same Apache release." I will make > > that split explicit in the next FIP revision. > > > > Proposed cut: > > > > First merge / first release (Phases 1–4) > > - Dual-backend (`MetadataStoreClient` + `MetadataBackend.from`) > > - Raft as an *opt-in* backend > > - ITs on both backends (including HA) > > - Migration CLI (`zk-to-raft-export` / import / live-migrate) > > - Docs: how to enable Raft, how to stay on ZK, how to migrate > > - ZK remains the supported production path for existing clusters > > > > Later release (Phase 5), after soak > > - Raft becomes the documented default > > - ZK marked deprecated (still supported until Phase 6) > > - Gated by HA/migration/K8s/downstream feedback, not by the > > prototype branch being green > > > > Future major (Phase 6) > > - Delete the ZK backend > > > > In-repo tests, JMH, and chaos ITs can stay on the branch and keep > > exercising Raft as if it were default. That is not the same as shipping > > Raft-as-default to users. > > > > --------------------------------------------------------------------- > > 2. Default backend: fail fast when neither key is set > > --------------------------------------------------------------------- > > > > Also agreed. This is the more important of the two, because it is an > > upgrade-safety issue. > > > > Today on the prototype branch, `MetadataBackend.from()` is: > > > > zookeeper.address only → ZK > > coordinator.seed.nodes only → Raft > > both → IllegalConfigurationException (already > > fail-fast) > > neither → RAFT (Phase 5 > > behavior) > > > > That last line is exactly the hazard you described. An incomplete > > `server.yaml`, a Helm value that dropped `zookeeper.address` without > > adding seeds, or a local-cluster script that "assumes Raft" would > > silently start a 1-voter metadata group instead of refusing to boot. > > Existing ZK clusters that *keep* `zookeeper.address` would be fine; > > the dangerous case is the config gap. > > > > For the first release I will change the selector to: > > > > zookeeper.address only → ZK > > coordinator.seed.nodes only → Raft > > both → IllegalConfigurationException > > neither → IllegalConfigurationException > > (fail fast: set exactly one) > > > > Raft is then an explicit opt-in via `coordinator.seed.nodes` (plus > > `coordinator.raft.voters`). No implicit default in either direction. > > > > That also matches how operators actually have to run Raft: TabletServers > > need a seed list to find the coordinator quorum anyway, so requiring > > the key is not extra ceremony. > > > > Consequences I will fold into the FIP / patch: > > > > - `MetadataBackend.from()` + `MetadataBackendTest`: neither → throw, > > not `return RAFT`. > > - Distro `conf/server.yaml`: first release keeps an explicit backend > > key in the template (ZK for the upgrade-compatible default file, > > with the Raft block commented and documented as opt-in). We should > > not ship a template that omits both. > > - `local-cluster.sh`: stop "assuming Raft" when `zookeeper.address` > > is absent; require one of the two keys. > > - Website / quickstart examples: every snippet sets exactly one key. > > - Upgrade note: a ZK cluster that already has `zookeeper.address` > > does not change behavior. A config that has neither fails at > > startup with a message naming both keys. > > > > I would even keep fail-fast after Phase 5. "Neither key → Raft" is > > convenient for empty unit-test configs; it is a poor production > > default. After Phase 6 removes ZK, missing `coordinator.seed.nodes` > > should still fail — you cannot discover the coordinator quorum without > > it. > > > > --------------------------------------------------------------------- > > > > I will update the FIP with: > > > > - a release-shape section (1–4 now, 5 after soak, 6 later major) > > - the fail-fast selector above > > - an upgrade matrix: existing `zookeeper.address` stays ZK; Raft > > only when `coordinator.seed.nodes` is set; neither is an error > > > > More eyes on HA tests, Kubernetes deploy, migration tooling, and > > downstream SDK test harnesses would be very welcome — those are the > > gates I want before Phase 5, not just "the branch compiles." > > > > Thank you again for the review offer. I will take you up on > > implementation review once the FIP revision and the fail-fast > > selector change are up. > > > > Best, > > Forward > > > > On 2026/08/21 12:08:28 han wrote: > > > Hi Forward Xu, > > > > > > Thanks for putting together this detailed proposal and for driving this > > work. The overall direction makes sense to me. Removing the external > > ZooKeeper dependency and moving coordination and metadata management into > > an embedded Raft quorum should simplify Fluss deployment and operations. > > > > > > From an implementation and upgrade perspective, I have two points that I > > would like to discuss: > > > > > > Merge and release shape > > > The proposal asks the community to accept Phases 1–5 together. Phases > > 1–4 — dual-backend support, testing, migration tools, and documentation — > > look like a reasonable initial scope to me. For Phase 5, I wonder whether > > making Raft the default should be considered separately. Would it be safer > > for the first release to provide Raft as an explicitly enabled backend, and > > then revisit the default after we have gained more experience from HA > > testing, migration testing, and real deployments? > > > > > > Default backend selection > > > I have a small concern about selecting Raft when neither > > zookeeper.address nor coordinator.seed.nodes is configured. During an > > upgrade, or when a configuration is incomplete, this could result in > > behavior that the operator did not explicitly intend. Would it make sense > > to require coordinator.seed.nodes to opt into Raft for at least one > > release, and fail fast when neither backend is configured? Once the Raft > > backend has received broader testing and operational feedback, we could > > revisit making it the implicit default. > > > > > > Since this change touches coordination, metadata storage, deployment, > > migration, and downstream test infrastructure, it would be great to involve > > more contributors in the discussion. In particular, feedback from people > > familiar with HA testing, Kubernetes deployments, migration tooling, and > > downstream integrations would be very helpful. > > > > > > Overall, I am supportive of the direction, and I would also be happy to > > help with implementation review and testing. > > > > > > Best, > > > Fei Han > > > > > > > > > > > > > > > > > > > > > At 2026-08-17 20:41:49, "Forward Xu" <[email protected]> wrote: > > > >Hi all, > > > > > > > >I would like to start a discussion for FIP-52: ZooKeeper Removal and > > > >Embedded Raft Metadata Backend: > > > > > > > > > > https://cwiki.apache.org/confluence/spaces/FLUSS/pages/449282216/FIP-52+ZooKeeper+Removal+and+Embedded+Raft+Metadata+Backend > > > > > > > >Issue: https://github.com/apache/fluss/issues/4014 > > > >Branch: feature/zookeeper-removal-raft > > > > > > > >Please keep the discussion on this list rather than commenting on the > > > >wiki. > > > > > > > >---------------------------------------------------------------------- > > > >Why > > > >---------------------------------------------------------------------- > > > > > > > >Fluss today depends on an external ZooKeeper ensemble for cluster > > > >coordination, metadata persistence, sequence IDs, TabletServer > > > >liveness (ephemeral znodes), dynamic config, ACLs, and related leases. > > > >Clients never talk to ZooKeeper (bootstrap.servers + Admin RPC only); > > > >servers do. Operators still have to run ZooKeeper for every Fluss > > > >cluster: > > > > > > > > - A production cluster cannot start without a healthy ZK ensemble > > > > (quorum, tick time, session timeout, SASL/ACL, backups). > > > > - ZK issues cascade into coordinator leadership, TabletServer > > > > registration, and metadata availability. > > > > - Extra network hops and a second failure domain sit on the > > > > metadata path. > > > > - Container, Kubernetes, edge, and single-node developer setups pay > > > > for an extra stateful service that exists only for coordination. > > > > > > > >This is already the published architecture direction: ZooKeeper is a > > > >transitional dependency, to be replaced by KvStore for metadata and > > > >Raft for coordination. > > > > > > > >---------------------------------------------------------------------- > > > >What this FIP proposes > > > >---------------------------------------------------------------------- > > > > > > > >Replace ZooKeeper with an embedded Apache Ratis Raft quorum on > > > >CoordinatorServers, plus a RocksDB-backed metadata KvStore. > > > > > > > >Only CoordinatorServers are Raft voters. TabletServers are observers: > > > >they discover a coordinator from coordinator.seed.nodes (optionally > > > >expanded by gossip), register/heartbeat over RPC, and consume a > > > >pull-based metadata stream. > > > > > > > > Today Target > > > > ----- ------ > > > > Coordinator ──► ZooKeeper Coordinator quorum > > > > TabletServer ──► ZooKeeper └── embedded Ratis + KvStore > > > > Clients ──► bootstrap.servers TabletServer ──► Register / > > > > Heartbeat / FetchMetadata > > > > Clients ──► bootstrap.servers > > > > (unchanged) > > > > > > > >Consensus engine is Apache Ratis. Metadata apply is KRaft-style > > > >commit-apply: every node, including the leader, mutates the store only > > > >after a log entry is committed. Leader-only serving for strongly > > > >consistent reads; followers redirect. The Raft term is the coordinator > > > >epoch. > > > > > > > >How each ZooKeeper role is replaced: > > > > > > > > - Coordinator leader election (Curator latch) > > > > -> Ratis election + RaftLeaderSelector (gainPrimacy / losePrimacy, > > > > catch-up before serving) > > > > - Database / table / partition / schema / assignment / LeaderAndIsr / > > > > ACL / config / leases > > > > -> typed MetadataRecords in the Raft log, applied into a > > > > RocksDB-backed metadata KvStore (same JSON serdes as ZK values) > > > > - Sequence IDs > > > > -> RaftSequenceIDCounter with pre-reserved blocks > > > > (coordinator.raft.id-block-size) > > > > - TabletServer ephemeral znode + session > > > > -> RegisterTabletServer + heartbeat lease; timeout appends a > > > > replicated FENCE_TABLET_SERVER > > > > - ZK watches > > > > -> in-process watches on the leader store; cross-process > > > > FetchMetadata stream (offset = Raft index) > > > > - zkCli observability > > > > -> fluss raft status / leader + Raft metrics > > > > > > > >We borrow KRaft patterns (commit-apply, record log, broker-style > > > >registration/heartbeat, observer tablets, id blocks, metadata stream) > > > >and reject KRaft-the-codebase. Fluss needs an embeddable Java Raft > > > >library with a pluggable state machine; that is Apache Ratis, shaded > > > >as fluss-shaded-ratis (org.apache.fluss.shaded.ratis.*). > > > > > > > >---------------------------------------------------------------------- > > > >What we are asking the community to accept > > > >---------------------------------------------------------------------- > > > > > > > >This FIP asks the community to accept Phases 1-5 as the merge/release > > > >shape: > > > > > > > > 1. Dual-support (ZK or Raft via config) > > > > 2. Testing (dual-backend HA, Raft correctness, chaos ITs) > > > > 3. Migration tools > > > > 4. Docs > > > > 5. Raft as the default > > > > > > > >Phase 6 (delete ZooKeeper mode and the shaded ZK dependency) is a > > > >future major and is out of the first merge. That deletion is not part > > > >of the first release of this work. > > > > > > > >Goals for this work: > > > > > > > > 1. Default deploy is Fluss-only: CoordinatorServers form a Raft > > > > group; no ZooKeeper process is required. > > > > 2. A supported dual-backend window so existing ZK clusters can keep > > > > running and migrate with tools, not a flag day. > > > > 3. No change to client APIs, log/KV format, or bootstrap.servers. > > > > 4. A later major (Phase 6) removes ZK mode after soak. > > > > > > > >---------------------------------------------------------------------- > > > >Compatibility (clients do not change) > > > >---------------------------------------------------------------------- > > > > > > > >Unchanged: > > > > > > > > - Client bootstrap is still bootstrap.servers. fluss-client has no > > > > ZooKeeper dependency. > > > > - Admin / DDL / produce / fetch / lookup public APIs and table RPCs. > > > > - Binary log / KV format. > > > > - Connector / SDK wire protocol. > > > > > > > >Downstream *test harnesses* that launch a ZK container and inject > > > >zookeeper.address must switch to seed nodes when the server image > > > >defaults to Raft. > > > > > > > >Existing ZK clusters keep working if zookeeper.address remains set and > > > >coordinator.seed.nodes is not set. Mixed ZK+Raft in one cluster is > > > >unsupported (no dual-write). > > > > > > > >---------------------------------------------------------------------- > > > >Backend selection and configuration > > > >---------------------------------------------------------------------- > > > > > > > >Backend selection is exclusive. Setting both keys fails fast > > > >(IllegalConfigurationException): > > > > > > > > 1. zookeeper.address only -> ZOOKEEPER (deprecated; warn at > > > > startup) > > > > 2. coordinator.seed.nodes only -> RAFT > > > > 3. Both -> reject > > > > 4. Neither -> RAFT (current default on the > > > > implementation branch) > > > > > > > >A silent default flip to Raft on upgrade would be a foot-gun; existing > > > >ZK clusters MUST keep an explicit zookeeper.address on upgrade. Docs > > > >and a startup warning exist for that reason. > > > > > > > >New Raft / discovery keys (defaults in ConfigOptions): > > > > > > > > coordinator.raft.voters static voter set > > host:raftPort > > > > coordinator.raft.node.id this node's Raft peer id > > > > coordinator.raft.port 9124 (dedicated Raft > > gRPC; > > > > not client-facing) > > > > coordinator.raft.dir local Raft log + metadata > > > > KvStore directory > > > > coordinator.raft.heartbeat.interval 500ms > > > > coordinator.raft.election.timeout 1s > > > > coordinator.raft.first-election.timeout 300ms > > > > coordinator.raft.catchup.timeout 30s (new leader must apply > > > > through leadership-gain > > > > index before serving) > > > > coordinator.raft.log.segment-size 8mb > > > > coordinator.raft.snapshot.trigger.threshold 10000 > > > > coordinator.raft.id-block-size 1000 > > > > coordinator.tablet-server.session-timeout 30s > > > > coordinator.metadata-stream.max-wait 500ms > > > > coordinator.gossip.enabled false (optional SWIM-style > > > > TCP gossip to expand a > > > > *partial* seed list; does > > > > not replace raft.voters) > > > > coordinator.gossip.port 9125 > > > > coordinator.raft.tls.enabled false > > > > > > > >Firewall coordinator.raft.port (and gossip, if enabled) to coordinator > > > >hosts. They are not client ports. Deprecated but kept until Phase 6: > > > >zookeeper.address, zookeeper.path.root, zookeeper.client.*. > > > > > > > >Gossip is not consensus. Static seeds still select the Raft backend. > > > > > > > >---------------------------------------------------------------------- > > > >New RPCs, CLI, and ops surface > > > >---------------------------------------------------------------------- > > > > > > > >New PRIVATE / PUBLIC RPCs in FlussApi.proto / ApiKeys. Mixed-version > > > >clients that only use existing Admin/data APIs are unaffected. > > > > > > > > FETCH_METADATA 1065 PRIVATE pull committed metadata > > > > records by offset > > > > FETCH_METADATA_SNAPSHOT 1066 PRIVATE bootstrap when from_offset > > > > was compacted > > > > REGISTER_TABLET_SERVER 1067 PRIVATE replaces ephemeral > > > > /tabletservers/ids/[id] > > > > TABLET_SERVER_HEARTBEAT 1068 PRIVATE in-memory lease renewal; > > > > expiry appends fence > > > > ALLOCATE_ID 1069 PRIVATE contiguous ID range from a > > > > Raft sequence counter > > > > GET_COORDINATOR_RAFT_STATUS 1070 PUBLIC leader identity + voter lag > > > > ADD_RAFT_VOTER 1071 PUBLIC quorum add; leader-only > > > > REMOVE_RAFT_VOTER 1072 PUBLIC quorum remove with safety > > > > checks > > > > > > > >On the ZooKeeper backend, the PRIVATE TabletServer-facing RPCs are not > > > >served (tablets still register via ZK). > > > > > > > >New bin/fluss umbrella (FlussCli): > > > > > > > > fluss raft bootstrap > > > > fluss raft status [--watch] [--interval ms] [--json] > > > > fluss raft leader > > > > fluss raft add-voter / remove-voter > > > > fluss raft unsafe-recover --i-understand-data-loss > > > > > > > > fluss zk-to-raft-export > > > > fluss zk-to-raft-import > > > > fluss zk-to-raft-validate > > > > fluss zk-to-raft-live-migrate > > > > > > > >unsafe-recover is offline, majority-loss recovery on the most > > > >up-to-date survivor only; it can lose unreplicated commits. It never > > > >runs automatically. > > > > > > > >Coordinators expose Raft gauges on the existing fluss-metrics > > > >MetricGroup (raftRole, raftTerm, raftCommitIndex, raftAppliedIndex, > > > >raftApplyLag, raftFollowerLag). No new public Java API for end users. > > > > > > > >---------------------------------------------------------------------- > > > >Migration > > > >---------------------------------------------------------------------- > > > > > > > >Recommended path (also documented in > > > >website/docs/install-deploy/migrate-zk-to-raft.md): > > > > > > > > 1. Export ZK (zk-to-raft-export). First export may run online. > > > > 2. Provision an empty Raft quorum; fluss raft bootstrap. > > > > 3. Import (zk-to-raft-import) and byte-validate > > > > (zk-to-raft-validate). Payloads reuse existing JSON serdes. > > > > 4. Cut over: either a short write-fence + re-export/import, or > > > > zk-to-raft-live-migrate (re-scan {path,mzxid} diffs until quiet, > > > > then fence + final scan). No ZooKeeper watches as CDC — watches > > > > coalesce and are not a reliable change feed. > > > > 5. Point bootstrap.servers / seed nodes at Raft coordinators; > > > > decommission ZK after soak. > > > > > > > >Live-migrate is a downtime shrink, not a correctness substitute for > > > >validation. Prefer stop-the-world if the metadata tree is huge or > > > >writes never quiet. > > > > > > > >Rollback while ZK data is still intact: stop Raft coordinators, > > > >restore zookeeper.address, restart. > > > > > > > >New / default clusters: no ZooKeeper. Format coordinator.raft.dir > > > >once, set seed nodes + voters, start coordinators then tablets. > > > > > > > >---------------------------------------------------------------------- > > > >Performance (JMH, 1-node loopback) > > > >---------------------------------------------------------------------- > > > > > > > >fluss-jmh compared embedded Curator TestingServer (ZK) vs 1-voter > > > >embedded Ratis on the same machine, same flags (2026-08-16 / 08-17). > > > >This is not a 3-voter same-AZ NVMe cluster. Production Raft writes > > > >still need majority fsync + RTT; the <5ms p50 write target in the > > > >design is that production bar, not this loopback run. > > > > > > > >The paths that dominate coordinator metadata traffic are faster on > > > >Raft: > > > > > > > > Reads (store, in-process KvStore vs ZK RPC) > > > > getTable ZK 102 µs Raft 1.6 µs ~64x > > > > getDatabase ZK 101 µs Raft 1.0 µs ~100x > > > > prefix list / scan ZK ~100 µs Raft 1-3 µs > > > > > > > > Reads (Admin / RPC, warm cluster) > > > > getTableInfo ZK 421 µs Raft 89 µs ~4.7x > > > > listTables ZK 283 µs Raft 70 µs ~4x > > > > listPartitionInfos ZK 18.5 ms Raft 500 µs ~37x > > > > > > > > Sequence IDs (Raft reserves a block per append; > > > > default coordinator.raft.id-block-size=1000) > > > > ZK 1:1 CAS 4871 µs/id (~205 ops/s) > > > > Raft blockSize=1 8194 µs/id (one append per id; loses) > > > > Raft blockSize=64 127 µs/id (~7.9k ops/s) > > > > Raft blockSize=1000 8.5 µs/id (~590x vs ZK) > > > > > > > > Multi-key / pipelined DDL (appendAll: several keys, one apply wait) > > > > Admin createTable ZK 40.3 ms Raft 9.3 ms ~4.3x > > > > Admin dropTable ZK 89.3 ms Raft 43.7 ms ~2x > > > > Admin createPartition ZK 28.8 ms Raft 17.0 ms ~1.7x > > > > store registerFirstSchema ZK 20.4 ms Raft 8.8 ms > > > > store registerPartition ZK 14.9 ms Raft 8.8 ms > > > > store registerLeaderAndIsr ZK 15.6 ms Raft 8.5 ms > > > > > > > >Single-key store writes on this 1-voter box are not faster today: > > > >Raft still fsyncs the local log, so registerTable is ZK 4.7 ms vs > > > >Raft 8.5 ms. A 3-voter quorum adds majority RTT on top of that. > > > >That is the physical floor of one durable Raft append (majority > > > >fsync + RTT), not a claim that the write path is finished. > > > > > > > >There is still room to close the gap without weakening > > > >commit-apply: > > > > > > > > - Pipeline / group-commit. JMH is @Threads(1) + append().get(), > > > > which empties the Ratis log queue and forces an fsync per > > > > record. Concurrent or batched appends can share a flush > > > > (force.sync.num / async-flush). We have not turned on > > > > unsafe-flush; durability stays on the Raft log fsync. > > > > - More appendAll. 2026-08-17 already folded multi-key DDL into > > > > one apply wait (createTable 24.7 ms -> 9.3 ms; > > > > registerPartition 17.0 ms -> 8.8 ms). Other multi-mutation > > > > coordinator paths can do the same. > > > > - Topology. The design write target is <5 ms p50 on 3-voter > > > > same-AZ NVMe (majority fsync + RTT <1 ms). The 8.5 ms number > > > > is 1-node loopback on a laptop disk, not that bar. > > > > - Not in v1: leader pre-apply (rejected for correctness; may be > > > > revisited post-GA behind a flag) and follower read-index / > > > > leader leases. > > > > > > > >So: reads, lists, block ID allocation, and pipelined DDL are > > > >already clearly faster than ZK. A single durable append is > > > >bounded by fsync/RTT and still has optimization headroom; it is > > > >not the reason to keep ZooKeeper. > > > > > > > >Leader election is not in the 1-node JMH. RaftLeaderElectionTimingITCase > > > >(3-voter loopback) soft-asserts <5s under CI; the design target is > > > ><1s with election.timeout=1s, vs historical ZK latch times around ~5s. > > > > > > > >Full tables and caveats: wiki design §8.4. > > > > > > > >---------------------------------------------------------------------- > > > >Rejected alternatives (brief) > > > >---------------------------------------------------------------------- > > > > > > > > - Keep ZooKeeper as the long-term default: does not solve deploy or > > > > ops cost; contradicts the published architecture direction. Kept > > > > only as a deprecated dual-support backend until Phase 6. > > > > - External etcd / Consul: same problem class (a second stateful > > > > system). The goal is embedded coordination inside > > > > CoordinatorServers. > > > > - Use Kafka KRaft as the engine: tightly coupled to Kafka's > > > > controller/record stack. We borrow patterns, not the codebase. > > > > - Hand-written Raft: high correctness risk when Ratis already > > > > provides transport and log. > > > > - Every node is a voter (tablets in the Raft group): elastic > > > > TabletServer scale would churn quorum membership. Small, stable > > > > coordinator quorum; tablets as observers. > > > > - GooseFS-style pre-apply (mutate memory before the append commits): > > > > rejected for v1. If leadership is lost mid-flight the in-memory > > > > store diverges from the log. Pure commit-apply only. > > > > - Follower-served strongly consistent reads (read-index / leader > > > > leases) in v1: possible later. v1 is leader-only serving plus an > > > > eventually consistent metadata stream for caches. > > > > > > > >---------------------------------------------------------------------- > > > >Implementation status on the branch > > > >---------------------------------------------------------------------- > > > > > > > >On feature/zookeeper-removal-raft, Phases 1-5 are implemented: > > > >dual-support, embedded Ratis + KvStore, metadata stream, TabletServer > > > >register/heartbeat/fence, bin/fluss, ZK-to-Raft export/import/validate > > > >and live-migrate, Raft TLS/metrics/add-remove-voter/unsafe-recover, > > > >optional gossip, preview docs, and Raft as default when neither key is > > > >set. > > > > > > > >Phase 6 (delete ZK mode) is not started. ZK mode remains supported > > > >(deprecated). > > > > > > > >Test coverage already on the branch includes dual-backend HA > > > >(DualBackendFlussClusters + @EnumSource(MetadataBackend)), Raft > > > >correctness (election timing, stale-leader append reject, minority > > > >cannot commit, metadata stream chaos, TabletServer liveness/fence), > > > >CLI guards, and migration-tool semantics. Release bar is ./mvnw verify > > > >on the affected modules; dual-backend ITs must stay green for both > > > >backends until Phase 6. > > > > > > > >Open (optional, not claimed done): corrupt-snapshot install, disk-full > > > >voter, 3-voter production-topology perf. The JMH in the design is > > > >1-node loopback (embedded Curator TestingServer vs single-voter > > > >Ratis), not a 3-voter same-AZ NVMe cluster. Production Raft writes > > > >still need majority fsync + RTT. > > > > > > > >---------------------------------------------------------------------- > > > >Points I would especially like feedback on > > > >---------------------------------------------------------------------- > > > > > > > > 1. Merge/release shape: is accepting Phases 1-5 now, and deferring > > > > Phase 6 (delete ZK) to a later major, the right contract? > > > > > > > > 2. Default backend: neither-key = Raft. Is the upgrade rule > > > > ("existing ZK clusters must keep an explicit zookeeper.address") > > > > clear enough, or should we require coordinator.seed.nodes to > > > > opt into Raft for one more release? > > > > > > > > 3. Engine choice: Apache Ratis, with KRaft patterns layered on top. > > > > Any objection to embedding Ratis vs another library? > > > > > > > > 4. Voter set: only CoordinatorServers. TabletServers stay observers > > > > and never join the Raft group. > > > > > > > > 5. v1 consistency: leader-only serving; no follower read-index / > > > > leader leases in the first release. > > > > > > > > 6. Migration: stop-the-world export/import as the safe default; > > > > live-migrate as a downtime shrink. Is that the ops story we want > > > > to document? > > > > > > > > 7. Gossip: default off; static coordinator.raft.voters remains the > > > > source of truth for quorum membership. > > > > > > > > 8. unsafe-recover: explicit --i-understand-data-loss, offline, > > > > majority-loss only. Any additional guardrails before we merge > > > > that CLI? > > > > > > > > 9. Performance: 1-node JMH shows Raft much faster on reads, > > > > prefix lists, block ID allocation, and multi-key DDL. A > > > > single-key append is slower on loopback because of local > > > > fsync + awaitApplied; that path is still open to pipeline / > > > > group-commit / same-AZ NVMe measurement, not a v1 pre-apply > > > > change. Does that match what we want to tell operators? > > > > > > > >The full class-level design, sequence diagrams, config appendix, test > > > >plan, and rejected alternatives are on the wiki page linked above. > > > > > > > >Looking forward to your feedback. > > > > > > > >-- forwardxu > > > > > >
