Hi Yang,
Thanks for the careful read, and for pushing this onto the list. Removing
ZooKeeper
is a long-term milestone and these four questions are exactly what the FIP needs
to settle in public. Below is how I currently see each point, grounded in the
code on `feature/zookeeper-removal-raft` and in a 1-node loopback JMH catalog
(ZK `TestingServer` vs 1-voter embedded Ratis, 2026-08-16/17). Those numbers
are a baseline, not a 3-voter production measurement — I will call that out
where it matters.
I would also welcome more people on this thread.
---------------------------------------------------------------------
1. Embedded ZooKeeper as an alternative
---------------------------------------------------------------------
I agree this is a real alternative and should be written up as "Alternatives
considered" in the FIP. Loading the ZK library into the Coordinator looks, at
first glance, similar to loading a Raft component, and it would have been the
smaller patch *before* we introduced a backend-neutral `MetadataStoreClient`.
I do not think it actually solves the problem we are trying to remove, for
four reasons.
(a) Embedded ZK is still ZooKeeper. It is still Zab, DataTree, sessions,
ephemeral znodes, tickTime, and Curator. The operational pain in the problem
statement is not only "an extra process". It is a second consensus system,
a second failure domain, ZK version/shading compatibility, and a data model
(watches + getChildren + 1:1 CAS) that Fluss has been stretching for years
(LeaderAndIsr, sequence ids, ACL change znodes, tablet ephemeral registration).
(b) HA does not get cheaper. A production embedded ZK still needs 3 in-process
quorum members. That is the same "at most one Coordinator unavailable under
the minimum HA deploy" constraint you raise in (4), except we would still be
operating ZooKeeper *inside* those Coordinators, with a separate disk layout
and a separate recovery story. We would not have removed a dependency; we
would have hidden it.
(c) We already keep ZK as a first-class backend. Dual-support is implemented:
`zookeeper.address` xor `coordinator.seed.nodes`. Existing clusters can stay
on ZK until they migrate. The FIP is not "ZK stops working next release"; it
is "Raft becomes the default, ZK is deprecated, then deleted after a window".
Embedded ZK would freeze us on the znode model for another generation.
(d) Even against the fastest ZK we can measure, user-visible paths do not
lose. The JMH catalog compares 1-voter Ratis to an in-process Curator
`TestingServer` — which is essentially the embedded-ZK shape. Honest numbers:
- Single durable metadata write (`registerTable`): ZK 4.7 ms, Raft 8.5 ms
(~1.7x). Root cause is design, not a botched implementation: pure
commit-apply (no leader pre-apply), `append().get()` waits for local
apply, Ratis fsyncs on an empty queue, then RocksDB put. We should not
pretend this is free.
- Point read (`getTable`): ZK ~100 µs, Raft ~1.6 µs (in-process KvStore).
- Prefix list (`listTables` / `listDatabases` / `getCurrentSchemaId`):
after a prefix-iterator fix, Raft ~1 µs vs ZK `getChildren` ~100 µs.
- Admin `createTable` (schema + assignment + table + tablet ISR/RPC):
ZK ~40 ms, Raft ~9.3 ms, after pipelining the three store appends into
one apply wait (`appendAll`).
- Sequence ids: ZK is one CAS per id (~5 ms). Raft `id-block-size=1000`
is 8.5 µs/id. Using Raft as 1:1 CAS is slower, which is why we do not.
So: Raft loses on isolated single-key durable writes; it wins on leader
reads, list*, block id allocation, and the DDL the user actually runs.
1-voter already fsyncs; 3-voter will add majority RTT + a follower fsync,
so we must not claim the production write p50 target (<5 ms same-AZ NVMe)
from this catalog. That is a follow-up measurement, not a talking point.
Kafka went from ZK to KRaft, not to "ZK inside the controller". I would
rather follow that split (small coordinator quorum, brokers/tablets stay
out) than re-embed Zab.
---------------------------------------------------------------------
2. Extending the PK-table replica mechanism to implement Raft
---------------------------------------------------------------------
This is the most interesting suggestion, and I want to separate three
things that look similar but are not the same protocol.
What a PK table replica actually is today (`Replica` + `ReplicaFetcherThread`
+ `KvTablet` + `PeriodicSnapshotManager`):
- Kafka-style leader/follower: followers *fetch* from the leader.
- Commit is ISR + high watermark, not majority quorum. min.insync can be 1.
- Leadership is *externally assigned* by the Coordinator via LeaderAndIsr.
The replica protocol does not elect a leader; it is told who the leader is.
- Fencing is leader-epoch / bucket-epoch, not Raft term + log matching.
- Snapshots are RocksDB checkpoints pushed to remote FS for replica catch-up.
What metadata consensus needs:
- Self-contained leader election among a small voter set. It cannot depend
on the Coordinator, because the Coordinator *is* what we are electing.
- Majority commit. Cluster metadata cannot afford unclean leader election.
- One Raft group for the whole cluster, not one group per table-bucket
(we have thousands of those).
That is why I do not want to "grow ISR into Raft" as the ZK-removal path:
- If we stored cluster metadata *in* a PK table, we get a circular
dependency: PK leadership needs Coordinator metadata (LeaderAndIsr),
and Coordinator metadata would need that PK table. This is the same
chicken-and-egg that forced Kafka to invent a dedicated metadata log
rather than putting the controller state in a user topic.
- The missing Raft pieces (votes, term, log matching, majority commit,
membership change) are not a small extension of `ReplicaFetcherThread`.
They *are* Raft. Hand-rolling that on the data-plane fetcher is more
risk than embedding Apache Ratis, which we already shade and which
gives us snapshot, `setConfiguration`, and gRPC transport.
What we *did* reuse from the PK stack, and I think this is the right
granularity:
- RocksDB as the metadata state machine (`RocksDBMetadataKvStore`), with
durability anchored on the Raft log fsync, not a second RocksDB sync.
- Snapshot + log catch-up (`FlussMetadataStateMachine.takeSnapshot` /
`loadSnapshot`), same idea as KV tablet snapshots, different API
(Ratis `StateMachine` vs `PeriodicSnapshotManager`).
- A typed record log applied deterministically (`MetadataRecord` →
`RaftMetadataStore.apply`), commit-apply only, no leader pre-apply.
Your "bonus" — a PK replica unit as strong as TiDB's Region — I agree is
worth a *separate* FIP: data-plane Raft per bucket, replacing ISR. That is
a much larger change (client produce path, high watermark, lake tiering,
rebalance). I would not couple it to ZK removal. If the community wants
that discussion, I am happy to start it after the metadata Raft group is
the source of truth, because a data-plane Raft still needs an independent
place to store membership.
---------------------------------------------------------------------
3. ZK-style tree vs system tables
---------------------------------------------------------------------
The prototype is already neither a filesystem tree nor SQL system tables.
It is a KRaft-style record log:
- One `MetadataRecord` per mutation (`REGISTER_TABLE`,
`UPDATE_LEADER_AND_ISR`,
`ALLOCATE_ID_RANGE`, …), codec `[version][type][key][payload]`.
- Applied into a KvStore whose keys are `[1B namespace][2B type][body]`
(`MetadataKeys`). The header layout is deliberately aligned with the
old znode tree so `ZkToRaftMigrationTool` is a mechanical path rewrite,
not a semantic redesign.
- TabletServers do not watch znodes; they pull a metadata stream by Raft
index (`FetchMetadata` / `FetchMetadataSnapshot`).
I would keep that for v1 of ZK removal.
A relational "system tables" layer on top of the same Raft log is attractive
for operators (SELECT from tables / assignments / isr). Coordinator today
has no SQL engine, and introducing one in this FIP would mix a catalog
redesign into a consensus replacement. The log is the source of truth;
the KvStore is a materialized view. We can later project that view into
system tables (or keep prefix scans, which after the iterator fix are ~1 µs)
without changing the Raft group.
What I would *not* do in v1 is a `MULTI_OP` record type that packs N
mutations into one log entry. That needs a metadata.version bump, stream
applier changes, and mixed-version quorum rules. Today's `createTable` is
already not a ZK multi-op transaction (schema can be left behind). We
pipelined independent entries (`appendAll`) instead; failure semantics
stay "prefix may already be committed", same class as ZK.
---------------------------------------------------------------------
4. Operational model of embedding consensus in the Coordinator
---------------------------------------------------------------------
I agree with you: this is a real trade-off, not a free simplification.
It needs an explicit ops section in the FIP, not a footnote.
What changes:
- Today, HA is "ZooKeeper ensemble (typically 3) + N Coordinators".
A single Coordinator is a valid production shape because ZK holds
the quorum.
- After, HA is "3 (or 5) Coordinator voters". TabletServers stay *out*
of the Raft group (KRaft-style observers). Clients still use
`bootstrap.servers`.
- Minimum HA deploy: 3 Coordinators, tolerate 1 down — the same
majority math as a 3-node ZK ensemble. We do not get to keep
"1 Coordinator + 3 ZK" and also drop ZK.
- Coordinators become more precious: rolling restart, upgrade, and
replace must never drop the live set below majority.
What we already specified / implemented so this is not hand-waving:
- Odd sizing only (1 / 3 / 5). 1-voter is dev/bootstrap. 2 and 4 are
refused as a recommendation.
- `fluss raft add-voter` / `remove-voter` via Ratis `setConfiguration`,
with a quorum-safety check (refuse a remove that would lose majority).
Dead-node runbook: remove-voter, provision, add-voter.
- Rolling upgrade: one Coordinator at a time; new binary must apply
all historical record versions; cluster `metadata.version` is a
feature gate (leader refuses a bump if any voter is behind).
- Dual-support + `fluss zk-to-raft-migrate` / live-migrate, so nobody
is forced off ZK in the same release that Raft becomes default.
- Guarded `fluss raft unsafe-recover` for true majority loss (explicit
data-loss flag).
- `fluss raft status [--watch]`, TLS on the Raft port, and Raft
metrics (`role`, `term`, `commitIndex`, `applyLag`, `appendLatency`).
- HA ITs on both backends (`CoordinatorHighAvailabilityITCase`: ZK
session-kill vs Raft 3-voter close+restart;
`RaftMinorityCannotCommitITCase`;
`RaftStaleLeaderRejectITCase`).
What I would still like the FIP discussion to pin down:
- Recommended production topology: 3 vs 5 Coordinators.
- Rolling restart order (followers first; never restart a second voter
until the first has caught up).
- Whether we document a "dev: 1 Coordinator" profile next to "HA: 3".
- A 3-voter same-AZ NVMe benchmark before we call write latency done.
1-voter `registerTable` is already ~8.5 ms; 3-voter cannot be faster.
Net: we trade "operate ZK + operate Fluss" for "operate a 3-node
Coordinator quorum with Raft rules". Fewer process types, stricter
Coordinator lifecycle. I think that is the right trade for the roadmap,
but it should be an explicit choice in the FIP, not an implied one.
---------------------------------------------------------------------
Proposal for the next FIP revision
----------------------------------
I will add three sections to the FIP based on this thread:
1. Alternatives considered: embedded ZK; "PK table as Raft"; KRaft-in-process.
2. Metadata data model: record log + KvStore now; system tables as a
possible later projection of the same log.
3. Operations: quorum sizing, rolling restart/upgrade, add/remove-voter,
migration, and what HA looks like compared to ZK today.
If you (or others) disagree with (1) or (2) in particular, I would rather
find that out here than in a vote. The prototype branch is available for
anyone who wants to poke at `MetadataStoreClient`, the record types, or
the JMH classes (`MetadataStoreBenchmark` / `MetadataPathBenchmark`).
Thanks again for the questions.
Best,
Xu
On 2026/08/17 14:42:39 Yang Wang wrote:
> Hi Xu,
> Removing ZooKeeper is a significant milestone in the long-term roadmap, and
> I think this is a truly exciting effort — I'm glad you've decided to drive
> it. Given how important and complex it is, I believe we should encourage
> more people to join the discussion on this proposal.
> I've read the first version of the FIP and have some thoughts and questions
> that I'd like to discuss:
> 1. An alternative worth considering: embedded ZooKeeper. Looking at both
> the problem statement and the proposed solution, there is actually another
> option: ZooKeeper supports embedded deployment, meaning the Coordinator
> could load the ZooKeeper library directly and run it as a dedicated role.
> This mode seems essentially no different from having the Coordinator load a
> Raft component, yet it would require far fewer code-level changes.
> 2. Extending Fluss's existing primary-key table mechanism to implement
> Raft. The
> core of Raft consists of: (1) a log replication protocol; (2) quorum-based
> commit; (3) a safety argument based on Term (Epoch); and (4) fast catch-up
> based on Snapshot + Log. In fact, many of Raft's components already have
> corresponding implementations in Fluss — for example, the primary-key table
> mechanism (multi-replica log replication + snapshot + KV state machine). So
> could we extend Fluss's existing primary-key table mechanism to implement
> an equivalent Raft protocol? This has obvious benefits: the code would
> become much easier to maintain and understand, and as a bonus we would gain
> a primary-key table replica unit as powerful as TiDB's. This may be a
> direction worth discussing.
> 3. Metadata state machine design: ZK-style tree vs. system tables. Should
> we continue with a ZK-style filesystem-tree metadata state machine, or
> consider following the style of database systems and introducing system
> tables implemented on top of the Raft protocol to store metadata?
> 4. New operational challenges from embedding consensus into the
> Coordinator. ZooKeeper
> is indeed hard to maintain, but embedding the consensus protocol directly
> into the Coordinator role may introduce new challenges: (1) the
> Coordinator's high availability becomes more delicate — it requires at
> least 3 replicas, and under the minimum deployment at most one Coordinator
> can be unavailable; (2) Coordinator upgrades, rolling restarts, and
> migrations start to carry specific constraints. It may also be worth
> discussing the operational and deployment models, including the trade-offs
> of different architectural choices.
>
> Best,
> Yang
>
> Forward Xu <[email protected]> 于2026年8月17日周一 20:42写道：
>
> > Hi all,
> >
> > I would like to start a discussion for FIP-52: ZooKeeper Removal and
> > Embedded Raft Metadata Backend:
> >
> >
> > https://cwiki.apache.org/confluence/spaces/FLUSS/pages/449282216/FIP-52+ZooKeeper+Removal+and+Embedded+Raft+Metadata+Backend
> >
> > Issue: https://github.com/apache/fluss/issues/4014
> > Branch: feature/zookeeper-removal-raft
> >
> > Please keep the discussion on this list rather than commenting on the
> > wiki.
> >
> > ----------------------------------------------------------------------
> > Why
> > ----------------------------------------------------------------------
> >
> > Fluss today depends on an external ZooKeeper ensemble for cluster
> > coordination, metadata persistence, sequence IDs, TabletServer
> > liveness (ephemeral znodes), dynamic config, ACLs, and related leases.
> > Clients never talk to ZooKeeper (bootstrap.servers + Admin RPC only);
> > servers do. Operators still have to run ZooKeeper for every Fluss
> > cluster:
> >
> > - A production cluster cannot start without a healthy ZK ensemble
> > (quorum, tick time, session timeout, SASL/ACL, backups).
> > - ZK issues cascade into coordinator leadership, TabletServer
> > registration, and metadata availability.
> > - Extra network hops and a second failure domain sit on the
> > metadata path.
> > - Container, Kubernetes, edge, and single-node developer setups pay
> > for an extra stateful service that exists only for coordination.
> >
> > This is already the published architecture direction: ZooKeeper is a
> > transitional dependency, to be replaced by KvStore for metadata and
> > Raft for coordination.
> >
> > ----------------------------------------------------------------------
> > What this FIP proposes
> > ----------------------------------------------------------------------
> >
> > Replace ZooKeeper with an embedded Apache Ratis Raft quorum on
> > CoordinatorServers, plus a RocksDB-backed metadata KvStore.
> >
> > Only CoordinatorServers are Raft voters. TabletServers are observers:
> > they discover a coordinator from coordinator.seed.nodes (optionally
> > expanded by gossip), register/heartbeat over RPC, and consume a
> > pull-based metadata stream.
> >
> > Today Target
> > ----- ------
> > Coordinator ──► ZooKeeper Coordinator quorum
> > TabletServer ──► ZooKeeper └── embedded Ratis + KvStore
> > Clients ──► bootstrap.servers TabletServer ──► Register /
> > Heartbeat / FetchMetadata
> > Clients ──► bootstrap.servers
> > (unchanged)
> >
> > Consensus engine is Apache Ratis. Metadata apply is KRaft-style
> > commit-apply: every node, including the leader, mutates the store only
> > after a log entry is committed. Leader-only serving for strongly
> > consistent reads; followers redirect. The Raft term is the coordinator
> > epoch.
> >
> > How each ZooKeeper role is replaced:
> >
> > - Coordinator leader election (Curator latch)
> > -> Ratis election + RaftLeaderSelector (gainPrimacy / losePrimacy,
> > catch-up before serving)
> > - Database / table / partition / schema / assignment / LeaderAndIsr /
> > ACL / config / leases
> > -> typed MetadataRecords in the Raft log, applied into a
> > RocksDB-backed metadata KvStore (same JSON serdes as ZK values)
> > - Sequence IDs
> > -> RaftSequenceIDCounter with pre-reserved blocks
> > (coordinator.raft.id-block-size)
> > - TabletServer ephemeral znode + session
> > -> RegisterTabletServer + heartbeat lease; timeout appends a
> > replicated FENCE_TABLET_SERVER
> > - ZK watches
> > -> in-process watches on the leader store; cross-process
> > FetchMetadata stream (offset = Raft index)
> > - zkCli observability
> > -> fluss raft status / leader + Raft metrics
> >
> > We borrow KRaft patterns (commit-apply, record log, broker-style
> > registration/heartbeat, observer tablets, id blocks, metadata stream)
> > and reject KRaft-the-codebase. Fluss needs an embeddable Java Raft
> > library with a pluggable state machine; that is Apache Ratis, shaded
> > as fluss-shaded-ratis (org.apache.fluss.shaded.ratis.*).
> >
> > ----------------------------------------------------------------------
> > What we are asking the community to accept
> > ----------------------------------------------------------------------
> >
> > This FIP asks the community to accept Phases 1-5 as the merge/release
> > shape:
> >
> > 1. Dual-support (ZK or Raft via config)
> > 2. Testing (dual-backend HA, Raft correctness, chaos ITs)
> > 3. Migration tools
> > 4. Docs
> > 5. Raft as the default
> >
> > Phase 6 (delete ZooKeeper mode and the shaded ZK dependency) is a
> > future major and is out of the first merge. That deletion is not part
> > of the first release of this work.
> >
> > Goals for this work:
> >
> > 1. Default deploy is Fluss-only: CoordinatorServers form a Raft
> > group; no ZooKeeper process is required.
> > 2. A supported dual-backend window so existing ZK clusters can keep
> > running and migrate with tools, not a flag day.
> > 3. No change to client APIs, log/KV format, or bootstrap.servers.
> > 4. A later major (Phase 6) removes ZK mode after soak.
> >
> > ----------------------------------------------------------------------
> > Compatibility (clients do not change)
> > ----------------------------------------------------------------------
> >
> > Unchanged:
> >
> > - Client bootstrap is still bootstrap.servers. fluss-client has no
> > ZooKeeper dependency.
> > - Admin / DDL / produce / fetch / lookup public APIs and table RPCs.
> > - Binary log / KV format.
> > - Connector / SDK wire protocol.
> >
> > Downstream *test harnesses* that launch a ZK container and inject
> > zookeeper.address must switch to seed nodes when the server image
> > defaults to Raft.
> >
> > Existing ZK clusters keep working if zookeeper.address remains set and
> > coordinator.seed.nodes is not set. Mixed ZK+Raft in one cluster is
> > unsupported (no dual-write).
> >
> > ----------------------------------------------------------------------
> > Backend selection and configuration
> > ----------------------------------------------------------------------
> >
> > Backend selection is exclusive. Setting both keys fails fast
> > (IllegalConfigurationException):
> >
> > 1. zookeeper.address only -> ZOOKEEPER (deprecated; warn at
> > startup)
> > 2. coordinator.seed.nodes only -> RAFT
> > 3. Both -> reject
> > 4. Neither -> RAFT (current default on the
> > implementation branch)
> >
> > A silent default flip to Raft on upgrade would be a foot-gun; existing
> > ZK clusters MUST keep an explicit zookeeper.address on upgrade. Docs
> > and a startup warning exist for that reason.
> >
> > New Raft / discovery keys (defaults in ConfigOptions):
> >
> > coordinator.raft.voters static voter set
> > host:raftPort
> > coordinator.raft.node.id this node's Raft peer id
> > coordinator.raft.port 9124 (dedicated Raft gRPC;
> > not client-facing)
> > coordinator.raft.dir local Raft log + metadata
> > KvStore directory
> > coordinator.raft.heartbeat.interval 500ms
> > coordinator.raft.election.timeout 1s
> > coordinator.raft.first-election.timeout 300ms
> > coordinator.raft.catchup.timeout 30s (new leader must apply
> > through leadership-gain
> > index before serving)
> > coordinator.raft.log.segment-size 8mb
> > coordinator.raft.snapshot.trigger.threshold 10000
> > coordinator.raft.id-block-size 1000
> > coordinator.tablet-server.session-timeout 30s
> > coordinator.metadata-stream.max-wait 500ms
> > coordinator.gossip.enabled false (optional SWIM-style
> > TCP gossip to expand a
> > *partial* seed list; does
> > not replace raft.voters)
> > coordinator.gossip.port 9125
> > coordinator.raft.tls.enabled false
> >
> > Firewall coordinator.raft.port (and gossip, if enabled) to coordinator
> > hosts. They are not client ports. Deprecated but kept until Phase 6:
> > zookeeper.address, zookeeper.path.root, zookeeper.client.*.
> >
> > Gossip is not consensus. Static seeds still select the Raft backend.
> >
> > ----------------------------------------------------------------------
> > New RPCs, CLI, and ops surface
> > ----------------------------------------------------------------------
> >
> > New PRIVATE / PUBLIC RPCs in FlussApi.proto / ApiKeys. Mixed-version
> > clients that only use existing Admin/data APIs are unaffected.
> >
> > FETCH_METADATA 1065 PRIVATE pull committed metadata
> > records by offset
> > FETCH_METADATA_SNAPSHOT 1066 PRIVATE bootstrap when from_offset
> > was compacted
> > REGISTER_TABLET_SERVER 1067 PRIVATE replaces ephemeral
> > /tabletservers/ids/[id]
> > TABLET_SERVER_HEARTBEAT 1068 PRIVATE in-memory lease renewal;
> > expiry appends fence
> > ALLOCATE_ID 1069 PRIVATE contiguous ID range from a
> > Raft sequence counter
> > GET_COORDINATOR_RAFT_STATUS 1070 PUBLIC leader identity + voter lag
> > ADD_RAFT_VOTER 1071 PUBLIC quorum add; leader-only
> > REMOVE_RAFT_VOTER 1072 PUBLIC quorum remove with safety
> > checks
> >
> > On the ZooKeeper backend, the PRIVATE TabletServer-facing RPCs are not
> > served (tablets still register via ZK).
> >
> > New bin/fluss umbrella (FlussCli):
> >
> > fluss raft bootstrap
> > fluss raft status [--watch] [--interval ms] [--json]
> > fluss raft leader
> > fluss raft add-voter / remove-voter
> > fluss raft unsafe-recover --i-understand-data-loss
> >
> > fluss zk-to-raft-export
> > fluss zk-to-raft-import
> > fluss zk-to-raft-validate
> > fluss zk-to-raft-live-migrate
> >
> > unsafe-recover is offline, majority-loss recovery on the most
> > up-to-date survivor only; it can lose unreplicated commits. It never
> > runs automatically.
> >
> > Coordinators expose Raft gauges on the existing fluss-metrics
> > MetricGroup (raftRole, raftTerm, raftCommitIndex, raftAppliedIndex,
> > raftApplyLag, raftFollowerLag). No new public Java API for end users.
> >
> > ----------------------------------------------------------------------
> > Migration
> > ----------------------------------------------------------------------
> >
> > Recommended path (also documented in
> > website/docs/install-deploy/migrate-zk-to-raft.md):
> >
> > 1. Export ZK (zk-to-raft-export). First export may run online.
> > 2. Provision an empty Raft quorum; fluss raft bootstrap.
> > 3. Import (zk-to-raft-import) and byte-validate
> > (zk-to-raft-validate). Payloads reuse existing JSON serdes.
> > 4. Cut over: either a short write-fence + re-export/import, or
> > zk-to-raft-live-migrate (re-scan {path,mzxid} diffs until quiet,
> > then fence + final scan). No ZooKeeper watches as CDC — watches
> > coalesce and are not a reliable change feed.
> > 5. Point bootstrap.servers / seed nodes at Raft coordinators;
> > decommission ZK after soak.
> >
> > Live-migrate is a downtime shrink, not a correctness substitute for
> > validation. Prefer stop-the-world if the metadata tree is huge or
> > writes never quiet.
> >
> > Rollback while ZK data is still intact: stop Raft coordinators,
> > restore zookeeper.address, restart.
> >
> > New / default clusters: no ZooKeeper. Format coordinator.raft.dir
> > once, set seed nodes + voters, start coordinators then tablets.
> >
> > ----------------------------------------------------------------------
> > Performance (JMH, 1-node loopback)
> > ----------------------------------------------------------------------
> >
> > fluss-jmh compared embedded Curator TestingServer (ZK) vs 1-voter
> > embedded Ratis on the same machine, same flags (2026-08-16 / 08-17).
> > This is not a 3-voter same-AZ NVMe cluster. Production Raft writes
> > still need majority fsync + RTT; the <5ms p50 write target in the
> > design is that production bar, not this loopback run.
> >
> > The paths that dominate coordinator metadata traffic are faster on
> > Raft:
> >
> > Reads (store, in-process KvStore vs ZK RPC)
> > getTable ZK 102 µs Raft 1.6 µs ~64x
> > getDatabase ZK 101 µs Raft 1.0 µs ~100x
> > prefix list / scan ZK ~100 µs Raft 1-3 µs
> >
> > Reads (Admin / RPC, warm cluster)
> > getTableInfo ZK 421 µs Raft 89 µs ~4.7x
> > listTables ZK 283 µs Raft 70 µs ~4x
> > listPartitionInfos ZK 18.5 ms Raft 500 µs ~37x
> >
> > Sequence IDs (Raft reserves a block per append;
> > default coordinator.raft.id-block-size=1000)
> > ZK 1:1 CAS 4871 µs/id (~205 ops/s)
> > Raft blockSize=1 8194 µs/id (one append per id; loses)
> > Raft blockSize=64 127 µs/id (~7.9k ops/s)
> > Raft blockSize=1000 8.5 µs/id (~590x vs ZK)
> >
> > Multi-key / pipelined DDL (appendAll: several keys, one apply wait)
> > Admin createTable ZK 40.3 ms Raft 9.3 ms ~4.3x
> > Admin dropTable ZK 89.3 ms Raft 43.7 ms ~2x
> > Admin createPartition ZK 28.8 ms Raft 17.0 ms ~1.7x
> > store registerFirstSchema ZK 20.4 ms Raft 8.8 ms
> > store registerPartition ZK 14.9 ms Raft 8.8 ms
> > store registerLeaderAndIsr ZK 15.6 ms Raft 8.5 ms
> >
> > Single-key store writes on this 1-voter box are not faster today:
> > Raft still fsyncs the local log, so registerTable is ZK 4.7 ms vs
> > Raft 8.5 ms. A 3-voter quorum adds majority RTT on top of that.
> > That is the physical floor of one durable Raft append (majority
> > fsync + RTT), not a claim that the write path is finished.
> >
> > There is still room to close the gap without weakening
> > commit-apply:
> >
> > - Pipeline / group-commit. JMH is @Threads(1) + append().get(),
> > which empties the Ratis log queue and forces an fsync per
> > record. Concurrent or batched appends can share a flush
> > (force.sync.num / async-flush). We have not turned on
> > unsafe-flush; durability stays on the Raft log fsync.
> > - More appendAll. 2026-08-17 already folded multi-key DDL into
> > one apply wait (createTable 24.7 ms -> 9.3 ms;
> > registerPartition 17.0 ms -> 8.8 ms). Other multi-mutation
> > coordinator paths can do the same.
> > - Topology. The design write target is <5 ms p50 on 3-voter
> > same-AZ NVMe (majority fsync + RTT <1 ms). The 8.5 ms number
> > is 1-node loopback on a laptop disk, not that bar.
> > - Not in v1: leader pre-apply (rejected for correctness; may be
> > revisited post-GA behind a flag) and follower read-index /
> > leader leases.
> >
> > So: reads, lists, block ID allocation, and pipelined DDL are
> > already clearly faster than ZK. A single durable append is
> > bounded by fsync/RTT and still has optimization headroom; it is
> > not the reason to keep ZooKeeper.
> >
> > Leader election is not in the 1-node JMH. RaftLeaderElectionTimingITCase
> > (3-voter loopback) soft-asserts <5s under CI; the design target is
> > <1s with election.timeout=1s, vs historical ZK latch times around ~5s.
> >
> > Full tables and caveats: wiki design §8.4.
> >
> > ----------------------------------------------------------------------
> > Rejected alternatives (brief)
> > ----------------------------------------------------------------------
> >
> > - Keep ZooKeeper as the long-term default: does not solve deploy or
> > ops cost; contradicts the published architecture direction. Kept
> > only as a deprecated dual-support backend until Phase 6.
> > - External etcd / Consul: same problem class (a second stateful
> > system). The goal is embedded coordination inside
> > CoordinatorServers.
> > - Use Kafka KRaft as the engine: tightly coupled to Kafka's
> > controller/record stack. We borrow patterns, not the codebase.
> > - Hand-written Raft: high correctness risk when Ratis already
> > provides transport and log.
> > - Every node is a voter (tablets in the Raft group): elastic
> > TabletServer scale would churn quorum membership. Small, stable
> > coordinator quorum; tablets as observers.
> > - GooseFS-style pre-apply (mutate memory before the append commits):
> > rejected for v1. If leadership is lost mid-flight the in-memory
> > store diverges from the log. Pure commit-apply only.
> > - Follower-served strongly consistent reads (read-index / leader
> > leases) in v1: possible later. v1 is leader-only serving plus an
> > eventually consistent metadata stream for caches.
> >
> > ----------------------------------------------------------------------
> > Implementation status on the branch
> > ----------------------------------------------------------------------
> >
> > On feature/zookeeper-removal-raft, Phases 1-5 are implemented:
> > dual-support, embedded Ratis + KvStore, metadata stream, TabletServer
> > register/heartbeat/fence, bin/fluss, ZK-to-Raft export/import/validate
> > and live-migrate, Raft TLS/metrics/add-remove-voter/unsafe-recover,
> > optional gossip, preview docs, and Raft as default when neither key is
> > set.
> >
> > Phase 6 (delete ZK mode) is not started. ZK mode remains supported
> > (deprecated).
> >
> > Test coverage already on the branch includes dual-backend HA
> > (DualBackendFlussClusters + @EnumSource(MetadataBackend)), Raft
> > correctness (election timing, stale-leader append reject, minority
> > cannot commit, metadata stream chaos, TabletServer liveness/fence),
> > CLI guards, and migration-tool semantics. Release bar is ./mvnw verify
> > on the affected modules; dual-backend ITs must stay green for both
> > backends until Phase 6.
> >
> > Open (optional, not claimed done): corrupt-snapshot install, disk-full
> > voter, 3-voter production-topology perf. The JMH in the design is
> > 1-node loopback (embedded Curator TestingServer vs single-voter
> > Ratis), not a 3-voter same-AZ NVMe cluster. Production Raft writes
> > still need majority fsync + RTT.
> >
> > ----------------------------------------------------------------------
> > Points I would especially like feedback on
> > ----------------------------------------------------------------------
> >
> > 1. Merge/release shape: is accepting Phases 1-5 now, and deferring
> > Phase 6 (delete ZK) to a later major, the right contract?
> >
> > 2. Default backend: neither-key = Raft. Is the upgrade rule
> > ("existing ZK clusters must keep an explicit zookeeper.address")
> > clear enough, or should we require coordinator.seed.nodes to
> > opt into Raft for one more release?
> >
> > 3. Engine choice: Apache Ratis, with KRaft patterns layered on top.
> > Any objection to embedding Ratis vs another library?
> >
> > 4. Voter set: only CoordinatorServers. TabletServers stay observers
> > and never join the Raft group.
> >
> > 5. v1 consistency: leader-only serving; no follower read-index /
> > leader leases in the first release.
> >
> > 6. Migration: stop-the-world export/import as the safe default;
> > live-migrate as a downtime shrink. Is that the ops story we want
> > to document?
> >
> > 7. Gossip: default off; static coordinator.raft.voters remains the
> > source of truth for quorum membership.
> >
> > 8. unsafe-recover: explicit --i-understand-data-loss, offline,
> > majority-loss only. Any additional guardrails before we merge
> > that CLI?
> >
> > 9. Performance: 1-node JMH shows Raft much faster on reads,
> > prefix lists, block ID allocation, and multi-key DDL. A
> > single-key append is slower on loopback because of local
> > fsync + awaitApplied; that path is still open to pipeline /
> > group-commit / same-AZ NVMe measurement, not a v1 pre-apply
> > change. Does that match what we want to tell operators?
> >
> > The full class-level design, sequence diagrams, config appendix, test
> > plan, and rejected alternatives are on the wiki page linked above.
> >
> > Looking forward to your feedback.
> >
> > -- forwardxu
> >
>
