Hi Zuo, Thank you for the detailed report. Let me see if I can provide a breakdown here.
1. NotifyLeaderAndIsrRequest and UpdateMetadataRequest from the Coordinator are both handled directly on the Netty/RPC worker thread pool in TabletService. The becomeLeaderOrFollower path inside those handlers can be slow (opening RocksDB, replaying logs, updating metadata cache), which causes the worker thread to block. Because the worker pool is shared, all concurrent RPC requests, including subsequent metadata updates, queue behind it. Under leader-election storms (e.g., rolling restarts, failover), this can stall the entire tablet server. This PR should help with this: https://github.com/apache/fluss/pull/3647 along with: https://github.com/apache/fluss/pull/3654 .. Until a new release is cut, maybe increasing *rpc.worker.threads* helps with this issue. 2. This is most likely a metadata propagation timing issue. When auto-partition creates a new partition in ZooKeeper, there is a window before the TabletServer's in-memory metadata cache receives the UpdateMetadata from the Coordinator. If the client resolves the partition and attempts to write during that window, the client's retry logic does not always recover cleanly. This PR probably helps with this: https://github.com/apache/fluss/pull/3511 . Some other configuration options that might help: - Increase *auto-partition.num-precreate* so partitions are pre-created ahead of time, reducing the race window. - Add retry logic in your producer application, for example, a transient PartitionNotExistException during auto-partition creation should be retried; the partition will eventually appear. - Ensure your auto-partition *time-interval *is not too aggressive relative to the time it takes for metadata to propagate. 3. When a TabletServer restarts abnormally, the Coordinator triggers leader elections for all KV buckets that were on that node. Each NotifyLeaderAndIsrRequest results in becomeLeaderOrFollower → RocksDB open() call (which acquires the LOCK file, reads SST files, replays WAL). This is inherently slow and happens on the RPC worker thread in 0.9.1, serializing all subsequent RPCs. PR #3654 should probably help with this issue as well, and PR https://github.com/apache/fluss/pull/3651 further reduces shutdown time so restarts are faster overall. So I believe the fixes are most likely there and targeted for the upcoming 1.0 release. So if your timeline allows, the safest path is to for the 1.0 release. PRs #3647, #3654, #3511, and #3651 will all be included. The goal is to release within the next month. Let me know if the above helps. Best, Giannis On Thu, Jul 23, 2026 at 3:22 PM zuo wei <[email protected]> wrote: > Dear Apache Fluss Community Developers, > > I’m a R&D engineer from Xiaomi. We’re currently testing *Apache Fluss > 0.9.1* > for our internal business scenarios, and ran into several severe unstable > issues during our tests. I’d like to share these problems and ask for your > guidance. > > Here are the some main issues we encountered: > *1. RPC stuck issue*: RPC requests (e.g. update metadata request) from > Coordinator Server to Tablet Server often get stuck for a long time, > causing service blocking. > *2. Partition not exist error*: Data writing throws partition-not-exist > exceptions even when the target partitions exist normally. > *3. RocksDB lock blocking issue*: After abnormal restart of Tablet Server > nodes, a lot of KV buckets leader switching occurs. The RocksDB open > process holds locks for a long time, seriously blocking other worker > threads. > > I'm not sure whether these problems are caused by our improper usage, > configuration mistakes, or existing known bugs of Fluss 0.9.1. And I’d like > to consult the community whether Fluss 0.9.1 currently is recommended for > production environment deployment? > We really want to adopt Fluss in our business environment. Thanks a lot for > your help! Looking forward to your reply. > > Best regards, > > Zuo Wei > Email: [email protected] >
