Hi Giannis, Thanks a lot for your clear and detailed reply! It helps me fully understand the root causes of the stability issues we met in Fluss 0.9.1. I will apply your recommended solutions on our 0.9.1 test cluster immediately. I'm looking forward to the 1.0 release with the related PR fixes.
I really appreciate the community’s efficient problem solving and technical support! During the follow-up testing and optimization process, if I encounter any new problems or have relevant verification results, I will actively synchronize with the community in a timely manner.Thanks again for your great help! Best regards, Zuo Wei Email: [email protected] Giannis Polyzos <[email protected]> 于2026年7月24日周五 13:52写道： > Hi Zuo, > > Thank you for the detailed report. Let me see if I can provide a breakdown > here. > > 1. NotifyLeaderAndIsrRequest and UpdateMetadataRequest from the Coordinator > are both handled directly on the Netty/RPC worker thread pool in > TabletService. The becomeLeaderOrFollower path inside those handlers can be > slow (opening RocksDB, replaying logs, updating metadata cache), which > causes the worker thread to block. Because the worker pool is shared, all > concurrent RPC requests, including subsequent metadata updates, queue > behind it. Under leader-election storms (e.g., rolling restarts, failover), > this can stall the entire tablet server. > This PR should help with this: https://github.com/apache/fluss/pull/3647 > along with: https://github.com/apache/fluss/pull/3654 .. Until a new > release is cut, maybe increasing *rpc.worker.threads* helps with this > issue. > > 2. This is most likely a metadata propagation timing issue. When > auto-partition creates a new partition in ZooKeeper, there is a window > before the TabletServer's in-memory metadata cache receives the > UpdateMetadata from the Coordinator. If the client resolves the partition > and attempts to write during that window, the client's retry logic does not > always recover cleanly. This PR probably helps with this: > https://github.com/apache/fluss/pull/3511 . > Some other configuration options that might help: > - Increase *auto-partition.num-precreate* so partitions are pre-created > ahead of time, reducing the race window. > - Add retry logic in your producer application, for example, a transient > PartitionNotExistException during auto-partition creation should be > retried; the partition will eventually appear. > - Ensure your auto-partition *time-interval *is not too aggressive relative > to the time it takes for metadata to propagate. > > 3. When a TabletServer restarts abnormally, the Coordinator triggers leader > elections for all KV buckets that were on that node. Each > NotifyLeaderAndIsrRequest results in becomeLeaderOrFollower → RocksDB > open() call (which acquires the LOCK file, reads SST files, replays WAL). > This is inherently slow and happens on the RPC worker thread in 0.9.1, > serializing all subsequent RPCs. PR #3654 should probably help with this > issue as well, and PR https://github.com/apache/fluss/pull/3651 further > reduces shutdown time so restarts are faster overall. > > So I believe the fixes are most likely there and targeted for the upcoming > 1.0 release. So if your timeline allows, the safest path is to for the 1.0 > release. PRs #3647, #3654, #3511, and #3651 will all be included. > > The goal is to release within the next month. > > Let me know if the above helps. > > Best, > Giannis > > On Thu, Jul 23, 2026 at 3:22 PM zuo wei <[email protected]> wrote: > > > Dear Apache Fluss Community Developers, > > > > I’m a R&D engineer from Xiaomi. We’re currently testing *Apache Fluss > > 0.9.1* > > for our internal business scenarios, and ran into several severe unstable > > issues during our tests. I’d like to share these problems and ask for > your > > guidance. > > > > Here are the some main issues we encountered: > > *1. RPC stuck issue*: RPC requests (e.g. update metadata request) from > > Coordinator Server to Tablet Server often get stuck for a long time, > > causing service blocking. > > *2. Partition not exist error*: Data writing throws partition-not-exist > > exceptions even when the target partitions exist normally. > > *3. RocksDB lock blocking issue*: After abnormal restart of Tablet Server > > nodes, a lot of KV buckets leader switching occurs. The RocksDB open > > process holds locks for a long time, seriously blocking other worker > > threads. > > > > I'm not sure whether these problems are caused by our improper usage, > > configuration mistakes, or existing known bugs of Fluss 0.9.1. And I’d > like > > to consult the community whether Fluss 0.9.1 currently is recommended for > > production environment deployment? > > We really want to adopt Fluss in our business environment. Thanks a lot > for > > your help! Looking forward to your reply. > > > > Best regards, > > > > Zuo Wei > > Email: [email protected] > > >
