Hi Zuo, Thank you for the detailed feedback. Results from real-world testing are very valuable to the Fluss community.
Fluss 0.9.1 was released nearly three months ago, so it does not include fixes for some issues discovered since then. Since you are currently testing Fluss, I recommend building the latest main branch and checking whether these issues can still be reproduced: https://fluss.apache.org/community/dev/building/ Testing the main branch now can help uncover remaining issues early and give the community time to address them before 1.0 is finalized. This will help ensure that the official 1.0 release can be reliably deployed in production environments such as Xiaomi’s. Thanks again for your support! Best regards, Jark On Fri, 24 Jul 2026 at 15:43, zuo wei <[email protected]> wrote: > > Hi Giannis, > > Thanks a lot for your clear and detailed reply! It helps me fully > understand > the root causes of the stability issues we met in Fluss 0.9.1. I will apply > your > recommended solutions on our 0.9.1 test cluster immediately. I'm looking > forward to the 1.0 release with the related PR fixes. > > I really appreciate the community’s efficient problem solving and technical > support! > During the follow-up testing and optimization process, if I encounter any > new problems or have relevant verification results, I will actively > synchronize > with the community in a timely manner.Thanks again for your great help! > > Best regards, > > Zuo Wei > Email: [email protected] > > > > Giannis Polyzos <[email protected]> 于2026年7月24日周五 13:52写道： > > > Hi Zuo, > > > > Thank you for the detailed report. Let me see if I can provide a breakdown > > here. > > > > 1. NotifyLeaderAndIsrRequest and UpdateMetadataRequest from the Coordinator > > are both handled directly on the Netty/RPC worker thread pool in > > TabletService. The becomeLeaderOrFollower path inside those handlers can be > > slow (opening RocksDB, replaying logs, updating metadata cache), which > > causes the worker thread to block. Because the worker pool is shared, all > > concurrent RPC requests, including subsequent metadata updates, queue > > behind it. Under leader-election storms (e.g., rolling restarts, failover), > > this can stall the entire tablet server. > > This PR should help with this: https://github.com/apache/fluss/pull/3647 > > along with: https://github.com/apache/fluss/pull/3654 .. Until a new > > release is cut, maybe increasing *rpc.worker.threads* helps with this > > issue. > > > > 2. This is most likely a metadata propagation timing issue. When > > auto-partition creates a new partition in ZooKeeper, there is a window > > before the TabletServer's in-memory metadata cache receives the > > UpdateMetadata from the Coordinator. If the client resolves the partition > > and attempts to write during that window, the client's retry logic does not > > always recover cleanly. This PR probably helps with this: > > https://github.com/apache/fluss/pull/3511 . > > Some other configuration options that might help: > > - Increase *auto-partition.num-precreate* so partitions are pre-created > > ahead of time, reducing the race window. > > - Add retry logic in your producer application, for example, a transient > > PartitionNotExistException during auto-partition creation should be > > retried; the partition will eventually appear. > > - Ensure your auto-partition *time-interval *is not too aggressive relative > > to the time it takes for metadata to propagate. > > > > 3. When a TabletServer restarts abnormally, the Coordinator triggers leader > > elections for all KV buckets that were on that node. Each > > NotifyLeaderAndIsrRequest results in becomeLeaderOrFollower → RocksDB > > open() call (which acquires the LOCK file, reads SST files, replays WAL). > > This is inherently slow and happens on the RPC worker thread in 0.9.1, > > serializing all subsequent RPCs. PR #3654 should probably help with this > > issue as well, and PR https://github.com/apache/fluss/pull/3651 further > > reduces shutdown time so restarts are faster overall. > > > > So I believe the fixes are most likely there and targeted for the upcoming > > 1.0 release. So if your timeline allows, the safest path is to for the 1.0 > > release. PRs #3647, #3654, #3511, and #3651 will all be included. > > > > The goal is to release within the next month. > > > > Let me know if the above helps. > > > > Best, > > Giannis > > > > On Thu, Jul 23, 2026 at 3:22 PM zuo wei <[email protected]> wrote: > > > > > Dear Apache Fluss Community Developers, > > > > > > I’m a R&D engineer from Xiaomi. We’re currently testing *Apache Fluss > > > 0.9.1* > > > for our internal business scenarios, and ran into several severe unstable > > > issues during our tests. I’d like to share these problems and ask for > > your > > > guidance. > > > > > > Here are the some main issues we encountered: > > > *1. RPC stuck issue*: RPC requests (e.g. update metadata request) from > > > Coordinator Server to Tablet Server often get stuck for a long time, > > > causing service blocking. > > > *2. Partition not exist error*: Data writing throws partition-not-exist > > > exceptions even when the target partitions exist normally. > > > *3. RocksDB lock blocking issue*: After abnormal restart of Tablet Server > > > nodes, a lot of KV buckets leader switching occurs. The RocksDB open > > > process holds locks for a long time, seriously blocking other worker > > > threads. > > > > > > I'm not sure whether these problems are caused by our improper usage, > > > configuration mistakes, or existing known bugs of Fluss 0.9.1. And I’d > > like > > > to consult the community whether Fluss 0.9.1 currently is recommended for > > > production environment deployment? > > > We really want to adopt Fluss in our business environment. Thanks a lot > > for > > > your help! Looking forward to your reply. > > > > > > Best regards, > > > > > > Zuo Wei > > > Email: [email protected] > > > > >
