Leverage multiple disks for kafka streams stores

Hi there, My kafka streams topology has one store that is particularly busy, that alongside other stores in the same topology is exhausting I/O which leads to write stalls and increased latency. The amount of compaction that this store does with regards to others is about 3/4 times more, so we we

Final reminder: ApacheCon North America call for presentations closing soon

[Note: You're receiving this because you are subscribed to one or more Apache Software Foundation project mailing lists.] This is your final reminder that the Call for Presetations for ApacheCon North America 2022 will close at 00:01 GMT on Monday, May 23rd, 2022. Please don't wait! Get your talk

Re: Leverage multiple disks for kafka streams stores

Hi Adrian, I am afraid that you cannot set the state directory for a single state store to a different directory than all other stores. Maybe the following blog post can help you debug and solve your issue: https://www.confluent.io/blog/how-to-tune-rocksdb-kafka-streams-state-stores-performan

Re: Leverage multiple disks for kafka streams stores

Hi Bruno, Thanks a lot for your answer. I have tried to tune store by store to the best of my ability, and indeed I have managed to improve considerably. We even changed the disk to a much faster one. But it's still not enough. Yes we can try dividing the application up into sub applications to

Re: Leverage multiple disks for kafka streams stores

Hi Adrian, Thank you for the additional information! One reason to have a single folder is that Streams also stores metadata that refers to all state stores in the state directory. That could be changed if we have a good reason. If you have a good idea to solve this issue, please feel free t

Connect task configuration limits?

Hi, Are there limits to the size of configuration data passed via the taskConfigs method of the Connector class? I'm observing a situation where if I use a large configuration no tasks are created, and no log messages appear in the connect log file. Using a smaller configuration works. If t

Re: [ANNOUNCE] Apache Kafka 3.2.0

Bruno, Congrats on the release! There is a small typo on the page. > KIP-791 > > adds method recordMetada() to the StateStoreContext, Should be > KIP-791 >