Re: High write operations rate on disk

2020-05-16 Thread Alexandre Dupriez
Hi Soumyajit, It is possible that due to the broker restart, you benefit from less I/O merges than under steady state. Intuitively, that would come from a shift from sequential workload with one more dispersed in nature. It is likely your broker generates more disk read than before the restart, es

Re: High write operations rate on disk

2020-05-16 Thread Ashutosh singh
I think this is expected behaviour. You will have to tune up IOPS accordingly. When you restart your brokers it tries to read metadata for all available topics and partitions (including all files) and that is where your read IOPS will shoot up. And if your cluster is busy one then it will try to

Re: data structures used by GlobalKTable, KTable

2020-05-16 Thread Pushkar Deole
Matthias, I would like to provide a suggestion here. Please check if this can be converted into a KIP. Since GlobalKTable holds complete topic data, and when the store underneath is in-memory store then the data in memory can quickly grow to a large value. I think it would be good if while using G

Re: data structures used by GlobalKTable, KTable

2020-05-16 Thread Pushkar Deole
If my thinking is correct then for some scenarios or use cases, the MRU for GlobalKTable might also work as a local store of data since the MRU will always store the data required by that application instance. On Sun, May 17, 2020 at 9:42 AM Pushkar Deole wrote: > Matthias, > > I would like to p