Hi Sophie, Thank you for your feedback!
Please find my comments inline. On Mon, May 11, 2020 at 9:29 PM Sophie Blee-Goldman <sop...@confluent.io> wrote: > > Hey Bruno, > > Thanks for the KIP! I have one high-level concern, which is that we should > consider > reporting these metrics on the per-store level rather than instance-wide. I > know I was > the one who first proposed making it instance-wide, so bear with me: > > While I would still argue that the instance-wide memory usage is probably > the most *useful*, > exposing them at the store-level does not prevent users from monitoring the > instance-wide > memory. They should be able to roll up all the store-level metrics on an > instance to > compute the total off-heap memory. But rolling it up for the users does > prevent them from > using this to debug rare cases where one store may be using significantly > more memory than > expected. > Having the metrics on store-level makes sense to me. I also agree that users can make the roll-up themselves. > It's also worth considering that some users may be using the bounded memory > config setter > to put a cap on the off-heap memory of the entire process, in which case > the memory usage > metric for any one store should reflect the memory usage of the entire > instance. In that case > any effort to roll up the memory usages ourselves would just be wasted. > We would not roll-up the metrics by ourselves but use the utilities of RocksDB. I guess those utilities would do the roll-up correctly and without wasting anything also in the case of the bounded memory config setter. Of course we would need to verify those utilities. > Sorry for the reversal, but after a second thought I'm pretty strongly in > favor of reporting these > at the store level. > > Best, > Sophie > > On Wed, May 6, 2020 at 8:41 AM Bruno Cadonna <br...@confluent.io> wrote: > > > Hi all, > > > > I'd like to discuss KIP-607 that aims to add RocksDB memory usage > > metrics to Kafka Streams. > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-607%3A+Add+Metrics+to+Record+the+Memory+Used+by+RocksDB+to+Kafka+Streams > > > > Best, > > Bruno > >
