Thanks for the updates. Personally, I'd be in favor of not going out on a limb with unsupported metrics APIs. We should take care to make sure that what we add in KIP-471 is stable and well supported, even if it's not the complete picture. We can always do follow-on work to tackle complex metrics as an isolated design exercise.
Just my two cents. Thanks, -John On Wed, Jun 19, 2019 at 6:02 AM Bruno Cadonna <br...@confluent.io> wrote: > > Hi Guozhang, > > Regarding your comments about the wiki page: > > 1) Exactly, I rephrased the paragraph to make it more clear. > > 2) Yes, I used the wrong term. All hit related metrics are ratios. I > corrected the names of the affected metrics. > > Regarding your meta comments: > > 1) The plan is to expose the hit ratio. I used the wrong term. The > formulas compute ratios. Regarding your question about a metric to > know from where a read is served when it is not in the memtable, there > are metrics in RocksDB that give you the number of get() queries that > are served from L0, L1, and L2_AND_UP. I could not find any metric > that give you information about whether a query was served from disk > vs. OS cache. One metric that could be used to indirectly measure > whether disk or OS cache is accessed seems to be READ_BLOCK_GET_MICROS > that gives you the time for an IO read of a block. If it is high, it > was read from disk, otherwise from the OS cache. A similar strategy to > monitor the performance is described in [1]. DISCLAIMER: > READ_BLOCK_GET_MICROS is not documented. I had to look into the C++ > code to understand its meaning. I could have missed something. > > 2) There are some additional compaction statistics that contain sizes > of files on disk and numbers about write amplification that you can > get programmatically in RocksDB, but they are for debugging purposes > [2]. To get this data and publish it into a metric, one has to parse a > string. Since this data is for debugging purposes, I do not know how > stable the output format is. One thing, we could do, is to dump the > string with the compaction statistics into our log files at DEBUG > level. But that is outside of the scope of this KIP. > > Best, > Bruno > > [1] > https://github.com/facebook/rocksdb/wiki/Perf-Context-and-IO-Stats-Context#block-cache-and-os-page-cache-efficiency > [2] > https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide#rocksdb-statistics > > On Tue, Jun 18, 2019 at 8:24 PM Guozhang Wang <wangg...@gmail.com> wrote: > > > > Hello Bruno, > > > > I've read through the aggregation section and I think they look good to me. > > There are a few minor comments about the wiki page itself: > > > > 1) A state store might consist of multiple state stores -> You mean a > > `logical` state store be consistent of multiple `physical` store instances? > > > > 2) The "Hit Rates" calculation seems to be referring to the `Hit Ratio` > > (which is a percentage) than `Hit Rate`? > > > > And a couple further meta comments: > > > > 1) For memtable / block cache, instead of the hit-rate do you think we > > should expose the hit-ratio? I felt it is more useful for users to debug > > what's the root cause of unexpected slow performance. > > > > And for block cache misses, is it easy to provide a metric as of "target > > read" of where a read is served (from which level, either in OS cache or in > > SST files), similar to Fig.11 in > > http://cidrdb.org/cidr2017/papers/p82-dong-cidr17.pdf? > > > > 2) As @Patrik mentioned, is there a good way we can expose the total amount > > of memory and disk usage for each state store as well? I think it would > > also be very helpful for users to understand their capacity needs and read > > / write amplifications. > > > > > > Guozhang > > > > On Fri, Jun 14, 2019 at 6:55 AM Bruno Cadonna <br...@confluent.io> wrote: > > > > > Hi, > > > > > > I decided to go for the option in which metrics are exposed for each > > > logical state store. I revisited the KIP correspondingly and added a > > > section on how to aggregate metrics over multiple physical RocksDB > > > instances within one logical state store. Would be great, if you could > > > take a look and give feedback. If nobody has complaints about the > > > chosen option I would proceed with voting on this KIP since this was > > > the last open question. > > > > > > Best, > > > Bruno > > > > > > On Fri, Jun 7, 2019 at 9:38 PM Patrik Kleindl <pklei...@gmail.com> wrote: > > > > > > > > Hi Sophie > > > > This will be a good change, I have been thinking about proposing > > > something similar or even passing the properties per store. > > > > RocksDB should probably know how much memory was reserved but maybe does > > > not expose it. > > > > We are limiting it already as you suggested but this is a rather crude > > > tool. > > > > Especially in a larger topology with mixed loads par topic it would be > > > helpful to get more insights which store puts a lot of load on memory. > > > > Regarding the limiting capability, I think I remember reading that those > > > only affect some parts of the memory and others can still exceed this > > > limit. I‘ll try to look up the difference. > > > > Best regards > > > > Patrik > > > > > > > > > Am 07.06.2019 um 21:03 schrieb Sophie Blee-Goldman < > > > sop...@confluent.io>: > > > > > > > > > > Hi Patrik, > > > > > > > > > > As of 2.3 you will be able to use the RocksDBConfigSetter to > > > effectively > > > > > bound the total memory used by RocksDB for a single app instance. You > > > > > should already be able to limit the memory used per rocksdb store, > > > though > > > > > as you mention there can be a lot of them. I'm not sure you can > > > monitor the > > > > > memory usage if you are not limiting it though. > > > > > > > > > >> On Fri, Jun 7, 2019 at 2:06 AM Patrik Kleindl <pklei...@gmail.com> > > > wrote: > > > > >> > > > > >> Hi > > > > >> Thanks Bruno for the KIP, this is a very good idea. > > > > >> > > > > >> I have one question, are there metrics available for the memory > > > consumption > > > > >> of RocksDB? > > > > >> As they are running outside the JVM we have run into issues because > > > they > > > > >> were using all the other memory. > > > > >> And with multiple streams applications on the same machine, each with > > > > >> several KTables and 10+ partitions per topic the number of stores can > > > get > > > > >> out of hand pretty easily. > > > > >> Or did I miss something obvious how those can be monitored better? > > > > >> > > > > >> best regards > > > > >> > > > > >> Patrik > > > > >> > > > > >>> On Fri, 17 May 2019 at 23:54, Bruno Cadonna <br...@confluent.io> > > > wrote: > > > > >>> > > > > >>> Hi all, > > > > >>> > > > > >>> this KIP describes the extension of the Kafka Streams' metrics to > > > include > > > > >>> RocksDB's internal statistics. > > > > >>> > > > > >>> Please have a look at it and let me know what you think. Since I am > > > not a > > > > >>> RocksDB expert, I am thankful for any additional pair of eyes that > > > > >>> evaluates this KIP. > > > > >>> > > > > >>> > > > > >>> > > > > >> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-471:+Expose+RocksDB+Metrics+in+Kafka+Streams > > > > >>> > > > > >>> Best regards, > > > > >>> Bruno > > > > >>> > > > > >> > > > > > > > > > -- > > -- Guozhang
