Hi Guozhang, Regarding your comments about the wiki page:
1) Exactly, I rephrased the paragraph to make it more clear. 2) Yes, I used the wrong term. All hit related metrics are ratios. I corrected the names of the affected metrics. Regarding your meta comments: 1) The plan is to expose the hit ratio. I used the wrong term. The formulas compute ratios. Regarding your question about a metric to know from where a read is served when it is not in the memtable, there are metrics in RocksDB that give you the number of get() queries that are served from L0, L1, and L2_AND_UP. I could not find any metric that give you information about whether a query was served from disk vs. OS cache. One metric that could be used to indirectly measure whether disk or OS cache is accessed seems to be READ_BLOCK_GET_MICROS that gives you the time for an IO read of a block. If it is high, it was read from disk, otherwise from the OS cache. A similar strategy to monitor the performance is described in [1]. DISCLAIMER: READ_BLOCK_GET_MICROS is not documented. I had to look into the C++ code to understand its meaning. I could have missed something. 2) There are some additional compaction statistics that contain sizes of files on disk and numbers about write amplification that you can get programmatically in RocksDB, but they are for debugging purposes [2]. To get this data and publish it into a metric, one has to parse a string. Since this data is for debugging purposes, I do not know how stable the output format is. One thing, we could do, is to dump the string with the compaction statistics into our log files at DEBUG level. But that is outside of the scope of this KIP. Best, Bruno [1] https://github.com/facebook/rocksdb/wiki/Perf-Context-and-IO-Stats-Context#block-cache-and-os-page-cache-efficiency [2] https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide#rocksdb-statistics On Tue, Jun 18, 2019 at 8:24 PM Guozhang Wang <wangg...@gmail.com> wrote: > > Hello Bruno, > > I've read through the aggregation section and I think they look good to me. > There are a few minor comments about the wiki page itself: > > 1) A state store might consist of multiple state stores -> You mean a > `logical` state store be consistent of multiple `physical` store instances? > > 2) The "Hit Rates" calculation seems to be referring to the `Hit Ratio` > (which is a percentage) than `Hit Rate`? > > And a couple further meta comments: > > 1) For memtable / block cache, instead of the hit-rate do you think we > should expose the hit-ratio? I felt it is more useful for users to debug > what's the root cause of unexpected slow performance. > > And for block cache misses, is it easy to provide a metric as of "target > read" of where a read is served (from which level, either in OS cache or in > SST files), similar to Fig.11 in > http://cidrdb.org/cidr2017/papers/p82-dong-cidr17.pdf? > > 2) As @Patrik mentioned, is there a good way we can expose the total amount > of memory and disk usage for each state store as well? I think it would > also be very helpful for users to understand their capacity needs and read > / write amplifications. > > > Guozhang > > On Fri, Jun 14, 2019 at 6:55 AM Bruno Cadonna <br...@confluent.io> wrote: > > > Hi, > > > > I decided to go for the option in which metrics are exposed for each > > logical state store. I revisited the KIP correspondingly and added a > > section on how to aggregate metrics over multiple physical RocksDB > > instances within one logical state store. Would be great, if you could > > take a look and give feedback. If nobody has complaints about the > > chosen option I would proceed with voting on this KIP since this was > > the last open question. > > > > Best, > > Bruno > > > > On Fri, Jun 7, 2019 at 9:38 PM Patrik Kleindl <pklei...@gmail.com> wrote: > > > > > > Hi Sophie > > > This will be a good change, I have been thinking about proposing > > something similar or even passing the properties per store. > > > RocksDB should probably know how much memory was reserved but maybe does > > not expose it. > > > We are limiting it already as you suggested but this is a rather crude > > tool. > > > Especially in a larger topology with mixed loads par topic it would be > > helpful to get more insights which store puts a lot of load on memory. > > > Regarding the limiting capability, I think I remember reading that those > > only affect some parts of the memory and others can still exceed this > > limit. I‘ll try to look up the difference. > > > Best regards > > > Patrik > > > > > > > Am 07.06.2019 um 21:03 schrieb Sophie Blee-Goldman < > > sop...@confluent.io>: > > > > > > > > Hi Patrik, > > > > > > > > As of 2.3 you will be able to use the RocksDBConfigSetter to > > effectively > > > > bound the total memory used by RocksDB for a single app instance. You > > > > should already be able to limit the memory used per rocksdb store, > > though > > > > as you mention there can be a lot of them. I'm not sure you can > > monitor the > > > > memory usage if you are not limiting it though. > > > > > > > >> On Fri, Jun 7, 2019 at 2:06 AM Patrik Kleindl <pklei...@gmail.com> > > wrote: > > > >> > > > >> Hi > > > >> Thanks Bruno for the KIP, this is a very good idea. > > > >> > > > >> I have one question, are there metrics available for the memory > > consumption > > > >> of RocksDB? > > > >> As they are running outside the JVM we have run into issues because > > they > > > >> were using all the other memory. > > > >> And with multiple streams applications on the same machine, each with > > > >> several KTables and 10+ partitions per topic the number of stores can > > get > > > >> out of hand pretty easily. > > > >> Or did I miss something obvious how those can be monitored better? > > > >> > > > >> best regards > > > >> > > > >> Patrik > > > >> > > > >>> On Fri, 17 May 2019 at 23:54, Bruno Cadonna <br...@confluent.io> > > wrote: > > > >>> > > > >>> Hi all, > > > >>> > > > >>> this KIP describes the extension of the Kafka Streams' metrics to > > include > > > >>> RocksDB's internal statistics. > > > >>> > > > >>> Please have a look at it and let me know what you think. Since I am > > not a > > > >>> RocksDB expert, I am thankful for any additional pair of eyes that > > > >>> evaluates this KIP. > > > >>> > > > >>> > > > >>> > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-471:+Expose+RocksDB+Metrics+in+Kafka+Streams > > > >>> > > > >>> Best regards, > > > >>> Bruno > > > >>> > > > >> > > > > > -- > -- Guozhang
