Hello, Igniters. I have implemented the profiling tool [1, 2]. It writes duration and other parameters of user operations (scan, SQL query, transactions, tasks, jobs, CQ, etc) to a local file. This info can be used in various cases. The main goal is to build the performance report to analyze the count and duration of user queries [3].
We already have the tracing with similar functionality but I think Ignite should have both tools - tracing and profiling. Because the tracing causes performance drop 52% [4] and can not be used for collecting statistics about all queries in production deployments. The performance drop of the profiling tool is less than 2% and it can be used in production. I have benchmarked the tracing and got the results: -2% when configured OpenCensusTracingSpi and all scopes disabled -52% for TX scope (IgnitePutTxBenchmark) -58% for SQL scope (IgniteSqlQueryBenchmark) Such a performance drop is significant to not use the tracing in production. I have considered the possibility to reuse the tracing API. If statistics collecting will be implemented with the TracingSpi then we get a performance drop due to: - Transferring tracing context over the network. - Using ThreadLocal for spans - Converting primitives and objects to string and vice versa. (API supports only String-based tags and values) - Generating span objects I have benchmarked implementations on the yardstick’s IgniteGetBenchmark. The tracing context transferring over the network was disabled. The results: - Tracing API implementation - 8% performance drop. - Proposed implementation - 2% performance drop. I think this is a significant drop and implementation with reuse tracing API should not be used. The cluster profiling should have as little performance drop as possible to be used in production. The tracing will be used for the detailed investigation. WDYT? The tool is ready to be reviewed [3, 5]. [1] https://issues.apache.org/jira/browse/IGNITE-12666 [2] https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool [3] https://github.com/apache/ignite-extensions/pull/16 [4] https://issues.apache.org/jira/secure/attachment/13016636/Tracing%20benchmarks.docx [5] https://github.com/apache/ignite/pull/7693 ср, 24 июн. 2020 г. в 23:31, Saikat Maitra <saikat.mai...@gmail.com>: > > Hi Nikita, > > The changes in this PR looks good. > > https://github.com/apache/ignite-extensions/pull/16 > > Regards, > Saikat > > On Mon, Jun 22, 2020 at 12:03 PM Nikolay Izhikov <nizhi...@apache.org> > wrote: > > > Hello, Igniters. > > > > I think that inside Ignite core we should name this feature as > > «performance statistics» > > We already have «cache statistics». > > Data that is collected by performance statistics can be used not only for > > profiling but to solve other tasks. > > > > > > > 22 июня 2020 г., в 14:00, Nikita Amelchev <nsamelc...@gmail.com> > > написал(а): > > > > > > Hi, guys. > > > > > > I have mentioned components under the MIT license in the LICENSE file. > > > > > > Saikat, I have fixed PR according to your suggestions. Thanks for taking > > a look. > > > > -- Best wishes, Amelchev Nikita
