Re: Tool for performance statistics reports

Guys, I have filed the ticket [1] to improve the TracingSpi interface and further reusing it for cluster profiling. I'll assign it to me and take it to work after profiling merge if nobody minds. [1] https://issues.apache.org/jira/browse/IGNITE-13850 пн, 14 дек. 2020 г. в 13:38, Alexander Lapin

Re: Tool for performance statistics reports

Ok from my side. Few more details about tracing spi updates, based on mentioned above discussion with Nikolay and Nikita. Tracing provides enough data for a performance profiling tool, actually only root spans are required. However, according to Nikinta, root-span-tracing has a 7-8% performance dr

Re: Tool for performance statistics reports

Hello, Igniters. We discussed this feature privately with Alexander and Nikita. Here are the results we want to share with the community: 0. In the end, both, performance statistic tool and tracing should use the same API. 1. We should improve the Tracing API, so it able to be used for gathering

Re: Tool for performance statistics reports

Hello Igniters, Because the tracing causes performance drop 52% [4] and can not be > used for collecting statistics about all queries in production > deployments. The performance drop of the profiling tool is less than > 2% and it can be used in production. I have benchmarked the tracing > and got

Re: Tool for performance statistics reports

Hello, Nikita. Makes sense. I will take a look. > 7 дек. 2020 г., в 15:29, Nikita Amelchev написал(а): > > Hello, Igniters. > > I have implemented the profiling tool [1, 2]. It writes duration and > other parameters of user operations (scan, SQL query, transactions, > tasks, jobs, CQ, etc) to

Re: Tool for performance statistics reports

Hello, Igniters. I have implemented the profiling tool [1, 2]. It writes duration and other parameters of user operations (scan, SQL query, transactions, tasks, jobs, CQ, etc) to a local file. This info can be used in various cases. The main goal is to build the performance report to analyze the c

Re: Tool for performance statistics reports

Hi Nikita, The changes in this PR looks good. https://github.com/apache/ignite-extensions/pull/16 Regards, Saikat On Mon, Jun 22, 2020 at 12:03 PM Nikolay Izhikov wrote: > Hello, Igniters. > > I think that inside Ignite core we should name this feature as > «performance statistics» > We alrea

Re: Tool for performance statistics reports

Hello, Igniters. I think that inside Ignite core we should name this feature as «performance statistics» We already have «cache statistics». Data that is collected by performance statistics can be used not only for profiling but to solve other tasks. > 22 июня 2020 г., в 14:00, Nikita Amelchev

Re: Tool for performance statistics reports

Hi, guys. I have mentioned components under the MIT license in the LICENSE file. Saikat, I have fixed PR according to your suggestions. Thanks for taking a look.

Re: Tool for performance statistics reports

According to [1], it's absolutely OK to include files under MIT licence. But we should mention these components or modules in LICENCE file. [1] -- https://www.apache.org/legal/resolved.html вс, 21 июн. 2020 г. в 23:34, Saikat Maitra : > Hi Nikita, > > I have reviewed the PR and shared comments.

Re: Tool for performance statistics reports

Hi Nikita, I have reviewed the PR and shared comments. I also had a question on including files with a MIT license. Are we ok to use both Apache and MIT licenses in our source files? Regards, Saikat On Fri, Jun 12, 2020 at 8:45 PM Saikat Maitra wrote: > Hello Nikita, > > Thank you for sharing

Re: Tool for performance statistics reports

Hello Nikita, Thank you for sharing the information, it is very helpful. Regards, Saikat On Thu, Jun 11, 2020 at 3:20 AM Nikita Amelchev wrote: > Hello, > > > Can you please share more info on how we can use the profiling tool with > > ignite-extensions modules? > > I have updated the module r

Re: Tool for performance statistics reports

Hello, > Can you please share more info on how we can use the profiling tool with > ignite-extensions modules? I have updated the module readme file. You can find instructions there. For now, I am working on review fixes and marked PR as a draft. чт, 11 июн. 2020 г. в 03:21, Saikat Maitra : > >

Re: Tool for performance statistics reports

Hello Nikita, I observed we have an open PR in ignite repo for this feature with different set of changes compared to ignite extensions repo. apache/ignite#7693 https://github.com/apache/ignite-extensions/pull/16 Can you please share more info on ho

Re: Tool for performance statistics reports

Hello, Alexey. Thanks for the review. My understanding if the following: We will have 3 in-depth tool to find issues in cluster: 1. Metrics + System views - data that describe Ignite entities very high-level. 2. Profiling - tool to know what specific query of transactions are slow. In many ca

Re: Tool for performance statistics reports

Nikita, Igniters, I left a few comments on the tool itself in the PR. However, I would like to reiterate and discuss why a user would prefer to use the profiling tool over tracing? Profiling tool only captures very high-level details of the operations (a single cache operation, for example), and

Re: Tool for performance statistics reports

Hi, Igniters. I have implemented cluster profiling and tool to build the performance report. It's ready to be reviewed. [1, 2] Profiling can be managed by JMX bean. I have plans to implement it to control.sh also. Nodes write statistics to the temporary off heap buffer and then one thread flushe

Re: Tool for performance statistics reports

Hello Nikolay, > Who deprecated visor and when? Maybe I miss something? On the one hand, there was technically no community consensus that this tool should be obsolete. On the other hand, my opinion based on the following topic: http://apache-ignite-developers.2346864.n4.nabble.com/Re-Visor-plugin

Re: Tool for performance statistics reports

Hello, Slava, Ilya, Denis. Thanks for joining this discussion! > - visor (which is deprecated) Who deprecated visor and when? Maybe I miss something? > - web-console (to be honest, I don't quite understand the status of this tool) +1. > I am not against the new tool, I just want to understan

Re: Tool for performance statistics reports

> > Tracing is more deeply takes statistics. If it will be possible, I'm for > reuse. Looks like we need to sync up on these activities/initiatives to ensure we don't do a duplicate job. If you think a separate discussion is necessary let's kick it off. - Denis On Fri, Apr 24, 2020 at 9:18 AM

Re: Tool for performance statistics reports

Denis, Ilya, I will try to integrate profiling functionality into control.sh utility. > Speaking of the probes/statistics collection approach, is it supposed to > reuse tracing capabilities that are to be added as part of IEP-35? Tracing is more deeply takes statistics. If it will be possible, I'

Re: Tool for performance statistics reports

Hello! I suggest that it's one of the places where it could be put instead of adding a new tool. Regards, -- Ilya Kasnacheev пт, 24 апр. 2020 г. в 18:56, Nikita Amelchev : > Ilya, > > You suggest using control.sh to build the report? > > пт, 24 апр. 2020 г. в 18:20, Ilya Kasnacheev : > > > >

Re: Tool for performance statistics reports

Nikita, Thanks for preparing the IEP and breaking the initiative down into pieces. I studied the IEP and it feels like the goal is to built a tool similar to Oracle AWR. If it's so then shouldn't we add this as an extra function to either visorcmd or control.sh. Hopefully, we can start moving in t

Re: Tool for performance statistics reports

Ilya, You suggest using control.sh to build the report? пт, 24 апр. 2020 г. в 18:20, Ilya Kasnacheev : > > Hello! > > You need to throw in control.sh also, which does some kind of statistics > too, such as idle_verify. > > Regards, > -- > Ilya Kasnacheev > > > пт, 24 апр. 2020 г. в 18:06, Вячесла

Re: Tool for performance statistics reports

Slava, The web-console is used to interactive configuration wizard, management and monitoring tool [1]. I want to implement a profiling tool. It's one of the monitoring parts and helps to answer the question: What will be running slow? See IEP-35 [2]. [1] https://apacheignite-tools.readme.io/doc

Re: Tool for performance statistics reports

Maxim, > Can you provide more information about the binary logging format? I started to investigate. It should be with minimal effect on performance. I haven't any details for now. > What the additional log category are you talking about? PoC is based on logging performance statistics to the "pro

Re: Tool for performance statistics reports

Hello! You need to throw in control.sh also, which does some kind of statistics too, such as idle_verify. Regards, -- Ilya Kasnacheev пт, 24 апр. 2020 г. в 18:06, Вячеслав Коптилин : > Hello Nikita, > > Perhaps, I am missing something... > Apache Ignite already has a web-console tool. Do we w

Re: Tool for performance statistics reports

Hello Nikita, Perhaps, I am missing something... Apache Ignite already has a web-console tool. Do we want to improve the existing tool instead of creating a new one? It seems to me, this can be confusing for users. - visor (which is deprecated) - web-console (to be honest, I don't quite understa

Re: Tool for performance statistics reports

Hello, Nikita Nice to have such a tool in Apache Ignite. Can you provide more information about the binary logging format? Do you have any examples? Is there are any problems you faced? >The Ignite will log some additional internal statistics using separate log >category of IgniteLogger. What t

Tool for performance statistics reports

Hi, Igniters. I'm working on cluster profiling and the tool for creating a performance report. [1] I have prepared PoC based on performance logging to a separate category of Ignite log. The report contains: - Cache operations and its distribution by types [2] - Transactions and histogram of durat