Anton. 1. Value exported based on SPI settings, not in the moment it changed.
2. Clock synchronisation - if we export start time, we should also export node local timestamp. пн, 22 июля 2019 г., 8:33 Anton Vinogradov <a...@apache.org>: > Folks, > > What's the reason for duration counting? > AFAIU, it's a monitoring system feature to count the durations. > Sine monitoring system checks metrics periodically it will know the > duration by its own log. > > On Fri, Jul 19, 2019 at 7:32 PM Pavel Kovalenko <jokse...@gmail.com> > wrote: > > > Nikita, > > > > Yes, I mean duration not timestamp. For the metric name, I suggest > > "cacheOperationsBlockingDuration", I think it cleaner represents what is > > blocked during PME. > > We can also combine both timestamp "cacheOperationsBlockingStartTs" and > > duration to have better correlation when cache operations were blocked > and > > how much time it's taken. > > For instant view (like in JMX bean) a calculated value as you mentioned > > can be used. > > For metrics are exported to some backend (IEP-35) a counter can be used. > > The counter is incremented by blocking time after blocking has ended. > > > > пт, 19 июл. 2019 г. в 19:10, Nikita Amelchev <nsamelc...@gmail.com>: > > > >> Pavel, > >> > >> The main purpose of this metric is > >> >> how much time we wait for resuming cache operations > >> > >> Seems I misunderstood you. Do you mean timestamp or duration here? > >> >> What do you think if we change the boolean value of metric to a long > >> value that represents time in milliseconds when operations were blocked? > >> > >> This time can be calculated as (currentTime - > >> timeSinceOperationsBlocked) in case of timestamp. > >> > >> Duration will be more understandable. It'll be something like > >> getCurrentBlockingPmeDuration. But I haven't come up with a better > >> name yet. > >> > >> пт, 19 июл. 2019 г. в 18:30, Pavel Kovalenko <jokse...@gmail.com>: > >> > > >> > Nikita, > >> > > >> > I think getCurrentPmeDuration doesn't show useful information. The > main > >> PME side effect for end-users is blocking cache operations. Not all PME > >> time blocks it. > >> > What information gives to an end-user timestamp of > >> "timeSinceOperationsBlocked"? For what analysis it can be used and how? > >> > > >> > пт, 19 июл. 2019 г. в 17:48, Nikita Amelchev <nsamelc...@gmail.com>: > >> >> > >> >> Hi Pavel, > >> >> > >> >> This time already can be obtained from the getCurrentPmeDuration and > >> >> new isOperationsBlockedByPme metrics. > >> >> > >> >> As an alternative solution, I can rework recently added > >> >> getCurrentPmeDuration metric (not released yet). Seems for users it > >> >> useless in case of non-blocking PME. > >> >> Lets name it timeSinceOperationsBlocked. It'll be timestamp when > >> >> blocking started (minimal value of cluster nodes) and 0 if blocking > >> >> ends (there is no running PME). > >> >> > >> >> WDYT? > >> >> > >> >> пт, 19 июл. 2019 г. в 15:56, Pavel Kovalenko <jokse...@gmail.com>: > >> >> > > >> >> > Hi Nikita, > >> >> > > >> >> > Thank you for working on this. What do you think if we change the > >> boolean > >> >> > value of metric to a long value that represents time in > milliseconds > >> when > >> >> > operations were blocked? > >> >> > Since we have not only JMX and now metrics are periodically > exported > >> to > >> >> > some backend it can give a more clear picture of how much time we > >> wait for > >> >> > resuming cache operations instead of instant boolean indicator. > >> >> > > >> >> > пт, 19 июл. 2019 г. в 14:41, Nikita Amelchev <nsamelc...@gmail.com > >: > >> >> > > >> >> > > Anton, Nikolay, > >> >> > > > >> >> > > Thanks for the support. > >> >> > > > >> >> > > For now, we have the getCurrentPmeDuration() metric that does not > >> show > >> >> > > influence on the cluster correctly. PME can be without blocking > >> >> > > operations. For example, client node join/leave events. > >> >> > > > >> >> > > I suggest add new metric - isOperationsBlockedByPme(). Together, > >> these > >> >> > > metrics will show influence of the PME on cluster and user > >> operations. > >> >> > > > >> >> > > I have prepared PR for this (Bot visa is green). [1] Can anyone > >> take a > >> >> > > look? > >> >> > > > >> >> > > [1] https://issues.apache.org/jira/browse/IGNITE-11961 > >> >> > > > >> >> > > вт, 16 июл. 2019 г. в 14:58, Nikolay Izhikov < > nizhi...@apache.org > >> >: > >> >> > > > >> >> > > > > >> >> > > > I think administator of Ignite cluster should be able to > monitor > >> all > >> >> > > Ignite process, including non blocking PME. > >> >> > > > > >> >> > > > В Вт, 16/07/2019 в 14:57 +0300, Anton Vinogradov пишет: > >> >> > > > > BTW, > >> >> > > > > Found PME metric - getCurrentPmeDuration(). > >> >> > > > > Seems, it shows exactly PME time and not so useful because of > >> this. > >> >> > > > > The goal it so show exactly blocking period. > >> >> > > > > When PME cause no blocking, it's a good PME and I see no > >> reason to have > >> >> > > > > monitoring related to it :) > >> >> > > > > > >> >> > > > > On Tue, Jul 16, 2019 at 2:50 PM Nikolay Izhikov < > >> nizhi...@apache.org> > >> >> > > wrote: > >> >> > > > > > >> >> > > > > > Anton. > >> >> > > > > > > >> >> > > > > > Why do we need to postpone implementation of this metrics? > >> >> > > > > > For now, implementation of new metric is very simple. > >> >> > > > > > > >> >> > > > > > I think we can implement this metrics as a single > >> contribution. > >> >> > > > > > > >> >> > > > > > В Вт, 16/07/2019 в 13:47 +0300, Anton Vinogradov пишет: > >> >> > > > > > > Nikita, > >> >> > > > > > > > >> >> > > > > > > Looks like all we need now is a 1 simple metric: are > >> operations > >> >> > > blocked? > >> >> > > > > > > Just a true or false. > >> >> > > > > > > Lest start from this. > >> >> > > > > > > All other metrics can be extracted from logs now and can > be > >> >> > > implemented > >> >> > > > > > > later. > >> >> > > > > > > > >> >> > > > > > > On Tue, Jul 16, 2019 at 12:46 PM Nikolay Izhikov < > >> >> > > nizhi...@apache.org> > >> >> > > > > > > wrote: > >> >> > > > > > > > >> >> > > > > > > > +1. > >> >> > > > > > > > > >> >> > > > > > > > Nikita, please, go ahead. > >> >> > > > > > > > > >> >> > > > > > > > > >> >> > > > > > > > вт, 16 июля 2019 г., 11:45 Nikita Amelchev < > >> nsamelc...@gmail.com > >> >> > > >: > >> >> > > > > > > > > >> >> > > > > > > > > Hello, Igniters. > >> >> > > > > > > > > > >> >> > > > > > > > > I suggest to add some useful metrics about the > >> partition map > >> >> > > exchange > >> >> > > > > > > > > (PME). For now, the duration of PME stages available > >> only in > >> >> > > log > >> >> > > > > > > >> >> > > > > > files > >> >> > > > > > > > > and cannot be obtained using JMX or other external > >> tools. [1] > >> >> > > > > > > > > > >> >> > > > > > > > > I made the list of local node metrics that help to > >> understand > >> >> > > the > >> >> > > > > > > > > actual status of current PME: > >> >> > > > > > > > > > >> >> > > > > > > > > 1. initialVersion. Topology version that initiates > the > >> >> > > exchange. > >> >> > > > > > > > > 2. initTime. Time PME was started. > >> >> > > > > > > > > 3. initEvent. Event that triggered PME. > >> >> > > > > > > > > 4. partitionReleaseTime. Time when a node has > finished > >> waiting > >> >> > > for > >> >> > > > > > > >> >> > > > > > all > >> >> > > > > > > > > updates and translations on a previous topology. > >> >> > > > > > > > > 5. sendSingleMessageTime. Time when a node sent a > >> single > >> >> > > message. > >> >> > > > > > > > > 6. recieveFullMessageTime. Time when a node received > a > >> full > >> >> > > message. > >> >> > > > > > > > > 7. finishTime. Time PME was ended. > >> >> > > > > > > > > > >> >> > > > > > > > > When new PME started all these metrics resets. > >> >> > > > > > > > > > >> >> > > > > > > > > These metrics help to understand: > >> >> > > > > > > > > - how long PME was (current or previous). > >> >> > > > > > > > > - how long awaited for all updates was completed. > >> >> > > > > > > > > - what node blocks PME (didn't send a single message) > >> >> > > > > > > > > - what triggered PME. > >> >> > > > > > > > > > >> >> > > > > > > > > Thoughts? > >> >> > > > > > > > > > >> >> > > > > > > > > [1] > https://issues.apache.org/jira/browse/IGNITE-11961 > >> >> > > > > > > > > > >> >> > > > > > > > > -- > >> >> > > > > > > > > Best wishes, > >> >> > > > > > > > > Amelchev Nikita > >> >> > > > > > > > > > >> >> > > > >> >> > > > >> >> > > > >> >> > > -- > >> >> > > Best wishes, > >> >> > > Amelchev Nikita > >> >> > > > >> >> > >> >> > >> >> > >> >> -- > >> >> Best wishes, > >> >> Amelchev Nikita > >> > >> > >> > >> -- > >> Best wishes, > >> Amelchev Nikita > >> > > >
