Re: Cache operations performance metrics

[Andrey Gura]

> The goal of the proposed metrics is to measure whole cache operations 
> behavior.
> It provides some kind of statistics(histograms) for it.

Nikolay, reformulating doesn't make metrics more meaningful. Seriously :)

> Yes, metrics will evaluate API call performance

And what? Where the sense in this value? I explained why this metrics
are relatively useless.

> These are metrics of client-side operation performance.

Again. It's just a number without any sense.

> I think a specific user has knowledge - what are his transactions.

May be. But user can't distinguish one transaction from another, so
his knowledge doesn't make sense definitely.

> From these metrics it can answer on the question «If my transaction includes 
> cacheXXX, how long it usually takes?»

Actually not. The same caches can be involved  in a dozen of
transactions and there are no ways to understand what transactions are
slow or fast. It is useless.

> I disagree here.
> If you have a better approach to measure cache operations performance - 
> please, share your vision.

I already wrote about better approach. Two main points:

1. Measure some important internals (WAL operations, checkpoint time,
etc) because it can talk about real problems.
2. Measure business operations in user context, not cache API operations.

So  what we have? We have useless metrics that are doubled by useless
histograms.

We should reconsider approach to metrics and performance measuring. It
is hard and long task. There are no need to commit tons of useless
metrics that just decrease performance.

Sorry for some sarcasm but I really believe in my opinion. Metrics
problem exists very very long time and existing metrics discussed many
times. No one can explain this metrics to users because it requires
too many additional knowledge about internals. And metric  value
itself depends on many aspects of internals. It leads to impossibility
of interpretation. And it's good time to remove it (in AI 3.0 due to a
backward compatibility).

On Thu, Dec 19, 2019 at 9:09 PM Николай Ижиков  wrote:
>
> Hello, Andrey.
>
> The goal of the proposed metrics is to measure whole cache operations 
> behavior.
> It provides some kind of statistics(histograms) for it.
> For more fine-grained analysis one will be use tracing or other «go deeper» 
> tools.
>
> > > Measured for API calls on the caller node side
> > Values will the same only for cases when node is remote relative to data
>
> Yes, metrics will evaluate API call performance.
> I think this is the most valuable information from a user's point of view.
>
> Regular user wants to know how fast his cache operation performs.
> And these metrics provide the answer.
>
> > For regular data node (server node) timing will depend on answers for 
> > question:
>
> I think these answers are always available.
> I barely can imagine a scenario when one monitor «black box» cluster and 
> don’t know it.
> Even so, all answers are provided through system view we brought to the 
> Ignite :)
>
> > What is transaction commit or rollback time?
>
> These are metrics of client-side operation performance.
>
> I think a specific user has knowledge - what are his transactions.
> From these metrics it can answer on the question «If my transaction includes 
> cacheXXX, how long it usually takes?»
> I think it’s very valuable knowledge.
>
> > It will be implemented for most types of messages.
>
> Good, let’s do it?
>
> > So, from my point of view, commits for get/put/remove and commit/rollback 
> > should be reverted.
>
> I disagree here.
> If you have a better approach to measure cache operations performance - 
> please, share your vision.
>
> > 19 дек. 2019 г., в 16:03, Andrey Gura  написал(а):
> >
> > From my point of view, Ignite should provide meaningful metrics for
> > internal components that could be useful for monitoring and analysis.
> > All suggested options are meaningless in a sense. Below I'll try
> > explain why.
> >
> >> * `get`, `put`, `remove` time histograms. Measured for API calls on the 
> >> caller node side.
> >>   Implemented in [1], commit [2].
> >
> > All cache operations in Ignite are distributed. So each value measured
> > for some cache operation will vary depending on where actually
> > operation is performed. Values will the same only for cases when node
> > is remote relative to data (e.g. client node).
> >
> > For regular data node (server node) timing will depend on answers for 
> > question:
> >
> > - is node primary for particular key or not? (for all operations)
> > - how many backups configured for the cache? (for put and remove)
> > - what write synchronization mode is configured for particular cache?
> > (for put and remove)
> > - is readFromBackup enabled for the cache? (for get)
> >
> > Both Ignite users and Ignite developers can't make any decision based
> > on this metrics.
> >
> >> * `commit`, `rollback` time histograms. Measured for API calls on the 
> >> caller node side [3].
> >
> > What is transaction comm

[jira] [Created] (IGNITE-12475) TDE - Phase-2. CLI process management

[Amelchev Nikita (Jira)]

Amelchev Nikita created IGNITE-12475:


 Summary: TDE - Phase-2. CLI process management
 Key: IGNITE-12475
 URL: https://issues.apache.org/jira/browse/IGNITE-12475
 Project: Ignite
  Issue Type: Sub-task
Reporter: Amelchev Nikita
Assignee: Amelchev Nikita


Provide the ability to manage the process from CLI:
{noformat}
# Starts master key rotation.
control.sh --encryption change_master_key newMasterKeyName

# Displays cluster's current master key name.
control.sh --encryption get_master_key 
{noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Thin Clients: Renaming of AffinityAwareness to PartitionAwareness

[Igor Sapego]

+1, sounds reasonable to me.

Best Regards,
Igor


On Fri, Dec 20, 2019 at 10:25 AM Pavel Tupitsyn 
wrote:

> +1, let's rename while we have a chance before 2.8 is released.
>

Re: Cache operations performance metrics

[Николай Ижиков]

Hello, Andrey.

> Where the sense in this value? I explained why this metrics are relatively 
> useless.

I don’t agree with you.
I believe they are not useless for a user.
And I try to explain why I think so.

> But user can't distinguish one transaction from another, so his knowledge 
> doesn't make sense definitely.

Users shouldn’t distinguish.
If a cache has some percent of the relatively slow transaction this is a 
trigger to make a deeper investigation.

> 1. Measure some important internals (WAL operations, checkpoint time, etc) 
> because it can talk about real problems.

We already implement it.
What metrics are missing for internal processes?

> 2. Measure business operations in user context, not cache API operations.

Why do you think these approaches should exclude one another?
Users definitely should measure whole business transaction performance.

I think we should provide a way to measure part of the business transaction 
that relates to the Ignite.


> 20 дек. 2019 г., в 13:02, Andrey Gura  написал(а):
> 
>> The goal of the proposed metrics is to measure whole cache operations 
>> behavior.
>> It provides some kind of statistics(histograms) for it.
> 
> Nikolay, reformulating doesn't make metrics more meaningful. Seriously :)
> 
>> Yes, metrics will evaluate API call performance
> 
> And what? Where the sense in this value? I explained why this metrics
> are relatively useless.
> 
>> These are metrics of client-side operation performance.
> 
> Again. It's just a number without any sense.
> 
>> I think a specific user has knowledge - what are his transactions.
> 
> May be. But user can't distinguish one transaction from another, so
> his knowledge doesn't make sense definitely.
> 
>> From these metrics it can answer on the question «If my transaction includes 
>> cacheXXX, how long it usually takes?»
> 
> Actually not. The same caches can be involved  in a dozen of
> transactions and there are no ways to understand what transactions are
> slow or fast. It is useless.
> 
>> I disagree here.
>> If you have a better approach to measure cache operations performance - 
>> please, share your vision.
> 
> I already wrote about better approach. Two main points:
> 
> 1. Measure some important internals (WAL operations, checkpoint time,
> etc) because it can talk about real problems.
> 2. Measure business operations in user context, not cache API operations.
> 
> So  what we have? We have useless metrics that are doubled by useless
> histograms.
> 
> We should reconsider approach to metrics and performance measuring. It
> is hard and long task. There are no need to commit tons of useless
> metrics that just decrease performance.
> 
> Sorry for some sarcasm but I really believe in my opinion. Metrics
> problem exists very very long time and existing metrics discussed many
> times. No one can explain this metrics to users because it requires
> too many additional knowledge about internals. And metric  value
> itself depends on many aspects of internals. It leads to impossibility
> of interpretation. And it's good time to remove it (in AI 3.0 due to a
> backward compatibility).
> 
> On Thu, Dec 19, 2019 at 9:09 PM Николай Ижиков  wrote:
>> 
>> Hello, Andrey.
>> 
>> The goal of the proposed metrics is to measure whole cache operations 
>> behavior.
>> It provides some kind of statistics(histograms) for it.
>> For more fine-grained analysis one will be use tracing or other «go deeper» 
>> tools.
>> 
 Measured for API calls on the caller node side
>>> Values will the same only for cases when node is remote relative to data
>> 
>> Yes, metrics will evaluate API call performance.
>> I think this is the most valuable information from a user's point of view.
>> 
>> Regular user wants to know how fast his cache operation performs.
>> And these metrics provide the answer.
>> 
>>> For regular data node (server node) timing will depend on answers for 
>>> question:
>> 
>> I think these answers are always available.
>> I barely can imagine a scenario when one monitor «black box» cluster and 
>> don’t know it.
>> Even so, all answers are provided through system view we brought to the 
>> Ignite :)
>> 
>>> What is transaction commit or rollback time?
>> 
>> These are metrics of client-side operation performance.
>> 
>> I think a specific user has knowledge - what are his transactions.
>> From these metrics it can answer on the question «If my transaction includes 
>> cacheXXX, how long it usually takes?»
>> I think it’s very valuable knowledge.
>> 
>>> It will be implemented for most types of messages.
>> 
>> Good, let’s do it?
>> 
>>> So, from my point of view, commits for get/put/remove and commit/rollback 
>>> should be reverted.
>> 
>> I disagree here.
>> If you have a better approach to measure cache operations performance - 
>> please, share your vision.
>> 
>>> 19 дек. 2019 г., в 16:03, Andrey Gura  написал(а):
>>> 
>>> From my point of view, Ignite should provide meaningful metrics for
>>> internal compon

Re: Cache operations performance metrics

[Andrey Gura]

> If a cache has some percent of the relatively slow transaction this is a 
> trigger to make a deeper investigation.

It also will be visible on other metrics. So cache operations metrics
still useless because it transitive values.

>> 1. Measure some important internals (WAL operations, checkpoint time, etc) 
>> because it can talk about real problems.

> We already implement it.

I don't talk that it isn't implemented. It is just example of things
that should be measured. All other metrics depends on internals.

>> 2. Measure business operations in user context, not cache API operations.

>Why do you think these approaches should exclude one another?

Because one of them is useless.

On Fri, Dec 20, 2019 at 1:43 PM Николай Ижиков  wrote:
>
> Hello, Andrey.
>
> > Where the sense in this value? I explained why this metrics are relatively 
> > useless.
>
> I don’t agree with you.
> I believe they are not useless for a user.
> And I try to explain why I think so.
>
> > But user can't distinguish one transaction from another, so his knowledge 
> > doesn't make sense definitely.
>
> Users shouldn’t distinguish.
> If a cache has some percent of the relatively slow transaction this is a 
> trigger to make a deeper investigation.
>
> > 1. Measure some important internals (WAL operations, checkpoint time, etc) 
> > because it can talk about real problems.
>
> We already implement it.
> What metrics are missing for internal processes?
>
> > 2. Measure business operations in user context, not cache API operations.
>
> Why do you think these approaches should exclude one another?
> Users definitely should measure whole business transaction performance.
>
> I think we should provide a way to measure part of the business transaction 
> that relates to the Ignite.
>
>
> > 20 дек. 2019 г., в 13:02, Andrey Gura  написал(а):
> >
> >> The goal of the proposed metrics is to measure whole cache operations 
> >> behavior.
> >> It provides some kind of statistics(histograms) for it.
> >
> > Nikolay, reformulating doesn't make metrics more meaningful. Seriously :)
> >
> >> Yes, metrics will evaluate API call performance
> >
> > And what? Where the sense in this value? I explained why this metrics
> > are relatively useless.
> >
> >> These are metrics of client-side operation performance.
> >
> > Again. It's just a number without any sense.
> >
> >> I think a specific user has knowledge - what are his transactions.
> >
> > May be. But user can't distinguish one transaction from another, so
> > his knowledge doesn't make sense definitely.
> >
> >> From these metrics it can answer on the question «If my transaction 
> >> includes cacheXXX, how long it usually takes?»
> >
> > Actually not. The same caches can be involved  in a dozen of
> > transactions and there are no ways to understand what transactions are
> > slow or fast. It is useless.
> >
> >> I disagree here.
> >> If you have a better approach to measure cache operations performance - 
> >> please, share your vision.
> >
> > I already wrote about better approach. Two main points:
> >
> > 1. Measure some important internals (WAL operations, checkpoint time,
> > etc) because it can talk about real problems.
> > 2. Measure business operations in user context, not cache API operations.
> >
> > So  what we have? We have useless metrics that are doubled by useless
> > histograms.
> >
> > We should reconsider approach to metrics and performance measuring. It
> > is hard and long task. There are no need to commit tons of useless
> > metrics that just decrease performance.
> >
> > Sorry for some sarcasm but I really believe in my opinion. Metrics
> > problem exists very very long time and existing metrics discussed many
> > times. No one can explain this metrics to users because it requires
> > too many additional knowledge about internals. And metric  value
> > itself depends on many aspects of internals. It leads to impossibility
> > of interpretation. And it's good time to remove it (in AI 3.0 due to a
> > backward compatibility).
> >
> > On Thu, Dec 19, 2019 at 9:09 PM Николай Ижиков  
> > wrote:
> >>
> >> Hello, Andrey.
> >>
> >> The goal of the proposed metrics is to measure whole cache operations 
> >> behavior.
> >> It provides some kind of statistics(histograms) for it.
> >> For more fine-grained analysis one will be use tracing or other «go 
> >> deeper» tools.
> >>
>  Measured for API calls on the caller node side
> >>> Values will the same only for cases when node is remote relative to data
> >>
> >> Yes, metrics will evaluate API call performance.
> >> I think this is the most valuable information from a user's point of view.
> >>
> >> Regular user wants to know how fast his cache operation performs.
> >> And these metrics provide the answer.
> >>
> >>> For regular data node (server node) timing will depend on answers for 
> >>> question:
> >>
> >> I think these answers are always available.
> >> I barely can imagine a scenario when one monitor «black box» cluster a

Re: Cache operations performance metrics

[Николай Ижиков]

> It also will be visible on other metrics

How will it be visible?

For example, the user saw «checkpoint time» metric becomes x2 bigger.
How it relates to business operations? Is it become slower or faster?
What does it mean for an application performance?

On the other hand - if `PuTime` increased - then we know for sure, all 
operation executing `put` becomes slower.

*Why* it’s become slower - is the essence of «go deeper» investigation.

> 20 дек. 2019 г., в 15:07, Andrey Gura  написал(а):
> 
>> If a cache has some percent of the relatively slow transaction this is a 
>> trigger to make a deeper investigation.
> 
> It also will be visible on other metrics. So cache operations metrics
> still useless because it transitive values.
> 
>>> 1. Measure some important internals (WAL operations, checkpoint time, etc) 
>>> because it can talk about real problems.
> 
>> We already implement it.
> 
> I don't talk that it isn't implemented. It is just example of things
> that should be measured. All other metrics depends on internals.
> 
>>> 2. Measure business operations in user context, not cache API operations.
> 
>> Why do you think these approaches should exclude one another?
> 
> Because one of them is useless.
> 
> On Fri, Dec 20, 2019 at 1:43 PM Николай Ижиков  wrote:
>> 
>> Hello, Andrey.
>> 
>>> Where the sense in this value? I explained why this metrics are relatively 
>>> useless.
>> 
>> I don’t agree with you.
>> I believe they are not useless for a user.
>> And I try to explain why I think so.
>> 
>>> But user can't distinguish one transaction from another, so his knowledge 
>>> doesn't make sense definitely.
>> 
>> Users shouldn’t distinguish.
>> If a cache has some percent of the relatively slow transaction this is a 
>> trigger to make a deeper investigation.
>> 
>>> 1. Measure some important internals (WAL operations, checkpoint time, etc) 
>>> because it can talk about real problems.
>> 
>> We already implement it.
>> What metrics are missing for internal processes?
>> 
>>> 2. Measure business operations in user context, not cache API operations.
>> 
>> Why do you think these approaches should exclude one another?
>> Users definitely should measure whole business transaction performance.
>> 
>> I think we should provide a way to measure part of the business transaction 
>> that relates to the Ignite.
>> 
>> 
>>> 20 дек. 2019 г., в 13:02, Andrey Gura  написал(а):
>>> 
 The goal of the proposed metrics is to measure whole cache operations 
 behavior.
 It provides some kind of statistics(histograms) for it.
>>> 
>>> Nikolay, reformulating doesn't make metrics more meaningful. Seriously :)
>>> 
 Yes, metrics will evaluate API call performance
>>> 
>>> And what? Where the sense in this value? I explained why this metrics
>>> are relatively useless.
>>> 
 These are metrics of client-side operation performance.
>>> 
>>> Again. It's just a number without any sense.
>>> 
 I think a specific user has knowledge - what are his transactions.
>>> 
>>> May be. But user can't distinguish one transaction from another, so
>>> his knowledge doesn't make sense definitely.
>>> 
 From these metrics it can answer on the question «If my transaction 
 includes cacheXXX, how long it usually takes?»
>>> 
>>> Actually not. The same caches can be involved  in a dozen of
>>> transactions and there are no ways to understand what transactions are
>>> slow or fast. It is useless.
>>> 
 I disagree here.
 If you have a better approach to measure cache operations performance - 
 please, share your vision.
>>> 
>>> I already wrote about better approach. Two main points:
>>> 
>>> 1. Measure some important internals (WAL operations, checkpoint time,
>>> etc) because it can talk about real problems.
>>> 2. Measure business operations in user context, not cache API operations.
>>> 
>>> So  what we have? We have useless metrics that are doubled by useless
>>> histograms.
>>> 
>>> We should reconsider approach to metrics and performance measuring. It
>>> is hard and long task. There are no need to commit tons of useless
>>> metrics that just decrease performance.
>>> 
>>> Sorry for some sarcasm but I really believe in my opinion. Metrics
>>> problem exists very very long time and existing metrics discussed many
>>> times. No one can explain this metrics to users because it requires
>>> too many additional knowledge about internals. And metric  value
>>> itself depends on many aspects of internals. It leads to impossibility
>>> of interpretation. And it's good time to remove it (in AI 3.0 due to a
>>> backward compatibility).
>>> 
>>> On Thu, Dec 19, 2019 at 9:09 PM Николай Ижиков  
>>> wrote:
 
 Hello, Andrey.
 
 The goal of the proposed metrics is to measure whole cache operations 
 behavior.
 It provides some kind of statistics(histograms) for it.
 For more fine-grained analysis one will be use tracing or other «go 
 deeper» tools.
 
>> Measured for A

[jira] [Created] (IGNITE-12476) Slow test in Continuous Query 1 test suite

[Ivan Pavlukhin (Jira)]

Ivan Pavlukhin created IGNITE-12476:
---

 Summary: Slow test in Continuous Query 1 test suite
 Key: IGNITE-12476
 URL: https://issues.apache.org/jira/browse/IGNITE-12476
 Project: Ignite
  Issue Type: Task
Reporter: Ivan Pavlukhin


"Continuous Query 1" suite execution time can be optimized.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-12477) Slow test in Queries 1 test suite

[Ivan Pavlukhin (Jira)]

Ivan Pavlukhin created IGNITE-12477:
---

 Summary: Slow test in Queries 1 test suite
 Key: IGNITE-12477
 URL: https://issues.apache.org/jira/browse/IGNITE-12477
 Project: Ignite
  Issue Type: Task
Reporter: Ivan Pavlukhin


"Queries 1" suite execution time can be optimized.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Apache Ignite 2.8 RELEASE [Time, Scope, Manager]

[Anton Vinogradov]

Singe, no objections here -> feature merged to 2.8.

On Thu, Dec 19, 2019 at 9:10 PM Николай Ижиков  wrote:

> Good news, Anton!
>
> Thanks for your work on PME feature!
>
>
> > 19 дек. 2019 г., в 21:00, Anton Vinogradov  написал(а):
> >
> > Folks,
> > "2.8 + cherrypicked pme-free switch" vs "2.8" check finished, no blockers
> > found.
> >
> https://mtcga.gridgain.com/pr.html?serverId=apache&suiteId=IgniteTests24Java8_RunAll&branchForTc=pull%2F7165%2Fhead&action=Latest&baseBranchForTc=pull%2F7102%2Fhead
> > Any objections to merging?
> >
> > On Thu, Dec 19, 2019 at 4:20 PM Вячеслав Коптилин <
> slava.kopti...@gmail.com>
> > wrote:
> >
> >> Hello Pavel,
> >>
> >> Good to hear from you!
> >>
> >>> I guess we should have a system property to have ability to turn off
> PME
> >> free switch behavior if something goes wrong after release.
> >>> After feature battle testing we can remove it in the next release.
> >> Sounds good to me.
> >>
> >> Thanks,
> >> S.
> >>
> >> чт, 19 дек. 2019 г. в 15:59, Pavel Kovalenko :
> >>
> >>> Anton, Slava
> >>>
> >>> I guess we should have a system property to have ability to turn off
> PME
> >>> free switch behavior if something goes wrong after release.
> >>> After feature battle testing we can remove it in the next release.
> >>>
> >>> чт, 19 дек. 2019 г. в 15:26, Anton Vinogradov :
> >>>
>  Slava,
> 
> >> It would be nice to cut off a new branch from the release and run
> >> all
>  the
> >> tests with an integrated feature and after that,
> >> if you don’t see new failures and the release engineer agrees with
> >> the
> >> result, then and only then this feature can be merged into the
> >>> release.
>  I fully agree and see no other way to perform this.
>  PR already created and testing check scheduled.
> 
>  But, Maxim's proposal was to perform checks at the master branch
> 
> >> 1. Merge the issue to the master branch.
> >> 2. Wait for two-three weeks of running tests.
> >> 3. Check that there are not flaky failures and fix them all
> >> otherwise.
> >> 4. Cherry-pick commit to the ignite-2.8 branch.
> 
>  and that's the point of discussion.
> 
>  On Thu, Dec 19, 2019 at 3:14 PM Вячеслав Коптилин <
>  slava.kopti...@gmail.com>
>  wrote:
> 
> > Hi,
> >
> >> We're releasing release branch, not master... why we're checking
> >> the
> > "wrong" branch? :)
> >> Performing the release verification we're checking not only how
>  features
> > work, but also how they work together.
> > Yep, I agree that we should do verification for both branches of
> >> corse.
> >
> >> Finally, my concerns mostly related to integration checks and fail
> > hidings, not feature checks.
> >> That's why I propose to start integration checks asap.
> > IMHO, that is not the right way in order to handle such cases.
> > Cherry-picking new features into the release branch directly without
> >>> the
> > integration branch is not about STABILITY (this is normal and
> >>> acceptable
>  to
> > cherry-pick obvious/trivial fixes into the release branch, though).
> > It would be nice to cut off a new branch from the release and run all
> >>> the
> > tests with an integrated feature and after that,
> > if you don’t see new failures and the release engineer agrees with
> >> the
> > result, then and only then this feature can be merged into the
> >> release.
> >
> > This is my own view of the matter, and I do not insist that you
> >> should
>  act
> > in this way, but it seems to me that it would be safer and more
> >>> correct.
> >
> > Thanks,
> > S.
> >
> > чт, 19 дек. 2019 г. в 14:20, Anton Vinogradov :
> >
> >> Feature already tested at the feature branch properly.
> >> Question is about master -> release merge.
> >>
> >> So:
> >> 1) Testing at master does not equal to testing at release.
> >> Code may fail in the master while it works at the release branch
> >> and
>  vice
> >> versa.
> >>
> >> 2) Master is flakier that release branch, so we able to miss
> >> problem
> > hidden
> >> by other failures.
> >>
> >> Summarizing,
> >> We're releasing release branch, not master... why we're checking
> >> the
> >> "wrong" branch? :)
> >> Performing the release verification we're checking not only how
>  features
> >> work, but also how they work together.
> >> We may have different race condition windows at different branched.
> >> The bug may never happen at master, but happen each 50th check at
> >>> 2.8,
> > and
> >> in case we'll delay feature merge and check 2.8 only 49 times
> >> (while
> >>> we
> >> have 150+ successful checks at master) we gain false-negative case
> >>> for
> > 2.8.
> >>
> >> Finally, my concerns mostly related to integration checks and fail
> > hidings,
> >> not featu

Re: Cache operations performance metrics

[Andrey Gura]

> For example, the user saw «checkpoint time» metric becomes x2 bigger.

I just quote your words: " this is a trigger to make a deeper
investigation". And if we have two metrics that are triggered for the
same then one of them is useless.

> How it relates to business operations?

Why it should be related with business operation? It is concrete
metrics for concrete process which can slowdown all operations in the
system. Obviously if you want know how fast or slow your business
operation you must measure latency of your business operation. What
could be easier?

> Is it become slower or faster?

Very correct question! User saw "cache put time" metric becomes x2
bigger. Does it become slower or faster? Or we just put into the cache
values that 4x bigger in size? Or all time before we put values
locally and now we put values on remote nodes. Or our operations
execute in transaction and then time will depend on transaction type,
actions in transaction and other transaction and actually will nothing
talk about real cache operation. We have more questions then answers.

> On the other hand - if `PuTime` increased - then we know for sure, all 
> operation executing `put` becomes slower.

Of course not :) See above.

On Fri, Dec 20, 2019 at 3:20 PM Николай Ижиков  wrote:
>
> > It also will be visible on other metrics
>
> How will it be visible?
>
> For example, the user saw «checkpoint time» metric becomes x2 bigger.
> How it relates to business operations? Is it become slower or faster?
> What does it mean for an application performance?
>
> On the other hand - if `PuTime` increased - then we know for sure, all 
> operation executing `put` becomes slower.
>
> *Why* it’s become slower - is the essence of «go deeper» investigation.
>
> > 20 дек. 2019 г., в 15:07, Andrey Gura  написал(а):
> >
> >> If a cache has some percent of the relatively slow transaction this is a 
> >> trigger to make a deeper investigation.
> >
> > It also will be visible on other metrics. So cache operations metrics
> > still useless because it transitive values.
> >
> >>> 1. Measure some important internals (WAL operations, checkpoint time, 
> >>> etc) because it can talk about real problems.
> >
> >> We already implement it.
> >
> > I don't talk that it isn't implemented. It is just example of things
> > that should be measured. All other metrics depends on internals.
> >
> >>> 2. Measure business operations in user context, not cache API operations.
> >
> >> Why do you think these approaches should exclude one another?
> >
> > Because one of them is useless.
> >
> > On Fri, Dec 20, 2019 at 1:43 PM Николай Ижиков  wrote:
> >>
> >> Hello, Andrey.
> >>
> >>> Where the sense in this value? I explained why this metrics are 
> >>> relatively useless.
> >>
> >> I don’t agree with you.
> >> I believe they are not useless for a user.
> >> And I try to explain why I think so.
> >>
> >>> But user can't distinguish one transaction from another, so his knowledge 
> >>> doesn't make sense definitely.
> >>
> >> Users shouldn’t distinguish.
> >> If a cache has some percent of the relatively slow transaction this is a 
> >> trigger to make a deeper investigation.
> >>
> >>> 1. Measure some important internals (WAL operations, checkpoint time, 
> >>> etc) because it can talk about real problems.
> >>
> >> We already implement it.
> >> What metrics are missing for internal processes?
> >>
> >>> 2. Measure business operations in user context, not cache API operations.
> >>
> >> Why do you think these approaches should exclude one another?
> >> Users definitely should measure whole business transaction performance.
> >>
> >> I think we should provide a way to measure part of the business 
> >> transaction that relates to the Ignite.
> >>
> >>
> >>> 20 дек. 2019 г., в 13:02, Andrey Gura  написал(а):
> >>>
>  The goal of the proposed metrics is to measure whole cache operations 
>  behavior.
>  It provides some kind of statistics(histograms) for it.
> >>>
> >>> Nikolay, reformulating doesn't make metrics more meaningful. Seriously :)
> >>>
>  Yes, metrics will evaluate API call performance
> >>>
> >>> And what? Where the sense in this value? I explained why this metrics
> >>> are relatively useless.
> >>>
>  These are metrics of client-side operation performance.
> >>>
> >>> Again. It's just a number without any sense.
> >>>
>  I think a specific user has knowledge - what are his transactions.
> >>>
> >>> May be. But user can't distinguish one transaction from another, so
> >>> his knowledge doesn't make sense definitely.
> >>>
>  From these metrics it can answer on the question «If my transaction 
>  includes cacheXXX, how long it usually takes?»
> >>>
> >>> Actually not. The same caches can be involved  in a dozen of
> >>> transactions and there are no ways to understand what transactions are
> >>> slow or fast. It is useless.
> >>>
>  I disagree here.
>  If you have a better approach to measure cache operations perfor

Re: Apache Ignite 2.8 RELEASE [Time, Scope, Manager]

[Alexey Goncharuk]

Anton,

Shouldn't we target IGNITE-12470 to 2.8? It kind of does not make sense to
add an ability to disable potentially dangerous change only in the next
release.

чт, 19 дек. 2019 г. в 16:18, Anton Vinogradov :

> Pavel,
> Issue created - https://issues.apache.org/jira/browse/IGNITE-12470
>
> Slava,
> Does it mean we able to perform merge once I'll provide check results?
>
> On Thu, Dec 19, 2019 at 4:04 PM Вячеслав Коптилин <
> slava.kopti...@gmail.com>
> wrote:
>
> > Hi Anton,
> >
> > >> It would be nice to cut off a new branch from the release and run all
> > the
> > > >> tests with an integrated feature and after that,
> > > >> if you don’t see new failures and the release engineer agrees with
> the
> > > >> result, then and only then this feature can be merged into the
> > release.
> > > I fully agree and see no other way to perform this.
> > > PR already created and testing check scheduled.
> > >
> > This is good news, this means that we are on the same page! It was not so
> > clear that the integration branch has been created and will be used to
> > check all the failures.
> >
> > Thanks,
> > S.
> >
> > чт, 19 дек. 2019 г. в 15:26, Anton Vinogradov :
> >
> > > Slava,
> > >
> > > >> It would be nice to cut off a new branch from the release and run
> all
> > > the
> > > >> tests with an integrated feature and after that,
> > > >> if you don’t see new failures and the release engineer agrees with
> the
> > > >> result, then and only then this feature can be merged into the
> > release.
> > > I fully agree and see no other way to perform this.
> > > PR already created and testing check scheduled.
> > >
> > > But, Maxim's proposal was to perform checks at the master branch
> > >
> > > >> 1. Merge the issue to the master branch.
> > > >> 2. Wait for two-three weeks of running tests.
> > > >> 3. Check that there are not flaky failures and fix them all
> otherwise.
> > > >> 4. Cherry-pick commit to the ignite-2.8 branch.
> > >
> > > and that's the point of discussion.
> > >
> > > On Thu, Dec 19, 2019 at 3:14 PM Вячеслав Коптилин <
> > > slava.kopti...@gmail.com>
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > > We're releasing release branch, not master... why we're checking
> the
> > > > "wrong" branch? :)
> > > > > Performing the release verification we're checking not only how
> > > features
> > > > work, but also how they work together.
> > > > Yep, I agree that we should do verification for both branches of
> corse.
> > > >
> > > > > Finally, my concerns mostly related to integration checks and fail
> > > > hidings, not feature checks.
> > > > > That's why I propose to start integration checks asap.
> > > > IMHO, that is not the right way in order to handle such cases.
> > > > Cherry-picking new features into the release branch directly without
> > the
> > > > integration branch is not about STABILITY (this is normal and
> > acceptable
> > > to
> > > > cherry-pick obvious/trivial fixes into the release branch, though).
> > > > It would be nice to cut off a new branch from the release and run all
> > the
> > > > tests with an integrated feature and after that,
> > > > if you don’t see new failures and the release engineer agrees with
> the
> > > > result, then and only then this feature can be merged into the
> release.
> > > >
> > > > This is my own view of the matter, and I do not insist that you
> should
> > > act
> > > > in this way, but it seems to me that it would be safer and more
> > correct.
> > > >
> > > > Thanks,
> > > > S.
> > > >
> > > > чт, 19 дек. 2019 г. в 14:20, Anton Vinogradov :
> > > >
> > > > > Feature already tested at the feature branch properly.
> > > > > Question is about master -> release merge.
> > > > >
> > > > > So:
> > > > > 1) Testing at master does not equal to testing at release.
> > > > > Code may fail in the master while it works at the release branch
> and
> > > vice
> > > > > versa.
> > > > >
> > > > > 2) Master is flakier that release branch, so we able to miss
> problem
> > > > hidden
> > > > > by other failures.
> > > > >
> > > > > Summarizing,
> > > > > We're releasing release branch, not master... why we're checking
> the
> > > > > "wrong" branch? :)
> > > > > Performing the release verification we're checking not only how
> > > features
> > > > > work, but also how they work together.
> > > > > We may have different race condition windows at different branched.
> > > > > The bug may never happen at master, but happen each 50th check at
> > 2.8,
> > > > and
> > > > > in case we'll delay feature merge and check 2.8 only 49 times
> (while
> > we
> > > > > have 150+ successful checks at master) we gain false-negative case
> > for
> > > > 2.8.
> > > > >
> > > > > Finally, my concerns mostly related to integration checks and fail
> > > > hidings,
> > > > > not feature checks.
> > > > > That's why I propose to start integration checks asap.
> > > > >
> > > > > On Thu, Dec 19, 2019 at 1:21 PM Вячеслав Коптилин <
> > > > > slava.kopti...@gmail.com>
> > > > > wrote:
> > 

Re: Apache Ignite 2.8 RELEASE [Time, Scope, Manager]

[Maxim Muzafarov]

Folks,

The issue [1] has pinned to 2.8 and raised as the release blocker.

[1] https://issues.apache.org/jira/browse/IGNITE-12470

On Fri, 20 Dec 2019 at 16:04, Alexey Goncharuk
 wrote:
>
> Anton,
>
> Shouldn't we target IGNITE-12470 to 2.8? It kind of does not make sense to
> add an ability to disable potentially dangerous change only in the next
> release.
>
> чт, 19 дек. 2019 г. в 16:18, Anton Vinogradov :
>
> > Pavel,
> > Issue created - https://issues.apache.org/jira/browse/IGNITE-12470
> >
> > Slava,
> > Does it mean we able to perform merge once I'll provide check results?
> >
> > On Thu, Dec 19, 2019 at 4:04 PM Вячеслав Коптилин <
> > slava.kopti...@gmail.com>
> > wrote:
> >
> > > Hi Anton,
> > >
> > > >> It would be nice to cut off a new branch from the release and run all
> > > the
> > > > >> tests with an integrated feature and after that,
> > > > >> if you don’t see new failures and the release engineer agrees with
> > the
> > > > >> result, then and only then this feature can be merged into the
> > > release.
> > > > I fully agree and see no other way to perform this.
> > > > PR already created and testing check scheduled.
> > > >
> > > This is good news, this means that we are on the same page! It was not so
> > > clear that the integration branch has been created and will be used to
> > > check all the failures.
> > >
> > > Thanks,
> > > S.
> > >
> > > чт, 19 дек. 2019 г. в 15:26, Anton Vinogradov :
> > >
> > > > Slava,
> > > >
> > > > >> It would be nice to cut off a new branch from the release and run
> > all
> > > > the
> > > > >> tests with an integrated feature and after that,
> > > > >> if you don’t see new failures and the release engineer agrees with
> > the
> > > > >> result, then and only then this feature can be merged into the
> > > release.
> > > > I fully agree and see no other way to perform this.
> > > > PR already created and testing check scheduled.
> > > >
> > > > But, Maxim's proposal was to perform checks at the master branch
> > > >
> > > > >> 1. Merge the issue to the master branch.
> > > > >> 2. Wait for two-three weeks of running tests.
> > > > >> 3. Check that there are not flaky failures and fix them all
> > otherwise.
> > > > >> 4. Cherry-pick commit to the ignite-2.8 branch.
> > > >
> > > > and that's the point of discussion.
> > > >
> > > > On Thu, Dec 19, 2019 at 3:14 PM Вячеслав Коптилин <
> > > > slava.kopti...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > > We're releasing release branch, not master... why we're checking
> > the
> > > > > "wrong" branch? :)
> > > > > > Performing the release verification we're checking not only how
> > > > features
> > > > > work, but also how they work together.
> > > > > Yep, I agree that we should do verification for both branches of
> > corse.
> > > > >
> > > > > > Finally, my concerns mostly related to integration checks and fail
> > > > > hidings, not feature checks.
> > > > > > That's why I propose to start integration checks asap.
> > > > > IMHO, that is not the right way in order to handle such cases.
> > > > > Cherry-picking new features into the release branch directly without
> > > the
> > > > > integration branch is not about STABILITY (this is normal and
> > > acceptable
> > > > to
> > > > > cherry-pick obvious/trivial fixes into the release branch, though).
> > > > > It would be nice to cut off a new branch from the release and run all
> > > the
> > > > > tests with an integrated feature and after that,
> > > > > if you don’t see new failures and the release engineer agrees with
> > the
> > > > > result, then and only then this feature can be merged into the
> > release.
> > > > >
> > > > > This is my own view of the matter, and I do not insist that you
> > should
> > > > act
> > > > > in this way, but it seems to me that it would be safer and more
> > > correct.
> > > > >
> > > > > Thanks,
> > > > > S.
> > > > >
> > > > > чт, 19 дек. 2019 г. в 14:20, Anton Vinogradov :
> > > > >
> > > > > > Feature already tested at the feature branch properly.
> > > > > > Question is about master -> release merge.
> > > > > >
> > > > > > So:
> > > > > > 1) Testing at master does not equal to testing at release.
> > > > > > Code may fail in the master while it works at the release branch
> > and
> > > > vice
> > > > > > versa.
> > > > > >
> > > > > > 2) Master is flakier that release branch, so we able to miss
> > problem
> > > > > hidden
> > > > > > by other failures.
> > > > > >
> > > > > > Summarizing,
> > > > > > We're releasing release branch, not master... why we're checking
> > the
> > > > > > "wrong" branch? :)
> > > > > > Performing the release verification we're checking not only how
> > > > features
> > > > > > work, but also how they work together.
> > > > > > We may have different race condition windows at different branched.
> > > > > > The bug may never happen at master, but happen each 50th check at
> > > 2.8,
> > > > > and
> > > > > > in case we'll delay feature merge and check 2.8 only

Re: Cache operations performance metrics

[Николай Ижиков]

> And if we have two metrics that are triggered for the same then one of them 
> is useless.

I don't understand what is the two metrics you are talking about.
I wrote about a single metric for a single cache operation.

> Obviously if you want know how fast or slow your business operation you must 
> measure latency of your business operation. What could be easier?

A business transaction includes work with several data sources, sending network 
requests, executing some remote services.
If it becomes slower then we should know - what basic API operations become 
slower.
So we have to measure  `PutTime` from Ignite, `InsertTime` from RDBMS and other 
parts of a transaction.

Ignite will provide this kind of value out of the box.
I think it’s useful values.

> User saw "cache put time" metric becomes x2 bigger. Does it become slower or 
> faster? Or we just put into the cache values that 4x bigger in size?

Ignite cache operations obviously becomes 2 times slower. 
*Why* they become slower is the question of an ongoing investigation.

I tried to look at other open-source products.
Here is an example of metrics provided by Apache Kafka [1] [2]

`request-latency-avg` - The average request latency in ms.
`records-lag-max` - The maximum lag in terms of number of records for any 
partition in this window. An increasing value over time is your best indication 
that the consumer group is not keeping up with the producers.
`fetch-latency-avg` - The average time taken for a fetch request.

It seems, they implement a similar approach to what I proposed.


[1] https://docs.confluent.io/current/kafka/monitoring.html#producer-metrics
[2] https://docs.confluent.io/current/kafka/monitoring.html#new-consumer-metrics

> 20 дек. 2019 г., в 15:53, Andrey Gura  написал(а):
> 
>> For example, the user saw «checkpoint time» metric becomes x2 bigger.
> 
> I just quote your words: " this is a trigger to make a deeper
> investigation". And if we have two metrics that are triggered for the
> same then one of them is useless.
> 
>> How it relates to business operations?
> 
> Why it should be related with business operation? It is concrete
> metrics for concrete process which can slowdown all operations in the
> system. Obviously if you want know how fast or slow your business
> operation you must measure latency of your business operation. What
> could be easier?
> 
>> Is it become slower or faster?
> 
> Very correct question! User saw "cache put time" metric becomes x2
> bigger. Does it become slower or faster? Or we just put into the cache
> values that 4x bigger in size? Or all time before we put values
> locally and now we put values on remote nodes. Or our operations
> execute in transaction and then time will depend on transaction type,
> actions in transaction and other transaction and actually will nothing
> talk about real cache operation. We have more questions then answers.
> 
>> On the other hand - if `PuTime` increased - then we know for sure, all 
>> operation executing `put` becomes slower.
> 
> Of course not :) See above.
> 
> On Fri, Dec 20, 2019 at 3:20 PM Николай Ижиков  wrote:
>> 
>>> It also will be visible on other metrics
>> 
>> How will it be visible?
>> 
>> For example, the user saw «checkpoint time» metric becomes x2 bigger.
>> How it relates to business operations? Is it become slower or faster?
>> What does it mean for an application performance?
>> 
>> On the other hand - if `PuTime` increased - then we know for sure, all 
>> operation executing `put` becomes slower.
>> 
>> *Why* it’s become slower - is the essence of «go deeper» investigation.
>> 
>>> 20 дек. 2019 г., в 15:07, Andrey Gura  написал(а):
>>> 
 If a cache has some percent of the relatively slow transaction this is a 
 trigger to make a deeper investigation.
>>> 
>>> It also will be visible on other metrics. So cache operations metrics
>>> still useless because it transitive values.
>>> 
> 1. Measure some important internals (WAL operations, checkpoint time, 
> etc) because it can talk about real problems.
>>> 
 We already implement it.
>>> 
>>> I don't talk that it isn't implemented. It is just example of things
>>> that should be measured. All other metrics depends on internals.
>>> 
> 2. Measure business operations in user context, not cache API operations.
>>> 
 Why do you think these approaches should exclude one another?
>>> 
>>> Because one of them is useless.
>>> 
>>> On Fri, Dec 20, 2019 at 1:43 PM Николай Ижиков  wrote:
 
 Hello, Andrey.
 
> Where the sense in this value? I explained why this metrics are 
> relatively useless.
 
 I don’t agree with you.
 I believe they are not useless for a user.
 And I try to explain why I think so.
 
> But user can't distinguish one transaction from another, so his knowledge 
> doesn't make sense definitely.
 
 Users shouldn’t distinguish.
 If a cache has some percent of the relatively slow transaction this is 

[jira] [Created] (IGNITE-12478) Add long running operation timeout to MXBean for manipulate in run-time

[Kirill Tkalenko (Jira)]

Kirill Tkalenko created IGNITE-12478:


 Summary: Add long running operation timeout to MXBean for 
manipulate in run-time
 Key: IGNITE-12478
 URL: https://issues.apache.org/jira/browse/IGNITE-12478
 Project: Ignite
  Issue Type: Improvement
Reporter: Kirill Tkalenko
Assignee: Kirill Tkalenko
 Fix For: 2.8


Previously we can change the timeout only through property 
({{IGNITE_LONG_OPERATIONS_DUMP_TIMEOUT}} default 60_000), but in a real 
environment, we do not want to stop cluster (or node) for modify this property.
We should have a MBean's property, which allow to fluent in run-time behavior 
of printing LRT/LRF (like a property 
{{SqlQueryMXBeanImpl#setLongQueryWarningTimeout}} which printing long running 
queries).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-12479) All binary types are registered twice

[Denis Mekhanikov (Jira)]

Denis Mekhanikov created IGNITE-12479:
-

 Summary: All binary types are registered twice
 Key: IGNITE-12479
 URL: https://issues.apache.org/jira/browse/IGNITE-12479
 Project: Ignite
  Issue Type: Bug
  Components: binary
Reporter: Denis Mekhanikov
Assignee: Denis Mekhanikov
 Fix For: 2.8


When a POJO is put into a cache, its binary type is registered twice during 
marshalling.

Example:
{code:java}
public class MetadataRegistrationExample {
public static void main(String[] args) {
Ignite ignite = Ignition.start("config/ignite.xml");
Person p = new Person("Denis");
ignite.getOrCreateCache("cache").put(1, p);
}

static class Person {
private String name;
public Person(String name) {
this.name = name;
}
}
}
{code}
 

Here is the generated debug log from the package
{noformat}
[23:31:14,020][DEBUG][main][CacheObjectBinaryProcessorImpl] Requesting metadata 
update [typeId=-1210012928, 
typeName=binary.NestedObjectMarshallingExample$Person, changedSchemas=[], 
holder=null, fut=MetadataUpdateResultFuture [key=SyncKey [typeId=-1210012928, 
ver=0]]]
[23:31:14,023][DEBUG][disco-notifier-worker-#41][CacheObjectBinaryProcessorImpl]
 Received MetadataUpdateProposedListener [typeId=-1210012928, 
typeName=binary.NestedObjectMarshallingExample$Person, pendingVer=0, 
acceptedVer=0, schemasCnt=0]
[23:31:14,024][DEBUG][disco-notifier-worker-#41][CacheObjectBinaryProcessorImpl]
 Versions are stamped on coordinator [typeId=-1210012928, changedSchemas=[], 
pendingVer=1, acceptedVer=0]
[23:31:14,024][DEBUG][disco-notifier-worker-#41][CacheObjectBinaryProcessorImpl]
 Updated metadata on originating node: [typeId=-1210012928, pendingVer=1, 
acceptedVer=0]
[23:31:14,025][DEBUG][disco-notifier-worker-#41][CacheObjectBinaryProcessorImpl]
 Received MetadataUpdateAcceptedMessage MetadataUpdateAcceptedMessage 
[id=599e0a86c61-183a790b-7038-4dd5-b99d-89f1483e3635, typeId=-1210012928, 
acceptedVer=1, duplicated=false]
[23:31:14,025][DEBUG][disco-notifier-worker-#41][CacheObjectBinaryProcessorImpl]
 Completing future MetadataUpdateResultFuture [key=SyncKey [typeId=-1210012928, 
ver=1]] for [typeId=-1210012928, pendingVer=1, acceptedVer=1]
[23:31:14,026][DEBUG][main][CacheObjectBinaryProcessorImpl] Completed metadata 
update [typeId=-1210012928, 
typeName=binary.NestedObjectMarshallingExample$Person, waitTime=4ms, 
fut=MetadataUpdateResultFuture [key=SyncKey [typeId=-1210012928, ver=1]], 
tx=null]
[23:31:14,027][DEBUG][main][CacheObjectBinaryProcessorImpl] Requesting metadata 
update [typeId=-1210012928, 
typeName=binary.NestedObjectMarshallingExample$Person, 
changedSchemas=[1975878747], holder=[typeId=-1210012928, pendingVer=1, 
acceptedVer=1], fut=MetadataUpdateResultFuture [key=SyncKey 
[typeId=-1210012928, ver=0]]]
[23:31:14,027][DEBUG][disco-notifier-worker-#41][CacheObjectBinaryProcessorImpl]
 Received MetadataUpdateProposedListener [typeId=-1210012928, 
typeName=binary.NestedObjectMarshallingExample$Person, pendingVer=0, 
acceptedVer=0, schemasCnt=1]
[23:31:14,028][DEBUG][disco-notifier-worker-#41][CacheObjectBinaryProcessorImpl]
 Versions are stamped on coordinator [typeId=-1210012928, 
changedSchemas=[1975878747], pendingVer=2, acceptedVer=1]
[23:31:14,028][DEBUG][disco-notifier-worker-#41][CacheObjectBinaryProcessorImpl]
 Updated metadata on originating node: [typeId=-1210012928, pendingVer=2, 
acceptedVer=1]
[23:31:14,028][DEBUG][disco-notifier-worker-#41][CacheObjectBinaryProcessorImpl]
 Received MetadataUpdateAcceptedMessage MetadataUpdateAcceptedMessage 
[id=d99e0a86c61-183a790b-7038-4dd5-b99d-89f1483e3635, typeId=-1210012928, 
acceptedVer=2, duplicated=false]
[23:31:14,028][DEBUG][disco-notifier-worker-#41][CacheObjectBinaryProcessorImpl]
 Completing future MetadataUpdateResultFuture [key=SyncKey [typeId=-1210012928, 
ver=2]] for [typeId=-1210012928, pendingVer=2, acceptedVer=2]
[23:31:14,029][DEBUG][main][CacheObjectBinaryProcessorImpl] Completed metadata 
update [typeId=-1210012928, 
typeName=binary.NestedObjectMarshallingExample$Person, waitTime=1ms, 
fut=MetadataUpdateResultFuture [key=SyncKey [typeId=-1210012928, ver=2]], 
tx=null]
{noformat}

You can see, that a type is registered twice. First it's registered without any 
fields, and only the second time the type is registered properly.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re[2]: Apache Ignite 2.8 RELEASE [Time, Scope, Manager]

[Zhenya Stanilovsky]

Folks,

The issue [1] already in master, submit minor fix with performance enhancements.
i think it`s not risky to gather it too.
thanks!

[1]  https://issues.apache.org/jira/browse/IGNITE-12442
  
>Пятница, 20 декабря 2019, 16:09 +03:00 от Maxim Muzafarov :
> 
>Folks,
>
>The issue [1] has pinned to 2.8 and raised as the release blocker.
>
>[1]  https://issues.apache.org/jira/browse/IGNITE-12470
>
>On Fri, 20 Dec 2019 at 16:04, Alexey Goncharuk
>< alexey.goncha...@gmail.com > wrote:
>>
>> Anton,
>>
>> Shouldn't we target IGNITE-12470 to 2.8? It kind of does not make sense to
>> add an ability to disable potentially dangerous change only in the next
>> release.
>>
>> чт, 19 дек. 2019 г. в 16:18, Anton Vinogradov < a...@apache.org >:
>>
>> > Pavel,
>> > Issue created -  https://issues.apache.org/jira/browse/IGNITE-12470
>> >
>> > Slava,
>> > Does it mean we able to perform merge once I'll provide check results?
>> >
>> > On Thu, Dec 19, 2019 at 4:04 PM Вячеслав Коптилин <
>> >  slava.kopti...@gmail.com >
>> > wrote:
>> >
>> > > Hi Anton,
>> > >
>> > > >> It would be nice to cut off a new branch from the release and run all
>> > > the
>> > > > >> tests with an integrated feature and after that,
>> > > > >> if you don’t see new failures and the release engineer agrees with
>> > the
>> > > > >> result, then and only then this feature can be merged into the
>> > > release.
>> > > > I fully agree and see no other way to perform this.
>> > > > PR already created and testing check scheduled.
>> > > >
>> > > This is good news, this means that we are on the same page! It was not so
>> > > clear that the integration branch has been created and will be used to
>> > > check all the failures.
>> > >
>> > > Thanks,
>> > > S.
>> > >
>> > > чт, 19 дек. 2019 г. в 15:26, Anton Vinogradov < a...@apache.org >:
>> > >
>> > > > Slava,
>> > > >
>> > > > >> It would be nice to cut off a new branch from the release and run
>> > all
>> > > > the
>> > > > >> tests with an integrated feature and after that,
>> > > > >> if you don’t see new failures and the release engineer agrees with
>> > the
>> > > > >> result, then and only then this feature can be merged into the
>> > > release.
>> > > > I fully agree and see no other way to perform this.
>> > > > PR already created and testing check scheduled.
>> > > >
>> > > > But, Maxim's proposal was to perform checks at the master branch
>> > > >
>> > > > >> 1. Merge the issue to the master branch.
>> > > > >> 2. Wait for two-three weeks of running tests.
>> > > > >> 3. Check that there are not flaky failures and fix them all
>> > otherwise.
>> > > > >> 4. Cherry-pick commit to the ignite-2.8 branch.
>> > > >
>> > > > and that's the point of discussion.
>> > > >
>> > > > On Thu, Dec 19, 2019 at 3:14 PM Вячеслав Коптилин <
>> > > >  slava.kopti...@gmail.com >
>> > > > wrote:
>> > > >
>> > > > > Hi,
>> > > > >
>> > > > > > We're releasing release branch, not master... why we're checking
>> > the
>> > > > > "wrong" branch? :)
>> > > > > > Performing the release verification we're checking not only how
>> > > > features
>> > > > > work, but also how they work together.
>> > > > > Yep, I agree that we should do verification for both branches of
>> > corse.
>> > > > >
>> > > > > > Finally, my concerns mostly related to integration checks and fail
>> > > > > hidings, not feature checks.
>> > > > > > That's why I propose to start integration checks asap.
>> > > > > IMHO, that is not the right way in order to handle such cases.
>> > > > > Cherry-picking new features into the release branch directly without
>> > > the
>> > > > > integration branch is not about STABILITY (this is normal and
>> > > acceptable
>> > > > to
>> > > > > cherry-pick obvious/trivial fixes into the release branch, though).
>> > > > > It would be nice to cut off a new branch from the release and run all
>> > > the
>> > > > > tests with an integrated feature and after that,
>> > > > > if you don’t see new failures and the release engineer agrees with
>> > the
>> > > > > result, then and only then this feature can be merged into the
>> > release.
>> > > > >
>> > > > > This is my own view of the matter, and I do not insist that you
>> > should
>> > > > act
>> > > > > in this way, but it seems to me that it would be safer and more
>> > > correct.
>> > > > >
>> > > > > Thanks,
>> > > > > S.
>> > > > >
>> > > > > чт, 19 дек. 2019 г. в 14:20, Anton Vinogradov < a...@apache.org >:
>> > > > >
>> > > > > > Feature already tested at the feature branch properly.
>> > > > > > Question is about master -> release merge.
>> > > > > >
>> > > > > > So:
>> > > > > > 1) Testing at master does not equal to testing at release.
>> > > > > > Code may fail in the master while it works at the release branch
>> > and
>> > > > vice
>> > > > > > versa.
>> > > > > >
>> > > > > > 2) Master is flakier that release branch, so we able to miss
>> > problem
>> > > > > hidden
>> > > > > > by other failures.
>> > > > > >
>> > > > > > Summarizing,
>> > > > > > We're r

Re[2]: Cache operations performance metrics

[Zhenya Stanilovsky]

>> Is it become slower or faster?
>
>Very correct question! User saw "cache put time" metric becomes x2
>bigger. Does it become slower or faster? Or we just put into the cache
>values that 4x bigger in size? Or all time before we put values
>locally and now we put values on remote nodes. Or our operations
>execute in transaction and then time will depend on transaction type,
>actions in transaction and other transaction and actually will nothing
>talk about real cache operation. We have more questions then answers.
 
Andrey, i hope i understand your point of view here, but between to have 
something and have nothing i choose — something, it sometimes really helpful. 
From real life case: i found 1 grid machine with very different io usage than 
others, «dig deeper» highlight cache with very different from other nodes cache 
put operations and final «dig deeper» help to found code bug, but to be clear — 
old mechanism works ok for me here, if new one would be more useful — why not ?
 
>> On the other hand - if `PuTime` increased - then we know for sure, all 
>> operation executing `put` becomes slower.
>
>Of course not :) See above.
>
>On Fri, Dec 20, 2019 at 3:20 PM Николай Ижиков < nizhi...@apache.org > wrote:
>>
>> > It also will be visible on other metrics
>>
>> How will it be visible?
>>
>> For example, the user saw «checkpoint time» metric becomes x2 bigger.
>> How it relates to business operations? Is it become slower or faster?
>> What does it mean for an application performance?
>>
>> On the other hand - if `PuTime` increased - then we know for sure, all 
>> operation executing `put` becomes slower.
>>
>> *Why* it’s become slower - is the essence of «go deeper» investigation.
>>
>> > 20 дек. 2019 г., в 15:07, Andrey Gura < ag...@apache.org > написал(а):
>> >
>> >> If a cache has some percent of the relatively slow transaction this is a 
>> >> trigger to make a deeper investigation.
>> >
>> > It also will be visible on other metrics. So cache operations metrics
>> > still useless because it transitive values.
>> >
>> >>> 1. Measure some important internals (WAL operations, checkpoint time, 
>> >>> etc) because it can talk about real problems.
>> >
>> >> We already implement it.
>> >
>> > I don't talk that it isn't implemented. It is just example of things
>> > that should be measured. All other metrics depends on internals.
>> >
>> >>> 2. Measure business operations in user context, not cache API operations.
>> >
>> >> Why do you think these approaches should exclude one another?
>> >
>> > Because one of them is useless.
>> >
>> > On Fri, Dec 20, 2019 at 1:43 PM Николай Ижиков < nizhi...@apache.org > 
>> > wrote:
>> >>
>> >> Hello, Andrey.
>> >>
>> >>> Where the sense in this value? I explained why this metrics are 
>> >>> relatively useless.
>> >>
>> >> I don’t agree with you.
>> >> I believe they are not useless for a user.
>> >> And I try to explain why I think so.
>> >>
>> >>> But user can't distinguish one transaction from another, so his 
>> >>> knowledge doesn't make sense definitely.
>> >>
>> >> Users shouldn’t distinguish.
>> >> If a cache has some percent of the relatively slow transaction this is a 
>> >> trigger to make a deeper investigation.
>> >>
>> >>> 1. Measure some important internals (WAL operations, checkpoint time, 
>> >>> etc) because it can talk about real problems.
>> >>
>> >> We already implement it.
>> >> What metrics are missing for internal processes?
>> >>
>> >>> 2. Measure business operations in user context, not cache API operations.
>> >>
>> >> Why do you think these approaches should exclude one another?
>> >> Users definitely should measure whole business transaction performance.
>> >>
>> >> I think we should provide a way to measure part of the business 
>> >> transaction that relates to the Ignite.
>> >>
>> >>
>> >>> 20 дек. 2019 г., в 13:02, Andrey Gura < ag...@apache.org > написал(а):
>> >>>
>>  The goal of the proposed metrics is to measure whole cache operations 
>>  behavior.
>>  It provides some kind of statistics(histograms) for it.
>> >>>
>> >>> Nikolay, reformulating doesn't make metrics more meaningful. Seriously :)
>> >>>
>>  Yes, metrics will evaluate API call performance
>> >>>
>> >>> And what? Where the sense in this value? I explained why this metrics
>> >>> are relatively useless.
>> >>>
>>  These are metrics of client-side operation performance.
>> >>>
>> >>> Again. It's just a number without any sense.
>> >>>
>>  I think a specific user has knowledge - what are his transactions.
>> >>>
>> >>> May be. But user can't distinguish one transaction from another, so
>> >>> his knowledge doesn't make sense definitely.
>> >>>
>>  From these metrics it can answer on the question «If my transaction 
>>  includes cacheXXX, how long it usually takes?»
>> >>>
>> >>> Actually not. The same caches can be involved in a dozen of
>> >>> transactions and there are no ways to understand what transactions are
>> >>> slow or fast. It i

Re: Cache operations performance metrics

[Andrey Gura]

>> And if we have two metrics that are triggered for the same then one of them 
>> is useless.
> I don't understand what is the two metrics you are talking about.

Please, don't loose context. In your example it was checkpoint time
and some cache operation time.

> A business transaction includes work with several data sources, sending 
> network requests, executing some remote services.
> If it becomes slower then we should know - what basic API operations become 
> slower.

No, we should now what basic operations become slower. It is problem
with network (net io), with disk (disk io), JVM (VM internal metrics),
etc. All this operations are bricks of API operations.

> So we have to measure  `PutTime` from Ignite, `InsertTime` from RDBMS and 
> other parts of a transaction.

We can't do it properly due to a transactions implementation specific.
I already wrote about it. It doesn't mean that we must not fix it. But
it also means that we should reconsider approach to metrics in Ignite.
Bigger doesn't mean better.

> Ignite cache operations obviously becomes 2 times slower.
> *Why* they become slower is the question of an ongoing investigation.

But business operations metrics will show the same. And many other
internals related metrics will show the same. It is transitive,
redundant and relatively useless metric if it doesn't bring something
new in information. 500 caches with similar configurations (the same
nodes, the same data region, the same affinity, etc) and 500 metrics
like put time will show the same trend.And the same trend will show a
couple of system internal metrics. A couple vs hundreds. Doesn't make
sense and useless.

> I tried to look at other open-source products.
> Here is an example of metrics provided by Apache Kafka [1] [2]

If somebody do something it doesn't mean that they do it properly.

On Fri, Dec 20, 2019 at 4:28 PM Николай Ижиков  wrote:
>
> > And if we have two metrics that are triggered for the same then one of them 
> > is useless.
>
> I don't understand what is the two metrics you are talking about.
> I wrote about a single metric for a single cache operation.
>
> > Obviously if you want know how fast or slow your business operation you 
> > must measure latency of your business operation. What could be easier?
>
> A business transaction includes work with several data sources, sending 
> network requests, executing some remote services.
> If it becomes slower then we should know - what basic API operations become 
> slower.
> So we have to measure  `PutTime` from Ignite, `InsertTime` from RDBMS and 
> other parts of a transaction.
>
> Ignite will provide this kind of value out of the box.
> I think it’s useful values.
>
> > User saw "cache put time" metric becomes x2 bigger. Does it become slower 
> > or faster? Or we just put into the cache values that 4x bigger in size?
>
> Ignite cache operations obviously becomes 2 times slower.
> *Why* they become slower is the question of an ongoing investigation.
>
> I tried to look at other open-source products.
> Here is an example of metrics provided by Apache Kafka [1] [2]
>
> `request-latency-avg` - The average request latency in ms.
> `records-lag-max` - The maximum lag in terms of number of records for any 
> partition in this window. An increasing value over time is your best 
> indication that the consumer group is not keeping up with the producers.
> `fetch-latency-avg` - The average time taken for a fetch request.
>
> It seems, they implement a similar approach to what I proposed.
>
>
> [1] https://docs.confluent.io/current/kafka/monitoring.html#producer-metrics
> [2] 
> https://docs.confluent.io/current/kafka/monitoring.html#new-consumer-metrics
>
> > 20 дек. 2019 г., в 15:53, Andrey Gura  написал(а):
> >
> >> For example, the user saw «checkpoint time» metric becomes x2 bigger.
> >
> > I just quote your words: " this is a trigger to make a deeper
> > investigation". And if we have two metrics that are triggered for the
> > same then one of them is useless.
> >
> >> How it relates to business operations?
> >
> > Why it should be related with business operation? It is concrete
> > metrics for concrete process which can slowdown all operations in the
> > system. Obviously if you want know how fast or slow your business
> > operation you must measure latency of your business operation. What
> > could be easier?
> >
> >> Is it become slower or faster?
> >
> > Very correct question! User saw "cache put time" metric becomes x2
> > bigger. Does it become slower or faster? Or we just put into the cache
> > values that 4x bigger in size? Or all time before we put values
> > locally and now we put values on remote nodes. Or our operations
> > execute in transaction and then time will depend on transaction type,
> > actions in transaction and other transaction and actually will nothing
> > talk about real cache operation. We have more questions then answers.
> >
> >> On the other hand - if `PuTime` increased - then we know for sure

Re: Re[2]: Cache operations performance metrics

[Andrey Gura]

> but between to have something and have nothing i choose — something

We already have "something". put, get, etc. metrics. As I told early
it relatively useless. But the same metrics with histograms doesn't
add any value.

> i found 1 grid machine with very different io usage than others, «dig deeper» 
> highlight cache with very different from other nodes cache put operations and 
> final «dig deeper» help to found code bug

I believe the same could be noticed using PK index stats.

> if new one would be more useful — why not ?

If some particular value is relatively useless then the same histogram
will be still relatively useless :) It's my point. Stop adding a dozen
of metrics, start thinking about benefits and meaning. Discuss it with
community.


On Fri, Dec 20, 2019 at 4:59 PM Zhenya Stanilovsky
 wrote:
>
>
> >> Is it become slower or faster?
> >
> >Very correct question! User saw "cache put time" metric becomes x2
> >bigger. Does it become slower or faster? Or we just put into the cache
> >values that 4x bigger in size? Or all time before we put values
> >locally and now we put values on remote nodes. Or our operations
> >execute in transaction and then time will depend on transaction type,
> >actions in transaction and other transaction and actually will nothing
> >talk about real cache operation. We have more questions then answers.
>
> Andrey, i hope i understand your point of view here, but between to have 
> something and have nothing i choose — something, it sometimes really helpful. 
> From real life case: i found 1 grid machine with very different io usage than 
> others, «dig deeper» highlight cache with very different from other nodes 
> cache put operations and final «dig deeper» help to found code bug, but to be 
> clear — old mechanism works ok for me here, if new one would be more useful — 
> why not ?
>
> >> On the other hand - if `PuTime` increased - then we know for sure, all 
> >> operation executing `put` becomes slower.
> >
> >Of course not :) See above.
> >
> >On Fri, Dec 20, 2019 at 3:20 PM Николай Ижиков < nizhi...@apache.org > wrote:
> >>
> >> > It also will be visible on other metrics
> >>
> >> How will it be visible?
> >>
> >> For example, the user saw «checkpoint time» metric becomes x2 bigger.
> >> How it relates to business operations? Is it become slower or faster?
> >> What does it mean for an application performance?
> >>
> >> On the other hand - if `PuTime` increased - then we know for sure, all 
> >> operation executing `put` becomes slower.
> >>
> >> *Why* it’s become slower - is the essence of «go deeper» investigation.
> >>
> >> > 20 дек. 2019 г., в 15:07, Andrey Gura < ag...@apache.org > написал(а):
> >> >
> >> >> If a cache has some percent of the relatively slow transaction this is 
> >> >> a trigger to make a deeper investigation.
> >> >
> >> > It also will be visible on other metrics. So cache operations metrics
> >> > still useless because it transitive values.
> >> >
> >> >>> 1. Measure some important internals (WAL operations, checkpoint time, 
> >> >>> etc) because it can talk about real problems.
> >> >
> >> >> We already implement it.
> >> >
> >> > I don't talk that it isn't implemented. It is just example of things
> >> > that should be measured. All other metrics depends on internals.
> >> >
> >> >>> 2. Measure business operations in user context, not cache API 
> >> >>> operations.
> >> >
> >> >> Why do you think these approaches should exclude one another?
> >> >
> >> > Because one of them is useless.
> >> >
> >> > On Fri, Dec 20, 2019 at 1:43 PM Николай Ижиков < nizhi...@apache.org > 
> >> > wrote:
> >> >>
> >> >> Hello, Andrey.
> >> >>
> >> >>> Where the sense in this value? I explained why this metrics are 
> >> >>> relatively useless.
> >> >>
> >> >> I don’t agree with you.
> >> >> I believe they are not useless for a user.
> >> >> And I try to explain why I think so.
> >> >>
> >> >>> But user can't distinguish one transaction from another, so his 
> >> >>> knowledge doesn't make sense definitely.
> >> >>
> >> >> Users shouldn’t distinguish.
> >> >> If a cache has some percent of the relatively slow transaction this is 
> >> >> a trigger to make a deeper investigation.
> >> >>
> >> >>> 1. Measure some important internals (WAL operations, checkpoint time, 
> >> >>> etc) because it can talk about real problems.
> >> >>
> >> >> We already implement it.
> >> >> What metrics are missing for internal processes?
> >> >>
> >> >>> 2. Measure business operations in user context, not cache API 
> >> >>> operations.
> >> >>
> >> >> Why do you think these approaches should exclude one another?
> >> >> Users definitely should measure whole business transaction performance.
> >> >>
> >> >> I think we should provide a way to measure part of the business 
> >> >> transaction that relates to the Ignite.
> >> >>
> >> >>
> >> >>> 20 дек. 2019 г., в 13:02, Andrey Gura < ag...@apache.org > написал(а):
> >> >>>
> >>  The goal of the proposed metrics is to measur

Re: Checkstyle "NewlineAtEndOfFile" rule issue

[Maxim Muzafarov]

Andrey,


I don't know why the issue [1] has an open status, but according to
release notes [2] and checkstyle-PR [3] the default behaviour for the
check `NewlineAtEndOfFile` has changed to  `lf_cr_crlf` and works.

Actually, you may probably find a long interesting discussion in this
PR [3] about which is better to check `lf` as line endings (same as
you suggested earlier) or `lf_cr_crlf` and folks have concluded to
change the rule default behaviour to `lf_cr_crlf`. So, I think it's
better for us to use their defaults.


I've prepared PR [4] with my proposal and tested it on TC and locally
(on Windows). Most of the changes related to EmptyLineSeparator rule
which has been fixed here (EmptyLineSeparator check does not validate
newlines before comments. [6]) and included to 8.20 release version.

Please, consider applying PR [4].
Also, let's add a recommendation to configure `autocrlf` option for git.

[1] https://issues.apache.org/jira/browse/MCHECKSTYLE-376
[2] https://checkstyle.org/releasenotes.html#Release_8.21
[3] https://github.com/checkstyle/checkstyle/issues/4073
[4] https://github.com/apache/ignite/pull/7174
[5] https://issues.apache.org/jira/browse/IGNITE-12472
[6] https://github.com/checkstyle/checkstyle/issues/5981
[7] https://git-scm.com/book/en/v2/Customizing-Git-Git-Configuration

On Thu, 19 Dec 2019 at 20:55, Andrey Gura  wrote:
>
> Agree with Pavel. But it could broke something in our pipeline.
>
> Maxim, if issue was fixed so it dependency update is better way. But
> it seems that this issue in Open status.
>
> On Thu, Dec 19, 2019 at 7:22 PM Pavel Tupitsyn  wrote:
> >
> > Maxim,
> >
> > The guidelines are not set in stone.
> > If we decide that some guideline does not bring any value and only wastes
> > our time (like this one),
> > we can (and should) remove it.
> >
> > On Thu, Dec 19, 2019 at 7:13 PM Maxim Muzafarov  wrote:
> >
> > > Pavel,
> > >
> > > It's configured according to accepted Coding Guidelines [1].
> > >
> > > [1]
> > > https://cwiki.apache.org/confluence/display/IGNITE/Coding+Guidelines#CodingGuidelines-Whitespacesandemptylines
> > >
> > > On Thu, 19 Dec 2019 at 18:59, Pavel Tupitsyn  wrote:
> > > >
> > > > Igniters,
> > > >
> > > > Does this rule bring any value whatsoever for us?
> > > > Let's just disable it.
> > > >
> > > > On Thu, Dec 19, 2019 at 6:25 PM Maxim Muzafarov 
> > > wrote:
> > > >
> > > > > Hello, Andrey
> > > > >
> > > > > Is it better to upgrade the checkstyle plugin version?
> > > > > It seems the issue has been fixed since 8.21 version (currently we 
> > > > > have
> > > > > 8.19)
> > > > >
> > > > > [1] https://checkstyle.org/releasenotes.html#Release_8.21
> > > > >
> > > > > On Thu, 19 Dec 2019 at 18:09, Andrey Gura  wrote:
> > > > > >
> > > > > > Igniters,
> > > > > >
> > > > > > recently I run build with checkstyle profile on Windows machine and
> > > > > > got 8 issues related to the "NewlineAtEndOfFile" rule while there 
> > > > > > are
> > > > > > no problems on my Linux machine.
> > > > > >
> > > > > > I investigated the problem and suggest explicitly configure this 
> > > > > > rule
> > > > > > by LF separator. See [1] for more details.
> > > > > >
> > > > > > I want merge this change if there are no any objections from
> > > community
> > > > > members.
> > > > > >
> > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-12472
> > > > >
> > >

Re: Let's remove ignite-schedule module and IgniteScheduler interface

[Ilya Kasnacheev]

Hello!

This module has two obvious downsides:

- It's LGPL.
- It can only schedule locally.

We could fix 1) by using other implementation, but given 2) this no longer
sounds feasible. If someone wants to use local scheduler, why not just use
it directly?

Regards,
-- 
Ilya Kasnacheev


пт, 20 дек. 2019 г. в 10:26, Ivan Pavlukhin :

> Denis,
>
> > The API is definitely used with even higher demand for the last months
> (overall the demand is comparable to Ignite Kafka and ML). See attachment.
> I do not see the attachement. Where can I find it?
>
> чт, 19 дек. 2019 г. в 20:01, Denis Magda :
> >
> > The API is definitely used with even higher demand for the last months
> (overall the demand is comparable to Ignite Kafka and ML). See attachment.
> >
> > If the module has some problems let's discuss them separately and see
> how to approach first. Do we have a list of the issues tracked anywhere?
> >
> >
> > -
> > Denis
> >
> >
> > On Thu, Dec 19, 2019 at 8:52 AM Valentin Kulichenko <
> valentin.kuliche...@gmail.com> wrote:
> >>
> >> Ivan,
> >>
> >> IGFS and Hadoop Accelerator had inherent architectural flaws - the vast
> >> majority of users who tried to use these features failed to achieve
> >> expected results. And yes, at the same time the interest was very high,
> so
> >> we really needed to take action :)
> >>
> >> Scheduler module, on the other hand, works as expected and might be
> used by
> >> someone. There is no need to hurry.
> >>
> >> It probably makes sense to deprecate the functionality in 2.8 so that
> users
> >> are aware of upcoming removal. But the removal itself should happen in
> the
> >> major release.
> >>
> >> -Val
> >>
> >> On Thu, Dec 19, 2019 at 12:09 AM Ivan Pavlukhin 
> wrote:
> >>
> >> > Guys,
> >> >
> >> > Why some of us are so critical regarding the subject? If I recall
> >> > correctly we decided to drop IGFS and Hadoop support before 2.8
> >> > without much debate. And it was a feature users were interested in. I
> >> > never saw an interest to IgniteSchedule. My statistics is based on our
> >> > User mailing list.
> >> >
> >> > чт, 19 дек. 2019 г. в 11:00, Alexey Kuznetsov  >:
> >> > >
> >> > > I will vote "+1" for 3.0
> >> > >
> >> > > On Thu, Dec 19, 2019 at 10:57 AM Anton Vinogradov 
> wrote:
> >> > >
> >> > > > My Vote was for 3.0
> >> > > >
> >> > > > On Thu, Dec 19, 2019 at 10:44 AM Valentin Kulichenko <
> >> > > > valentin.kuliche...@gmail.com> wrote:
> >> > > >
> >> > > > > Is this suggested for 3.0 or 2.8?
> >> > > > >
> >> > > > > I tend to agree with Alexey - API compatibility should be
> preserved
> >> > > > within
> >> > > > > a major version. I would oppose doing such a change in 2.x.
> >> > > > >
> >> > > > > If this is planned for 3.0, then it's a definite +1 from me.
> >> > > > >
> >> > > > > -Val
> >> > > > >
> >> > > > > On Wed, Dec 18, 2019 at 11:34 PM Alexey Kuznetsov <
> >> > akuznet...@apache.org
> >> > > > >
> >> > > > > wrote:
> >> > > > >
> >> > > > > > Hi!
> >> > > > > >
> >> > > > > > What if some users already using this module?
> >> > > > > > What they should do? Rewrite code?
> >> > > > > > I do not think it is a good idea.
> >> > > > > >
> >> > > > > > My "-1" here.
> >> > > > > >
> >> > > > > > On Thu, Dec 19, 2019 at 8:53 AM Anton Vinogradov <
> a...@apache.org>
> >> > > > wrote:
> >> > > > > >
> >> > > > > > > ignite-schedule does not look to be properly located or
> useful.
> >> > > > > > > My +1 here.
> >> > > > > > >
> >> > > > > > > On Thu, Dec 19, 2019 at 8:35 AM Ivan Pavlukhin <
> >> > vololo...@gmail.com>
> >> > > > > > > wrote:
> >> > > > > > >
> >> > > > > > > > Ilya,
> >> > > > > > > >
> >> > > > > > > > I think it is a good initiative! Do we really need to keep
> >> > > > > > > > run/callLocall methods at all?
> >> > > > > > > >
> >> > > > > > > > ср, 18 дек. 2019 г. в 17:59, Ilya Kasnacheev <
> il...@apache.org
> >> > >:
> >> > > > > > > > >
> >> > > > > > > > > Hello!
> >> > > > > > > > >
> >> > > > > > > > > Since 2.8 is branched, I want to initiate the discussion
> >> > about
> >> > > > > > removal
> >> > > > > > > of
> >> > > > > > > > > ignite-schedule module.
> >> > > > > > > > >
> >> > > > > > > > > My plan as follows:
> >> > > > > > > > >
> >> > > > > > > > > Remove ignite-schedule module entirely.
> >> > > > > > > > > Move runLocal and callLocal methods from
> IgniteScheduler to
> >> > > > > > > > IgniteCompute.
> >> > > > > > > > > Delete IgniteScheduler interface with its remaining
> >> > > > scheduleLocal()
> >> > > > > > > > methods.
> >> > > > > > > > >
> >> > > > > > > > > Rationale: Ignite is not a tool for local scheduling,
> >> > > > > IgniteScheduler
> >> > > > > > > > does
> >> > > > > > > > > not provide any means of remote scheduling, and I don't
> think
> >> > > > > anybody
> >> > > > > > > is
> >> > > > > > > > > using that (especially since ignite-schedule is
> unpublished
> >> > LGPL
> >> > > > > > > module).
> >> > > > > > > > >
> >> > > > > > > > > I would like to hear opinions as well as positive an

Re: Checkstyle "NewlineAtEndOfFile" rule issue

[Ivan Pavlukhin]

Can we use "autocrlf" safely? AFAIR there were problems (in other
project) with some Windows specific stuff if CRLF is replaced to LF
automatically.

пт, 20 дек. 2019 г. в 17:31, Maxim Muzafarov :
>
> Andrey,
>
>
> I don't know why the issue [1] has an open status, but according to
> release notes [2] and checkstyle-PR [3] the default behaviour for the
> check `NewlineAtEndOfFile` has changed to  `lf_cr_crlf` and works.
>
> Actually, you may probably find a long interesting discussion in this
> PR [3] about which is better to check `lf` as line endings (same as
> you suggested earlier) or `lf_cr_crlf` and folks have concluded to
> change the rule default behaviour to `lf_cr_crlf`. So, I think it's
> better for us to use their defaults.
>
>
> I've prepared PR [4] with my proposal and tested it on TC and locally
> (on Windows). Most of the changes related to EmptyLineSeparator rule
> which has been fixed here (EmptyLineSeparator check does not validate
> newlines before comments. [6]) and included to 8.20 release version.
>
> Please, consider applying PR [4].
> Also, let's add a recommendation to configure `autocrlf` option for git.
>
> [1] https://issues.apache.org/jira/browse/MCHECKSTYLE-376
> [2] https://checkstyle.org/releasenotes.html#Release_8.21
> [3] https://github.com/checkstyle/checkstyle/issues/4073
> [4] https://github.com/apache/ignite/pull/7174
> [5] https://issues.apache.org/jira/browse/IGNITE-12472
> [6] https://github.com/checkstyle/checkstyle/issues/5981
> [7] https://git-scm.com/book/en/v2/Customizing-Git-Git-Configuration
>
> On Thu, 19 Dec 2019 at 20:55, Andrey Gura  wrote:
> >
> > Agree with Pavel. But it could broke something in our pipeline.
> >
> > Maxim, if issue was fixed so it dependency update is better way. But
> > it seems that this issue in Open status.
> >
> > On Thu, Dec 19, 2019 at 7:22 PM Pavel Tupitsyn  wrote:
> > >
> > > Maxim,
> > >
> > > The guidelines are not set in stone.
> > > If we decide that some guideline does not bring any value and only wastes
> > > our time (like this one),
> > > we can (and should) remove it.
> > >
> > > On Thu, Dec 19, 2019 at 7:13 PM Maxim Muzafarov  wrote:
> > >
> > > > Pavel,
> > > >
> > > > It's configured according to accepted Coding Guidelines [1].
> > > >
> > > > [1]
> > > > https://cwiki.apache.org/confluence/display/IGNITE/Coding+Guidelines#CodingGuidelines-Whitespacesandemptylines
> > > >
> > > > On Thu, 19 Dec 2019 at 18:59, Pavel Tupitsyn  
> > > > wrote:
> > > > >
> > > > > Igniters,
> > > > >
> > > > > Does this rule bring any value whatsoever for us?
> > > > > Let's just disable it.
> > > > >
> > > > > On Thu, Dec 19, 2019 at 6:25 PM Maxim Muzafarov 
> > > > wrote:
> > > > >
> > > > > > Hello, Andrey
> > > > > >
> > > > > > Is it better to upgrade the checkstyle plugin version?
> > > > > > It seems the issue has been fixed since 8.21 version (currently we 
> > > > > > have
> > > > > > 8.19)
> > > > > >
> > > > > > [1] https://checkstyle.org/releasenotes.html#Release_8.21
> > > > > >
> > > > > > On Thu, 19 Dec 2019 at 18:09, Andrey Gura  wrote:
> > > > > > >
> > > > > > > Igniters,
> > > > > > >
> > > > > > > recently I run build with checkstyle profile on Windows machine 
> > > > > > > and
> > > > > > > got 8 issues related to the "NewlineAtEndOfFile" rule while there 
> > > > > > > are
> > > > > > > no problems on my Linux machine.
> > > > > > >
> > > > > > > I investigated the problem and suggest explicitly configure this 
> > > > > > > rule
> > > > > > > by LF separator. See [1] for more details.
> > > > > > >
> > > > > > > I want merge this change if there are no any objections from
> > > > community
> > > > > > members.
> > > > > > >
> > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-12472
> > > > > >
> > > >



-- 
Best regards,
Ivan Pavlukhin

[jira] [Created] (IGNITE-12480) Add BinaryFieldExtractionSelfTest to the Binary Objects test suite

[Denis Mekhanikov (Jira)]

Denis Mekhanikov created IGNITE-12480:
-

 Summary: Add BinaryFieldExtractionSelfTest to the Binary Objects 
test suite
 Key: IGNITE-12480
 URL: https://issues.apache.org/jira/browse/IGNITE-12480
 Project: Ignite
  Issue Type: Test
  Components: binary
Reporter: Denis Mekhanikov
Assignee: Denis Mekhanikov
 Fix For: 2.8


BinaryFieldExtractionSelfTest is not run on TeamCity because it's not included 
into any test suite.
It should be added into IgniteBinaryObjectsTestSuite



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-12481) .NET: NuGet verification script does not handle multi-digit version parts

[Pavel Tupitsyn (Jira)]

Pavel Tupitsyn created IGNITE-12481:
---

 Summary: .NET: NuGet verification script does not handle 
multi-digit version parts
 Key: IGNITE-12481
 URL: https://issues.apache.org/jira/browse/IGNITE-12481
 Project: Ignite
  Issue Type: Bug
  Components: platforms
Affects Versions: 2.9
Reporter: Pavel Tupitsyn
Assignee: Pavel Tupitsyn
 Fix For: 2.9


The following regex in verify-nuget.ps1 does not support multi-digit versions 
like 8.7.99

{code}
$packageId = $_.Name -replace '(.*?)\.\d\.\d\.\d\.nupkg', '$1'
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Contribution page & easy/moderate tickets for newcomers

[Denis Magda]

Ignite,

I've done a major overhaul of our contribution page that promotes
involvement in Ignite development:
https://ignite.apache.org/community/contribute.html

Many sections were simplified, some legacy stuff was removed and I'll
appreciate if some of you check the page and advise further improvements.

Apart from that, one of the sections (Pick a Ticket) lists of  easy and
moderate tickets

for
newcomers. I do believe that the list needs to be reviewed. Could each of
you recall any tickets of easy/moderate complexity that are still pending
but will be great to implement? Put the "newbie" label for those and
mention "component". I've already highlighted some of the components on the
webpage (ML, SQL, platforms) and we can expand the list.


Thanks!

-
Denis

Re: [Webinar] Data Streaming Using Apache Flink and Apache Ignite

[Saikat Maitra]

Hi Denis, Ivan

Thank you, I am glad you liked the webinar.

Please let me know if there are other opportunities where I can present on
Data Streaming and I would be happy to do a talk.

Regards,
Saikat

On Fri, Dec 20, 2019 at 1:44 AM Ivan Pavlukhin  wrote:

> Saikat,
>
> I watched a recording and found it great! Thank you for that! Would
> love to see more materials how Ignite can be used in practice.
>
> пт, 20 дек. 2019 г. в 10:19, Denis Magda :
> >
> > Saikat, excellent start! I wasn't able to attend but watched the
> > recording later. Thanks for sharing your experience.
> >
> > -
> > Denis
> >
> >
> > On Wed, Dec 18, 2019 at 5:12 PM Saikat Maitra 
> > wrote:
> >
> > > Hi,
> > >
> > >
> > > The page with the embedded recording and slides link is
> > >
> https://www.gridgain.com/resources/webinars/data-streaming-using-apache-flink-and-apache-ignite/view
> > > .
> > >
> > >
> > >
> > > You can go directly to the YouTube video at
> > > https://www.youtube.com/watch?v=wM-YVYVU8Lk.
> > >
> > >
> > >
> > > You can go directly to the slides at
> > >
> https://www.gridgain.com/resources/technical%20presentations/data-streaming-using-apache-flink-and-apache-ignite/view
> > > <
> https://www.gridgain.com/resources/technical%20presentations/data-streaming-using-apache-flink-and-apache-ignite/view?mkt_tok=eyJpIjoiWldKa05UZzVabVprWmpBeSIsInQiOiJwSDR1Yjd4alJSc1NcL3UzVlBqU1pqbXNwUHFqMUQrXC90OHZcL0lvSzRGS2dwajlIb1RTa0haM1d1bENFc1FuUkJZZk5JSzFjcVp1Y3dBXC93SEg0cDlkUXhTQ2EwQnlXVVZ2RVdlZUgybHhhakFrb2srWlVucVZyQkZZU3RJSXJjeTMifQ%3D%3D
> >
> > >
> > >
> > > Regards,
> > >
> > > Saikat
> > >
> > > On Mon, Dec 9, 2019 at 9:33 PM Saikat Maitra 
> > > wrote:
> > >
> > >> Thank you Alexey, yes I think all the webinar videos are recorded and
> > >> shared in youtube.
> > >>
> > >> https://www.youtube.com/results?search_query=gridgain+webinar
> > >>
> > >> I can share the webinar video once it is available.
> > >>
> > >> On Sun, Dec 8, 2019 at 10:28 PM Alexey Zinoviev <
> zaleslaw@gmail.com>
> > >> wrote:
> > >>
> > >>> Great to get a video after to watch and share later, very ineresting
> case
> > >>>
> > >>> вс, 8 дек. 2019 г., 20:39 Saikat Maitra :
> > >>>
> >  Hi,
> > 
> >  I will be presenting in a webinar about Data Streaming Using Apache
> >  Flink
> >  and Apache Ignite on Wednesday, Dec 18th.
> > 
> > 
> > 
> https://www.gridgain.com/resources/webinars/data-streaming-using-apache-flink-and-apache-ignite
> > 
> >  Apache Ignite is a powerful in-memory computing platform. The Apache
> >  IgniteSink streaming connector enables users to inject Flink data
> into
> >  the
> >  Ignite cache. Join Saikat Maitra to learn how to build a simple data
> >  streaming application using Apache Flink and Apache Ignite. This
> stream
> >  processing topology will allow data streaming in a distributed,
> >  scalable,
> >  and fault-tolerant manner, which can process data sets consisting of
> >  virtually unlimited streams of events.
> > 
> >  Please feel free to join the webinar.
> > 
> >  Regards,
> >  Saikat
> > 
> > >>>
>
>
>
> --
> Best regards,
> Ivan Pavlukhin
>

Re: Let's remove ignite-schedule module and IgniteScheduler interface

[Denis Magda]

Ilya, good points, then support the idea of the API removal in 3.0.

Ivan, downloaded the screenshot to Google Drive:
https://drive.google.com/file/d/1N21N7yqCbeZtCNs1sHvJLiJfHF_Hp0wd/view?usp=sharing


-
Denis


On Fri, Dec 20, 2019 at 7:09 AM Ilya Kasnacheev 
wrote:

> Hello!
>
> This module has two obvious downsides:
>
> - It's LGPL.
> - It can only schedule locally.
>
> We could fix 1) by using other implementation, but given 2) this no longer
> sounds feasible. If someone wants to use local scheduler, why not just use
> it directly?
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> пт, 20 дек. 2019 г. в 10:26, Ivan Pavlukhin :
>
> > Denis,
> >
> > > The API is definitely used with even higher demand for the last months
> > (overall the demand is comparable to Ignite Kafka and ML). See
> attachment.
> > I do not see the attachement. Where can I find it?
> >
> > чт, 19 дек. 2019 г. в 20:01, Denis Magda :
> > >
> > > The API is definitely used with even higher demand for the last months
> > (overall the demand is comparable to Ignite Kafka and ML). See
> attachment.
> > >
> > > If the module has some problems let's discuss them separately and see
> > how to approach first. Do we have a list of the issues tracked anywhere?
> > >
> > >
> > > -
> > > Denis
> > >
> > >
> > > On Thu, Dec 19, 2019 at 8:52 AM Valentin Kulichenko <
> > valentin.kuliche...@gmail.com> wrote:
> > >>
> > >> Ivan,
> > >>
> > >> IGFS and Hadoop Accelerator had inherent architectural flaws - the
> vast
> > >> majority of users who tried to use these features failed to achieve
> > >> expected results. And yes, at the same time the interest was very
> high,
> > so
> > >> we really needed to take action :)
> > >>
> > >> Scheduler module, on the other hand, works as expected and might be
> > used by
> > >> someone. There is no need to hurry.
> > >>
> > >> It probably makes sense to deprecate the functionality in 2.8 so that
> > users
> > >> are aware of upcoming removal. But the removal itself should happen in
> > the
> > >> major release.
> > >>
> > >> -Val
> > >>
> > >> On Thu, Dec 19, 2019 at 12:09 AM Ivan Pavlukhin 
> > wrote:
> > >>
> > >> > Guys,
> > >> >
> > >> > Why some of us are so critical regarding the subject? If I recall
> > >> > correctly we decided to drop IGFS and Hadoop support before 2.8
> > >> > without much debate. And it was a feature users were interested in.
> I
> > >> > never saw an interest to IgniteSchedule. My statistics is based on
> our
> > >> > User mailing list.
> > >> >
> > >> > чт, 19 дек. 2019 г. в 11:00, Alexey Kuznetsov <
> akuznet...@apache.org
> > >:
> > >> > >
> > >> > > I will vote "+1" for 3.0
> > >> > >
> > >> > > On Thu, Dec 19, 2019 at 10:57 AM Anton Vinogradov 
> > wrote:
> > >> > >
> > >> > > > My Vote was for 3.0
> > >> > > >
> > >> > > > On Thu, Dec 19, 2019 at 10:44 AM Valentin Kulichenko <
> > >> > > > valentin.kuliche...@gmail.com> wrote:
> > >> > > >
> > >> > > > > Is this suggested for 3.0 or 2.8?
> > >> > > > >
> > >> > > > > I tend to agree with Alexey - API compatibility should be
> > preserved
> > >> > > > within
> > >> > > > > a major version. I would oppose doing such a change in 2.x.
> > >> > > > >
> > >> > > > > If this is planned for 3.0, then it's a definite +1 from me.
> > >> > > > >
> > >> > > > > -Val
> > >> > > > >
> > >> > > > > On Wed, Dec 18, 2019 at 11:34 PM Alexey Kuznetsov <
> > >> > akuznet...@apache.org
> > >> > > > >
> > >> > > > > wrote:
> > >> > > > >
> > >> > > > > > Hi!
> > >> > > > > >
> > >> > > > > > What if some users already using this module?
> > >> > > > > > What they should do? Rewrite code?
> > >> > > > > > I do not think it is a good idea.
> > >> > > > > >
> > >> > > > > > My "-1" here.
> > >> > > > > >
> > >> > > > > > On Thu, Dec 19, 2019 at 8:53 AM Anton Vinogradov <
> > a...@apache.org>
> > >> > > > wrote:
> > >> > > > > >
> > >> > > > > > > ignite-schedule does not look to be properly located or
> > useful.
> > >> > > > > > > My +1 here.
> > >> > > > > > >
> > >> > > > > > > On Thu, Dec 19, 2019 at 8:35 AM Ivan Pavlukhin <
> > >> > vololo...@gmail.com>
> > >> > > > > > > wrote:
> > >> > > > > > >
> > >> > > > > > > > Ilya,
> > >> > > > > > > >
> > >> > > > > > > > I think it is a good initiative! Do we really need to
> keep
> > >> > > > > > > > run/callLocall methods at all?
> > >> > > > > > > >
> > >> > > > > > > > ср, 18 дек. 2019 г. в 17:59, Ilya Kasnacheev <
> > il...@apache.org
> > >> > >:
> > >> > > > > > > > >
> > >> > > > > > > > > Hello!
> > >> > > > > > > > >
> > >> > > > > > > > > Since 2.8 is branched, I want to initiate the
> discussion
> > >> > about
> > >> > > > > > removal
> > >> > > > > > > of
> > >> > > > > > > > > ignite-schedule module.
> > >> > > > > > > > >
> > >> > > > > > > > > My plan as follows:
> > >> > > > > > > > >
> > >> > > > > > > > > Remove ignite-schedule module entirely.
> > >> > > > > > > > > Move runLocal and callLocal methods from
> > IgniteScheduler to
> > >> > > > > > > > IgniteCompute.
> > >> > > > > > > > > Dele