Re: Choosing historical rebalance heuristics

[Vladislav Pyatkov]

I completely forget about another promise to favor of using historical
rebalance where it is possible. When cluster decided to use a full balance,
demander nodes should clear not empty partitions.
This can to consume a long time, in some cases that may be compared with a
time of rebalance.
It also accepts a side of heuristics above.

On Thu, Jul 16, 2020 at 12:09 AM Vladislav Pyatkov 
wrote:

> Ivan,
>
> I agree with a combined approach: threshold for small partitions and count
> of update for partition that outgrew it.
> This helps to avoid partitions that update not frequently.
>
> Reading of a big WAL piece (more than 100Gb) it can happen, when a client
> configured it intentionally.
> There are no doubts we can to read it, otherwise WAL space was not
> configured that too large.
>
> I don't see a connection optimization of iterator and issue in atomic
> protocol.
> Reordering in WAL, that happened in checkpoint where counter was not
> changing, is an extremely rare case and the issue will not solve for
> generic case, this should be fixed in bound of protocol.
>
> I think we can modify the heuristic so
> 1) Exclude partitions by threshold (IGNITE_PDS_WAL_REBALANCE_THRESHOLD -
> reduce it to 500)
> 2) Select only that partition for historical rebalance where difference
> between counters less that partition size.
>
> Also implement mentioned optimization for historical iterator, that may
> reduce a time on reading large WAL interval.
>
> On Wed, Jul 15, 2020 at 3:15 PM Ivan Rakov  wrote:
>
>> Hi Vladislav,
>>
>> Thanks for raising this topic.
>> Currently present IGNITE_PDS_WAL_REBALANCE_THRESHOLD (default is 500_000)
>> is controversial. Assuming that the default number of partitions is 1024,
>> cache should contain a really huge amount of data in order to make WAL
>> delta rebalancing possible. In fact, it's currently disabled for most
>> production cases, which makes rebalancing of persistent caches
>> unreasonably
>> long.
>>
>> I think, your approach [1] makes much more sense than the current
>> heuristic, let's move forward with the proposed solution.
>>
>> Though, there are some other corner cases, e.g. this one:
>> - Configured size of WAL archive is big (>100 GB)
>> - Cache has small partitions (e.g. 1000 entries)
>> - Infrequent updates (e.g. ~100 in the whole WAL history of any node)
>> - There is another cache with very frequent updates which allocate >99% of
>> WAL
>> In such scenario we may need to iterate over >100 GB of WAL in order to
>> fetch <1% of needed updates. Even though the amount of network traffic is
>> still optimized, it would be more effective to transfer partitions with
>> ~1000 entries fully instead of reading >100 GB of WAL.
>>
>> I want to highlight that your heuristic definitely makes the situation
>> better, but due to possible corner cases we should keep the fallback lever
>> to restrict or limit historical rebalance as before. Probably, it would be
>> handy to keep IGNITE_PDS_WAL_REBALANCE_THRESHOLD property with a low
>> default value (1000, 500 or even 0) and apply your heuristic only for
>> partitions with bigger size.
>>
>> Regarding case [2]: it looks like an improvement that can mitigate some
>> corner cases (including the one that I have described). I'm ok with it as
>> long as it takes data updates reordering on backup nodes into account. We
>> don't track skipped updates for atomic caches. As a result, detection of
>> the absence of updates between two checkpoint markers with the same
>> partition counter can be false positive.
>>
>> --
>> Best Regards,
>> Ivan Rakov
>>
>> On Tue, Jul 14, 2020 at 3:03 PM Vladislav Pyatkov 
>> wrote:
>>
>> > Hi guys,
>> >
>> > I want to implement a more honest heuristic for historical rebalance.
>> > Before, a cluster makes a choice between the historical rebalance or
>> not it
>> > only from a partition size. This threshold more known by a name of
>> property
>> > IGNITE_PDS_WAL_REBALANCE_THRESHOLD.
>> > It might prevent a historical rebalance when a partition is too small,
>> but
>> > not if WAL contains more updates than a size of partition, historical
>> > rebalance still can be chosen.
>> > There is a ticket where need to implement more fair heuristic[1].
>> >
>> > My idea for implementation is need to estimate a size of data which
>> will be
>> > transferred owe network. In other word if need to rebalance a part of
>> WAL
>> > that contains N updates, for recover a partition on another node, which
>> > have to contain M rows at all, need chooses a historical rebalance on
>> the
>> > case where N < M (WAL history should be presented as well).
>> >
>> > This approach is easy implemented, because a coordinator node has the
>> size
>> > of partitions and counters' interval. But in this case cluster still can
>> > find not many updates in too long WAL history. I assume a possibility to
>> > work around it, if rebalance historical iterator will not handle
>> > checkpoints where not contains updates of particular cache. Check

[jira] [Created] (IGNITE-13262) Remove GridClientUtils#shutdownNow() as full duplicate of U#shutdownNow

[Alexey Kuznetsov (Jira)]

Alexey Kuznetsov created IGNITE-13262:
-

 Summary: Remove GridClientUtils#shutdownNow() as full duplicate of 
U#shutdownNow
 Key: IGNITE-13262
 URL: https://issues.apache.org/jira/browse/IGNITE-13262
 Project: Ignite
  Issue Type: Task
  Components: general
Affects Versions: 2.8.1
Reporter: Alexey Kuznetsov
Assignee: Alexey Kuznetsov


Method o.a.i.internal.client.util.GridClientUtils#shutdownNow is a full 
duplicate of U#shutdownNow.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

New committer: Artem Budnikov

[Ilya Kasnacheev]

Hello!

The Project Management Committee for Apache Ignite project has invited
Artem Budnikov to become a committer and we are pleased to announce that he
has accepted.

He is the person behind a monumental effort to keep Apache Ignite
documentation neat and tidy, which he does by writing, reviewing and
versioning most of docs on apacheignite.readme.io.

Being a committer enables easier contribution to the project since it makes
possible to merge your own, or other contributors', changes after the
review. This should enable better productivity.

On behalf of Apache Ignite PMC,

Re: New committer: Artem Budnikov

[Dmitriy Pavlov]

Hi Artem,

Congrats! That's good to know what we've invited committer because of
documentation effort, not just code.

Igniters,

This makes me happy especially because of CoPDoC principle (
https://community.apache.org/contributors/) which describes you may
contribute in different ways and all your efforts are appreciated..

Sincerely,
Dmitriy Pavlov

чт, 16 июл. 2020 г. в 14:19, Ilya Kasnacheev :

> Hello!
>
> The Project Management Committee for Apache Ignite project has invited
> Artem Budnikov to become a committer and we are pleased to announce that he
> has accepted.
>
> He is the person behind a monumental effort to keep Apache Ignite
> documentation neat and tidy, which he does by writing, reviewing and
> versioning most of docs on apacheignite.readme.io.
>
> Being a committer enables easier contribution to the project since it makes
> possible to merge your own, or other contributors', changes after the
> review. This should enable better productivity.
>
> On behalf of Apache Ignite PMC,
>

Re: New committer: Artem Budnikov

[Roman Kondakov]

Congrats, Artem!

--
Roman Kondakov

On 16.07.2020 14:19, Ilya Kasnacheev wrote:

Hello!

The Project Management Committee for Apache Ignite project has invited
Artem Budnikov to become a committer and we are pleased to announce that he
has accepted.

He is the person behind a monumental effort to keep Apache Ignite
documentation neat and tidy, which he does by writing, reviewing and
versioning most of docs on apacheignite.readme.io.

Being a committer enables easier contribution to the project since it makes
possible to merge your own, or other contributors', changes after the
review. This should enable better productivity.

On behalf of Apache Ignite PMC,

Re: New committer: Artem Budnikov

[Ivan Pavlukhin]

Artem, my congratulations!

2020-07-16 15:10 GMT+03:00, Roman Kondakov :
> Congrats, Artem!
>
> --
> Roman Kondakov
>
> On 16.07.2020 14:19, Ilya Kasnacheev wrote:
>> Hello!
>>
>> The Project Management Committee for Apache Ignite project has invited
>> Artem Budnikov to become a committer and we are pleased to announce that
>> he
>> has accepted.
>>
>> He is the person behind a monumental effort to keep Apache Ignite
>> documentation neat and tidy, which he does by writing, reviewing and
>> versioning most of docs on apacheignite.readme.io.
>>
>> Being a committer enables easier contribution to the project since it
>> makes
>> possible to merge your own, or other contributors', changes after the
>> review. This should enable better productivity.
>>
>> On behalf of Apache Ignite PMC,
>>
>


-- 

Best regards,
Ivan Pavlukhin

Re: New committer: Artem Budnikov

[Вячеслав Коптилин]

Hello Artem,

Congratulations! You definitely deserve it.

Thanks,
S.

чт, 16 июл. 2020 г. в 15:26, Ivan Pavlukhin :

> Artem, my congratulations!
>
> 2020-07-16 15:10 GMT+03:00, Roman Kondakov :
> > Congrats, Artem!
> >
> > --
> > Roman Kondakov
> >
> > On 16.07.2020 14:19, Ilya Kasnacheev wrote:
> >> Hello!
> >>
> >> The Project Management Committee for Apache Ignite project has invited
> >> Artem Budnikov to become a committer and we are pleased to announce that
> >> he
> >> has accepted.
> >>
> >> He is the person behind a monumental effort to keep Apache Ignite
> >> documentation neat and tidy, which he does by writing, reviewing and
> >> versioning most of docs on apacheignite.readme.io.
> >>
> >> Being a committer enables easier contribution to the project since it
> >> makes
> >> possible to merge your own, or other contributors', changes after the
> >> review. This should enable better productivity.
> >>
> >> On behalf of Apache Ignite PMC,
> >>
> >
>
>
> --
>
> Best regards,
> Ivan Pavlukhin
>

Re: New committer: Artem Budnikov

[Artem Budnikov]

Thanks, guys.

-Artem

On 16.07.2020 15:47, Вячеслав Коптилин wrote:

Hello Artem,

Congratulations! You definitely deserve it.

Thanks,
S.

чт, 16 июл. 2020 г. в 15:26, Ivan Pavlukhin :


Artem, my congratulations!

2020-07-16 15:10 GMT+03:00, Roman Kondakov :

Congrats, Artem!

--
Roman Kondakov

On 16.07.2020 14:19, Ilya Kasnacheev wrote:

Hello!

The Project Management Committee for Apache Ignite project has invited
Artem Budnikov to become a committer and we are pleased to announce that
he
has accepted.

He is the person behind a monumental effort to keep Apache Ignite
documentation neat and tidy, which he does by writing, reviewing and
versioning most of docs on apacheignite.readme.io.

Being a committer enables easier contribution to the project since it
makes
possible to merge your own, or other contributors', changes after the
review. This should enable better productivity.

On behalf of Apache Ignite PMC,



--

Best regards,
Ivan Pavlukhin

[jira] [Created] (IGNITE-13263) PDS 4 suite timeout

[Nikita Tolstunov (Jira)]

Nikita Tolstunov created IGNITE-13263:
-

 Summary: PDS 4 suite timeout
 Key: IGNITE-13263
 URL: https://issues.apache.org/jira/browse/IGNITE-13263
 Project: Ignite
  Issue Type: Bug
Reporter: Nikita Tolstunov
Assignee: Nikita Tolstunov
 Attachments: Ignite_Tests_2.4_Java_8_9_10_11_PDS_4_18257.log.zip

[Build 
log|[https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_Pds4?branch=&buildTypeTab=overview&mode=builds]]
 fails sometimes with timeout

log in attach



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-13264) DataStreamerImplSelfTest#testAllOperationFinishedBeforeFutureCompletion is broken

[Alexey Scherbakov (Jira)]

Alexey Scherbakov created IGNITE-13264:
--

 Summary: 
DataStreamerImplSelfTest#testAllOperationFinishedBeforeFutureCompletion is 
broken
 Key: IGNITE-13264
 URL: https://issues.apache.org/jira/browse/IGNITE-13264
 Project: Ignite
  Issue Type: Bug
Reporter: Alexey Scherbakov
 Fix For: 2.10
 Attachments: bad.patch

Apply a patch and run several times to get this assertion:Apply a patch and run 
several times to get this assertion:
{noformat}java.lang.AssertionError: Expected :2Actual   :null 
 at org.junit.Assert.fail(Assert.java:88) at 
org.junit.Assert.failNotEquals(Assert.java:834) at 
org.junit.Assert.assertEquals(Assert.java:118) at 
org.junit.Assert.assertEquals(Assert.java:144) at 
org.apache.ignite.testframework.junits.JUnitAssertAware.assertEquals(JUnitAssertAware.java:58)
 at 
org.apache.ignite.internal.processors.datastreamer.DataStreamerImplSelfTest.access$100(DataStreamerImplSelfTest.java:68)
 at 
org.apache.ignite.internal.processors.datastreamer.DataStreamerImplSelfTest$2.apply(DataStreamerImplSelfTest.java:320)
 at 
org.apache.ignite.internal.processors.datastreamer.DataStreamerImplSelfTest$2.apply(DataStreamerImplSelfTest.java:314)
 at 
org.apache.ignite.internal.util.future.IgniteFutureImpl$InternalFutureListener.apply(IgniteFutureImpl.java:213)
 at 
org.apache.ignite.internal.util.future.IgniteFutureImpl$InternalFutureListener.apply(IgniteFutureImpl.java:178)
 at 
org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:398)
 at 
org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:346)
 at 
org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:334)
 at 
org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:510)
 at 
org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:489)
 at 
org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:455)
 at 
org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl$5.apply(DataStreamerImpl.java:948)
 at 
org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl$5.apply(DataStreamerImpl.java:938)
 at 
org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:398)
 at 
org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:346)
 at 
org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:334)
 at 
org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:510)
 at 
org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:489)
 at 
org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:466)
 at 
org.apache.ignite.internal.util.future.GridCompoundFuture.checkComplete(GridCompoundFuture.java:284)
 at 
org.apache.ignite.internal.util.future.GridCompoundFuture.apply(GridCompoundFuture.java:142)
 at 
org.apache.ignite.internal.util.future.GridCompoundFuture.apply(GridCompoundFuture.java:43)
 at 
org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:398)
 at 
org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:346)
 at 
org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:334)
 at 
org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:510)
 at 
org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:489)
 at 
org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl$Buffer.onResponse(DataStreamerImpl.java:2072)
 at 
org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl$3.onMessage(DataStreamerImpl.java:359)
 at 
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1711)
 at 
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1331)
 at 
org.apache.ignite.internal.managers.communication.GridIoManager.access$4800(GridIoManager.java:153)
 at 
org.apache.ignite.internal.managers.communication.GridIoManager$8.execute(GridIoManager.java:1216)
 at 
org.apache.ignite.internal.managers.communication.TraceRunnable.run(TraceRunnable.java:54)
 at 
org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:559)
 at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:119) 
at java.lang.Thread.run(Thread.java:748)\{noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

request for contributors permissions

[Konstantin Sirotkin]

Helloi've registered on https://issues.apache.org/jira/secure/Dashboard.jsp
 and
according to
https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute I'm
requesting contributors permissionsthank you

Konstantin Sirotkin

Re: Choosing historical rebalance heuristics

[Ivan Rakov]

>
>  I think we can modify the heuristic so
> 1) Exclude partitions by threshold (IGNITE_PDS_WAL_REBALANCE_THRESHOLD -
> reduce it to 500)
> 2) Select only that partition for historical rebalance where difference
> between counters less that partition size.

Agreed, let's go this way.

On Thu, Jul 16, 2020 at 11:03 AM Vladislav Pyatkov 
wrote:

> I completely forget about another promise to favor of using historical
> rebalance where it is possible. When cluster decided to use a full balance,
> demander nodes should clear not empty partitions.
> This can to consume a long time, in some cases that may be compared with a
> time of rebalance.
> It also accepts a side of heuristics above.
>
> On Thu, Jul 16, 2020 at 12:09 AM Vladislav Pyatkov 
> wrote:
>
> > Ivan,
> >
> > I agree with a combined approach: threshold for small partitions and
> count
> > of update for partition that outgrew it.
> > This helps to avoid partitions that update not frequently.
> >
> > Reading of a big WAL piece (more than 100Gb) it can happen, when a client
> > configured it intentionally.
> > There are no doubts we can to read it, otherwise WAL space was not
> > configured that too large.
> >
> > I don't see a connection optimization of iterator and issue in atomic
> > protocol.
> > Reordering in WAL, that happened in checkpoint where counter was not
> > changing, is an extremely rare case and the issue will not solve for
> > generic case, this should be fixed in bound of protocol.
> >
> > I think we can modify the heuristic so
> > 1) Exclude partitions by threshold (IGNITE_PDS_WAL_REBALANCE_THRESHOLD -
> > reduce it to 500)
> > 2) Select only that partition for historical rebalance where difference
> > between counters less that partition size.
> >
> > Also implement mentioned optimization for historical iterator, that may
> > reduce a time on reading large WAL interval.
> >
> > On Wed, Jul 15, 2020 at 3:15 PM Ivan Rakov 
> wrote:
> >
> >> Hi Vladislav,
> >>
> >> Thanks for raising this topic.
> >> Currently present IGNITE_PDS_WAL_REBALANCE_THRESHOLD (default is
> 500_000)
> >> is controversial. Assuming that the default number of partitions is
> 1024,
> >> cache should contain a really huge amount of data in order to make WAL
> >> delta rebalancing possible. In fact, it's currently disabled for most
> >> production cases, which makes rebalancing of persistent caches
> >> unreasonably
> >> long.
> >>
> >> I think, your approach [1] makes much more sense than the current
> >> heuristic, let's move forward with the proposed solution.
> >>
> >> Though, there are some other corner cases, e.g. this one:
> >> - Configured size of WAL archive is big (>100 GB)
> >> - Cache has small partitions (e.g. 1000 entries)
> >> - Infrequent updates (e.g. ~100 in the whole WAL history of any node)
> >> - There is another cache with very frequent updates which allocate >99%
> of
> >> WAL
> >> In such scenario we may need to iterate over >100 GB of WAL in order to
> >> fetch <1% of needed updates. Even though the amount of network traffic
> is
> >> still optimized, it would be more effective to transfer partitions with
> >> ~1000 entries fully instead of reading >100 GB of WAL.
> >>
> >> I want to highlight that your heuristic definitely makes the situation
> >> better, but due to possible corner cases we should keep the fallback
> lever
> >> to restrict or limit historical rebalance as before. Probably, it would
> be
> >> handy to keep IGNITE_PDS_WAL_REBALANCE_THRESHOLD property with a low
> >> default value (1000, 500 or even 0) and apply your heuristic only for
> >> partitions with bigger size.
> >>
> >> Regarding case [2]: it looks like an improvement that can mitigate some
> >> corner cases (including the one that I have described). I'm ok with it
> as
> >> long as it takes data updates reordering on backup nodes into account.
> We
> >> don't track skipped updates for atomic caches. As a result, detection of
> >> the absence of updates between two checkpoint markers with the same
> >> partition counter can be false positive.
> >>
> >> --
> >> Best Regards,
> >> Ivan Rakov
> >>
> >> On Tue, Jul 14, 2020 at 3:03 PM Vladislav Pyatkov  >
> >> wrote:
> >>
> >> > Hi guys,
> >> >
> >> > I want to implement a more honest heuristic for historical rebalance.
> >> > Before, a cluster makes a choice between the historical rebalance or
> >> not it
> >> > only from a partition size. This threshold more known by a name of
> >> property
> >> > IGNITE_PDS_WAL_REBALANCE_THRESHOLD.
> >> > It might prevent a historical rebalance when a partition is too small,
> >> but
> >> > not if WAL contains more updates than a size of partition, historical
> >> > rebalance still can be chosen.
> >> > There is a ticket where need to implement more fair heuristic[1].
> >> >
> >> > My idea for implementation is need to estimate a size of data which
> >> will be
> >> > transferred owe network. In other word if need to rebalance a part of
> >> WAL
> >> > that contains N updates, for r

Re: request for contributors permissions

[Denis Magda]

Hello Konstantin and welcome to the community!

Please share your JIRA id so that we can add you to the contributors' list
there and you can select a ticket of your interest.

-
Denis


On Thu, Jul 16, 2020 at 8:29 AM Konstantin Sirotkin <
konstantins...@gmail.com> wrote:

> Helloi've registered on
> https://issues.apache.org/jira/secure/Dashboard.jsp
>  and
> according to
> https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute I'm
> requesting contributors permissionsthank you
>
> Konstantin Sirotkin
>

Re: A few small pull requests

[Denis Magda]

Hi Stephen,

Thanks for reminding us about these pending contributions.

IGNITE-12192 Allow ignitevisorcmd to quit when pressing ^D <
> https://github.com/apache/ignite/pull/6877>


Merged the changes. Thanks!

IGNITE-12182 ExecutorService defaults to only server nodes <
> https://github.com/apache/ignite/pull/6876>


I'm afraid we can't merge this change until Ignite 3.0. Otherwise, it will
break backward compatibility.

IGNITE-11715 Allow extra options to be passed in to ignite.sh from Docker <
> https://github.com/apache/ignite/pull/6552>


Could you confirm that the JIRA ticket is correct? Whenever I try to open
IGNITE-11715 in JIRA it redirects me to
https://issues.apache.org/jira/projects/IGNITE/issues/IGNITE-13264


-
Denis


On Wed, Jul 15, 2020 at 6:42 AM Stephen Darlington <
stephen.darling...@gridgain.com> wrote:

> Hi,
>
> I have a few small quality-of-life pull requests languishing in the
> backlog. Can anyone take a look or suggest what else I need to do to
> progress them?
>
> IGNITE-12192 Allow ignitevisorcmd to quit when pressing ^D <
> https://github.com/apache/ignite/pull/6877>
> IGNITE-12182 ExecutorService defaults to only server nodes <
> https://github.com/apache/ignite/pull/6876>
> IGNITE-11715 Allow extra options to be passed in to ignite.sh from Docker <
> https://github.com/apache/ignite/pull/6552>
>
> Thanks,
> Stephen
>
>

Re: Proposal of new event QUERY_EXECUTION_EVENT

[Denis Magda]

Taras, Yury, Ivan,

Could you please join this thread and share your thoughts? Do we already
have any plans on tracking of the DDL and DML queries?

-
Denis


On Wed, Jul 15, 2020 at 12:09 AM Max Timonin 
wrote:

> Hi Denis, thanks for the answer!
>
> We already checked EVT_CACHE_QUERY_EXECUTED and found that it works only in
> cases:
> 1. Scan queries and Select queries (common pattern is access to cache
> data);
> 2. This event triggers only if query execution succeeds, in case of failure
> while execution this event won't fire.
>
> Our additional requirements are to protocol queries:
> 1. that aren't cache related (for example, alter user);
> 2. that relate to multiple caches (while EVT_CACHE_QUERY_EXECUTED have
> field cacheName related to specific cache);
> 3. we need to protocol also DDL and DML queries.
>
> Regards,
> Maksim
>
> On Tue, Jul 14, 2020 at 10:20 PM Denis Magda  wrote:
>
> > Hi Max,
> >
> > Could you check if the EVT_CACHE_QUERY_EXECUTED event is what you're
> > looking for?
> >
> >
> https://www.gridgain.com/docs/latest/developers-guide/events/events#cache-query-events
> >
> > -
> > Denis
> >
> >
> > On Fri, Jul 10, 2020 at 3:54 AM Max Timonin 
> > wrote:
> >
> > > Hi Igniters!
> > >
> > > We're going to protocol all input SQL queries from our users. Currently
> > > there is no such mechanism in Ignite to use for it. So we're proposing
> to
> > > add a new event: QUERY_EXECUITION_EVENT.
> > >
> > > Requirements for the event:
> > > 1. If this event fires it means that a query is correct and will be
> > > executed (and failed only in exceptional cases);
> > >
> > > 2. Event fires for all query types;
> > >
> > > 3. Required fields are:
> > > - text of a query (with hidden arguments);
> > > - arguments of query;
> > > - query type;
> > > - node id.
> > >
> > > Looks that this event should go along with `runningQryMgr::register` in
> > > class `IgniteH2Indexing` as this method invoked for all input queries
> > too.
> > >
> > > What do you think?
> > >
> > > Regards,
> > > Maksim
> > >
> >
>

Re: A few small pull requests

[Stephen Darlington]

This link works for me: https://issues.apache.org/jira/browse/IGNITE-11715 
 

> On 16 Jul 2020, at 16:47, Denis Magda  wrote:
> 
> Hi Stephen,
> 
> Thanks for reminding us about these pending contributions.
> 
> IGNITE-12192 Allow ignitevisorcmd to quit when pressing ^D <
>> https://github.com/apache/ignite/pull/6877>
> 
> 
> Merged the changes. Thanks!
> 
> IGNITE-12182 ExecutorService defaults to only server nodes <
>> https://github.com/apache/ignite/pull/6876>
> 
> 
> I'm afraid we can't merge this change until Ignite 3.0. Otherwise, it will
> break backward compatibility.
> 
> IGNITE-11715 Allow extra options to be passed in to ignite.sh from Docker <
>> https://github.com/apache/ignite/pull/6552>
> 
> 
> Could you confirm that the JIRA ticket is correct? Whenever I try to open
> IGNITE-11715 in JIRA it redirects me to
> https://issues.apache.org/jira/projects/IGNITE/issues/IGNITE-13264
> 
> 
> -
> Denis
> 
> 
> On Wed, Jul 15, 2020 at 6:42 AM Stephen Darlington <
> stephen.darling...@gridgain.com> wrote:
> 
>> Hi,
>> 
>> I have a few small quality-of-life pull requests languishing in the
>> backlog. Can anyone take a look or suggest what else I need to do to
>> progress them?
>> 
>> IGNITE-12192 Allow ignitevisorcmd to quit when pressing ^D <
>> https://github.com/apache/ignite/pull/6877>
>> IGNITE-12182 ExecutorService defaults to only server nodes <
>> https://github.com/apache/ignite/pull/6876>
>> IGNITE-11715 Allow extra options to be passed in to ignite.sh from Docker <
>> https://github.com/apache/ignite/pull/6552>
>> 
>> Thanks,
>> Stephen
>> 
>> 

Re: Re[2]: Apache Ignite 2.9.0 RELEASE [Time, Scope, Manager]

[Ivan Rakov]

Alex,

Tracing is merged to master:
https://issues.apache.org/jira/browse/IGNITE-13060

Can you please port it to 2.9?
For you convenience, there's PR versus 2.9 with conflicts resolved:
https://github.com/apache/ignite/pull/8046/files

--
Best Regards,
Ivan Rakov

On Wed, Jul 15, 2020 at 5:33 PM Alex Plehanov 
wrote:

> Ivan,
>
> Looks like master is broken after IGNITE-13246 (but everything is ok in 2.9
> branch)
>
> ср, 15 июл. 2020 г. в 18:54, Alex Plehanov :
>
> > Zhenya, Ivan,
> >
> > I've cherry-picked IGNITE-13229 and IGNITE-13246 to ignite-2.9 branch.
> > Thank you.
> >
> > ср, 15 июл. 2020 г. в 18:31, Ivan Bessonov :
> >
> >> Guys,
> >>
> >> can you please backport
> >> https://issues.apache.org/jira/browse/IGNITE-13246
> >> to ignite-2.9? Me and Alexey Kuznetsov really want these new events in
> >> release.
> >>
> >> This time I prepared PR with resolved conflicts:
> >> https://github.com/apache/ignite/pull/8042
> >>
> >> Thank you!
> >>
> >> вт, 14 июл. 2020 г. в 19:39, Zhenya Stanilovsky
> >>  >> >:
> >>
> >> >
> >> >
> >> >
> >> > Alex, i also suggest to merge this
> >> > https://issues.apache.org/jira/browse/IGNITE-13229 too, GridClient
> >> > leakage and further TC OOM preventing.
> >> >
> >> > >Ivan,
> >> > >
> >> > >It was already in release scope as discussed in this thread.
> >> > >
> >> > >вт, 14 июл. 2020 г. в 14:31, Ivan Rakov < ivan.glu...@gmail.com >:
> >> > >
> >> > >> Hi,
> >> > >>
> >> > >> We are still waiting for a final review of Tracing functionality
> [1]
> >> > until
> >> > >> the end of tomorrow (July 15).
> >> > >> We anticipate that it will be merged to Ignite master no later than
> >> July
> >> > >> 16.
> >> > >>
> >> > >> Sorry for being a bit late here. Alex P., can you include [1] to
> the
> >> > >> release scope?
> >> > >>
> >> > >> [1]:  https://issues.apache.org/jira/browse/IGNITE-13060
> >> > >>
> >> > >> --
> >> > >> Best Regards,
> >> > >> Ivan Rakov
> >> > >>
> >> > >> On Tue, Jul 14, 2020 at 6:16 AM Alexey Kuznetsov <
> >> > akuznet...@gridgain.com >
> >> > >> wrote:
> >> > >>
> >> > >>> Alex,
> >> > >>>
> >> > >>> Can you cherry-pick to Ignite 2.9 this issue:
> >> > >>>  https://issues.apache.org/jira/browse/IGNITE-13246 ?
> >> > >>>
> >> > >>> This issue is about BASELINE events and it is very useful for
> >> > notification
> >> > >>> external tools about changes in baseline.
> >> > >>>
> >> > >>> Thank you!
> >> > >>>
> >> > >>> ---
> >> > >>> Alexey Kuznetsov
> >> > >>>
> >> > >>
> >> >
> >> >
> >> >
> >> >
> >>
> >>
> >>
> >> --
> >> Sincerely yours,
> >> Ivan Bessonov
> >>
> >
>

Re: A few small pull requests

[Denis Magda]

Weird, works now. Seems to be some browser-related issue on my end.

Overall, thanks a lot for the contribution. I'm suggesting us to merge
IGNITE-11715 into the Ignite 2.9 release branch. Left details in JIRA.

-
Denis


On Thu, Jul 16, 2020 at 9:44 AM Stephen Darlington <
stephen.darling...@gridgain.com> wrote:

> This link works for me: https://issues.apache.org/jira/browse/IGNITE-11715
> 
>
> > On 16 Jul 2020, at 16:47, Denis Magda  wrote:
> >
> > Hi Stephen,
> >
> > Thanks for reminding us about these pending contributions.
> >
> > IGNITE-12192 Allow ignitevisorcmd to quit when pressing ^D <
> >> https://github.com/apache/ignite/pull/6877>
> >
> >
> > Merged the changes. Thanks!
> >
> > IGNITE-12182 ExecutorService defaults to only server nodes <
> >> https://github.com/apache/ignite/pull/6876>
> >
> >
> > I'm afraid we can't merge this change until Ignite 3.0. Otherwise, it
> will
> > break backward compatibility.
> >
> > IGNITE-11715 Allow extra options to be passed in to ignite.sh from
> Docker <
> >> https://github.com/apache/ignite/pull/6552>
> >
> >
> > Could you confirm that the JIRA ticket is correct? Whenever I try to open
> > IGNITE-11715 in JIRA it redirects me to
> > https://issues.apache.org/jira/projects/IGNITE/issues/IGNITE-13264
> >
> >
> > -
> > Denis
> >
> >
> > On Wed, Jul 15, 2020 at 6:42 AM Stephen Darlington <
> > stephen.darling...@gridgain.com> wrote:
> >
> >> Hi,
> >>
> >> I have a few small quality-of-life pull requests languishing in the
> >> backlog. Can anyone take a look or suggest what else I need to do to
> >> progress them?
> >>
> >> IGNITE-12192 Allow ignitevisorcmd to quit when pressing ^D <
> >> https://github.com/apache/ignite/pull/6877>
> >> IGNITE-12182 ExecutorService defaults to only server nodes <
> >> https://github.com/apache/ignite/pull/6876>
> >> IGNITE-11715 Allow extra options to be passed in to ignite.sh from
> Docker <
> >> https://github.com/apache/ignite/pull/6552>
> >>
> >> Thanks,
> >> Stephen
> >>
> >>
>
>
>

Re: IEP-50 Thin Client Continuous Queries

[Igor Sapego]

I've reviewed changes made to IEP and they look good to me.

Best Regards,
Igor


On Wed, Jul 15, 2020 at 1:03 PM Pavel Tupitsyn  wrote:

> Alex,
>
> You are correct, OP_RESOURCE_CLOSE is enough.
> Removed the extra op.
>
> > If client closes CQ it doesn't want to receive any new events. Why can't
> we
> > just ignore events for this CQ after that moment?
> I don't think that our protocol should involve ignoring messages.
> If the client stops the query, the server should guarantee that no events
> will be sent
> to the client after the OP_RESOURCE_CLOSE response.
>
> I had some concerns about this guarantee, but after reviewing GridNioServer
> logic,
> the current implementation with OP_RESOURCE_CLOSE seems to be fine.
>
>
>
> On Wed, Jul 15, 2020 at 10:09 AM Alex Plehanov 
> wrote:
>
> > Pavel,
> >
> > > OP_QUERY_CONTINUOUS_END_NOTIFICATION is another client -> server
> message
> > I think you mean "server -> client" here.
> >
> > But I still didn't get why do we need it.
> > I've briefly looked to the POC implementation and, as far as I
> understand,
> > OP_QUERY_CONTINUOUS_END_NOTIFICATION can be sent only when
> > OP_RESOURCE_CLOSE is received by server (client closes the CQ
> explicitly).
> > If client closes CQ it doesn't want to receive any new events. Why can't
> we
> > just ignore events for this CQ after that moment?
> > Also, in current implementation OP_QUERY_CONTINUOUS_END_NOTIFICATION is
> > sent before OP_RESOURCE_CLOSE response, so OP_RESOURCE_CLOSE response can
> > be used the same way as OP_QUERY_CONTINUOUS_END_NOTIFICATION.
> >
> > Such notification (or something more generalized like OP_RESOURCE_CLOSED)
> > can be helpful if CQ is closed by someone else (for example if
> > administrator call QueryMXBean.cancelContinuous), but AFAIK now we don't
> > have callbacks for this action on user side.
> >
> >
> > ср, 15 июл. 2020 г. в 01:26, Pavel Tupitsyn :
> >
> > > Igniters,
> > >
> > > I've made an important change to the IEP (and the POC):
> > > OP_QUERY_CONTINUOUS_END_NOTIFICATION is another client -> server
> message
> > > that notifies the client that the query has stopped and no more events
> > > should be expected.
> > >
> > > This is important because client can't immediately stop listening
> > > for OP_QUERY_CONTINUOUS_EVENT_NOTIFICATION
> > > after sending OP_RESOURCE_CLOSE - some more events can be present in
> one
> > of
> > > the buffers/queues of the server and/or the OS.
> > >
> > > Let me know if this makes sense.
> > >
> > > On Tue, Jul 14, 2020 at 3:25 PM Pavel Tupitsyn 
> > > wrote:
> > >
> > > > I've removed Initial Query from the POC and IEP (left a note there
> > about
> > > > the decision).
> > > >
> > > > Since there are no other comments and concerns, I'll move on with the
> > > > final implementation.
> > > >
> > > > On Fri, Jul 10, 2020 at 4:14 PM Pavel Tupitsyn  >
> > > > wrote:
> > > >
> > > >> Igor, Alex,
> > > >>
> > > >> I was aware of the duplicates issue with the initial query, but I
> did
> > > not
> > > >> give it a second thought.
> > > >>
> > > >> Now I see that Vladimir was right - Initial query seems to be
> > pointless,
> > > >> since users can
> > > >> achieve the same by simply invoking the regular query.
> > > >>
> > > >> I will remove Initial Query from the IEP and POC next week if there
> > are
> > > >> no objections by then.
> > > >>
> > > >>
> > > >> On Fri, Jul 10, 2020 at 3:58 PM Alex Plehanov <
> > plehanov.a...@gmail.com>
> > > >> wrote:
> > > >>
> > > >>> Igor, Pavel,
> > > >>>
> > > >>> Here is discussion about removal: [1]
> > > >>>
> > > >>> [1] :
> > > >>>
> > > >>>
> > >
> >
> http://apache-ignite-developers.2346864.n4.nabble.com/ContinuousQueryWithTransformer-implementation-questions-2-td21418i20.html#a22041
> > > >>>
> > > >>> пт, 10 июл. 2020 г. в 17:44, Igor Sapego :
> > > >>>
> > > >>> > Can not find proposal to remove them, so maybe it was not on
> > devlist,
> > > >>> > but here is discussion about the problem: [1]
> > > >>> >
> > > >>> > [1] -
> > > >>> >
> > > >>> >
> > > >>>
> > >
> >
> http://apache-ignite-developers.2346864.n4.nabble.com/Continuous-queries-and-duplicates-td39444.html
> > > >>> >
> > > >>> > Best Regards,
> > > >>> > Igor
> > > >>> >
> > > >>> >
> > > >>> > On Fri, Jul 10, 2020 at 3:27 PM Pavel Tupitsyn <
> > ptupit...@apache.org
> > > >
> > > >>> > wrote:
> > > >>> >
> > > >>> > > > What's about "stop" message? How can user unsubscribe from
> > > >>> receiving
> > > >>> > > notifications?
> > > >>> > > OP_RESOURCE_CLOSE is used for that. I've updated the IEP in an
> > > >>> attempt to
> > > >>> > > make this cleaner.
> > > >>> > >
> > > >>> > > >  I've seen discussions on removing initial query from
> > continuous
> > > >>> > queries
> > > >>> > > Interesting, I'm not aware of this. Can you please link those
> > > >>> > discussions?
> > > >>> > >
> > > >>> > > On Fri, Jul 10, 2020 at 2:04 PM Igor Sapego <
> isap...@apache.org>
> > > >>> wrote:
> > > >>> > >
> > > >>> > > > Pavel,
> > > >>> > > >
> > > >>> 

[MTCGA]: new failures in builds [5466890] needs to be handled

[dpavlov . tasks]

Hi Igniters,

 I've detected some new issue on TeamCity to be handled. You are more than 
welcomed to help.

 *New test failure in master 
TcpDiscoveryCoordinatorFailureTest.testClusterFailedNewCoordinatorInitialized 
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8&testNameId=4374507191666502409&branch=%3Cdefault%3E&tab=testDetails
 No changes in the build

 - Here's a reminder of what contributors were agreed to do 
https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute 
 - Should you have any questions please contact dev@ignite.apache.org 

Best Regards,
Apache Ignite TeamCity Bot 
https://github.com/apache/ignite-teamcity-bot
Notification generated at 06:29:23 17-07-2020 

Re: Re[2]: Apache Ignite 2.9.0 RELEASE [Time, Scope, Manager]

[Alex Plehanov]

Ivan,

Merged to 2.9.

Thanks

пт, 17 июл. 2020 г. в 01:35, Ivan Rakov :

> Alex,
>
> Tracing is merged to master:
> https://issues.apache.org/jira/browse/IGNITE-13060
>
> Can you please port it to 2.9?
> For you convenience, there's PR versus 2.9 with conflicts resolved:
> https://github.com/apache/ignite/pull/8046/files
>
> --
> Best Regards,
> Ivan Rakov
>
> On Wed, Jul 15, 2020 at 5:33 PM Alex Plehanov 
> wrote:
>
>> Ivan,
>>
>> Looks like master is broken after IGNITE-13246 (but everything is ok in
>> 2.9
>> branch)
>>
>> ср, 15 июл. 2020 г. в 18:54, Alex Plehanov :
>>
>> > Zhenya, Ivan,
>> >
>> > I've cherry-picked IGNITE-13229 and IGNITE-13246 to ignite-2.9 branch.
>> > Thank you.
>> >
>> > ср, 15 июл. 2020 г. в 18:31, Ivan Bessonov :
>> >
>> >> Guys,
>> >>
>> >> can you please backport
>> >> https://issues.apache.org/jira/browse/IGNITE-13246
>> >> to ignite-2.9? Me and Alexey Kuznetsov really want these new events in
>> >> release.
>> >>
>> >> This time I prepared PR with resolved conflicts:
>> >> https://github.com/apache/ignite/pull/8042
>> >>
>> >> Thank you!
>> >>
>> >> вт, 14 июл. 2020 г. в 19:39, Zhenya Stanilovsky
>> >> > >> >:
>> >>
>> >> >
>> >> >
>> >> >
>> >> > Alex, i also suggest to merge this
>> >> > https://issues.apache.org/jira/browse/IGNITE-13229 too, GridClient
>> >> > leakage and further TC OOM preventing.
>> >> >
>> >> > >Ivan,
>> >> > >
>> >> > >It was already in release scope as discussed in this thread.
>> >> > >
>> >> > >вт, 14 июл. 2020 г. в 14:31, Ivan Rakov < ivan.glu...@gmail.com >:
>> >> > >
>> >> > >> Hi,
>> >> > >>
>> >> > >> We are still waiting for a final review of Tracing functionality
>> [1]
>> >> > until
>> >> > >> the end of tomorrow (July 15).
>> >> > >> We anticipate that it will be merged to Ignite master no later
>> than
>> >> July
>> >> > >> 16.
>> >> > >>
>> >> > >> Sorry for being a bit late here. Alex P., can you include [1] to
>> the
>> >> > >> release scope?
>> >> > >>
>> >> > >> [1]:  https://issues.apache.org/jira/browse/IGNITE-13060
>> >> > >>
>> >> > >> --
>> >> > >> Best Regards,
>> >> > >> Ivan Rakov
>> >> > >>
>> >> > >> On Tue, Jul 14, 2020 at 6:16 AM Alexey Kuznetsov <
>> >> > akuznet...@gridgain.com >
>> >> > >> wrote:
>> >> > >>
>> >> > >>> Alex,
>> >> > >>>
>> >> > >>> Can you cherry-pick to Ignite 2.9 this issue:
>> >> > >>>  https://issues.apache.org/jira/browse/IGNITE-13246 ?
>> >> > >>>
>> >> > >>> This issue is about BASELINE events and it is very useful for
>> >> > notification
>> >> > >>> external tools about changes in baseline.
>> >> > >>>
>> >> > >>> Thank you!
>> >> > >>>
>> >> > >>> ---
>> >> > >>> Alexey Kuznetsov
>> >> > >>>
>> >> > >>
>> >> >
>> >> >
>> >> >
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> Sincerely yours,
>> >> Ivan Bessonov
>> >>
>> >
>>
>