Re: TeamCity Helper's wiki page

[Dmitrii Ryabov]

Thank you, I'll do it today.

2018-09-24 18:19 GMT+03:00 Dmitriy Pavlov :

> Awesome, thanks! Added you to list of contributors.
>
> Feel free to create a new page under https://cwiki.apache.
> org/confluence/display/IGNITE/Testing+and+benchmarking
>
> Sincerely,
> Dmitriy Pavlov
>
> пн, 24 сент. 2018 г. в 15:39, Dmitrii Ryabov :
>
>> I signed up. My username - somefire.
>>
>> 2018-09-24 15:08 GMT+03:00 Dmitriy Pavlov :
>>
>>> Dmitrii,
>>>
>>> could you please sign up to the wiki and share your ID. I will set
>>> necessary permissions.
>>>
>>> Sincerely,
>>> Dmitriy Pavlov
>>>
>>> пн, 24 сент. 2018 г. в 14:57, Dmitrii Ryabov :
>>>
>>> > Hello, Igniters!
>>> >
>>> > I propose to create page about TeamCity Helper on the wiki and move
>>> here
>>> > information about Helper from the MTCGA page.
>>> >
>>> > Also, I want to add instruction about how to use TCH to comment JIRA:
>>> >
>>> > 1. Start "Run All" build for PR on TeamCity.
>>> >
>>> > 2. Check failures and fix possible blockers.
>>> > * Go to https://mtcga.gridgain.com/
>>> > * Fill the form "Check branch/PR" with branch from TeamCity
>>> > ("pull//head"). Leave base branch empty and press
>>> > "Latest" button to compare with the latest master.
>>> >
>>> > 3. Comment JIRA ticket with TeamCity Helper message containing analisys
>>> > results:
>>> > * Go to https://mtcga.gridgain.com/services.html
>>> > * Fill the form "Notify JIRA" with branch from TeamCity (or
>>> > "" only). If PR is named according to contributing
>>> > guide (name starts with "IGNITE-"), you can leave ticket field empty.
>>> > Otherwise - enter JIRA ticket number.
>>> >
>>>
>>
>>

[jira] [Created] (IGNITE-9681) Wrong GIT config in Team City release archive

[Nikolay Izhikov (JIRA)]

Nikolay Izhikov created IGNITE-9681:
---

 Summary: Wrong GIT config in Team City release archive
 Key: IGNITE-9681
 URL: https://issues.apache.org/jira/browse/IGNITE-9681
 Project: Ignite
  Issue Type: Bug
Affects Versions: 2.6
Reporter: Nikolay Izhikov
 Fix For: 2.7


Release archive created by "[Prepare Vote #1] Java &.Net & C++ (Complete 
assembly)" [1] contains wrong .git/config. It includes local Team City path and 
doesn't work properly on release manager local environment.

Example of config file(lfs section is Team City specifi):

{noformat}
[core]
repositoryformatversion = 0
filemode = false
bare = false
logallrefupdates = true
sparseCheckout = true
[remote "origin"]
url = https://git-wip-us.apache.org/repos/asf/ignite
fetch = +refs/heads/*:refs/remotes/origin/*
[lfs]
storage = /data/teamcity/system/git/git-E4D58B67.git/lfs
[branch "master"]
remote = origin
merge = refs/heads/master

{noformat}


[1] 
https://ci.ignite.apache.org/viewType.html?buildTypeId=ApacheIgniteReleaseJava8_PrepareVote



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Page IO statistics for Ignite

[Vladimir Ozerov]

Hi Yuriy,

I think this is great idea. But we need to collect more details on how and
what to collect. I think one of the most interesting parts for us would be
index and data page usages, split by different "dimensions":
1) Global node statistics
2) Per-cache statistics
3) Per-index statistics

We can start with a short summary of what is collected by other database
vendors.

Vladimir.


On Sat, Sep 22, 2018 at 1:07 AM Denis Magda  wrote:

> Hello Yuri,
>
> I might give useful feedback if see how the metrics will look like from the
> API standpoint. If it's not difficult please create a draft.
>
> AS for the interface, in addition to JMX and SQL we need to ensure Visor
> CMD and Web Console gets updated. *Alex K.*, please join the thread and
> share your requirements.
>
> --
> Denis
>
> On Fri, Sep 21, 2018 at 8:16 AM Юрий  wrote:
>
> > Hi Igniters,
> >
> > I started IGNITE-8580
> >  ticket
> > related to print page read/write metrics and did some investigation what
> > other databases provide for the similar purposes.
> >
> > Based on the investigation I want to propose my raw vision of how to
> IGNITE
> > can be more transparent from performance perspective.
> >
> > Need to collect statistics for logical (from memory) and physical (from
> > storage) page reads/writes. All these metrics should be separated by next
> > dimensions:
> > 1) index/cache
> > 2) query level
> > 3) node/cluster
> > ...
> >
> > Seems the statistics should be limited by time.
> >
> > If we will have such statistics we could realize such things as:
> > 1) Get IO statistics per SQL query, global or/and splitted by
> > indexes/caches.
> > 2) Have ability to understand why performance goes down in case it
> related
> > to IO. For example on concrete node or cache.
> > 3) Evaluate effectiveness of use indexes. Find unused indexes.
> > 4) Keep TOP queries with aggressive physical reads
> > 
> >
> >
> > Such statistics could be available least at JMX and SQL interfaces.
> >
> > Let's discuss. In case it will be interested for you I can dig deeper
> into
> > the area and prepare IEP based on our discussion.
> >
> >
> > Igniters, what do you think?
> >
> >
> >
> >
> > --
> > Живи с улыбкой! :D
> >
>

Re: [TC Bot] Proposal of improvement

[Nikolay Izhikov]

Hello, Dmitriy

> What about the case when committer creates ignite-9679 branch and tests it> 
> without PR?

It means, committer is experienced enough to run tests via Team City interface 
:).

> So scanning seems to be possible only in JIRA

I don't understand you here.
You can retrieve comments filtered by *date*.
You don't have to scan all 1000 PR's one by one.
Anyway 1000 PR doesn't sound like big issue for me.

My vote goes strong to GiHub user interface.
I think we should have closer integration with GitHub, not Jira.

Jira is about tickets and project management.
GitHub is about code, commits and patches.
We test patch, not ticket.


В Вт, 25/09/2018 в 00:06 +0300, Dmitriy Pavlov пишет:
> Hi Nikolay,
> 
> What about the case when committer creates ignite-9679 branch and tests it
> without PR?
> 
> We have 1100+ open PRs and less than 100 open tickets. So scanning seems to
> be possible only in JIRA. Mention probably will work for GitHub, but it
> needs to be researched.
> 
> Two open PRs is not a valid situation in the majority of cases and How To
> Contribute asks to avoid it. The bot can ignore closed PRs and the bot can
> expect there is only one open PR per ticket.
> 
> Sincerely,
> Dmitriy Pavlov
> 
> пн, 24 сент. 2018 г. в 23:41, Nikolay Izhikov :
> 
> > Hello, Dmitriy.
> > 
> > > But it could be a lot of work to handle mentions (probably from the
> > 
> > email> account and inbox).
> > 
> > Actually, it can be done via GitHub REST API [1].
> > It has 'since' param, so getting new GitHub comments is a very basic task.
> > 
> > > Patch available ticket
> > 
> > I think we shouldn't take a ticket as an entity that should be tested.
> > For me, it's a PR.
> > 
> > Moreover, it's a common case when we have several PR in a ticket.
> > And it's a common case when both of them has to be tested.
> > 
> > My vote goes to the closer integration with GitHub.
> > 
> > [1]
> > https://developer.github.com/v3/pulls/comments/#list-comments-in-a-repository
> > 
> > В Пн, 24/09/2018 в 22:36 +0300, Dmitriy Pavlov пишет:
> > > Hi Nikolay,
> > > 
> > > The idea makes perfect sense for me, and we should definitely take the
> > 
> > best
> > > practices from other big Apache projects.
> > > 
> > > But it could be a lot of work to handle mentions (probably from the email
> > > account and inbox).
> > > 
> > > I would like to suggest the following algorithm:
> > > 
> > > Patch available ticket, which was never checked by the bot will be
> > > processed in the following steps:
> > > 1. check existing run all (by PR or by branch name), if found go to the
> > > step 3
> > > 2. run-all to be triggered by PR
> > > 3. results should be analyzed for the presence of possible blockers. If
> > > there is no blockers go to step 5.
> > > 4. re-run of particular suites containing possible blockers should be
> > > applied to try to get success for very rare flaky failures (<1%). Go to 3
> > > (this go to should be done only once).
> > > 5. comment should be added to JIRA ticket containing information about
> > > results.
> > > 
> > > If a ticket was processed by bot early (probably author added some fixes)
> > > but still in PA state, the bot will check comments list and find possible
> > > new mentions (made after the previous build complete date). If it finds
> > > such comments it goes to step 1 (trying to find only new builds
> > 
> > available).
> > > 
> > > What do you think?
> > > 
> > > Sincerely,
> > > Dmitriy Pavlov
> > > 
> > > пн, 24 сент. 2018 г. в 21:43, Nikolay Izhikov :
> > > 
> > > > Hello, Igniters.
> > > > 
> > > > I propose to implement following behaviour:
> > > > 
> > > > 1. Execute "Run all" suite for specific PR when the author of PR makes
> > 
> > a
> > > > comment
> > > > "@mtcga.bot Run Tests!" in GitHub comments.
> > > > 
> > > > 2. Send a comment with "Run All" results both: to a Jira ticket and
> > 
> > GitHub
> > > > comment.
> > > > 
> > > > 3. Label PR based on "Run All" results like it done in Apache Kafka [1]
> > > > 
> > > > I've create ticket for this proposal [2]
> > > > 
> > > > Thoughts?
> > > > 
> > > > [1] https://github.com/apache/kafka/pulls
> > > > [2] https://issues.apache.org/jira/browse/IGNITE-9678

signature.asc
Description: This is a digitally signed message part

[GitHub] ignite pull request #4806: IGNITE-9675: Deadlock on Ignite:active() and stop...

[avplatonov]

Github user avplatonov closed the pull request at:

https://github.com/apache/ignite/pull/4806


---

[GitHub] ignite pull request #4822: IGNITE-9675: Deadlock on Ignite:active() and stop...

[avplatonov]

GitHub user avplatonov opened a pull request:

https://github.com/apache/ignite/pull/4822

IGNITE-9675: Deadlock on Ignite:active() and stopping grid simultaneously 
calling



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-gg-14215

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4822.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4822


commit 987df83cdec0049259d4b86c3a9df1d991205c26
Author: Alexey Platonov 
Date:   2018-09-21T14:51:48Z

add activeAsync iface

commit 9234c433e4d4911be4c21215c54c24b1c2d9cbaf
Author: Alexey Platonov 
Date:   2018-09-21T14:51:55Z

Merge branch 'master' of https://github.com/apache/ignite into gg-14215-1

commit d32233a1050fb88e06aa2005d01ebeb3ec5750f6
Author: Alexey Platonov 
Date:   2018-09-21T15:02:03Z

avoid extra operation when awaitForTransition=false

commit f143f38538ce4b354d60e373caa30c9f49e21b60
Author: Alexey Platonov 
Date:   2018-09-24T15:06:55Z

avoid extra operation when awaitForTransition=false




---

[jira] [Created] (IGNITE-9682) Update full partition map in parallel

[Oleg Ostanin (JIRA)]

Oleg Ostanin created IGNITE-9682:


 Summary: Update full partition map in parallel
 Key: IGNITE-9682
 URL: https://issues.apache.org/jira/browse/IGNITE-9682
 Project: Ignite
  Issue Type: Improvement
Reporter: Oleg Ostanin
Assignee: Oleg Ostanin


I've noticed that we are updating full partition map for each cache 
sequentially and at the same time we cannot run much of other tasks before the 
updating is completed, so there must be some threads available for parallel 
updating. Sometimes it takes 20-30 seconds to update map for 100+ caches before 
rebalancing can even start:

[2018-09-24 16:38:49,016][INFO ][sys-#219] Received full message, will finish 
exchange [node=4d6b5921-0a61-4bfa-bf11-b8a1b8332dce, 
resVer=AffinityTopologyVersion [topVer=19, minorTopVer=2]]

...

[2018-09-24 16:39:01,217][INFO ][sys-#219] Finish exchange future 
[startVer=AffinityTopologyVersion [topVer=19, minorTopVer=2], 
resVer=AffinityTopologyVersion [topVer=19, minorTopVer=2], err=null]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ignite pull request #4823: Ignite-8580

[ygerzhedovich]

GitHub user ygerzhedovich opened a pull request:

https://github.com/apache/ignite/pull/4823

Ignite-8580



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-8580

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4823.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4823


commit ead1ebc782a563992d7ce02a7327ab59f328f3c1
Author: Yury Gerzhedovich 
Date:   2018-09-25T09:09:34Z

ignite-8580: Print page read/write metrics splitted by type

commit 5154940d4d62a9c30a9669b40f663869824b2922
Author: Yury Gerzhedovich 
Date:   2018-09-25T09:11:29Z

Merge branch 'master' into ignite-8580




---

[GitHub] ignite pull request #4824: IGNITE-9682 changed updatePartitionFullMap() meth...

[oleg-ostanin]

GitHub user oleg-ostanin opened a pull request:

https://github.com/apache/ignite/pull/4824

IGNITE-9682 changed updatePartitionFullMap() method to update map in …

…parallel

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-9682

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4824.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4824


commit ab31afe6a234db08b5455b3b40f64d895f4c614b
Author: oleg-ostanin 
Date:   2018-09-25T09:40:25Z

IGNITE-9682 changed updatePartitionFullMap() method to update map in 
parallel




---

Re: [TC Bot] Proposal of improvement

[Dmitriy Pavlov]

JIRA ticket is some elementary change that needs to be reviewed. We don't
review the patch, we review ticket (with motivation, implementation
details, history of discussion), so reviewer will look at a ticket first.

PR does not have a mark, that it is ready to be merged. Some PRs are
created just for research, but Patch Available ticket is something that is
ready to be in the product.

So if we concentrate on a ticket, from the point of view of a new
contributor,
- he or she creates a branch, PR and sets ticket to PA,
- and the bot will do all necessary mechanic work.

No issues with asking newcomers to run (proper) tests. A newbie needs only
to follow the first steps in How To Contribute. A reviewer will see a
ticket with the bot Visa after 2-3 hours after setting of PA state.

But only one concern I have here, I'm not sure I have spare cycles to do
this project. I'd rather move towards it in step-by-step mode, doing small
changes in each version. Any assistance is appreciated here.

вт, 25 сент. 2018 г. в 11:25, Nikolay Izhikov :

> Hello, Dmitriy
>
> > What about the case when committer creates ignite-9679 branch and tests
> it> without PR?
>
> It means, committer is experienced enough to run tests via Team City
> interface :).
>
> > So scanning seems to be possible only in JIRA
>
> I don't understand you here.
> You can retrieve comments filtered by *date*.
> You don't have to scan all 1000 PR's one by one.
> Anyway 1000 PR doesn't sound like big issue for me.
>
> My vote goes strong to GiHub user interface.
> I think we should have closer integration with GitHub, not Jira.
>
> Jira is about tickets and project management.
> GitHub is about code, commits and patches.
> We test patch, not ticket.
>
>
> В Вт, 25/09/2018 в 00:06 +0300, Dmitriy Pavlov пишет:
> > Hi Nikolay,
> >
> > What about the case when committer creates ignite-9679 branch and tests
> it
> > without PR?
> >
> > We have 1100+ open PRs and less than 100 open tickets. So scanning seems
> to
> > be possible only in JIRA. Mention probably will work for GitHub, but it
> > needs to be researched.
> >
> > Two open PRs is not a valid situation in the majority of cases and How To
> > Contribute asks to avoid it. The bot can ignore closed PRs and the bot
> can
> > expect there is only one open PR per ticket.
> >
> > Sincerely,
> > Dmitriy Pavlov
> >
> > пн, 24 сент. 2018 г. в 23:41, Nikolay Izhikov :
> >
> > > Hello, Dmitriy.
> > >
> > > > But it could be a lot of work to handle mentions (probably from the
> > >
> > > email> account and inbox).
> > >
> > > Actually, it can be done via GitHub REST API [1].
> > > It has 'since' param, so getting new GitHub comments is a very basic
> task.
> > >
> > > > Patch available ticket
> > >
> > > I think we shouldn't take a ticket as an entity that should be tested.
> > > For me, it's a PR.
> > >
> > > Moreover, it's a common case when we have several PR in a ticket.
> > > And it's a common case when both of them has to be tested.
> > >
> > > My vote goes to the closer integration with GitHub.
> > >
> > > [1]
> > >
> https://developer.github.com/v3/pulls/comments/#list-comments-in-a-repository
> > >
> > > В Пн, 24/09/2018 в 22:36 +0300, Dmitriy Pavlov пишет:
> > > > Hi Nikolay,
> > > >
> > > > The idea makes perfect sense for me, and we should definitely take
> the
> > >
> > > best
> > > > practices from other big Apache projects.
> > > >
> > > > But it could be a lot of work to handle mentions (probably from the
> email
> > > > account and inbox).
> > > >
> > > > I would like to suggest the following algorithm:
> > > >
> > > > Patch available ticket, which was never checked by the bot will be
> > > > processed in the following steps:
> > > > 1. check existing run all (by PR or by branch name), if found go to
> the
> > > > step 3
> > > > 2. run-all to be triggered by PR
> > > > 3. results should be analyzed for the presence of possible blockers.
> If
> > > > there is no blockers go to step 5.
> > > > 4. re-run of particular suites containing possible blockers should be
> > > > applied to try to get success for very rare flaky failures (<1%). Go
> to 3
> > > > (this go to should be done only once).
> > > > 5. comment should be added to JIRA ticket containing information
> about
> > > > results.
> > > >
> > > > If a ticket was processed by bot early (probably author added some
> fixes)
> > > > but still in PA state, the bot will check comments list and find
> possible
> > > > new mentions (made after the previous build complete date). If it
> finds
> > > > such comments it goes to step 1 (trying to find only new builds
> > >
> > > available).
> > > >
> > > > What do you think?
> > > >
> > > > Sincerely,
> > > > Dmitriy Pavlov
> > > >
> > > > пн, 24 сент. 2018 г. в 21:43, Nikolay Izhikov :
> > > >
> > > > > Hello, Igniters.
> > > > >
> > > > > I propose to implement following behaviour:
> > > > >
> > > > > 1. Execute "Run all" suite for specific PR when the author of PR
> makes
> > >
> > > a
> > > > > comment
>

[GitHub] ignite pull request #4769: IGNITE-9501 Exchange latch optimizations

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/4769


---

[GitHub] SomeFire opened a new pull request #16: IGNITE-9668 Comment JIRA from pr.html

[GitBox]

SomeFire opened a new pull request #16: IGNITE-9668 Comment JIRA from pr.html
URL: https://github.com/apache/ignite-teamcity-bot/pull/16
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (IGNITE-9683) Create manual pinger for ZK client

[Pavel Kovalenko (JIRA)]

Pavel Kovalenko created IGNITE-9683:
---

 Summary: Create manual pinger for ZK client
 Key: IGNITE-9683
 URL: https://issues.apache.org/jira/browse/IGNITE-9683
 Project: Ignite
  Issue Type: Improvement
  Components: cache
Affects Versions: 2.5
Reporter: Pavel Kovalenko
 Fix For: 2.8


Connection loss with Zookeeper more than ZK session timeout for server nodes is 
unacceptable. To improve durability of connrction, we need to keep session with 
ZK as long possible. We need to introduce manual pinger additionally to ZK 
client  and ping ZK server with simple request each tick time.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ignite pull request #4449: IGNITE-8559 extract segment index storage to out ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/4449


---

[jira] [Created] (IGNITE-9684) JDK9: pass JVM options to created process at HadoopCommandLineTest#createProcessBuilder

[Taras Ledkov (JIRA)]

Taras Ledkov created IGNITE-9684:


 Summary: JDK9: pass JVM options to created process at 
HadoopCommandLineTest#createProcessBuilder
 Key: IGNITE-9684
 URL: https://issues.apache.org/jira/browse/IGNITE-9684
 Project: Ignite
  Issue Type: Task
Reporter: Taras Ledkov
Assignee: Taras Ledkov


To support JDK9 the JVM options must be passed to created process: 
see {{HadoopCommandLineTest#createProcessBuilder}}
Because lauched process fails with

IllegalAccessException: class org.apache.ignite.internal.util.GridUnsafe cannot 
access class jdk.internal.misc.SharedSecrets (in module java.base) because 
module java.base does not export jdk.internal.misc to unnamed module 




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-9685) [ML] Add ignite-tensorflow module to build artifacts

[Yury Babak (JIRA)]

Yury Babak created IGNITE-9685:
--

 Summary: [ML] Add ignite-tensorflow module to build artifacts
 Key: IGNITE-9685
 URL: https://issues.apache.org/jira/browse/IGNITE-9685
 Project: Ignite
  Issue Type: New Feature
  Components: ml
Reporter: Yury Babak
Assignee: Peter Ivanov
 Fix For: 2.7


We want to release Apache Ignite TensorFlow Integration Module with other 
Ignite tools.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-9686) JDK9: pass JVM options to new process when start remote test node

[Taras Ledkov (JIRA)]

Taras Ledkov created IGNITE-9686:


 Summary: JDK9: pass JVM options to new process when start remote 
test node
 Key: IGNITE-9686
 URL: https://issues.apache.org/jira/browse/IGNITE-9686
 Project: Ignite
  Issue Type: Task
Affects Versions: 2.6
Reporter: Taras Ledkov
Assignee: Taras Ledkov


The JVM options must be passed to new process when remote test node is started.
See {{GridAbstractTest#startRemoteGrid}}

Affects tests:
{code}
 IgniteHadoopFileSystemClientBasedOpenTest
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-9687) JDK9: JTA tests failed

[Taras Ledkov (JIRA)]

Taras Ledkov created IGNITE-9687:


 Summary: JDK9: JTA tests failed
 Key: IGNITE-9687
 URL: https://issues.apache.org/jira/browse/IGNITE-9687
 Project: Ignite
  Issue Type: Task
Affects Versions: 2.6
Reporter: Taras Ledkov
Assignee: Taras Ledkov


JTA tests fail on JDK9 with error:
{{java.lang.NoClassDefFoundError: javax/rmi/PortableRemoteObject}}
the option {{--add-modules java.se.ee}}
changes the error to:
{{java.lang.NoClassDefFoundError: javax/transaction/UserTransaction}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ignite pull request #4788: IGNITE-9514: Refactor tests and common test time

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/4788


---

[GitHub] ignite pull request #4825: Ignite 8559 2.5

[akalash]

GitHub user akalash opened a pull request:

https://github.com/apache/ignite/pull/4825

Ignite 8559 2.5



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-8559-2.5

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4825.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4825


commit 82cbf44f745dffbb0bc1cd5348a1bb61af742a92
Author: Sergey Chugunov 
Date:   2018-06-06T14:11:12Z

GG-13865 filling wal segments with non-zero values

commit 918f748879182dbfe2a96681775ebddc7504e46d
Author: Alexey Stelmak 
Date:   2018-06-07T09:51:57Z

gg-13865 JVM crash if there is no space for wal storage

commit 5d47fed8c0d6237be13172552b48b113632deb73
Author: Alexey Stelmak 
Date:   2018-06-07T10:13:03Z

gg-13865 JVM crash if there is no space for wal storage

commit 1da633da7e83ddab92f13372cdf22695b53141c8
Author: Alexey Stelmak 
Date:   2018-06-09T12:37:51Z

Add failure handler

commit 2e0af995b1a377349b696673a5bab640324881de
Author: Alexey Stelmak 
Date:   2018-06-14T12:56:38Z

ignite-8748 Review fix

commit e9e274e8999e176a8d6fb9778dadfa11398b0c13
Author: Alexey Stelmak 
Date:   2018-06-14T14:34:37Z

ignite-8748 Review fix

commit 550df383d09b56ea570dd63d0361228a325e1b20
Author: Alexey Stelmak 
Date:   2018-06-14T15:03:24Z

ignite-8748 Code style fix

commit 59cf87ea0f021400f33df7cd82cc94b79263e13d
Author: Alexey Stelmak 
Date:   2018-06-14T15:14:17Z

ignite-8748 Use configured failure handler

commit 1ce344098e8bfd3031d8e3afde74efdb115d66bf
Author: Alexey Stelmak 
Date:   2018-06-15T18:03:34Z

ignite-8740 Return

commit d4c6a8be200db83a62b7b4679511b377fc1879f9
Author: Andrey Kuznetsov 
Date:   2018-06-18T09:47:25Z

IGNITE-8562 Turn system-critical Ignite threads into GridWorkers

Signed-off-by: agura 

commit f071b67bfebd2a7a1ac11f890cc009560fb66321
Author: Dmitriy Sorokin 
Date:   2018-06-18T10:48:11Z

IGNITE-8749 Exception for no space left situation should be propagated to 
FailureHandler.

Signed-off-by: agura 

commit 914dbbb0852bab40ddeae36bf6c68e7101f69877
Author: Ivan Rakov 
Date:   2018-06-18T12:07:48Z

IGNITE-8757 idle_verify utility doesn't show both update counter and hash 
conflicts

(cherry picked from commit 9131e4d)

commit 80c7cecd146fca25e71c621e3b18086271937881
Author: Anton Kalashnikov 
Date:   2018-06-18T15:53:43Z

IGNITE-8707 DataStorageMetrics.getTotalAllocatedSize metric does not 
account reserved partition page header - Fixes #4202.

Signed-off-by: Ivan Rakov 

(cherry picked from commit ceade87)

commit 2cb10a7ccbf18d7893450a0529f95743c0e395df
Author: Sergey Kosarev 
Date:   2018-06-18T15:48:05Z

IGNITE-8815 Ease enabling WAL management in control.sh - Fixes #4212.

Signed-off-by: Alexey Goncharuk 
(cherry picked from commit b80482d125d08a4779c8f7b1ea75e4dd1a4c4bde)

commit 1d34cf3f4ce3880b4ce3ca8829b49d08710773e2
Author: devozerov 
Date:   2018-06-15T08:43:01Z

IGNITE-8800: Binary: print type name and existing schema IDs in case of 
exception due to missing schema. This closes #4190.

(cherry picked from commit 7a39efb)

commit 2d554b704a365023a5c910d312786de8286cf219
Author: a-polyakov 
Date:   2018-06-18T16:11:32Z

IGNITE-8601 Add to control.sh utility information about transaction start 
time - Fixes #4070.

(cherry picked from commit 4776a1a)

commit 84582af1df76063ff66be1a2ad2049b09d45e393
Author: Andrey Gura 
Date:   2018-06-15T14:12:09Z

IGNITE-8789 Invoke failure processor in case of message processing error

commit e4f6a4fa8f65d7cea2a67e980c4d408591896eb8
Author: Ivan Daschinskiy 
Date:   2018-06-18T17:44:00Z

IGNITE-8755 Throw a correct error message when client optimized marshaller 
is trying to serialize too large object

(cherry picked from commit 61b3897)

commit 8779e1180aa20e67de0afc55c920cd713e5bef52
Author: Ilya Lantukh 
Date:   2018-04-26T15:31:47Z

GG-13652 PITR : handle situation when WAL got temporarily disabled for 
rebalancing

Signed-off-by: EdShangGG 

(cherry picked from commit 92dc88d)
Signed-off-by: EdShangGG 

commit c89a60089026df0cfa0fa83376986e2b278ade16
Author: Ivan Rakov 
Date:   2018-06-19T09:38:24Z

IGNITE-8749 Exception for no space left situation should be propagated to 
FailureHandler - rollback

commit 802ddafd3113421aef5cc2159521868ed80e302f
Author: Dmitriy Sorokin 
Date:   2018-06-19T12:35:51Z

IGNITE-8769 JVM crash in Basic1 suite in master branch on TC - Fixes #4206.

Signed-off-by: Ivan Rakov 

(cherry picked from commit b37f8a2)

commit 77f857e15e59535d5715e0022e9d8caf38ef8bee
Author: Alexander Menshikov 
Date:   2018-04-25T11:03:24Z

IGNITE-6565 Use long type for size and keySize in cache metr

[GitHub] asfgit closed pull request #9: IGNITE-9541 Add the comparison for two general statistics "RunAll" for master in the date interval

[GitBox]

asfgit closed pull request #9: IGNITE-9541 Add the comparison for two general 
statistics "RunAll" for master in the date interval
URL: https://github.com/apache/ignite-teamcity-bot/pull/9
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/ITeamcity.java 
b/ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/ITeamcity.java
index 6be0446..4151793 100644
--- a/ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/ITeamcity.java
+++ b/ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/ITeamcity.java
@@ -18,6 +18,7 @@
 package org.apache.ignite.ci;
 
 import java.io.File;
+import java.util.Date;
 import java.util.List;
 import java.util.concurrent.CompletableFuture;
 import java.util.concurrent.ExecutorService;
@@ -54,6 +55,7 @@
 public interface ITeamcity extends AutoCloseable {
 
 String DEFAULT = "";
+
 long DEFAULT_BUILDS_COUNT = 1000;
 
 CompletableFuture> getProjectSuites(String projectId);
@@ -61,45 +63,45 @@
 String serverId();
 
 /**
- * @param projectId suite ID (string without spaces)
- * @param branch
- * @return list of builds in historical order, recent builds coming last
+ * @param projectId Suite ID (string without spaces).
+ * @param branch Branch in TC identification.
+ * @return List of builds in historical order, recent builds coming last.
  */
 default List getFinishedBuilds(String projectId, String branch) {
-return getFinishedBuilds(projectId, branch, null, null);
+return getFinishedBuilds(projectId, branch, null, null, null);
 };
 
 /**
  * @param projectId suite ID (string without spaces).
  * @param branch Branch name in TC identification.
- * @param cnt builds count.
+ * @param sinceDate Since date.
+ * @param untilDate Until date.
  * @param sinceBuildId Some build ID in the past to to use as minimal 
build to export.
  * @return list of builds in historical order, recent builds coming last.
  */
-List getFinishedBuilds(String projectId, String branch, Long 
cnt, Integer sinceBuildId);
+List getFinishedBuilds(String projectId, String branch, Date 
sinceDate, Date untilDate, Integer sinceBuildId);
 
 /**
- * Includes snapshot dependencies failed builds into list
+ * Includes snapshot dependencies failed builds into list.
  *
- * @param projectId suite ID (string without spaces)
- * @param branch branch in TC identification
- * @return list of builds in historical order, recent builds coming last
+ * @param projectId suite ID (string without spaces).
+ * @param branch branch in TC identification.
+ * @return list of builds in historical order, recent builds coming last.
  */
 default List getFinishedBuildsIncludeSnDepFailed(String 
projectId, String branch){
-return getFinishedBuildsIncludeSnDepFailed(projectId, branch, null, 
null);
+return getFinishedBuildsIncludeSnDepFailed(projectId, branch, null);
 };
 
 /**
  * Includes 'snapshot dependencies failed' builds into list.
  * loads build history with following parameter: 
defaultFilter:false,state:finished
  *
- * @param projectId suite ID (string without spaces)
- * @param branch branch in TC identification
- * @param cnt builds count
- * @param sinceBuildId limit builds export with some build number, not 
operational for Persistent connection
- * @return list of builds in historical order, recent builds coming last
+ * @param projectId suite ID (string without spaces).
+ * @param branch branch in TC identification.
+ * @param sinceBuildId limit builds export with some build number, not 
operational for Persistent connection.
+ * @return list of builds in historical order, recent builds coming last.
  */
-List getFinishedBuildsIncludeSnDepFailed(String projectId, 
String branch, Long cnt, Integer sinceBuildId);
+List getFinishedBuildsIncludeSnDepFailed(String projectId, 
String branch, Integer sinceBuildId);
 
 /**   */
 CompletableFuture> getRunningBuilds(@Nullable String 
branch);
@@ -107,8 +109,24 @@
 /**   */
 CompletableFuture> getQueuedBuilds(@Nullable String branch);
 
-default int[] getBuildNumbersFromHistory(String projectId, String 
branchNameForHist, Long cnt) {
-return getFinishedBuilds(projectId, branchNameForHist, cnt, 
null).stream().mapToInt(BuildRef::getId).toArray();
+/**
+ * @param projectId Suite ID (string without spaces).
+ * @param branchNameForHist Branch in TC identification.
+ * @return List of build numbers in historical order, recent builds coming 
last.
+ */
+default int[] getBuildNumbersFromHistory(String p

[jira] [Created] (IGNITE-9688) MVCC: Implement out-of-order enlist optimization for bulk cache operations.

[Andrew Mashenkov (JIRA)]

Andrew Mashenkov created IGNITE-9688:


 Summary: MVCC: Implement out-of-order enlist optimization for bulk 
cache operations.
 Key: IGNITE-9688
 URL: https://issues.apache.org/jira/browse/IGNITE-9688
 Project: Ignite
  Issue Type: Improvement
  Components: cache, mvcc
Reporter: Andrew Mashenkov


For now, we always enlist updates in given order via setting 
"GridNearTxEnlistFuture.sequential" flag to true.
This flag is always true for query updates as we do not know full data set at a 
time future has been created.
For putAll (and other batch operations) full update map is known and we can 
make per-node batches full filled before send.

E.g. we can send batches to nodes one by one regarding primary node order.
This optimization has to be disscussed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ignite pull request #4826: IGNITE-8146: JDK9: fix gathering class loader's U...

[tledkov-gridgain]

GitHub user tledkov-gridgain opened a pull request:

https://github.com/apache/ignite/pull/4826

IGNITE-8146: JDK9: fix gathering class loader's URLs



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-8146

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4826.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4826


commit 8424d7c66972985dbfea57943c97bc77b4d12f30
Author: tledkov-gridgain 
Date:   2018-09-25T13:10:17Z

IGNITE-8146: JDK9: fix gathering class loader's URLs




---

[GitHub] asfgit closed pull request #16: IGNITE-9668 Comment JIRA from pr.html

[GitBox]

asfgit closed pull request #16: IGNITE-9668 Comment JIRA from pr.html
URL: https://github.com/apache/ignite-teamcity-bot/pull/16
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/TcHelper.java 
b/ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/TcHelper.java
index e1b709e..0adf3fb 100644
--- a/ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/TcHelper.java
+++ b/ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/TcHelper.java
@@ -189,10 +189,7 @@ private BranchesTracked getTrackedBranches() {
 return false;
 }
 
-if ("finished".equals(build.state))
-return teamcity.sendJiraComment(ticket, comment);
-
-return false;
+return teamcity.sendJiraComment(ticket, comment);
 }
 }
 
diff --git 
a/ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/web/rest/TriggerBuild.java
 
b/ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/web/rest/TriggerBuild.java
index f5619ef..a4c66e6 100644
--- 
a/ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/web/rest/TriggerBuild.java
+++ 
b/ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/web/rest/TriggerBuild.java
@@ -98,7 +98,6 @@ public SimpleResult commentJira(
 
 @NotNull
 public SimpleResult commentJiraEx(@QueryParam("serverId") @Nullable String 
srvId, @QueryParam("branchName") @Nullable String branchForTc, 
@QueryParam("suiteId") @Nullable String suiteId, @QueryParam("ticketId") 
@Nullable String ticketId) {
-System.out.println("commentJira ");
 final ICredentialsProv prov = ICredentialsProv.get(req);
 
 if (!prov.hasAccess(srvId))
diff --git a/ignite-tc-helper-web/src/main/webapp/js/testfails-2.1.js 
b/ignite-tc-helper-web/src/main/webapp/js/testfails-2.1.js
index 67d5c18..7974fca 100644
--- a/ignite-tc-helper-web/src/main/webapp/js/testfails-2.1.js
+++ b/ignite-tc-helper-web/src/main/webapp/js/testfails-2.1.js
@@ -35,6 +35,10 @@ class Settings {
 isGithubAvailable() {
 return this.javaFlags & 2
 };
+
+isJiraAvailable() {
+return this.javaFlags & 4
+};
 }
 
 //@param results - TestFailuresSummary
@@ -159,13 +163,22 @@ function showChainCurrentStatusData(server, settings) {
 }
 
 res += "";
-if (settings.isGithubAvailable()) {
-g_srv_to_notify_git = server;
-res += "Update PR status";
+
+// if (settings.isGithubAvailable()) {
+// g_srv_to_notify_git = server;
+// res += "Update PR status";
+// }
+
+if (settings.isJiraAvailable()) {
+res += "Comment JIRA";
 }
 
 if (isDefinedAndFilled(server.baseBranchForTc)) {
-if (settings.isGithubAvailable())
+// if (settings.isGithubAvailable())
+// res+="";
+
+if (settings.isJiraAvailable())
 res+="";
 
 res += "Base branch";
@@ -409,19 +422,43 @@ function commentJira(serverId, suiteId, branchName, 
ticketId) {
 "branchName": branchName,
 "ticketId": ticketId
 },
-success: function(result) {$("#notifyJira").html("");
+success: function(result) {
+$("#notifyJira").html("");
+
+var needTicketId = result.result.lastIndexOf("enter ticket id") 
!== -1;
+
+if (needTicketId) {
+var buttons = {
+"Retry": function () {
+$(this).dialog("close");
+
+ticketId = $("#enterTicketId").val();
+
+commentJira(serverId, suiteId, branchName, ticketId)
+},
+"Cancel": function () {
+$(this).dialog("close");
+}
+}
+}
+else {
+buttons = {
+"Ok": function () {
+$(this).dialog("close");
+}
+}
+}
+
 var dialog = $("#triggerDialog");
 
 dialog.html("Trigger builds at server: " + serverId + "" +
-" Suite: " + suiteId + "Branch:" + branchName + "Top: 
" +
-" Result: " + result.result);
+" Suite: " + suiteId + "Branch:" + branchName +
+" Result: " + result.result +
+(needTicketId ? ("Enter JIRA ticket number: ") : ""));
+
 dialog.dialog({
 modal: true,
-buttons: {
-"Ok": function() {
-$(this).dialog("close");
-}
-}
+buttons: buttons
 });
 
 loadData(); // should be defined by page

[jira] [Created] (IGNITE-9689) MVCC: Optimize filter usage in MvccUpdateDataRow.

[Andrew Mashenkov (JIRA)]

Andrew Mashenkov created IGNITE-9689:


 Summary: MVCC: Optimize filter usage in MvccUpdateDataRow.
 Key: IGNITE-9689
 URL: https://issues.apache.org/jira/browse/IGNITE-9689
 Project: Ignite
  Issue Type: Improvement
Reporter: Andrew Mashenkov


PutIfAbsent and all Replace operation uses filter for previous values checks.

When filter has provided then we have to retrieve full row (instead of header) 
just to apply the filter.
However, in most of cases filter doesn't need a value itself, but just a fact 
if previous value exists.

There is unused class 
org.apache.ignite.internal.processors.cache.CacheOperationFilter enum that can 
be used for optimization. We can just compare filter type and visitor 
resultType to make a decision in CacheDataStore.mvccUpdate\mvccRemove methods.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-9690) MVCC: Check mvcc operations behavior with LOST partitions.

[Andrew Mashenkov (JIRA)]

Andrew Mashenkov created IGNITE-9690:


 Summary: MVCC: Check mvcc operations behavior with LOST partitions.
 Key: IGNITE-9690
 URL: https://issues.apache.org/jira/browse/IGNITE-9690
 Project: Ignite
  Issue Type: Task
  Components: cache, mvcc
Reporter: Andrew Mashenkov
 Fix For: 2.8


We have to analyze mvcc behaviour in case of lost partition for different lost 
policies  for both cases SQL and KV API.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-9691) AuthenticationProcessorNodeRestartTest.testConcurrentAuthorize uses outdated assumption about exception message

[Oleg Ignatenko (JIRA)]

Oleg Ignatenko created IGNITE-9691:
--

 Summary: 
AuthenticationProcessorNodeRestartTest.testConcurrentAuthorize uses outdated 
assumption about exception message
 Key: IGNITE-9691
 URL: https://issues.apache.org/jira/browse/IGNITE-9691
 Project: Ignite
  Issue Type: Bug
Affects Versions: 2.6
Reporter: Oleg Ignatenko


Test {{AuthenticationProcessorNodeRestartTest.testConcurrentAuthorize}} that 
was introduced per IGNITE-7436 uses particular assumption about exception 
message thrown from method 
[GridIoManager.send|https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/managers/communication/GridIoManager.java]:
{code}
// Skip exception if server down.
if (!e.getMessage().contains("Failed to send message (node 
may have left the grid or "
+ "TCP connection cannot be established due to firewall 
issues)")) {
e.printStackTrace();
fail("Unexpected exception: " + e.getMessage());
}
// ...{code}

This expectation appears to be broken by changes introduced per IGNITE-4191 
which added yet another exception message that may occur in above piece of test 
code:
{code}
if (!ctx.discovery().alive(node))
throw new ClusterTopologyCheckedException("Failed to send 
message, node left: " + node.id(), e);{code}
(above code was added at line 1664 in {{GridIoManager.java}})

Regression wasn't immediately discovered because of indeterministic test 
scenario which made new failures appear randomly and mixed with passes when 
particular condition was missed in the course of test execution.

Test needs to be updated to accommodate the changes in codebase.




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-9692) Cache creation request may be missed on coordinator change

[Vyacheslav Daradur (JIRA)]

Vyacheslav Daradur created IGNITE-9692:
--

 Summary: Cache creation request may be missed on coordinator change
 Key: IGNITE-9692
 URL: https://issues.apache.org/jira/browse/IGNITE-9692
 Project: Ignite
  Issue Type: Bug
  Components: cache
Affects Versions: 2.6
Reporter: Vyacheslav Daradur
 Attachments: CacheCreationOnCoordinatorChange.java

Request of cache creation may be missed in case of coordinator change.

Reproducer: [^CacheCreationOnCoordinatorChange.java]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ignite pull request #4827: IGNITE-9418 initialize cache persistent storage c...

[zstan]

GitHub user zstan opened a pull request:

https://github.com/apache/ignite/pull/4827

IGNITE-9418 initialize cache persistent storage concurrently.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-9418

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4827.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4827


commit 14a06ea3dea5cfc7cab9e499e8b54759554b36fd
Author: Evgeny Stanilovskiy 
Date:   2018-09-25T13:49:29Z

IGNITE-9418 initialize cache persistent storage concurrently.




---

[GitHub] ignite pull request #4795: MVCC TX: Send partition update counters to backup...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/4795


---

[jira] [Created] (IGNITE-9693) Scale up wal compression workers to increase perormance

[Sergey Kosarev (JIRA)]

Sergey Kosarev created IGNITE-9693:
--

 Summary: Scale up wal compression workers to increase perormance
 Key: IGNITE-9693
 URL: https://issues.apache.org/jira/browse/IGNITE-9693
 Project: Ignite
  Issue Type: Improvement
Reporter: Sergey Kosarev






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ignite-release pull request #2: Fix gpg key regexp

[nizhikov]

GitHub user nizhikov opened a pull request:

https://github.com/apache/ignite-release/pull/2

Fix gpg key regexp



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/nizhikov/ignite-release vote-step-fix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite-release/pull/2.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2


commit b881e2a91a85bbc81f938560dceb89bf31a78e33
Author: Nikolay Izhikov 
Date:   2018-09-25T14:20:50Z

Fix gpg key regexp




---

[GitHub] ignite-release pull request #2: Fix gpg key regexp

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite-release/pull/2


---

[GitHub] zzzadruga opened a new pull request #17: IGNITE-9541 Hotfix

[GitBox]

zzzadruga opened a new pull request #17: IGNITE-9541 Hotfix
URL: https://github.com/apache/ignite-teamcity-bot/pull/17
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (IGNITE-9694) Do not block reading queries on exchange events that don't change data visibility

[Ivan Bessonov (JIRA)]

Ivan Bessonov created IGNITE-9694:
-

 Summary: Do not block reading queries on exchange events that 
don't change data visibility
 Key: IGNITE-9694
 URL: https://issues.apache.org/jira/browse/IGNITE-9694
 Project: Ignite
  Issue Type: Improvement
Reporter: Ivan Bessonov
Assignee: Ivan Bessonov


In current implementation there might be situations where reading operation 
waits, for example, exchange of client join event. Such events should not block 
read operations.

In theory - the only operation that has to block reading (except for writing) 
is "node left" for server (or baseline server in case of persistent setup).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[DISCUSSION] Add reviewer field to Apache Ignite JIRA project

[Dmitriy Pavlov]

Hi Ignite Enthusiasts,

During the planning of release 2.7, I've faced with the situation when it
is completely not clear who is going to review ticket.

Usually, we do not reassign tickets to a reviewer, but info about planned
reviewer can be very useful for all reviewers, who select some contribution
to pick up into a review.

Please share your vision about the idea of adding a reviewer field (type:
user) in addition to the assignee field.

If we agree I will try to ask the Infra team on Friday 28.09.

Sincerely,
Dmitriy Pavlov

[GitHub] ignite pull request #4828: IGNITE-9691 testConcurrentAuthorize uses outdated...

[oignatenko]

GitHub user oignatenko opened a pull request:

https://github.com/apache/ignite/pull/4828

IGNITE-9691 testConcurrentAuthorize uses outdated assumption about 
exception message



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-9691

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4828.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4828


commit 1eeca908a8076a8317947dac8a46845964d1d7ea
Author: Oleg Ignatenko 
Date:   2018-08-23T13:13:28Z

IGNITE-9348 ML examples improvements
- wip (logging improved)
-- verified with diffs overview, executing the examples and studying 
execution logs

commit e50b89c392568ba9b93935c4fa6c7f7f93f5ec6f
Author: Oleg Ignatenko 
Date:   2018-08-23T14:45:57Z

Revert "IGNITE-9348 ML examples improvements"

This reverts commit 1eeca908a8076a8317947dac8a46845964d1d7ea.

commit 474024b4c5bbdb3c0a4ed2f0a66238c8054c6674
Author: Oleg Ignatenko 
Date:   2018-08-23T14:57:34Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 9642b233b5701bdad47ebea163079160227c582a
Author: Oleg Ignatenko 
Date:   2018-08-28T14:01:11Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 7fc16d013ab725d2ff2e1a1b042c983f11d0c4d4
Author: Oleg Ignatenko 
Date:   2018-08-28T15:13:02Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit d2caba67b156674f051f50faebeafe0871bf0914
Author: Oleg Ignatenko 
Date:   2018-08-29T13:14:07Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 16775dff51d71ea68b4a3dea98be552130c493ed
Author: Oleg Ignatenko 
Date:   2018-08-30T09:00:56Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit aedb59929974fe205b949225c1a338c68c60cfc8
Author: Oleg Ignatenko 
Date:   2018-08-30T09:42:38Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 39c6482fcdca506aa33011ed21c98060b4a8c68b
Author: Oleg Ignatenko 
Date:   2018-08-30T11:28:05Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 33b32a2760a6559c78283b927e3191180d8ed9e1
Author: Oleg Ignatenko 
Date:   2018-08-30T12:31:16Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 9531028ddd1aef9e95f7e8c8b528086739bbb1b0
Author: Oleg Ignatenko 
Date:   2018-08-30T14:06:34Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 28f22c6e2fffcb82717ba5da7be2cfd39715c4e3
Author: Oleg Ignatenko 
Date:   2018-08-30T16:41:51Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit aacac88db519187413b0fc5ff9d0e55b8f8cff22
Author: Oleg Ignatenko 
Date:   2018-08-31T10:12:32Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 897f920dde46022849b13f9fb86dba8e54119a56
Author: Oleg Ignatenko 
Date:   2018-08-31T13:57:14Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 114c79e54c1b316006ccc3ff22d20d902f9313df
Author: Oleg Ignatenko 
Date:   2018-08-31T17:39:16Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit fd6b659bb8be1992618ce4ce91f568a0988b3afa
Author: Oleg Ignatenko 
Date:   2018-09-02T06:11:42Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 6ae6d1d3cf9743d8d466be0330511ddc8589e944
Author: Oleg Ignatenko 
Date:   2018-09-03T10:27:35Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit e8b27dbd3d0c1cbdb7a7659175f5c7bb447482bf
Author: Oleg Ignatenko 
Date:   2018-09-04T09:49:44Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 622c82efdd0aa182fadea6b7ffa5d4615521a3f5
Author: Oleg Ignatenko 
Date:   2018-09-05T10:50:28Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit fb844fe3751e2e8ae87e6b8030b0e4bd659df9c2
Author: Oleg Ignatenko 
Date:   2018-09-05T11:45:58Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 480ed78869277d7e32f725ab71bec9621f1ac03a
Author: Oleg Ignatenko 
Date:   2018-09-06T07:52:55Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit c99762498f617c0e98ea3062a43c0b30092166ef
Author: Oleg Ignatenko 
Date:   2018-09-06T14:45:04Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 2e17175225c72f747d370b5fee96f8be69d6d2cb
Author: Oleg Ignatenko 
Date:   2018-09-06T17:33:54Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 9ebcd9a2fe5966b0bf42a95484395867c15d863f
Author: Oleg Ignatenko 
Date:   2018-09-07T13:38:51Z

Merge branch 'mas

[GitHub] ignite pull request #4829: IGNITE-9686: JDK9: pass jdk9 specific JVM options...

[tledkov-gridgain]

GitHub user tledkov-gridgain opened a pull request:

https://github.com/apache/ignite/pull/4829

IGNITE-9686: JDK9: pass jdk9 specific JVM options to new process when…

… start Ignite test node in separate JVM

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-9686

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4829.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4829


commit 43c6300d1b226c6d0e0952311cff019ecc986d7a
Author: tledkov-gridgain 
Date:   2018-09-25T16:07:23Z

IGNITE-9686: JDK9: pass jdk9 specific JVM options to new process when start 
Ignite test node in separate JVM




---

[GitHub] asfgit closed pull request #17: IGNITE-9541 Hotfix

[GitBox]

asfgit closed pull request #17: IGNITE-9541 Hotfix
URL: https://github.com/apache/ignite-teamcity-bot/pull/17
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/IgniteTeamcityConnection.java
 
b/ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/IgniteTeamcityConnection.java
index 9ddd68a..65ece8d 100644
--- 
a/ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/IgniteTeamcityConnection.java
+++ 
b/ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/IgniteTeamcityConnection.java
@@ -486,7 +486,8 @@ public String basicAuthToken() {
 String branchFilter = isNullOrEmpty(branchName) ? "" :",branch:" + 
branchName;
 String sinceDateFilter = sinceDate == null ? "" : ",sinceDate:" + 
getDateYyyyMmDdTHhMmSsZ(sinceDate);
 String untilDateFilter = untilDate == null ? "" : ",untilDate:" + 
getDateYyyyMmDdTHhMmSsZ(untilDate);
-String buildNoFilter = sinceBuildId == null ? "" : ",sinceBuild:(id:" 
+ sinceBuildId + ")";
+String buildNoFilter = sinceBuildId == null || 
!(sinceDateFilter.isEmpty() && untilDateFilter.isEmpty()) ?
+"" : ",sinceBuild:(id:" + sinceBuildId + ")";
 
 return sendGetXmlParseJaxb(host + "app/rest/latest/builds"
 + "?locator="


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ignite pull request #4830: IGNITE-9649 Debug logs enhacement

[Jokser]

GitHub user Jokser opened a pull request:

https://github.com/apache/ignite/pull/4830

IGNITE-9649 Debug logs enhacement



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-9649

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4830.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4830


commit 2a20c08467a4a205fbe4ef901f6035439a0f2f4f
Author: Pavel Kovalenko 
Date:   2018-09-20T14:55:16Z

IGNITE-9649 WIP

commit 4af3db3d844e56847cf59250c6e095526c643799
Author: Pavel Kovalenko 
Date:   2018-09-24T12:44:05Z

IGNITE-9649 WIP

commit 90f484c13ce4f55c450e6fcfaba720343af1a822
Author: Pavel Kovalenko 
Date:   2018-09-24T15:06:36Z

IGNITE-9649 Move partition related abstractions to separate package.

commit ff3754b6a07da68db55facec9a18f42b9e6bdb7d
Author: Pavel Kovalenko 
Date:   2018-09-25T12:09:41Z

IGNITE-9492 More logs.

commit 21084513ac3b145c3068f63bd8cb9b1667223518
Author: Pavel Kovalenko 
Date:   2018-09-25T16:29:14Z

IGNITE-9649 WIP




---

Re: [DISCUSSION] Add reviewer field to Apache Ignite JIRA project

[Dmitriy Setrakyan]

I like the idea.

On Tue, Sep 25, 2018 at 8:25 AM Dmitriy Pavlov 
wrote:

> Hi Ignite Enthusiasts,
>
> During the planning of release 2.7, I've faced with the situation when it
> is completely not clear who is going to review ticket.
>
> Usually, we do not reassign tickets to a reviewer, but info about planned
> reviewer can be very useful for all reviewers, who select some contribution
> to pick up into a review.
>
> Please share your vision about the idea of adding a reviewer field (type:
> user) in addition to the assignee field.
>
> If we agree I will try to ask the Infra team on Friday 28.09.
>
> Sincerely,
> Dmitriy Pavlov
>

[GitHub] zzzadruga opened a new pull request #18: IGNITE-9541 Hotfix for old cache

[GitBox]

zzzadruga opened a new pull request #18: IGNITE-9541 Hotfix for old cache
URL: https://github.com/apache/ignite-teamcity-bot/pull/18
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] asfgit closed pull request #18: IGNITE-9541 Hotfix for old cache

[GitBox]

asfgit closed pull request #18: IGNITE-9541 Hotfix for old cache
URL: https://github.com/apache/ignite-teamcity-bot/pull/18
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/IgnitePersistentTeamcity.java
 
b/ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/IgnitePersistentTeamcity.java
index 33497bb..837a9dc 100644
--- 
a/ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/IgnitePersistentTeamcity.java
+++ 
b/ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/IgnitePersistentTeamcity.java
@@ -443,50 +443,59 @@ private boolean isHistoryAgeLessThanSecs(SuiteInBranch 
key, int seconds, Expirab
 idUntil = idUntil == -2 ? buildRefs.size() - 1 : idUntil;
 }
 
-if (idSince == -1 || idUntil == -1)
-return Collections.emptyList();
-else if (idSince == -3 || idUntil == -3) {
+if (idSince == -3 || idUntil == -3) {
 AtomicBoolean stopFilter = new AtomicBoolean();
 AtomicBoolean addBuild = new AtomicBoolean();
 
-return buildRefs.stream()
-.filter(b -> {
-if (stopFilter.get())
-return addBuild.get();
-
-Build build = getBuild(b.href);
-
-if (build == null || build.isFakeStub())
-return false;
+return buildRefs.stream()
+.filter(b -> {
+if (stopFilter.get())
+return addBuild.get();
 
-Date date = build.getFinishDate();
+Build build = getBuild(b.href);
 
-if (sinceDate != null && untilDate != null)
-return (date.after(sinceDate) || 
date.equals(sinceDate)) &&
-(date.before(untilDate) || 
date.equals(untilDate));
-else if (sinceDate != null) {
-if (date.after(sinceDate) || 
date.equals(sinceDate)) {
-stopFilter.set(true);
-addBuild.set(true);
+if (build == null || build.isFakeStub())
+return false;
 
-return true;
-}
+Date date = build.getFinishDate();
 
-return false;
-}
+if (sinceDate != null && untilDate != null)
+if ((date.after(sinceDate) || 
date.equals(sinceDate)) &&
+(date.before(untilDate) || 
date.equals(untilDate)))
+return true;
 else {
 if (date.after(untilDate)) {
 stopFilter.set(true);
 addBuild.set(false);
-
-return false;
 }
 
+return false;
+}
+else if (sinceDate != null) {
+if (date.after(sinceDate) || 
date.equals(sinceDate)) {
+stopFilter.set(true);
+addBuild.set(true);
+
 return true;
 }
-})
-.collect(Collectors.toList());
-}
+
+return false;
+}
+else {
+if (date.after(untilDate)) {
+stopFilter.set(true);
+addBuild.set(false);
+
+return false;
+}
+
+return true;
+}
+})
+.collect(Collectors.toList());
+}
+else if (idSince == -1 || idUntil == -1)
+return Collections.emptyList();
 else
 return buildRefs.subList(idSince, idUntil + 1);
 }


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apach

Re: IgniteSet implementation: changes required

[Pavel Pereslegin]

Hello Igniters.

As was discussed, IgniteSet implementation was based on on-heap data
duplication (setDataMap), as a result, the data was not recovered after
cluster restart and in the case of large data sets, this led to a
significant heap growing and gc pressure.

We changed the implementation so that this structure works well without
duplicating the data [1]. To reduce performance drop and speed up large
data sets, non-collocated version of IgniteSet now uses separate cache [2].

[1] https://issues.apache.org/jira/browse/IGNITE-5553
[2] https://issues.apache.org/jira/browse/IGNITE-7823



ср, 27 июн. 2018 г. в 23:26, Amir Akhmedov :

> Yes, you are right.
>
> Thanks,
> Amir
>
>
> On Wed, Jun 27, 2018 at 1:15 PM Denis Magda  wrote:
>
> > Got you. If it's about redundant data duplication in onheap region then
> no
> > any concerns from my side.
> >
> > Anyway, considering that the data structure will be interacting with the
> > page memory directly then its entries can be stored in Ignite persistence
> > automatically (if the latter is on). Does it mean that the data structure
> > will be fully recovered after a restart and its entries can be pulled
> from
> > disk on demand?
> >
> > --
> > Denis
> >
> >
> > On Tue, Jun 26, 2018 at 1:49 PM Amir Akhmedov 
> > wrote:
> >
> > > I also think it will better to remove setDataMap support cause
> > > 1. It's making extra pressure on GC by keeping entries on heap
> > > 2. It has difficult logic to support with lots of nuances
> > > 3. To maintain setDataMap today GridCacheMapEntry calls
> > > cctx.dataStructures().onEntryUpdated() on each entry mutation. I think
> > it's
> > > unnecessary cohesion.
> > > 4. For the case with single Ignite cache for all collocated
> > datastructure,
> > > an iterator creation will not be much slower than current
> implementation
> > > since we can run affinity call on the node where all entries reside.
> > Also,
> > > we can create a better affinity mapper to fairly distribute
> > datastructures
> > > across a cluster rather than mapping by datastructure's name.
> > >
> > > Thanks,
> > > Amir
> > >
> > >
> > > On Tue, Jun 26, 2018 at 8:10 AM Anton Vinogradov 
> wrote:
> > >
> > > > Denis,
> > > >
> > > > I think that better case is to remove onheap
> optimisation/duplication.
> > > > This brings no drop to frequently used operations (put/remove), but
> > even
> > > > will make it slightly faster.
> > > >
> > > > The only one question we have here is "is it possible to restore
> onheap
> > > map
> > > > in easy way?".
> > > > Seems that answer is no, so, I vote for setDataMap removal.
> > > >
> > > > вт, 26 июн. 2018 г. в 15:00, Denis Magda :
> > > >
> > > > > Anton,
> > > > >
> > > > > Will it be possible to reuse such a functionality for the rest of
> > data
> > > > > structures? I would invest our time in this if all data structures
> > > would
> > > > be
> > > > > able to work with Ignite persistence this way.
> > > > >
> > > > > --
> > > > > Denis
> > > > >
> > > > > On Tue, Jun 26, 2018 at 1:53 AM Anton Vinogradov 
> > > wrote:
> > > > >
> > > > > > >> Why don't we read data straight from the persistence layer
> > warming
> > > > RAM
> > > > > > up
> > > > > > >> in the background?
> > > > > > Because it's not a trivial task to finish such loading on
> unstable
> > > > > > topology.
> > > > > > That's possible, ofcourse, but solution and complexity will be
> > almost
> > > > > > equals to WAL enable/disable.
> > > > > >
> > > > > > пн, 25 июн. 2018 г. в 22:13, Denis Magda :
> > > > > >
> > > > > > > Folks,
> > > > > > >
> > > > > > > Why don't we read data straight from the persistence layer
> > warming
> > > > RAM
> > > > > up
> > > > > > > in the background? (like we do for SQL and other APIs). If
> it's a
> > > > > > question
> > > > > > > of time, then I would suggest us not to hurry up and do it in a
> > > right
> > > > > > way.
> > > > > > >
> > > > > > > --
> > > > > > > Denis
> > > > > > >
> > > > > > > On Mon, Jun 25, 2018 at 6:20 AM Anton Vinogradov <
> a...@apache.org>
> > > > > wrote:
> > > > > > >
> > > > > > > > +1 to removal in case there is no easy, fast and consistent
> way
> > > to
> > > > > > > restore
> > > > > > > > setDataMap on node restart.
> > > > > > > > I see that we'll gain some performance drop on size() or
> > keys(),
> > > > but
> > > > > > > these
> > > > > > > > methods are rarely used.
> > > > > > > >
> > > > > > > > пн, 25 июн. 2018 г. в 16:07, Pavel Pereslegin <
> > xxt...@gmail.com
> > > >:
> > > > > > > >
> > > > > > > > > Hello, Igniters.
> > > > > > > > >
> > > > > > > > > I tried to implement IgniteSet data recovery when
> persistence
> > > > > enabled
> > > > > > > > > [1] using trivial cache scanning, however I cannot find
> > optimal
> > > > way
> > > > > > to
> > > > > > > > > do that because of the following reasons:
> > > > > > > > > - Performing operations on IgniteSet requires completion of
> > > data
> > > > > > > > > loading (restoring of setDataMap) on all nodes. Do this
> > 

[GitHub] ignite pull request #4755: IGNITE-9573 Fix ZooKeeper SASL authentication tes...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/4755


---

[GitHub] ignite pull request #4831: IGNITE-9693 wip.

[ivandasch]

GitHub user ivandasch opened a pull request:

https://github.com/apache/ignite/pull/4831

IGNITE-9693 wip.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-9693

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4831.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4831


commit 905315d0bafb37dfb188945ae7c873490de24f8f
Author: Ivan Daschinskiy 
Date:   2018-09-25T17:31:27Z

IGNITE-9693 wip.




---

[GitHub] ignite pull request #4832: IGNITE-9540: MVCC TX: make cache invoke\invokeAll...

[AMashenkov]

GitHub user AMashenkov opened a pull request:

https://github.com/apache/ignite/pull/4832

IGNITE-9540: MVCC TX: make cache invoke\invokeAll operations support Mvcc 
tx mode.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-9540

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4832.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4832


commit b7d84760eb8dde4b1a922b4ee54885e86644e858
Author: Andrey V. Mashenkov 
Date:   2018-09-06T07:45:56Z

IGNITE-7764: WIP. Tests added.

Signed-off-by: Andrey V. Mashenkov 

commit 98ba5dc4e99e571056f4c0d7061c87f68c0d0aff
Author: Andrey V. Mashenkov 
Date:   2018-09-06T14:28:53Z

IGNITE-7764: WIP. Fix Get\GetAll operations.

Signed-off-by: Andrey V. Mashenkov 

commit b88c8b8aa37a493adb64e572e82e7e15c8be88e7
Author: Andrey V. Mashenkov 
Date:   2018-09-07T19:26:49Z

IGNITE-7764: WIP. Fix Put\PutAll operations. Naive implementation, 
optimization needed.

Signed-off-by: Andrey V. Mashenkov 

commit 09c0a2e8d321da968900af247f41b281cf505599
Author: Andrey V. Mashenkov 
Date:   2018-09-08T15:01:38Z

IGNITE-7764: WIP. Fix test.

Signed-off-by: Andrey V. Mashenkov 

commit 7971694048ab344353808072cbd8826ac958a48d
Author: Andrey V. Mashenkov 
Date:   2018-09-10T10:06:06Z

IGNITE-7764: WIP. Refactored query enlist futures.

Generify query enlist futures.

Signed-off-by: Andrey V. Mashenkov 

commit a06e80d0a3040e467163359016d5fb2cab8eb8a4
Author: Andrey V. Mashenkov 
Date:   2018-09-12T14:12:45Z

IGNITE-7764: WIP. Implemented MVCC protocol for basic 
put\putAll\remove\removeAll operations.

Signed-off-by: Andrey V. Mashenkov 

commit 043d42cda73c8d11ec270c8c5739f2d34d58d9fe
Author: Andrey V. Mashenkov 
Date:   2018-09-12T16:33:10Z

IGNITE-7764: WIP. Added tests to suite.

Signed-off-by: Andrey V. Mashenkov 

commit 3dffd70991cd782197749575770187705f780c52
Author: Andrey V. Mashenkov 
Date:   2018-09-13T10:06:06Z

IGNITE-7764: WIP. Added tests for remove\removeAll.

Signed-off-by: Andrey V. Mashenkov 

commit bfeed7184bece6b05fac3f2dbcee9e221f50
Author: Andrey V. Mashenkov 
Date:   2018-09-13T12:20:14Z

IGNITE-7764: WIP. Added tests for getAndPut\getAndRemove\putIfAbsent.

Added muted test replace/getAndReplace.

Signed-off-by: Andrey V. Mashenkov 

commit 7133ee487ce23a18b60f5fd9980db9a1bf204f25
Author: Andrey V. Mashenkov 
Date:   2018-09-13T14:07:10Z

IGNITE-7764: WIP. Fix tests.

Signed-off-by: Andrey V. Mashenkov 

commit 04925e8795a0db146aa9caf53e3aafadae4dea8c
Author: Andrey V. Mashenkov 
Date:   2018-09-13T15:14:28Z

IGNITE-7764: WIP. Mute tests.

Signed-off-by: Andrey V. Mashenkov 

commit f06157575650f5948f5c94a696f546b087d6a455
Author: Andrey V. Mashenkov 
Date:   2018-09-13T17:52:54Z

IGNITE-7764: WIP. Fixed putIfAbsent/replace/getAndReplace.

Signed-off-by: Andrey V. Mashenkov 

commit 8f4bafadeb22e3a420090a674e1c803d3add16c9
Author: Andrey V. Mashenkov 
Date:   2018-09-14T15:19:39Z

IGNITE-7764: WIP. Fixed putIfAbsent/replace/getAndReplace.

Signed-off-by: Andrey V. Mashenkov 

commit 8de70951b5864484fb552087eecf08fa45ace39b
Author: Andrey V. Mashenkov 
Date:   2018-09-14T15:36:00Z

IGNITE-7764: WIP. Minor.

Signed-off-by: Andrey V. Mashenkov 

commit bb8f375c9aed9d61a56b75513108e256b9c31a84
Author: Andrey V. Mashenkov 
Date:   2018-09-14T16:00:10Z

IGNITE-7764: WIP. Enable Full API MVCC tests.

Signed-off-by: Andrey V. Mashenkov 

commit a96a6e618bba269ef8c83185a352e5f6df2a1464
Author: Andrey V. Mashenkov 
Date:   2018-09-14T16:18:50Z

IGNITE-7764: WIP. Minors & Styles.

Signed-off-by: Andrey V. Mashenkov 

commit 7e5ca350360b56696f4238c97b5b8b7ee41d976f
Author: Andrey V. Mashenkov 
Date:   2018-09-14T17:41:03Z

IGNITE-7764: WIP. Minors.

Signed-off-by: Andrey V. Mashenkov 

commit f65324b1c69458128b5f5a2a0917bc1fe49e2cc1
Author: Andrey V. Mashenkov 
Date:   2018-09-14T17:53:12Z

IGNITE-7764: WIP. Minors after rebase.

Signed-off-by: Andrey V. Mashenkov 

commit 83116012412a4f24de8df2b64d953385060e8724
Author: Andrey V. Mashenkov 
Date:   2018-09-17T09:25:37Z

IGNITE-7764: WIP. Minors.

Signed-off-by: Andrey V. Mashenkov 

commit 0bd02e8939010891854b856c8d77f7f49f13d651
Author: Andrey V. Mashenkov 
Date:   2018-09-17T10:10:17Z

IGNITE-7764: WIP. Minors.

Signed-off-by: Andrey V. Mashenkov 

commit aafbe9c1a9d33086ff78c062f1a06381126e123d
Author: Andrey V. Mashenkov 
Date:   2018-09-17T10:49:22Z

IGNITE-7764: WIP. Fix non-versioned read.

Signed-off-by: Andrey V. Mashenkov 

commit 7cb3c54dfac32ba4eea5cb3e03e3daa5b4b7d9bf
Author: Andrey

[jira] [Created] (IGNITE-9695) Add a way to prevent per-cache WAL disabling in WalStateManager

[Andrey Kuznetsov (JIRA)]

Andrey Kuznetsov created IGNITE-9695:


 Summary: Add a way to prevent per-cache WAL disabling in 
WalStateManager
 Key: IGNITE-9695
 URL: https://issues.apache.org/jira/browse/IGNITE-9695
 Project: Ignite
  Issue Type: Task
Affects Versions: 2.6
Reporter: Andrey Kuznetsov
Assignee: Andrey Kuznetsov
 Fix For: 2.8


When this prevention is on, {{WalStateManager.init()}} should return an 
error-holding future immediately.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Page IO statistics for Ignite

[Alex Plehanov]

Hi,

I've made some investigation a couple of months ago about a statistics
collected by some RDBMS vendors (Oracle, Postgres, MySQL). These databases
collect detailed IO statistics in dimensions such as queries, database
objects (tables and indexes), files, sessions, users, event types etc.

Some views where you can get IO statistics:
Oracle: v$filestat, v$segment_statistics, v$sqlarea, v$sysstat, v$sesstat
Postgres: pg_stat_database, pg_statio_all_tables, pg_statio_all_indexes,
pg_statio_all_sequences
MySQL: table_io_waits_summary_by_table,
table_io_waits_summary_by_index_usage, io_global_by_file_by_bytes,
io_global_by_wait_by_bytes, file_summary_by_event_name,
file_summary_by_instance, host_summary_by_file_io, user_summary_by_file_io,
metrics, etc.

I think we can start by collecting statistics per FilePageStore (updating
counters on read(...) and write(...) methods). Each FilePageStore is
bounded to cache and partition or cache index, so we can easily aggregate
values and get IO statistics per cache/index/node.

вт, 25 сент. 2018 г. в 10:57, Vladimir Ozerov :

> Hi Yuriy,
>
> I think this is great idea. But we need to collect more details on how and
> what to collect. I think one of the most interesting parts for us would be
> index and data page usages, split by different "dimensions":
> 1) Global node statistics
> 2) Per-cache statistics
> 3) Per-index statistics
>
> We can start with a short summary of what is collected by other database
> vendors.
>
> Vladimir.
>
>
> On Sat, Sep 22, 2018 at 1:07 AM Denis Magda  wrote:
>
> > Hello Yuri,
> >
> > I might give useful feedback if see how the metrics will look like from
> the
> > API standpoint. If it's not difficult please create a draft.
> >
> > AS for the interface, in addition to JMX and SQL we need to ensure Visor
> > CMD and Web Console gets updated. *Alex K.*, please join the thread and
> > share your requirements.
> >
> > --
> > Denis
> >
> > On Fri, Sep 21, 2018 at 8:16 AM Юрий 
> wrote:
> >
> > > Hi Igniters,
> > >
> > > I started IGNITE-8580
> > >  ticket
> > > related to print page read/write metrics and did some investigation
> what
> > > other databases provide for the similar purposes.
> > >
> > > Based on the investigation I want to propose my raw vision of how to
> > IGNITE
> > > can be more transparent from performance perspective.
> > >
> > > Need to collect statistics for logical (from memory) and physical (from
> > > storage) page reads/writes. All these metrics should be separated by
> next
> > > dimensions:
> > > 1) index/cache
> > > 2) query level
> > > 3) node/cluster
> > > ...
> > >
> > > Seems the statistics should be limited by time.
> > >
> > > If we will have such statistics we could realize such things as:
> > > 1) Get IO statistics per SQL query, global or/and splitted by
> > > indexes/caches.
> > > 2) Have ability to understand why performance goes down in case it
> > related
> > > to IO. For example on concrete node or cache.
> > > 3) Evaluate effectiveness of use indexes. Find unused indexes.
> > > 4) Keep TOP queries with aggressive physical reads
> > > 
> > >
> > >
> > > Such statistics could be available least at JMX and SQL interfaces.
> > >
> > > Let's discuss. In case it will be interested for you I can dig deeper
> > into
> > > the area and prepare IEP based on our discussion.
> > >
> > >
> > > Igniters, what do you think?
> > >
> > >
> > >
> > >
> > > --
> > > Живи с улыбкой! :D
> > >
> >
>

[ML] TensorFlow intergration module release

[Юрий Бабак]

Hello, Igniters.

For release 2.7 we will introduce integration between TensorFlow and Apache
Ignite. This integration contains changes on Apache Ignite side and on
TensorFlow side.

Apache Ignite part is the command line tool which allows create and
maintain TensorFlow clusters over Apache Ignite and submit TF jobs to those
clusters.

For TensorFlow we implemented "ignite dataset". More details in related PR
[1]

As Apache Ignite part is done and TensorFlow part is ready for the merge I
suggest to add module "ignite-tensorflow" to other Ignite deliverables. So
I've created the ticket in JIRA for this [2]. In that case, we will be able
to release this feature with Apache Ignite binary release includes deb/rpm
packages.

[1] https://github.com/tensorflow/tensorflow/pull/22210
[2] https://issues.apache.org/jira/browse/IGNITE-9685

Regards,
Yury

Re: [ML] TensorFlow intergration module release

[Denis Magda]

Great addition, Yuriy!

Adding the "ignite-tensorflow" module makes total sense to me.

--
Denis

On Tue, Sep 25, 2018 at 12:53 PM Юрий Бабак  wrote:

> Hello, Igniters.
>
> For release 2.7 we will introduce integration between TensorFlow and Apache
> Ignite. This integration contains changes on Apache Ignite side and on
> TensorFlow side.
>
> Apache Ignite part is the command line tool which allows create and
> maintain TensorFlow clusters over Apache Ignite and submit TF jobs to those
> clusters.
>
> For TensorFlow we implemented "ignite dataset". More details in related PR
> [1]
>
> As Apache Ignite part is done and TensorFlow part is ready for the merge I
> suggest to add module "ignite-tensorflow" to other Ignite deliverables. So
> I've created the ticket in JIRA for this [2]. In that case, we will be able
> to release this feature with Apache Ignite binary release includes deb/rpm
> packages.
>
> [1] https://github.com/tensorflow/tensorflow/pull/22210
> [2] https://issues.apache.org/jira/browse/IGNITE-9685
>
> Regards,
> Yury
>

Re: [ML] TensorFlow intergration module release

[Dmitriy Setrakyan]

Distributed TensorFlow, awesome!

On Tue, Sep 25, 2018 at 12:53 PM Юрий Бабак  wrote:

> Hello, Igniters.
>
> For release 2.7 we will introduce integration between TensorFlow and Apache
> Ignite. This integration contains changes on Apache Ignite side and on
> TensorFlow side.
>
> Apache Ignite part is the command line tool which allows create and
> maintain TensorFlow clusters over Apache Ignite and submit TF jobs to those
> clusters.
>
> For TensorFlow we implemented "ignite dataset". More details in related PR
> [1]
>
> As Apache Ignite part is done and TensorFlow part is ready for the merge I
> suggest to add module "ignite-tensorflow" to other Ignite deliverables. So
> I've created the ticket in JIRA for this [2]. In that case, we will be able
> to release this feature with Apache Ignite binary release includes deb/rpm
> packages.
>
> [1] https://github.com/tensorflow/tensorflow/pull/22210
> [2] https://issues.apache.org/jira/browse/IGNITE-9685
>
> Regards,
> Yury
>