[jira] [Created] (IGNITE-10820) Clean up static Ignite instances in tests

[Alexey Goncharuk (JIRA)]

Alexey Goncharuk created IGNITE-10820:
-

 Summary: Clean up static Ignite instances in tests
 Key: IGNITE-10820
 URL: https://issues.apache.org/jira/browse/IGNITE-10820
 Project: Ignite
  Issue Type: Improvement
Reporter: Alexey Goncharuk


>From the recent TC OOME it can be seen that some tests contain static Ignite 
>instances that are not cleaned up on test end, which leads to memory leak. As 
>a first phase, we need to nullify them. Ultimately, we need to get rid of 
>static fields in tests after migration to JUnit5.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-10821) Caching affinity with affinity similarity key is broken

[Pavel Kovalenko (JIRA)]

Pavel Kovalenko created IGNITE-10821:


 Summary: Caching affinity with affinity similarity key is broken
 Key: IGNITE-10821
 URL: https://issues.apache.org/jira/browse/IGNITE-10821
 Project: Ignite
  Issue Type: Bug
  Components: cache
Affects Versions: 2.8
Reporter: Pavel Kovalenko
Assignee: Pavel Kovalenko
 Fix For: 2.8


When some cache groups have the same affinity function, number of partitions, 
backups and the same node filter they can use the same affinity distribution 
without needs for explicit recalculating. These parameters are called as 
"Affinity similarity key". 

In case of affinity recalculation caching affinity using this key may speed-up 
the process.

However, after https://issues.apache.org/jira/browse/IGNITE-9561 merge this 
mechanishm become broken, because parallell execution of affinity recalculation 
for the similar affinity groups leads to caching affinity misses.

To fix it we should couple together similar affinity groups and run affinity 
recalculation for them in one thread, caching previous results.




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-10822) AbstractFreelist init reused page in wrong way.

[Andrew Mashenkov (JIRA)]

Andrew Mashenkov created IGNITE-10822:
-

 Summary: AbstractFreelist init reused page in wrong way.
 Key: IGNITE-10822
 URL: https://issues.apache.org/jira/browse/IGNITE-10822
 Project: Ignite
  Issue Type: Bug
  Components: persistence
Reporter: Andrew Mashenkov
 Fix For: 2.8


This is similar to IGNITE-9303.
In IGNITE-9303 we missed wrong page flag check that hides this issue.

After fixing page flag check, one of mvcc tests fails with assertion as Ignite 
can't lock page for write due to unknown reason: smth goes wrong with page tag.

ExplicitWalDeltaConsistencyTest.testNotEmptyPds() fails in mvcc mode with next 
error
{noformat}
java.lang.AssertionError
at 
org.apache.ignite.internal.processors.cache.persistence.freelist.AbstractFreeList.insertDataRow(AbstractFreeList.java:507)
at 
org.apache.ignite.internal.processors.cache.persistence.metastorage.MetastorageRowStore.addRow(MetastorageRowStore.java:73)
at 
org.apache.ignite.internal.processors.cache.persistence.metastorage.MetaStorage.putData(MetaStorage.java:377)
at 
org.apache.ignite.internal.processors.cache.persistence.metastorage.MetaStorage.write(MetaStorage.java:353)
at 
org.apache.ignite.internal.processors.cluster.GridClusterStateProcessor.writeBaselineTopology(GridClusterStateProcessor.java:293)
at 
org.apache.ignite.internal.processors.cluster.GridClusterStateProcessor.onReadyForReadWrite(GridClusterStateProcessor.java:250)
at 
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.notifyMetastorageReadyForReadWrite(GridCacheDatabaseSharedManager.java:430)
at 
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.finishRecovery(GridCacheDatabaseSharedManager.java:884){noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-10823) Close all open cursors for running queries on Ignite stop

[Yury Gerzhedovich (JIRA)]

Yury Gerzhedovich created IGNITE-10823:
--

 Summary: Close all open cursors for running queries on Ignite stop
 Key: IGNITE-10823
 URL: https://issues.apache.org/jira/browse/IGNITE-10823
 Project: Ignite
  Issue Type: Task
  Components: sql
Affects Versions: 2.8
Reporter: Yury Gerzhedovich
Assignee: Yury Gerzhedovich
 Fix For: 2.8


Currently we can have situation when user doesn't close a cursors after use it 
or don't full read it. In this case we have potential memory leak with opened 
cursors. Need to close and set as null for all such running queries.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-10824) SQL: mixing _key and key columns in the DML queries must be disallowed

[Pavel Kuznetsov (JIRA)]

Pavel Kuznetsov created IGNITE-10824:


 Summary: SQL: mixing _key and key columns in the DML queries must 
be disallowed
 Key: IGNITE-10824
 URL: https://issues.apache.org/jira/browse/IGNITE-10824
 Project: Ignite
  Issue Type: Bug
  Components: sql
Reporter: Pavel Kuznetsov
Assignee: Pavel Kuznetsov


DML should contain either placeholder for _key (_val) or subset of key (value) 
columns but not both. Also we should keep in mind _key/_value aliases

Given table with primary key column {{id}}. Next query should be validated to 
parsing error:
{code:sql}
INSERT INTO TEST_TABLE (_key, id, salary) VALUES (1, 2, 42);
UPDATE TEST_TABLE SET _val = 1, salary = 2;
{code}   



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [MTCGA]: new failures in builds [2635316] needs to be handled

[Vyacheslav Daradur]

The fix has been approved by Bot's visa and merged to master.

Anton, thanks for your help.

On Tue, Dec 25, 2018 at 3:41 PM Anton Vinogradov  wrote:
>
> Slava,
>
> please fill the issue with fix and tcbot visa.
>
> On Tue, Dec 25, 2018 at 12:18 PM Vyacheslav Daradur  
> wrote:
>>
>> I prepared PR [1] to fix the issue, the test became green.
>>
>> Anton, could you assist with the merge, please?
>>
>> [1] https://github.com/apache/ignite/pull/5739
>> [2] https://ci.ignite.apache.org/viewQueued.html?itemId=2641732
>>
>> On Tue, Dec 25, 2018 at 10:53 AM Vyacheslav Daradur  
>> wrote:
>> >
>> > This is my fault, introduced within
>> > https://issues.apache.org/jira/browse/IGNITE-10715
>> >
>> > I have no idea why it had not been failed during testing on TC.
>> >
>> > I will prepare the fix quickly.
>> >
>> > On Tue, Dec 25, 2018 at 2:37 AM  wrote:
>> > >
>> > > Hi Igniters,
>> > >
>> > >  I've detected some new issue on TeamCity to be handled. You are more 
>> > > than welcomed to help.
>> > >
>> > >  If your changes can lead to this failure(s): We're grateful that you 
>> > > were a volunteer to make the contribution to this project, but things 
>> > > change and you may no longer be able to finalize your contribution.
>> > >  Could you respond to this email and indicate if you wish to continue 
>> > > and fix test failures or step down and some committer may revert you 
>> > > commit.
>> > >
>> > >  *New stable failure of a flaky test in master 
>> > > IgniteClientConnectTest.testClientConnectToBigTopology 
>> > > https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8&testNameId=9021926721143358489&branch=%3Cdefault%3E&tab=testDetails
>> > >  Changes may lead to failure were done by
>> > >  - ilya.kasnacheev 
>> > > https://ci.ignite.apache.org/viewModification.html?modId=850683
>> > >  - daradurvs 
>> > > https://ci.ignite.apache.org/viewModification.html?modId=850663
>> > >  - antonovsergey93 
>> > > https://ci.ignite.apache.org/viewModification.html?modId=850675
>> > >
>> > >  - Here's a reminder of what contributors were agreed to do 
>> > > https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute
>> > >  - Should you have any questions please contact 
>> > > dev@ignite.apache.org
>> > >
>> > > Best Regards,
>> > > Apache Ignite TeamCity Bot
>> > > https://github.com/apache/ignite-teamcity-bot
>> > > Notification generated at 02:36:58 25-12-2018
>> >
>> >
>> >
>> > --
>> > Best Regards, Vyacheslav D.
>>
>>
>>
>> --
>> Best Regards, Vyacheslav D.



-- 
Best Regards, Vyacheslav D.

[jira] [Created] (IGNITE-10825) After node restart and and new node to BLT due load - some partition inconsistent

[ARomantsov (JIRA)]

ARomantsov created IGNITE-10825:
---

 Summary: After node restart and and new node to BLT due load - 
some partition inconsistent
 Key: IGNITE-10825
 URL: https://issues.apache.org/jira/browse/IGNITE-10825
 Project: Ignite
  Issue Type: Bug
  Components: data structures
Affects Versions: 2.8
Reporter: ARomantsov






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-10826) Control.sh add logging to file

[Sergey Antonov (JIRA)]

Sergey Antonov created IGNITE-10826:
---

 Summary: Control.sh add logging to file
 Key: IGNITE-10826
 URL: https://issues.apache.org/jira/browse/IGNITE-10826
 Project: Ignite
  Issue Type: Improvement
Reporter: Sergey Antonov
 Fix For: 2.8






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-10827) auto close iterator for query cursor when all data has been read

[Yury Gerzhedovich (JIRA)]

Yury Gerzhedovich created IGNITE-10827:
--

 Summary: auto close iterator for query cursor when all data has 
been read
 Key: IGNITE-10827
 URL: https://issues.apache.org/jira/browse/IGNITE-10827
 Project: Ignite
  Issue Type: Task
  Components: sql
Reporter: Yury Gerzhedovich
Assignee: Yury Gerzhedovich


There is QueryCursorImpl class which we use as main realization of cursor for 
H2 iterators. As of now we call close method explicit when we suppose that all 
data already read. Will be better call close() method when iterator hasNext() 
method return false or when next() method of iterator throw any Exception. Such 
realization will be more safely and simple to use.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Let's force -Dfile.encoding=UTF-8

[Ilya Kasnacheev]

Hello!

It was recently discovered that there's some issue in H2 which leads to
differing string encodings when they come into reducer from nodes with
different system encoding. Even if we were able to fix this, I suspect that
user code may too be bitten by mismatch of this setting.

I propose to force UTF-8 as system encoding at all times when we control
how JVM is launched.
This includes ignite.sh, ignite.bat, Apache.Ignite.exe and C++'s ./ignite.

This will mainly affect Windows systems as I expect that Linux will most
always use UTF-8 locale and Mac OS X should always be UTF-8.

file.encoding is somewhat misleading name since it specifies the default
string encoding, such as the one used for String.getBytes(). It is a common
convention to set ut to UTF-8, for example, IDEA will do that.

WDYT?

There's a pull request: https://github.com/apache/ignite/pull/5725
If somebody could contribute C++ and .Net tests I would be also grateful.

Regards,
-- 
Ilya Kasnacheev

Re: Let's force -Dfile.encoding=UTF-8

[Dmitriy Pavlov]

+1 from my side. I think it is reasonable

ср, 26 дек. 2018 г. в 19:20, Ilya Kasnacheev :

> Hello!
>
> It was recently discovered that there's some issue in H2 which leads to
> differing string encodings when they come into reducer from nodes with
> different system encoding. Even if we were able to fix this, I suspect that
> user code may too be bitten by mismatch of this setting.
>
> I propose to force UTF-8 as system encoding at all times when we control
> how JVM is launched.
> This includes ignite.sh, ignite.bat, Apache.Ignite.exe and C++'s ./ignite.
>
> This will mainly affect Windows systems as I expect that Linux will most
> always use UTF-8 locale and Mac OS X should always be UTF-8.
>
> file.encoding is somewhat misleading name since it specifies the default
> string encoding, such as the one used for String.getBytes(). It is a common
> convention to set ut to UTF-8, for example, IDEA will do that.
>
> WDYT?
>
> There's a pull request: https://github.com/apache/ignite/pull/5725
> If somebody could contribute C++ and .Net tests I would be also grateful.
>
> Regards,
> --
> Ilya Kasnacheev
>

Re: Packt wants to feature books

[Prachi Garg]

Hi,

I've added a link to the book under the Docs menu item on the website.

On Tue, Dec 11, 2018 at 10:41 AM Denis Magda  wrote:

> Hello Hishit,
>
> Thanks for sharing the news, great!
>
> Prachi, could you please add a reference to the to our website menu?
>
> --
> Denis
>
> On Tue, Dec 11, 2018 at 10:14 AM Nishit Rajput 
> wrote:
>
>> Hi team,
>>
>>
>> I am Nishit from Packt. We publish over 50 books related to trending and
>> upcoming technologies every month. We also feature various books on Project
>> websites.
>>
>>
>> We would be interested in featuring our recently published book Apache
>> Ignite Quick Start Guide<
>> https://www.packtpub.com/big-data-and-business-intelligence/apache-ignite-quick-start-guide>
>> by Sujoy Acharya
>>
>>
>> The book takes you through the basics of Apache Ignite and in-memory
>> technologies. You will learn about installation and clustering Ignite
>> nodes, caching topologies, and various caching strategies, such as cache
>> aside, read and write through, and write behind. Next, you will delve into
>> detailed aspects of Ignite’s data grid: web session clustering and querying
>> data.
>>
>> You will learn how to process large volumes of data using compute grid
>> and Ignite’s map-reduce and executor service. You will learn about the
>> memory architecture of Apache Ignite and monitoring memory and caches. You
>> will use Ignite for complex event processing, event streaming, and the
>> time-series predictions of opportunities and threats. Additionally, you
>> will go through off-heap and on-heap caching, swapping, and native and
>> Spring framework integration with Apache Ignite.
>>
>> By the end of this book, you will be confident with all the features of
>> Apache Ignite 2.x that can be used to build a high-performance system
>> architecture.
>>
>>
>> We can shorten the description as per your website's requirement.
>>
>>
>> I am certain this book will prove to be an insightful resource to the
>> developers.
>>
>>
>> Do let me know if we can feature this book on your website and if you can
>> help us featuring it. If not, can you please direct me to the right person
>> for this requirement.
>>
>>
>> Looking forward to your reply.
>>
>>
>> Thanks and regards,
>>
>> [https://s3-eu-west-1.amazonaws.com/email-signature-packt/128x128px.png]
>>
>>
>>
>>
>>
>> Nishit Rajput
>>
>> Author Marketing Executive
>>
>>
>>
>>  [https://s3-eu-west-1.amazonaws.com/email-signature-packt/ph.png]
>>  022-28328148
>>
>>  [https://s3-eu-west-1.amazonaws.com/email-signature-packt/web.png]
>> www.packt.com
>>  [https://s3-eu-west-1.amazonaws.com/email-signature-packt/msg.png]
>> nish...@packt.com
>> [https://s3-eu-west-1.amazonaws.com/email-signature-packt/loc.png]  Plot
>> No.103, Arena House, 3rd Floor,12th Road, MIDC, Andheri (E), Mumbai -
>> 400093 Mumbai .
>>
>>
>> [https://www.facebook.com/PacktPub/?ref=br_rs]<
>> https://www.facebook.com/PacktPub/?ref=br_rs>  [
>> https://twitter.com/PacktPub]    [
>> https://www.linkedin.com/company/packt-publishing/] <
>> https://www.linkedin.com/company/packt-publishing/>   [
>> https://www.youtube.com/channel/UC3VydBGBl132baPCLeDspMQ] <
>> https://www.youtube.com/channel/UC3VydBGBl132baPCLeDspMQ>
>>
>>
>>
>>
>>
>>
>>
>> Packt Publishing Private Limited. Registered Address: Plot No.103, Arena
>> House, 3rd Floor,12th Road, MIDC, Andheri (E), Mumbai - 400093.
>> CIN:U22100MH2005PTC152766
>> This E-mail is confidential. It may also be legally privileged. If you
>> are not the addressee you may not copy, forward, disclose or use any part
>> of it. If you have received this message in error, please delete it and all
>> copies from your system and notify the sender immediately by return E-mail.
>> Whilst Packt Publishing Ltd take every reasonable precaution to avoid the
>> transfer of software viruses or defects that may cause damage to computer
>> systems; it is the responsibility of the recipient to ensure that all
>> emails and attachments received are free from infection.
>> The word 'Packt'® and the Packt logo are registered trademarks which
>> belong to Packt Publishing Limited.
>>
>

[jira] [Created] (IGNITE-10828) GridAbstractTest wrongly calculates test count

[Eduard Shangareev (JIRA)]

Eduard Shangareev created IGNITE-10828:
--

 Summary: GridAbstractTest wrongly calculates test count
 Key: IGNITE-10828
 URL: https://issues.apache.org/jira/browse/IGNITE-10828
 Project: Ignite
  Issue Type: Bug
Reporter: Eduard Shangareev


I have found that our naive approach doesn't work. In my case, it counted while 
112 while IDEA 113, what cause wrong isLastTest result.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Service grid redesign

[Nikolay Izhikov]

Hello, Igniters.

I've merged Service Grid Redesign - Phase 1 to the master.
Vyacheslav, great contribution!

Thanks for all Ignite veterans both for the code and design review.

В Пн, 24/12/2018 в 20:50 +0300, Nikolay Izhikov пишет:
> Hello, Igniters.
> 
> Please, let us know, if someone want to do additional review of this PR.
> 
> В Пн, 24/12/2018 в 20:23 +0300, Vyacheslav Daradur пишет:
> > Igniters, especially future reviewers,
> > 
> > Discovery listener registered by 'IgniteServiceProcessor' become
> > implemented 'HighPriorityListener', seems it's best lock-free
> > solutions discussed during the review. This change is covered by
> > `ServiceDeploymentDiscoveryListenerNotificationOrderTest` which should
> > protect us if the order of listeners will be changed.
> > 
> > It's about the problem of custom messages which are nullified by PME
> > [1] and are listened by service deployment to manage the lifecycle of
> > affinity services. This guarantees that service deployment discovery
> > listener will be notified earlier than PME's discovery listener and
> > will be able to capture custom messages which may be nullified in PME
> > process.
> > 
> > Looks like we do not have any controversial questions now.
> > 
> > Thanks!
> > 
> > [1] 
> > http://apache-ignite-developers.2346864.n4.nabble.com/Danger-change-of-DiscoveryCustomEvent-in-GridDhtPartitionsExchangeFuture-onDone-td35946.html
> > 
> > 
> > On Mon, Dec 24, 2018 at 4:23 PM Vyacheslav Daradur  
> > wrote:
> > > 
> > > Stanislav, thank you for the notes, most of them have been resolved. I
> > > answered on GitHub.
> > > 
> > > 
> > > On Sun, Dec 23, 2018 at 9:34 PM Stanislav Lukyanov
> > >  wrote:
> > > > 
> > > > I’ve done a quick superficial review. Didn’t look at the tests, didn’t 
> > > > dive into the design, etc, just the code.
> > > > I’ve left some comments – almost all are about minor issues, grammar 
> > > > and code style.
> > > > 
> > > > Stan
> > > > 
> > > > From: Vyacheslav Daradur
> > > > Sent: 21 декабря 2018 г. 14:58
> > > > To: dev@ignite.apache.org
> > > > Subject: Re: Service grid redesign
> > > > 
> > > > Igniters,
> > > > 
> > > > Please, let us know if someone is going to do an additional review?
> > > > 
> > > > We should know can we merge the PR since it has been approved by
> > > > Nikolay Izhikov and Denis Mekhanikov or we should wait for other
> > > > community members.
> > > > 
> > > > On Thu, Dec 20, 2018 at 7:52 PM Vyacheslav Daradur 
> > > >  wrote:
> > > > > 
> > > > > I think I found names which should satisfy me and Denis, and possibly 
> > > > > Nikolay )
> > > > > 
> > > > > See the following names (Actual name <- Previously used):
> > > > > 
> > > > > - ServiceDeploymentManager <- ServicesDeploymentManager
> > > > > - ServiceDeploymentActions <- ServicesDeploymentActions
> > > > > - ServiceDeploymentProcessId <- ServicesDeploymentProcessId
> > > > > - ServiceDeploymentTask <- ServicesDeploymentTask
> > > > > 
> > > > > - ServiceDeploymentRequest <- ServiceDeploymentChange
> > > > > - ServiceUndeploymentRequest <- ServiceUndeploymentChange
> > > > > - ServiceChangeAbstractRequest <- ServiceAbstractChange
> > > > > 
> > > > > - ServiceSingleNodeDeploymentResult <- ServiceSingleDeploymentsResults
> > > > > - ServiceSingleNodeDeploymentResultBatch <- 
> > > > > ServicesSingleDeploymentsMessage
> > > > > 
> > > > > - ServiceClusterDeploymentResult <- ServiceFullDeploymentsResults
> > > > > - ServiceClusterDeploymentResultBatch <- 
> > > > > ServicesFullDeploymentsMessage
> > > > > 
> > > > > - ServiceProcessorCommonDiscoveryData <- ServicesCommonDiscoveryData
> > > > > - ServiceProcessorJoinNodeDiscoveryData <- 
> > > > > ServicesJoinNodeDiscoveryData
> > > > > 
> > > > > Also, I had a short talk with Alexey Goncharuk about the problem of
> > > > > nullified custom messages. I changed the implementation to a lock-free
> > > > > solution which allows us to nullify messages depend on an using
> > > > > counter.
> > > > > 
> > > > > In comparison with high priority listener, this allows us to not copy
> > > > > custom discovery event in service deployment manager and work with the
> > > > > original object.
> > > > > 
> > > > > On Thu, Dec 20, 2018 at 8:57 AM Nikolay Izhikov  
> > > > > wrote:
> > > > > > 
> > > > > > Denis, great news!
> > > > > > 
> > > > > > Alexey, Vova, Yakov, do you want to take a look at this PR?
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > В Ср, 19/12/2018 в 18:47 +0300, Denis Mekhanikov пишет:
> > > > > > > Guys,
> > > > > > > 
> > > > > > > I finished my code review. The pool request looks good to me.
> > > > > > > 
> > > > > > > Does anybody else want to look at the changes?
> > > > > > > There are a few points, that we didn't meet an agreement on,
> > > > > > > though they don't affect the behaviour in any way:
> > > > > > > 
> > > > > > >- *Class naming. * See the discussion above.
> > > > > > >- *Unnecessary task object cleaning. *
> > > > > > >IMO, ServicesDeployment