Re: Transparent Data Encryption (TDE) in Apache Ignite

[Дмитрий Рябов]

Hi, Igniters! I think it would be nice to implement encryption in
Ignite. Even SQLite and H2 have encryption so why Ignite don't have
it?

I'd like to propose a design for discussion.

Configurations:
IgniteConfiguration:
- KeyStore tdeKeyStore - contain encryption keys.
- Encryptor encryptor - interface for encryption algorythm with
methods `encrypt()`, `decrypt()`.
CacheConfiguration:
- TransparentDataEncryption tde - configures where or how we will
encrypt. We can encrypt pages (PDS or PDS&offheap) or cached objects
(PDS, PDS&offheap and even heap if we encrypt just after key
validation and security check). And, of course, TDE can be turned off.

Encryption keys:
Every node has keystore with Cache Encryption Keys (CEKs), one key for
one cache, they are stored in encrypted form.
CEKs are encrypted by Master Encryption Key (MEK), which stored on the
coordinator or somewhere out of cluster (this way is safer). Also, we
need backups for MEK on other nodes or somewhere out of the cluster.

Encryption:
To make any operation with encrypting cache node gets MEK from
coordinator (or somewhere out of the cluster). Then it decrypts needed
CEK. With decrypted CEK we encrypt/decrypt data. When transaction
finished we must destroy received MEK.

TransparentDataEncryption enum:
NONE - no encryption.
ALL_OBJECTS - data will be encrypted on tx initiator just after key
validation and security check, so most of time it would be just a
byte[] object. It is good, but on the other hand we have additional
encryption for any node with listeners for data changing.
PDS_ONLY_OBJECTS - data will be encrypted on primary and backups when
storing to PDS.
OFFHEAP_PDS_OBJECTS - data will be encrypted when storing to offheap
storage and PDS.
PDS_ONLY_PAGES - data pages will be encrypted on primary and backups
when storing to PDS.
OFFHEAP_PDS_PAGES - data pages will be encrypted when storing to
offheap storage and PDS.

But I'm not sure about objects/pages. Do we really need both of them?


2017-06-26 14:39 GMT+03:00 Sergi Vladykin :
> No, we don't have plans for it.
>
> Sergi
>
> 2017-06-26 14:20 GMT+03:00 Vyacheslav Daradur :
>
>> Sergi, thanks for the answer.
>>
>> >> see TDE is just an option for PCI DSS compliancy but not a requirement.
>> Requirement: "Protect stored cardholder data"
>> Encryption is required.
>> TDE - is one of ways to implement it at the database level.
>>
>> Sure, an implementation at the application level solve it.
>>
>> I meant another.
>> I thought maybe this feature is in the Ignite roadmap?
>>
>>
>> 2017-06-26 13:53 GMT+03:00 Sergi Vladykin :
>>
>> > I think no one is interested in this stuff right now.
>> >
>> > Also as far as I see TDE is just an option for PCI DSS compliancy but
>> not a
>> > requirement.
>> >
>> > Your system should pass PCI DSS if you will do the required encryption at
>> > the application level and will properly manage encryption keys.
>> >
>> > Sergi
>> >
>> > 2017-06-26 11:40 GMT+03:00 Vyacheslav Daradur :
>> >
>> > > Guys, any thoughts?
>> > >
>> > > 2017-06-20 11:02 GMT+03:00 Vyacheslav Daradur :
>> > >
>> > > > Hi Igniters.
>> > > >
>> > > > I have some user cases where I need fast storage with TDE support.
>> > > > It is requered for PCI DSS certification.
>> > > >
>> > > > As far as I know AI doesn't support it.
>> > > >
>> > > > I looked at other storages.
>> > > > Many storages support it or are engaged in development this feature.
>> > > >
>> > > > Cassandra community are working on TDE support.[1]
>> > > >
>> > > > Oracle support it.[2] Moreover it supports indexing and querying on
>> > > > encrypted data.
>> > > >
>> > > > I think it will be very usefull to support TDE by AI.
>> > > >
>> > > > What do you think? Maybe development is already under way?
>> > > >
>> > > > [1] https://issues.apache.org/jira/browse/CASSANDRA-9945
>> > > > [2] https://docs.oracle.com/cd/B19306_01/network.102/b14268/
>> > > > asotrans.htm#ASOAG600
>> > > >
>> > > > --
>> > > > Best Regards, Vyacheslav D.
>> > > >
>> > >
>> > >
>> > >
>> > > --
>> > > Best Regards, Vyacheslav D.
>> > >
>> >
>>
>>
>>
>> --
>> Best Regards, Vyacheslav D.
>>

[jira] [Created] (IGNITE-7621) Review current builds on TeamCity

[Peter Ivanov (JIRA)]

Peter Ivanov created IGNITE-7621:


 Summary: Review current builds on TeamCity
 Key: IGNITE-7621
 URL: https://issues.apache.org/jira/browse/IGNITE-7621
 Project: Ignite
  Issue Type: Task
Reporter: Peter Ivanov
Assignee: Peter Ivanov


There is a little under 600 builds currently configured on teamcity. It is 
required to review each of them and:
* revise and clean obsolete build plans
* revise and clean duplicated VCS roots
* revise and reconfigure corresponding triggers
* revise and strict build configuration access permissions
* document everything



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-7622) SQL: Subquery fails in non-collocated mode

[Roman Shtykh (JIRA)]

Roman Shtykh created IGNITE-7622:


 Summary: SQL: Subquery fails in non-collocated mode
 Key: IGNITE-7622
 URL: https://issues.apache.org/jira/browse/IGNITE-7622
 Project: Ignite
  Issue Type: Bug
  Components: sql
Affects Versions: 2.3
Reporter: Roman Shtykh
 Attachments: TPCH_Q8.java

When trying to execute query 8 from TPC-H benchmarks with a non-collocated 
mode, I get _java.lang.ArrayIndexOutOfBoundsException_.
{noformat}
General error: "java.lang.ArrayIndexOutOfBoundsException" [5-195]
    at org.h2.message.DbException.getJdbcSQLException(DbException.java:345)
    at org.h2.message.DbException.get(DbException.java:168)
    at org.h2.message.DbException.convert(DbException.java:295)
    at org.h2.message.DbException.toSQLException(DbException.java:268)
    at org.h2.message.TraceObject.logAndConvert(TraceObject.java:352)
    at org.h2.jdbc.JdbcConnection.prepareStatement(JdbcConnection.java:292)
    at 
org.apache.ignite.internal.processors.query.h2.sql.GridSqlQuerySplitter.optimize(GridSqlQuerySplitter.java:1627)
    at 
org.apache.ignite.internal.processors.query.h2.sql.GridSqlQuerySplitter.split(GridSqlQuerySplitter.java:236)
    at 
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.queryDistributedSqlFields(IgniteH2Indexing.java:1307)
    ... 17 more
Caused by: java.lang.ArrayIndexOutOfBoundsException
    at java.lang.System.arraycopy(Native Method)
    at 
org.apache.ignite.internal.processors.query.h2.opt.GridH2CollocationModel.childFilters(GridH2CollocationModel.java:227)
    at 
org.apache.ignite.internal.processors.query.h2.opt.GridH2CollocationModel.buildCollocationModel(GridH2CollocationModel.java:689)
    at 
org.apache.ignite.internal.processors.query.h2.opt.GridH2IndexBase.getDistributedMultiplier(GridH2IndexBase.java:295)
    at 
org.apache.ignite.internal.processors.query.h2.opt.GridH2PrimaryScanIndex.getCost(GridH2PrimaryScanIndex.java:76)
    at org.h2.table.Table.getBestPlanItem(Table.java:708)
    at org.h2.table.TableFilter.getBestPlanItem(TableFilter.java:221)
    at org.h2.table.Plan.calculateCost(Plan.java:123)
    at org.h2.command.dml.Optimizer.calculateBruteForceSome(Optimizer.java:142)
    at org.h2.command.dml.Optimizer.calculateBestPlan(Optimizer.java:87)
    at org.h2.command.dml.Optimizer.optimize(Optimizer.java:244)
    at org.h2.command.dml.Select.preparePlan(Select.java:1002)
    at org.h2.command.dml.Select.prepare(Select.java:865)
    at org.h2.command.Parser.prepare(Parser.java:242)
    at org.h2.engine.Session.prepare(Session.java:538)
    at org.h2.index.ViewIndex.prepareSubQuery(ViewIndex.java:176)
    at org.h2.index.ViewIndex.getQuery(ViewIndex.java:312)
    at org.h2.index.ViewIndex.(ViewIndex.java:104)
    at org.h2.table.TableView.getBestPlanItem(TableView.java:248)
    at org.h2.table.TableView.getScanIndex(TableView.java:459)
    at org.h2.table.TableFilter.getBestPlanItem(TableFilter.java:202)
    at org.h2.table.Plan.calculateCost(Plan.java:123)
    at org.h2.command.dml.Optimizer.testPlan(Optimizer.java:185)
    at org.h2.command.dml.Optimizer.calculateBestPlan(Optimizer.java:81)
    at org.h2.command.dml.Optimizer.optimize(Optimizer.java:244)
    at org.h2.command.dml.Select.preparePlan(Select.java:1002)
    at org.h2.command.dml.Select.prepare(Select.java:865)
    at org.h2.command.Parser.prepareCommand(Parser.java:262)
    at org.h2.engine.Session.prepareLocal(Session.java:573)
    at org.h2.engine.Session.prepareCommand(Session.java:514)
    at org.h2.jdbc.JdbcConnection.prepareCommand(JdbcConnection.java:1204)
    at org.h2.jdbc.JdbcPreparedStatement.(JdbcPreparedStatement.java:73)
    at org.h2.jdbc.JdbcConnection.prepareStatement(JdbcConnection.java:288)
    ... 20 more{noformat}
I realize that for such queries collocation has to be done, but it should 
probably produce a hint or an error with better explanations.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Annoying extra steps for enabling metrics

[Alex Plehanov]

Ok, then I will rename methods to setEventsDisabled/getEventsDisabled.


2018-02-01 12:46 GMT+03:00 Anton Vinogradov :

> Folks,
>
> .setEventsDisabled looks to be a good solution, since we will always call
> it like .setEventsDisabled(true).
>
> Calling .setEventsEnabled(false), since true is a default, looks odd to me.
>
> On Wed, Jan 31, 2018 at 11:02 PM, Valentin Kulichenko <
> valentin.kuliche...@gmail.com> wrote:
>
> > Alex,
> >
> > As a workaround, you can use boxed Boolean as a field type, and then
> return
> > true from getEventsEnabled in case it's null. Will this work?
> >
> > -Val
> >
> > On Wed, Jan 31, 2018 at 7:31 AM, Alex Plehanov 
> > wrote:
> >
> > > Denis, there is a question about IGNITE-7346.
> > >
> > > CacheConfiguration fields are set to they default values when cache
> > > configuration is created by constructor. When CacheConfiguration is
> > created
> > > by deserialization (from another node or from PDS), constructor is not
> > > called. If it was serialized by older version of Ignite, a new added
> > > boolean fields are set to false after deserialization by a new Ignite
> > > versions. We can't change this behavior without methods for custom
> > > serialization/deserialization (which are not implemented now in
> > > CacheConfiguration class). It will differ from requirements for "Eve
> > > ntsEnabled" property (events must be enabled for caches by default).
> > >
> > > I think we can reverse the logic and rename method
> > > CacheConfiguration.setEventsEnabled
> > > to CacheConfiguration.setEventsDisabled, in this case the field value
> > will
> > > always be "false" by default.
> > >
> > > What do you think about it?
> > >
> > > 2017-12-30 10:12 GMT+03:00 Alex Plehanov :
> > >
> > > > Due to holidays I can start work on this ticket only after 8 jan 2018
> > > >
> > > > 2017-12-30 2:12 GMT+03:00 Denis Magda :
> > > >
> > > >> Good, closed the original ticket.
> > > >>
> > > >> Alex P, do you have time to work on IGNITE-7346 instead to address
> the
> > > >> issue with the cache events per cache in 2.4 release?
> > > >>
> > > >> —
> > > >> Denis
> > > >>
> > > >> > On Dec 29, 2017, at 3:10 PM, Valentin Kulichenko <
> > > >> valentin.kuliche...@gmail.com> wrote:
> > > >> >
> > > >> > Agree.
> > > >> >
> > > >> > -Val
> > > >> >
> > > >> > On Fri, Dec 29, 2017 at 3:08 PM, Denis Magda 
> > > wrote:
> > > >> >
> > > >> >> Now I see. Seems I was doing something wrong in my initial
> > > reproducer.
> > > >> >>
> > > >> >> Updated cache metrics readme doc by purging any events related
> > > >> parameters
> > > >> >> from there:
> > > >> >> https://apacheignite.readme.io/v2.3/docs/cache-metrics <
> > > >> >> https://apacheignite.readme.io/v2.3/docs/cache-metrics>
> > > >> >>
> > > >> >> The events readme doc looks good to me. Just updated the javadoc
> of
> > > >> >> IgniteEvents#*Query methods to bring MemoryEventStorageSpi to
> users
> > > >> >> attention.
> > > >> >>
> > > >> >> As for the cache events, it’s an oversight that there is no
> > parameter
> > > >> that
> > > >> >> enables/disables the events per cache. Let’s add
> > > >> CacheConfiguration.setEventsEnabled
> > > >> >> method and have it enabled by default (current behavior). If the
> > user
> > > >> >> deploys hundreds of caches or just interested in the events of
> > > specific
> > > >> >> ones then he can always set this property to ‘false’ in
> > configuration
> > > >> of
> > > >> >> the caches of no interest:
> > > >> >> https://issues.apache.org/jira/browse/IGNITE-7346 <
> > > >> >> https://issues.apache.org/jira/browse/IGNITE-7346>
> > > >> >>
> > > >> >> Agree?
> > > >> >>
> > > >> >> —
> > > >> >> Denis
> > > >> >>
> > > >> >>
> > > >> >>
> > > >> >>> On Dec 29, 2017, at 11:10 AM, Valentin Kulichenko <
> > > >> >> valentin.kuliche...@gmail.com> wrote:
> > > >> >>>
> > > >> >>> Guys,
> > > >> >>>
> > > >> >>> I'm not sure what issue we're trying to solve. Basically, we
> have
> > > >> three
> > > >> >>> different functionality parts here:
> > > >> >>>
> > > >> >>> 1. Cache metrics exposed via CacheMetrics interface and MBeans
> > > >> (number of
> > > >> >>> puts, average put time, this kind of stuff). These are
> controlled
> > on
> > > >> per
> > > >> >>> cache level by CacheConfiguration#statisticsEnabled property.
> > > >> >>> 2. Listening to cache events. To achieve this you only need to
> > > enable
> > > >> >>> particular event types and register listeners, this does not
> > depend
> > > on
> > > >> >>> CacheConfiguration#statisticsEnabled. Here is the test
> confirming
> > > >> this:
> > > >> >>> https://gist.github.com/vkulichenko/
> > 54043c7b75c69eca5515e7d7692b52
> > > 76
> > > >> >>> 3. Querying cache events via IgniteEvents#*Query methods. This
> is
> > > the
> > > >> >> ONLY
> > > >> >>> piece that requires MemoryEventStorageSpi to be configured.
> Since
> > > >> >> querying
> > > >> >>> is not very frequently used, and event storage introduces
> > > significant
> > > >> >>> memory overhead, I don

Subject: new committer: Nikolay Izhikov

[Anton Vinogradov]

The Project Management Committee (PMC) for Apache Ignite
has invited Nikolay Izhikov to become a committer and we are pleased
to announce that he has accepted.

Nikolay is an author of the Spark Data Frame API integration [1].
>From what I see, Nikolay one of the most strongest and active contributors
for Spark integration.

Also, Nikolay is one who made the last mile in Context switch
implementation for optimistic transactions [2].

[1] https://issues.apache.org/jira/browse/IGNITE-3084
[2] https://issues.apache.org/jira/browse/IGNITE-5712

Re: Subject: new committer: Nikolay Izhikov

[Vyacheslav Daradur]

That is good news! Congrats, Nikolay!

On Mon, Feb 5, 2018 at 12:21 PM, Anton Vinogradov  wrote:
> The Project Management Committee (PMC) for Apache Ignite
> has invited Nikolay Izhikov to become a committer and we are pleased
> to announce that he has accepted.
>
> Nikolay is an author of the Spark Data Frame API integration [1].
> From what I see, Nikolay one of the most strongest and active contributors
> for Spark integration.
>
> Also, Nikolay is one who made the last mile in Context switch
> implementation for optimistic transactions [2].
>
> [1] https://issues.apache.org/jira/browse/IGNITE-3084
> [2] https://issues.apache.org/jira/browse/IGNITE-5712



-- 
Best Regards, Vyacheslav D.

Re: Transparent Data Encryption (TDE) in Apache Ignite

[Dmitry Pavlov]

Hi Igniters,

Encryption will ensure security only if keys are stored in a secure
storage. Otherwise having access to Ignite node storage, we can extract
encryption master keys values and read data as plain text (page's clear
content).

Where are we going to store keys (MEK) physically? Whould it be PKCS#11
storage? Where we will store passwords to unlock storage or it will be
responibilty of user?

One more thing is how "node gets MEK from coordinator", if we send
cleartext MEK, such security becomes useless also.

Encryption will have negative influence to performance, so I think we
shouldn't do this thing as first priority. And we need to clearly realize
whole security scheme.

Sincerely,
Dmitriy Pavlov

пн, 5 февр. 2018 г. в 11:23, Дмитрий Рябов :

> Hi, Igniters! I think it would be nice to implement encryption in
> Ignite. Even SQLite and H2 have encryption so why Ignite don't have
> it?
>
> I'd like to propose a design for discussion.
>
> Configurations:
> IgniteConfiguration:
> - KeyStore tdeKeyStore - contain encryption keys.
> - Encryptor encryptor - interface for encryption algorythm with
> methods `encrypt()`, `decrypt()`.
> CacheConfiguration:
> - TransparentDataEncryption tde - configures where or how we will
> encrypt. We can encrypt pages (PDS or PDS&offheap) or cached objects
> (PDS, PDS&offheap and even heap if we encrypt just after key
> validation and security check). And, of course, TDE can be turned off.
>
> Encryption keys:
> Every node has keystore with Cache Encryption Keys (CEKs), one key for
> one cache, they are stored in encrypted form.
> CEKs are encrypted by Master Encryption Key (MEK), which stored on the
> coordinator or somewhere out of cluster (this way is safer). Also, we
> need backups for MEK on other nodes or somewhere out of the cluster.
>
> Encryption:
> To make any operation with encrypting cache node gets MEK from
> coordinator (or somewhere out of the cluster). Then it decrypts needed
> CEK. With decrypted CEK we encrypt/decrypt data. When transaction
> finished we must destroy received MEK.
>
> TransparentDataEncryption enum:
> NONE - no encryption.
> ALL_OBJECTS - data will be encrypted on tx initiator just after key
> validation and security check, so most of time it would be just a
> byte[] object. It is good, but on the other hand we have additional
> encryption for any node with listeners for data changing.
> PDS_ONLY_OBJECTS - data will be encrypted on primary and backups when
> storing to PDS.
> OFFHEAP_PDS_OBJECTS - data will be encrypted when storing to offheap
> storage and PDS.
> PDS_ONLY_PAGES - data pages will be encrypted on primary and backups
> when storing to PDS.
> OFFHEAP_PDS_PAGES - data pages will be encrypted when storing to
> offheap storage and PDS.
>
> But I'm not sure about objects/pages. Do we really need both of them?
>
>
> 2017-06-26 14:39 GMT+03:00 Sergi Vladykin :
> > No, we don't have plans for it.
> >
> > Sergi
> >
> > 2017-06-26 14:20 GMT+03:00 Vyacheslav Daradur :
> >
> >> Sergi, thanks for the answer.
> >>
> >> >> see TDE is just an option for PCI DSS compliancy but not a
> requirement.
> >> Requirement: "Protect stored cardholder data"
> >> Encryption is required.
> >> TDE - is one of ways to implement it at the database level.
> >>
> >> Sure, an implementation at the application level solve it.
> >>
> >> I meant another.
> >> I thought maybe this feature is in the Ignite roadmap?
> >>
> >>
> >> 2017-06-26 13:53 GMT+03:00 Sergi Vladykin :
> >>
> >> > I think no one is interested in this stuff right now.
> >> >
> >> > Also as far as I see TDE is just an option for PCI DSS compliancy but
> >> not a
> >> > requirement.
> >> >
> >> > Your system should pass PCI DSS if you will do the required
> encryption at
> >> > the application level and will properly manage encryption keys.
> >> >
> >> > Sergi
> >> >
> >> > 2017-06-26 11:40 GMT+03:00 Vyacheslav Daradur :
> >> >
> >> > > Guys, any thoughts?
> >> > >
> >> > > 2017-06-20 11:02 GMT+03:00 Vyacheslav Daradur  >:
> >> > >
> >> > > > Hi Igniters.
> >> > > >
> >> > > > I have some user cases where I need fast storage with TDE support.
> >> > > > It is requered for PCI DSS certification.
> >> > > >
> >> > > > As far as I know AI doesn't support it.
> >> > > >
> >> > > > I looked at other storages.
> >> > > > Many storages support it or are engaged in development this
> feature.
> >> > > >
> >> > > > Cassandra community are working on TDE support.[1]
> >> > > >
> >> > > > Oracle support it.[2] Moreover it supports indexing and querying
> on
> >> > > > encrypted data.
> >> > > >
> >> > > > I think it will be very usefull to support TDE by AI.
> >> > > >
> >> > > > What do you think? Maybe development is already under way?
> >> > > >
> >> > > > [1] https://issues.apache.org/jira/browse/CASSANDRA-9945
> >> > > > [2] https://docs.oracle.com/cd/B19306_01/network.102/b14268/
> >> > > > asotrans.htm#ASOAG600
> >> > > >
> >> > > > --
> >> > > > Best Regards, Vyacheslav D.
> >> > > >
> >> > >
> 

Re: Orphaned, duplicate, and main-class tests!

[Anton Vinogradov]

Ilya,

1) I don't think it's a good idea to rename classes to *AbstractTest.java
since they already have abstract word at definition.
We can perform such renaming only in case whole project will be refactored,
but I see no reason to do this.

2) All not included test should be included to appropriate siutes.
Creating IgniteLostAndFoundTestSuite,java is not acceptable.

3) In case you're not sure what to do with particular tests, please provide
lists of such tests. Please group tests by "problem".


On Fri, Feb 2, 2018 at 12:28 AM, Dmitry Pavlov 
wrote:

> Hi Ilya,
>
> Thank you for this research. I think it is useful for community to identify
> and remove obsolete tests (if any), and include lost test into CI run chain
> (if applicable).
>
> For test with main() methods I suggest to ask authors (git annotate) and if
> there is no response probably we should remove such code.
>
> Since I am not sure all tests in this lost&found suite are quite stable I
> suggest to create standalone TC Run configuration for such tests.
>
> Earlier I've removed most of tests causing timeouts from basic suite.
> Ideally Basic suite should contain fast run quite stable tests ( and 0
> flaky ) because it is included into RunAllBasic sub set to brief commit
> check  (
> https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_
> RunBasicTests
>  ).
>
> Sincerely,
> Dmitriy Pavlov
>
> чт, 1 февр. 2018 г. в 20:22, Ilya Kasnacheev :
>
> > Hello!
> >
> > While working on Ignite, I have noticed that not all tests are in any
> test
> > suite, hence I expect they are ignored. I have also noticed some files in
> > src/test and named *Test.java are actually runnable main-classes and not
> > tests. I think they're ignored to. Also I've noticed that 6 tests repeat
> > twice.
> >
> > I have tried to fix it by introducing "lost and found" test suite. Not
> sure
> > what to do with main-classes. I have also renamed abstract test classes
> to
> > *AbstractTest.
> >
> > Please consider pull request https://github.com/apache/ignite/pull/3464
> >
> > I have started this suite on TC but I expect it to hang or worse.
> >
> > https://ci.ignite.apache.org/viewLog.html?buildId=1071504&;
> tab=queuedBuildOverviewTab
> >
> > Regards,
> > --
> > Ilya Kasnacheev
> >
>

Re: Transport compression (not store compression)

[Nikita Amelchev]

Thanks for your comments,

I will try to separate network compression for clients and servers.

It makes sense to enable compression on servers if we have SSL turned on. I
tested rebalancing time and compression+ssl is faster. SSL throughput is
limited by 800 Mbits/sec per connection and if enable compression, it
boosted up to 1100 Mbits.

2018-02-02 18:52 GMT+03:00 Alexey Kuznetsov :

> I think Igor is right.
>
> Ususally servers connected via fast local network.
> But clients could be in external and slow network.
> In this scenario compression will be very useful.
>
> Once I had such scenario - client connected to cluster via 300 kb/s network
> and tries to transfer ~10Mb of uncumpressed data.
> So it takse ~30 seconds.
> After I implemented compression it becamed 1M and transfered for ~3
> seconds.
>
> I think we should take care of all mentioned problems with NIO threads in
> order to not slow down whole cluster.
>
>
> On Fri, Feb 2, 2018 at 10:05 PM, gvvinblade  wrote:
>
> > Nikita,
> >
> > Yes, you're right. Maybe I wasn't clear enough.
> >
> > Usually server nodes are placed in the same fast network segment (one
> > datacenter); in any case we need an ability to setup compression per
> > connection using some filter like useCompression(ClusterNode,
> ClusterNode)
> > to compress traffic only between servers and client nodes.
> >
> > But issue is still there, since the same NIO worker serves both client
> and
> > server connections, enabled compression may impact whole cluster
> > performance
> > because NIO threads will compress client messages instead of processing
> > servers' compute requests. That was my concern.
> >
> > Compression for clients is really cool feature and usefull in some cases.
> > Probably it makes sense to have two NIO servers with and without
> > compression
> > to process server and client requests separately or pin somehow worker
> > threads to client or server sessions...
> >
> > Also we have to think about client connections (JDBC, ODBC, .Net thin
> > client, etc) and setup compression for them separately.
> >
> > Anyway I would compare put, get, putAll, getAll and SQL SELECT operations
> > for strings and POJOs, one server, several clients with and without
> > compression, setting up the server to utilize all cores by NIO workers,
> > just
> > to get know possible impact.
> >
> > Possible configuration for servers with 16 cores:
> >
> > Selectors cnt = 16
> > Connections per node = 4
> >
> > Where client nodes perform operations in 16 threads
> >
> > Regards,
> > Igor
> >
> >
> >
> >
> > --
> > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
> >
>
>
>
> --
> Alexey Kuznetsov
>



-- 
Best wishes,
Amelchev Nikita

[jira] [Created] (IGNITE-7623) Thin client Java API - async API

[Alexey Kukushkin (JIRA)]

Alexey Kukushkin created IGNITE-7623:


 Summary: Thin client Java API - async API
 Key: IGNITE-7623
 URL: https://issues.apache.org/jira/browse/IGNITE-7623
 Project: Ignite
  Issue Type: Sub-task
  Components: clients
Reporter: Alexey Kukushkin


Implement Async version of all the client APIs. This is a big task - consider 
splitting it into multiple tasks. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Orphaned, duplicate, and main-class tests!

[Ilya Kasnacheev]

Anton,

1) We already have ~100 files named "*AbstractTest.java". Renaming these
several files will help checking for orphaned tests in the future, as well
as increasing code base consistency.

2) This is huge work that is not doable by any single developer. While
IgniteLostAndFoundTestSuite can be slowly refactored away
This is unless you are OK with putting all these tests, most of which are
red and some are hanging, in production test suites and therefore breaking
productivity for a couple months while this gets sorted.
Are you OK with that? Anybody else?

3) I think I *could* put them in some test suite or another, but I'm pretty
sure I can't fix them all, not in one commit, not ever. Nobody can do that
single-handedly. We need a plan here.

Ilya.


-- 
Ilya Kasnacheev

2018-02-05 13:00 GMT+03:00 Anton Vinogradov :

> Ilya,
>
> 1) I don't think it's a good idea to rename classes to *AbstractTest.java
> since they already have abstract word at definition.
> We can perform such renaming only in case whole project will be refactored,
> but I see no reason to do this.
>
> 2) All not included test should be included to appropriate siutes.
> Creating IgniteLostAndFoundTestSuite,java is not acceptable.
>
> 3) In case you're not sure what to do with particular tests, please provide
> lists of such tests. Please group tests by "problem".
>
>
> On Fri, Feb 2, 2018 at 12:28 AM, Dmitry Pavlov 
> wrote:
>
> > Hi Ilya,
> >
> > Thank you for this research. I think it is useful for community to
> identify
> > and remove obsolete tests (if any), and include lost test into CI run
> chain
> > (if applicable).
> >
> > For test with main() methods I suggest to ask authors (git annotate) and
> if
> > there is no response probably we should remove such code.
> >
> > Since I am not sure all tests in this lost&found suite are quite stable I
> > suggest to create standalone TC Run configuration for such tests.
> >
> > Earlier I've removed most of tests causing timeouts from basic suite.
> > Ideally Basic suite should contain fast run quite stable tests ( and 0
> > flaky ) because it is included into RunAllBasic sub set to brief commit
> > check  (
> > https://ci.ignite.apache.org/viewType.html?buildTypeId=
> IgniteTests24Java8_
> > RunBasicTests
> >  ).
> >
> > Sincerely,
> > Dmitriy Pavlov
> >
> > чт, 1 февр. 2018 г. в 20:22, Ilya Kasnacheev  >:
> >
> > > Hello!
> > >
> > > While working on Ignite, I have noticed that not all tests are in any
> > test
> > > suite, hence I expect they are ignored. I have also noticed some files
> in
> > > src/test and named *Test.java are actually runnable main-classes and
> not
> > > tests. I think they're ignored to. Also I've noticed that 6 tests
> repeat
> > > twice.
> > >
> > > I have tried to fix it by introducing "lost and found" test suite. Not
> > sure
> > > what to do with main-classes. I have also renamed abstract test classes
> > to
> > > *AbstractTest.
> > >
> > > Please consider pull request https://github.com/apache/
> ignite/pull/3464
> > >
> > > I have started this suite on TC but I expect it to hang or worse.
> > >
> > > https://ci.ignite.apache.org/viewLog.html?buildId=1071504&;
> > tab=queuedBuildOverviewTab
> > >
> > > Regards,
> > > --
> > > Ilya Kasnacheev
> > >
> >
>

Re: Ignite work directory usage?

[Dmitry Pavlov]

Hi Igniters,

As far as I remember, matching workDir for different nodes in marshaller
context are protected by file locks to avoid concurrent file modifications.

Sergey C. what do you think? Or am I confuse Marhsaller Context with Binary
Meta?

Sincerely,
Dmitry Pavlov

чт, 1 февр. 2018 г. в 5:18, Dmitriy Setrakyan :

> Cross sending to dev@.
>
> Would be nice to hear from the Ignite community. How critical is this issue
> and does it still exist?
>
> D.
>
> -- Forwarded message --
> From: Denis Magda 
> Date: Tue, Jan 9, 2018 at 2:28 PM
> Subject: Re: Ignite work directory usage?
> To: u...@ignite.apache.org
>
>
> It needs to be fixed if the users constantly bump into this issue. However,
> I guess it’s not a matter of urgency now.
>
> —
> Denis
>
> > On Jan 8, 2018, at 6:30 AM, ilya.kasnacheev 
> wrote:
> >
> > Hello Denis!
> >
> > There seems to be cases where matching workDir for different nodes
> > (including client nodes) will cause problems:
> >
> > http://apache-ignite-users.70518.x6.nabble.com/MarshallerCon
> textImpl-Failed-to-write-class-name-to-file-td18439.html
> >
> > There, on Windows, two nodes try to update binary marshaller cache on
> disk
> > at the same time, but, this being on Windows, one of them fails to open
> file
> > for writing.
> >
> > On other OSes there also exists case where one instance would open file
> for
> > writing, and other one, seeing that there is already a file present,
> tries
> > to read it, reads zero bytes and fails at that.
> >
> > WDYT about this case?
> >
> > Regards,
> >
> >
> >
> > --
> > Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Orphaned, duplicate, and main-class tests!

[Anton Vinogradov]

Ilya,

1) Still see no reason for such changes. Does this break something?

2) Looks like you're trying to add Trash*TestSuite.java which will never be
refactored.
We should do everything in proper way now, not sometime.

3) Your comments looks odd to me.
Issue should be resolved in proper way.

On Mon, Feb 5, 2018 at 1:07 PM, Ilya Kasnacheev 
wrote:

> Anton,
>
> 1) We already have ~100 files named "*AbstractTest.java". Renaming these
> several files will help checking for orphaned tests in the future, as well
> as increasing code base consistency.
>
> 2) This is huge work that is not doable by any single developer. While
> IgniteLostAndFoundTestSuite can be slowly refactored away
> This is unless you are OK with putting all these tests, most of which are
> red and some are hanging, in production test suites and therefore breaking
> productivity for a couple months while this gets sorted.
> Are you OK with that? Anybody else?
>
> 3) I think I *could* put them in some test suite or another, but I'm pretty
> sure I can't fix them all, not in one commit, not ever. Nobody can do that
> single-handedly. We need a plan here.
>
> Ilya.
>
>
> --
> Ilya Kasnacheev
>
> 2018-02-05 13:00 GMT+03:00 Anton Vinogradov :
>
> > Ilya,
> >
> > 1) I don't think it's a good idea to rename classes to *AbstractTest.java
> > since they already have abstract word at definition.
> > We can perform such renaming only in case whole project will be
> refactored,
> > but I see no reason to do this.
> >
> > 2) All not included test should be included to appropriate siutes.
> > Creating IgniteLostAndFoundTestSuite,java is not acceptable.
> >
> > 3) In case you're not sure what to do with particular tests, please
> provide
> > lists of such tests. Please group tests by "problem".
> >
> >
> > On Fri, Feb 2, 2018 at 12:28 AM, Dmitry Pavlov 
> > wrote:
> >
> > > Hi Ilya,
> > >
> > > Thank you for this research. I think it is useful for community to
> > identify
> > > and remove obsolete tests (if any), and include lost test into CI run
> > chain
> > > (if applicable).
> > >
> > > For test with main() methods I suggest to ask authors (git annotate)
> and
> > if
> > > there is no response probably we should remove such code.
> > >
> > > Since I am not sure all tests in this lost&found suite are quite
> stable I
> > > suggest to create standalone TC Run configuration for such tests.
> > >
> > > Earlier I've removed most of tests causing timeouts from basic suite.
> > > Ideally Basic suite should contain fast run quite stable tests ( and 0
> > > flaky ) because it is included into RunAllBasic sub set to brief commit
> > > check  (
> > > https://ci.ignite.apache.org/viewType.html?buildTypeId=
> > IgniteTests24Java8_
> > > RunBasicTests
> > >  ).
> > >
> > > Sincerely,
> > > Dmitriy Pavlov
> > >
> > > чт, 1 февр. 2018 г. в 20:22, Ilya Kasnacheev <
> ilya.kasnach...@gmail.com
> > >:
> > >
> > > > Hello!
> > > >
> > > > While working on Ignite, I have noticed that not all tests are in any
> > > test
> > > > suite, hence I expect they are ignored. I have also noticed some
> files
> > in
> > > > src/test and named *Test.java are actually runnable main-classes and
> > not
> > > > tests. I think they're ignored to. Also I've noticed that 6 tests
> > repeat
> > > > twice.
> > > >
> > > > I have tried to fix it by introducing "lost and found" test suite.
> Not
> > > sure
> > > > what to do with main-classes. I have also renamed abstract test
> classes
> > > to
> > > > *AbstractTest.
> > > >
> > > > Please consider pull request https://github.com/apache/
> > ignite/pull/3464
> > > >
> > > > I have started this suite on TC but I expect it to hang or worse.
> > > >
> > > > https://ci.ignite.apache.org/viewLog.html?buildId=1071504&;
> > > tab=queuedBuildOverviewTab
> > > >
> > > > Regards,
> > > > --
> > > > Ilya Kasnacheev
> > > >
> > >
> >
>

Re: Get TTL of the specific (K,V) entry

[Dmitry Pavlov]

Hi,

Previously, in real applications, I needed such information about what is
the remaining lifetime for the object (at least for debug and monitoring).

So I am also +1 for providing such information in the API. Otherwise, user
have to duplicate this information in an entry field.

Sincerely,
Dmitriy Pavlov

ср, 31 янв. 2018 г. в 23:39, Valentin Kulichenko <
valentin.kuliche...@gmail.com>:

> Cross-posting to dev.
>
> Igniters,
>
> This actually makes sense to me. Why don't we add IgniteCache#ttl(K key)
> method that would return current TTL for the key? Looks like this is
> already provided by GridCacheMapEntry#ttl() method, so we only need to
> properly expose it to public API. Am I right?
>
> If there are no objections, I will create a ticket for this improvement.
>
> -Val
>
> On Wed, Jan 31, 2018 at 9:28 AM, Ariel Tubaltsev 
> wrote:
>
>> Hi Val
>>
>> Thank you for confirming this.
>>
>> The use case is to assure correctness of TTL refresh with regard to
>> defined
>> expiry policies and especially views, where different views can have
>> different expiry policies.
>> From what I saw in examples, to validate that TTL was refreshed, we wait
>> for
>> some period of time and check that entries have gone from the cache. Such
>> kind of validation can be cumbersome for long (hours) TTL values.
>>
>> BR
>> Ariel
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>
>

[jira] [Created] (IGNITE-7624) Cluster Activation from Client Node hangs up in specific configuration

[Sergey Kosarev (JIRA)]

Sergey Kosarev created IGNITE-7624:
--

 Summary: Cluster Activation from Client Node hangs up in specific 
configuration
 Key: IGNITE-7624
 URL: https://issues.apache.org/jira/browse/IGNITE-7624
 Project: Ignite
  Issue Type: Bug
  Components: general
Affects Versions: 2.3
Reporter: Sergey Kosarev
 Fix For: 2.5


if we start cluster in inactive state GridTaskProcessor is not initiated fully:
{code:java}
@Override public void onKernalStart(boolean active) throws 
IgniteCheckedException {
if (!active)
return;

tasksMetaCache = ctx.security().enabled() && !ctx.isDaemon() ?
ctx.cache().utilityCache() : null;

startLatch.countDown();
}{code}
 

and those startLatch is still up!

 

Later on if we try activate cluster from client node async task is trying to be 
invoked

(see 
org.apache.ignite.internal.processors.cluster.GridClusterStateProcessorImpl#sendComputeChangeGlobalState)

and if ctx.security().enabled == true then Task is can't start as he waits 
indefinitely for those startLatch in

org.apache.ignite.internal.processors.task.GridTaskProcessor#taskMetaCache
{code:java}
private IgniteInternalCache taskMetaCache() {
assert ctx.security().enabled();

if (tasksMetaCache == null)
U.awaitQuiet(startLatch);

return tasksMetaCache;
}{code}
 

stacktrace of the waiting thread:
{code:java}
"pool-1-thread-1@3160" prio=5 tid=0x68 nid=NA waiting
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Unsafe.java:-1)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
at org.apache.ignite.internal.util.IgniteUtils.awaitQuiet(IgniteUtils.java:7491)
at 
org.apache.ignite.internal.processors.task.GridTaskProcessor.taskMetaCache(GridTaskProcessor.java:269)
at 
org.apache.ignite.internal.processors.task.GridTaskProcessor.saveTaskMetadata(GridTaskProcessor.java:845)
at 
org.apache.ignite.internal.processors.task.GridTaskProcessor.startTask(GridTaskProcessor.java:703)
at 
org.apache.ignite.internal.processors.task.GridTaskProcessor.execute(GridTaskProcessor.java:448)
at 
org.apache.ignite.internal.processors.closure.GridClosureProcessor.runAsync(GridClosureProcessor.java:244)
at 
org.apache.ignite.internal.processors.closure.GridClosureProcessor.runAsync(GridClosureProcessor.java:216)
at 
org.apache.ignite.internal.IgniteComputeImpl.runAsync0(IgniteComputeImpl.java:704)
at 
org.apache.ignite.internal.IgniteComputeImpl.runAsync(IgniteComputeImpl.java:689)
at 
org.apache.ignite.internal.processors.cluster.GridClusterStateProcessorImpl.sendComputeChangeGlobalState(GridClusterStateProcessorImpl.java:837)
at 
org.apache.ignite.internal.processors.cluster.GridClusterStateProcessorImpl.changeGlobalState0(GridClusterStateProcessorImpl.java:684)
at 
org.apache.ignite.internal.processors.cluster.GridClusterStateProcessorImpl.changeGlobalState(GridClusterStateProcessorImpl.java:618)
at 
org.apache.ignite.internal.cluster.IgniteClusterImpl.active(IgniteClusterImpl.java:306)
at org.apache.ignite.internal.IgniteKernal.active(IgniteKernal.java:3541)
at 
org.apache.ignite.internal.processors.cache.IgniteClusterActivateDeactivateTest$7.run(IgniteClusterActivateDeactivateTest.java:627)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
at java.util.concurrent.FutureTask.run(FutureTask.java:-1)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748){code}
 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-7625) Fix links for Javadoc

[Peter Ivanov (JIRA)]

Peter Ivanov created IGNITE-7625:


 Summary: Fix links for Javadoc
 Key: IGNITE-7625
 URL: https://issues.apache.org/jira/browse/IGNITE-7625
 Project: Ignite
  Issue Type: Task
Reporter: Peter Ivanov
Assignee: Peter Ivanov


# {{/parent/pom.xml}}
Remove obsolete linking to http://docs.oracle.com/javase/7/docs/api/ -- linking 
to corresponding JavaSe documentation is automatic due to {{detectJavaApiLink}} 
flag of {{maven-javadoc-plugin}} set in {{true}} by default.
# {{/modules/yarn/src/main/java/org/apache/ignite/yarn/IgniteProvider.java}}
Fix direct link to {{Logger.java}} Javadoc documentation.
{code}
 * Please take a look at http://docs.oracle.com/javase/7/docs/api/java/util/logging/Logger.html";>Logger
 javadoc
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ignite pull request #3472: IGNITE-7437 Fix javadoc in partition based datase...

[dmitrievanthony]

GitHub user dmitrievanthony opened a pull request:

https://github.com/apache/ignite/pull/3472

IGNITE-7437 Fix javadoc in partition based dataset.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-7437-fix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3472.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3472


commit 9624f00a1f9a33af87050d82b1682bbb7f842d08
Author: dmitrievanthony 
Date:   2018-02-05T11:29:47Z

IGNITE-7437 Fix javadoc.




---

[GitHub] ignite pull request #3473: IGNITE-7625 Fix links for Javadoc

[vveider]

GitHub user vveider opened a pull request:

https://github.com/apache/ignite/pull/3473

IGNITE-7625 Fix links for Javadoc



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-7625

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3473.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3473


commit cd873da46dc425f30d9e206c1a36e9f2cbb21002
Author: Ivanov Petr 
Date:   2018-02-05T11:38:46Z

IGNITE-7625 Fix links for Javadoc




---

Re: Abandoned Patch Available JIRA tickets

[Pavel Tupitsyn]

Bumping the thread.

Denis Magda, as a PMC chair, can you please look into this?

On Tue, Jan 30, 2018 at 5:26 PM, Pavel Tupitsyn 
wrote:

> Igniters,
>
> We have *almost 60* of PATCH AVAILABLE tickets that are inactive for more
> than a month:
> https://issues.apache.org/jira/issues/?filter=12333285&;
> jql=project%3Dignite%20AND%20status%3D%22Patch%20Available%22%20AND%
> 20updated%20%3C%20-30d
>
> (JQL: project=ignite AND status="Patch Available" AND updated < -30d)
>
>
> Some of these tickets ask specific people for review without any answer.
> You can check if you have any pending reviews with this link:
> https://issues.apache.org/jira/issues/?filter=12333285&;
> jql=project%3Dignite%20AND%20status%3D%22Patch%20Available%22%20AND%
> 20updated%20%3C%20-30d%20AND%20comment%20%20~%20currentUser()
>
> (JQL: project=ignite AND status="Patch Available" AND updated < -30d AND
> comment  ~ currentUser())
>
>
> I encourage everyone to walk through these tickets and see if you can help
> with reviewing and merging some of these contributions.
>
> Thanks,
> Pavel
>

Re: Abandoned Patch Available JIRA tickets

[Dmitry Pavlov]

Hi Denis,

There is an interesting aspect of this issue for me.

How close is Ignite community in this metric (not reviewed/not merged
tickets) to other Apache communities? For example, to Cassandra.

Sincerely,
Dmitriy Pavlov


пн, 5 февр. 2018 г. в 14:51, Pavel Tupitsyn :

> Bumping the thread.
>
> Denis Magda, as a PMC chair, can you please look into this?
>
> On Tue, Jan 30, 2018 at 5:26 PM, Pavel Tupitsyn 
> wrote:
>
> > Igniters,
> >
> > We have *almost 60* of PATCH AVAILABLE tickets that are inactive for more
> > than a month:
> > https://issues.apache.org/jira/issues/?filter=12333285&;
> > jql=project%3Dignite%20AND%20status%3D%22Patch%20Available%22%20AND%
> > 20updated%20%3C%20-30d
> >
> > (JQL: project=ignite AND status="Patch Available" AND updated < -30d)
> >
> >
> > Some of these tickets ask specific people for review without any answer.
> > You can check if you have any pending reviews with this link:
> > https://issues.apache.org/jira/issues/?filter=12333285&;
> > jql=project%3Dignite%20AND%20status%3D%22Patch%20Available%22%20AND%
> > 20updated%20%3C%20-30d%20AND%20comment%20%20~%20currentUser()
> >
> > (JQL: project=ignite AND status="Patch Available" AND updated < -30d AND
> > comment  ~ currentUser())
> >
> >
> > I encourage everyone to walk through these tickets and see if you can
> help
> > with reviewing and merging some of these contributions.
> >
> > Thanks,
> > Pavel
> >
>

Re: Abandoned Patch Available JIRA tickets

[Pavel Tupitsyn]

Fixed links:

https://issues.apache.org/jira/issues/?jql=project%3Dignite%20AND%20status%3D%22Patch%20Available%22%20AND%20updated%20%3C%20-30d

https://issues.apache.org/jira/issues/?jql=project%3Dignite%20AND%20status%3D%22Patch%20Available%22%20AND%20updated%20%3C%20-30d%20AND%20comment%20%20~%20currentUser()

On Mon, Feb 5, 2018 at 3:05 PM, Dmitry Pavlov  wrote:

> Hi Denis,
>
> There is an interesting aspect of this issue for me.
>
> How close is Ignite community in this metric (not reviewed/not merged
> tickets) to other Apache communities? For example, to Cassandra.
>
> Sincerely,
> Dmitriy Pavlov
>
>
> пн, 5 февр. 2018 г. в 14:51, Pavel Tupitsyn :
>
> > Bumping the thread.
> >
> > Denis Magda, as a PMC chair, can you please look into this?
> >
> > On Tue, Jan 30, 2018 at 5:26 PM, Pavel Tupitsyn 
> > wrote:
> >
> > > Igniters,
> > >
> > > We have *almost 60* of PATCH AVAILABLE tickets that are inactive for
> more
> > > than a month:
> > > https://issues.apache.org/jira/issues/?filter=12333285&;
> > > jql=project%3Dignite%20AND%20status%3D%22Patch%20Available%22%20AND%
> > > 20updated%20%3C%20-30d
> > >
> > > (JQL: project=ignite AND status="Patch Available" AND updated < -30d)
> > >
> > >
> > > Some of these tickets ask specific people for review without any
> answer.
> > > You can check if you have any pending reviews with this link:
> > > https://issues.apache.org/jira/issues/?filter=12333285&;
> > > jql=project%3Dignite%20AND%20status%3D%22Patch%20Available%22%20AND%
> > > 20updated%20%3C%20-30d%20AND%20comment%20%20~%20currentUser()
> > >
> > > (JQL: project=ignite AND status="Patch Available" AND updated < -30d
> AND
> > > comment  ~ currentUser())
> > >
> > >
> > > I encourage everyone to walk through these tickets and see if you can
> > help
> > > with reviewing and merging some of these contributions.
> > >
> > > Thanks,
> > > Pavel
> > >
> >
>

[GitHub] ignite pull request #3472: IGNITE-7437 Fix javadoc in partition based datase...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/3472


---

Re: Eviction policies with persistence

[Ivan Rakov]

Guys,

+1 for renaming "eviction" for PDS mode.
I'd choose "page replacement" over "page swapping" for two reasons:
1) We already have "replace" metrics in DataRegionMetrics. The less we 
change existing namings, the better.
2) "Page swapping" term is common in operating systems, and that's why 
we'll confuse our "page swapping" (which is essential of Ignite native 
persistence) with actual OS page swapping (which usually signals about 
ineffective hardware resources usage and is better to be avoided) in 
many discussions later.


Best Regards,
Ivan Rakov

On 04.02.2018 23:42, Dmitry Pavlov wrote:

Hi Alex,

Thank you for explanation. Do you mind if I copy this explanation to wiki
page?

Strongly +1 for new term for page replacement mechanism for PDS mode. I
like 'page swapping' because it has analogue in operating systems.

New term will also identify feature by class name PageEviction... and
PageSwap... will point to class purpose.

Sincerely,
Dmitriy Pavlov

вс, 4 февр. 2018 г. в 23:05, Alexey Goncharuk :


Guys,

To clarify the questions, I would like to reiterate over the mechanics of
evictions and then suggest a renaming that should resolve such things in
the future.

First of all, Lucas is right - eviction policy only makes sense when native
persistence is disabled because what it does is actually freeing up memory
when a user hits the memory limit. The only way to do this is to destroy
inserted data because there is no other way to free memory. Now as per the
eviction mechanism, it is both per-page and per-entry: first, we choose a
page which fits most for eviction, however, we cannot simply erase the page
because quite a lot of data structures are referring to that page. To deal
with it, we read keys that the chosen page contains and then clear
individual cache entries, which in turn will clear up all necessary links.
If there are no concurrent updates, the page becomes empty and will be
reused for other user data. This is exactly what is explained on the wiki
page (at least in my reading of the page).

Second, at this point, I would rename page management mechanism with
enabled persistence from 'eviction' to 'page replacement' or 'page
swapping', because it does not destroy any data, but rather replaces a
chosen buffer in memory from one page to another. The content of neither
pages does not change during page replacement. This mechanism is unlikely
to be selected by a user because the effectiveness of page replacements
heavily depends on internal data structures and may change from version to
version, and may even be adaptive depending on the load pattern.

Hope this resolves the confusion.

--AG

2018-02-03 1:03 GMT+03:00 Denis Magda :


Dmitriy,

I’m surprised to hear that Random-LRU works at the entry level when

Ignite

persistence is off. Frankly, this section confuses a lot:
https://cwiki.apache.org/confluence/display/IGNITE/
Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-
underthehood-Entryeviction 

While it was assumed that the entry-based eviction works only for on-heap
caches:
https://apacheignite.readme.io/docs/evictions <
https://apacheignite.readme.io/docs/evictions>

Alex G., please chime in and clarify the misunderstanding.

—
Denis


On Feb 2, 2018, at 4:01 AM, Dmitry Pavlov 

wrote:

Hi Val,

I think it is quite accurate because eviction in case PDS enabled or

not

has quite different purposes.

1. Let us consider PDS enabled and page eviction occurs. First of all

it

is

page based eviction, but not entry-based. Actually data is not removed,

but

only written to disk. We can address this page later by ID.
PDS eviction is primarily the replacement of pages from memory to page

on

disk. This eviction policy should ensure the maximum performance of the
solution in the future. There is no data removal from grid in this

case.

And Ignite does not allow users to configure the policy. If benchmarks

show

that a change in policy results in increased performance, then we can
switch policy.

2. Entry-based eviction (if there is no persistence is available) is
slithly different. User will need to reload data into grid in case

entry

is

evicted but required in cache. In that case Ignite provides selection

of

policies.

Sincerely,
Dmirtiy Pavlov



чт, 1 февр. 2018 г. в 22:24, Valentin Kulichenko <
valentin.kuliche...@gmail.com>:


Guys,

Thanks for responses. But I still fail to understand how it affects a

user.

Are you saying that with persistence enabled Random-LRU is always used
regardless of what is specified in pageEvictionMode, while in

in-memory

only scenario we can also utilize Random-2-LRU for eviction? Is this
accurate? Are there any other differences?

-Val

On Thu, Feb 1, 2018 at 5:01 AM, Dmitry Pavlov 
wrote:


Hi Vyacheslav,

Yes, this is right, but for now PDS page based eviction uses

Random-LRU,

not Random-

Re: Abandoned Patch Available JIRA tickets

[Дмитрий Рябов]

Dmitry, there are 103 tickets for Cassandra [1] and 120 for Zookeeper [2].

[1] 
https://issues.apache.org/jira/browse/CASSANDRA-14132?jql=project%3Dcassandra%20AND%20status%3D"Patch%20Available"%20AND%20updated%20<%20-30d
[2] 
https://issues.apache.org/jira/issues/?jql=project%3Dzookeeper%20AND%20status%3D"Patch%20Available"%20AND%20updated%20<%20-30d

2018-02-05 15:08 GMT+03:00 Pavel Tupitsyn :
> Fixed links:
>
> https://issues.apache.org/jira/issues/?jql=project%3Dignite%20AND%20status%3D%22Patch%20Available%22%20AND%20updated%20%3C%20-30d
>
> https://issues.apache.org/jira/issues/?jql=project%3Dignite%20AND%20status%3D%22Patch%20Available%22%20AND%20updated%20%3C%20-30d%20AND%20comment%20%20~%20currentUser()
>
> On Mon, Feb 5, 2018 at 3:05 PM, Dmitry Pavlov  wrote:
>
>> Hi Denis,
>>
>> There is an interesting aspect of this issue for me.
>>
>> How close is Ignite community in this metric (not reviewed/not merged
>> tickets) to other Apache communities? For example, to Cassandra.
>>
>> Sincerely,
>> Dmitriy Pavlov
>>
>>
>> пн, 5 февр. 2018 г. в 14:51, Pavel Tupitsyn :
>>
>> > Bumping the thread.
>> >
>> > Denis Magda, as a PMC chair, can you please look into this?
>> >
>> > On Tue, Jan 30, 2018 at 5:26 PM, Pavel Tupitsyn 
>> > wrote:
>> >
>> > > Igniters,
>> > >
>> > > We have *almost 60* of PATCH AVAILABLE tickets that are inactive for
>> more
>> > > than a month:
>> > > https://issues.apache.org/jira/issues/?filter=12333285&;
>> > > jql=project%3Dignite%20AND%20status%3D%22Patch%20Available%22%20AND%
>> > > 20updated%20%3C%20-30d
>> > >
>> > > (JQL: project=ignite AND status="Patch Available" AND updated < -30d)
>> > >
>> > >
>> > > Some of these tickets ask specific people for review without any
>> answer.
>> > > You can check if you have any pending reviews with this link:
>> > > https://issues.apache.org/jira/issues/?filter=12333285&;
>> > > jql=project%3Dignite%20AND%20status%3D%22Patch%20Available%22%20AND%
>> > > 20updated%20%3C%20-30d%20AND%20comment%20%20~%20currentUser()
>> > >
>> > > (JQL: project=ignite AND status="Patch Available" AND updated < -30d
>> AND
>> > > comment  ~ currentUser())
>> > >
>> > >
>> > > I encourage everyone to walk through these tickets and see if you can
>> > help
>> > > with reviewing and merging some of these contributions.
>> > >
>> > > Thanks,
>> > > Pavel
>> > >
>> >
>>

Re: Eviction policies with persistence

[Dmitry Pavlov]

Hi Folks,

Thank you for your replies! I've updated wiki according Lucas & Alexey's
notes. Thank you!

https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-underthehood-Pagereplacement(rotationwithdisk)

I've named this section 'Page replacement (rotation with disk)'. I'm going
to update also class naming in my current PR (3469
)

Sincerely,
Dmitriy Pavlov

пн, 5 февр. 2018 г. в 15:25, Ivan Rakov :

> Guys,
>
> +1 for renaming "eviction" for PDS mode.
> I'd choose "page replacement" over "page swapping" for two reasons:
> 1) We already have "replace" metrics in DataRegionMetrics. The less we
> change existing namings, the better.
> 2) "Page swapping" term is common in operating systems, and that's why
> we'll confuse our "page swapping" (which is essential of Ignite native
> persistence) with actual OS page swapping (which usually signals about
> ineffective hardware resources usage and is better to be avoided) in
> many discussions later.
>
> Best Regards,
> Ivan Rakov
>
> On 04.02.2018 23:42, Dmitry Pavlov wrote:
> > Hi Alex,
> >
> > Thank you for explanation. Do you mind if I copy this explanation to wiki
> > page?
> >
> > Strongly +1 for new term for page replacement mechanism for PDS mode. I
> > like 'page swapping' because it has analogue in operating systems.
> >
> > New term will also identify feature by class name PageEviction... and
> > PageSwap... will point to class purpose.
> >
> > Sincerely,
> > Dmitriy Pavlov
> >
> > вс, 4 февр. 2018 г. в 23:05, Alexey Goncharuk <
> alexey.goncha...@gmail.com>:
> >
> >> Guys,
> >>
> >> To clarify the questions, I would like to reiterate over the mechanics
> of
> >> evictions and then suggest a renaming that should resolve such things in
> >> the future.
> >>
> >> First of all, Lucas is right - eviction policy only makes sense when
> native
> >> persistence is disabled because what it does is actually freeing up
> memory
> >> when a user hits the memory limit. The only way to do this is to destroy
> >> inserted data because there is no other way to free memory. Now as per
> the
> >> eviction mechanism, it is both per-page and per-entry: first, we choose
> a
> >> page which fits most for eviction, however, we cannot simply erase the
> page
> >> because quite a lot of data structures are referring to that page. To
> deal
> >> with it, we read keys that the chosen page contains and then clear
> >> individual cache entries, which in turn will clear up all necessary
> links.
> >> If there are no concurrent updates, the page becomes empty and will be
> >> reused for other user data. This is exactly what is explained on the
> wiki
> >> page (at least in my reading of the page).
> >>
> >> Second, at this point, I would rename page management mechanism with
> >> enabled persistence from 'eviction' to 'page replacement' or 'page
> >> swapping', because it does not destroy any data, but rather replaces a
> >> chosen buffer in memory from one page to another. The content of neither
> >> pages does not change during page replacement. This mechanism is
> unlikely
> >> to be selected by a user because the effectiveness of page replacements
> >> heavily depends on internal data structures and may change from version
> to
> >> version, and may even be adaptive depending on the load pattern.
> >>
> >> Hope this resolves the confusion.
> >>
> >> --AG
> >>
> >> 2018-02-03 1:03 GMT+03:00 Denis Magda :
> >>
> >>> Dmitriy,
> >>>
> >>> I’m surprised to hear that Random-LRU works at the entry level when
> >> Ignite
> >>> persistence is off. Frankly, this section confuses a lot:
> >>> https://cwiki.apache.org/confluence/display/IGNITE/
> >>> Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-
> >>> underthehood-Entryeviction  >>> confluence/display/IGNITE/Ignite+Durable+Memory+-+under+
> >>> the+hood#IgniteDurableMemory-underthehood-Entryeviction>
> >>>
> >>> While it was assumed that the entry-based eviction works only for
> on-heap
> >>> caches:
> >>> https://apacheignite.readme.io/docs/evictions <
> >>> https://apacheignite.readme.io/docs/evictions>
> >>>
> >>> Alex G., please chime in and clarify the misunderstanding.
> >>>
> >>> —
> >>> Denis
> >>>
>  On Feb 2, 2018, at 4:01 AM, Dmitry Pavlov 
> >> wrote:
>  Hi Val,
> 
>  I think it is quite accurate because eviction in case PDS enabled or
> >> not
>  has quite different purposes.
> 
>  1. Let us consider PDS enabled and page eviction occurs. First of all
> >> it
> >>> is
>  page based eviction, but not entry-based. Actually data is not
> removed,
> >>> but
>  only written to disk. We can address this page later by ID.
>  PDS eviction is primarily the replacement of pages from memory to page
> >> on
>  disk. This eviction policy should ensure the maximum performance of
> the
>  solution in the future. There is no data removal from grid in this
> >>

Re: Abandoned Patch Available JIRA tickets

[Vyacheslav Daradur]

Hi Pavel,

I've noticed that usually abandoned tickets are tickets without
specified "Fix Version", otherwise tickets should be viewed once at
least by release manager for moving to next release.

Should we do "Fix Version" as required field?


On Mon, Feb 5, 2018 at 3:44 PM, Дмитрий Рябов  wrote:
> Dmitry, there are 103 tickets for Cassandra [1] and 120 for Zookeeper [2].
>
> [1] 
> https://issues.apache.org/jira/browse/CASSANDRA-14132?jql=project%3Dcassandra%20AND%20status%3D"Patch%20Available"%20AND%20updated%20<%20-30d
> [2] 
> https://issues.apache.org/jira/issues/?jql=project%3Dzookeeper%20AND%20status%3D"Patch%20Available"%20AND%20updated%20<%20-30d
>
> 2018-02-05 15:08 GMT+03:00 Pavel Tupitsyn :
>> Fixed links:
>>
>> https://issues.apache.org/jira/issues/?jql=project%3Dignite%20AND%20status%3D%22Patch%20Available%22%20AND%20updated%20%3C%20-30d
>>
>> https://issues.apache.org/jira/issues/?jql=project%3Dignite%20AND%20status%3D%22Patch%20Available%22%20AND%20updated%20%3C%20-30d%20AND%20comment%20%20~%20currentUser()
>>
>> On Mon, Feb 5, 2018 at 3:05 PM, Dmitry Pavlov  wrote:
>>
>>> Hi Denis,
>>>
>>> There is an interesting aspect of this issue for me.
>>>
>>> How close is Ignite community in this metric (not reviewed/not merged
>>> tickets) to other Apache communities? For example, to Cassandra.
>>>
>>> Sincerely,
>>> Dmitriy Pavlov
>>>
>>>
>>> пн, 5 февр. 2018 г. в 14:51, Pavel Tupitsyn :
>>>
>>> > Bumping the thread.
>>> >
>>> > Denis Magda, as a PMC chair, can you please look into this?
>>> >
>>> > On Tue, Jan 30, 2018 at 5:26 PM, Pavel Tupitsyn 
>>> > wrote:
>>> >
>>> > > Igniters,
>>> > >
>>> > > We have *almost 60* of PATCH AVAILABLE tickets that are inactive for
>>> more
>>> > > than a month:
>>> > > https://issues.apache.org/jira/issues/?filter=12333285&;
>>> > > jql=project%3Dignite%20AND%20status%3D%22Patch%20Available%22%20AND%
>>> > > 20updated%20%3C%20-30d
>>> > >
>>> > > (JQL: project=ignite AND status="Patch Available" AND updated < -30d)
>>> > >
>>> > >
>>> > > Some of these tickets ask specific people for review without any
>>> answer.
>>> > > You can check if you have any pending reviews with this link:
>>> > > https://issues.apache.org/jira/issues/?filter=12333285&;
>>> > > jql=project%3Dignite%20AND%20status%3D%22Patch%20Available%22%20AND%
>>> > > 20updated%20%3C%20-30d%20AND%20comment%20%20~%20currentUser()
>>> > >
>>> > > (JQL: project=ignite AND status="Patch Available" AND updated < -30d
>>> AND
>>> > > comment  ~ currentUser())
>>> > >
>>> > >
>>> > > I encourage everyone to walk through these tickets and see if you can
>>> > help
>>> > > with reviewing and merging some of these contributions.
>>> > >
>>> > > Thanks,
>>> > > Pavel
>>> > >
>>> >
>>>



-- 
Best Regards, Vyacheslav D.

[GitHub] ignite pull request #3474: IGNITE-5265 Eviction Rate memory metric

[alex-plekhanov]

GitHub user alex-plekhanov opened a pull request:

https://github.com/apache/ignite/pull/3474

IGNITE-5265 Eviction Rate memory metric



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/alex-plekhanov/ignite ignite-5265

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3474.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3474


commit 54249b1950a274bb7b7919af6c72ceafa0059aec
Author: Aleksey Plekhanov 
Date:   2018-01-29T13:39:44Z

IGNITE-5265 Eviction rate memory metric

commit cee5c762c424ee090041290e786734d214cb1873
Author: Aleksey Plekhanov 
Date:   2018-02-01T13:06:27Z

IGNITE-5265 Page eviction metric

commit 4758333e7567a370fa85ad8287e291712cf9b1f4
Author: Aleksey Plekhanov 
Date:   2018-02-05T13:27:58Z

IGNITE-5265 Unit test




---

Re: Abandoned Patch Available JIRA tickets

[Pavel Tupitsyn]

Fix Version should not be a required field, uncheduled tickets are
perfectly normal.

On Mon, Feb 5, 2018 at 4:20 PM, Vyacheslav Daradur 
wrote:

> Hi Pavel,
>
> I've noticed that usually abandoned tickets are tickets without
> specified "Fix Version", otherwise tickets should be viewed once at
> least by release manager for moving to next release.
>
> Should we do "Fix Version" as required field?
>
>
> On Mon, Feb 5, 2018 at 3:44 PM, Дмитрий Рябов 
> wrote:
> > Dmitry, there are 103 tickets for Cassandra [1] and 120 for Zookeeper
> [2].
> >
> > [1] https://issues.apache.org/jira/browse/CASSANDRA-14132?
> jql=project%3Dcassandra%20AND%20status%3D"Patch%20Available"
> %20AND%20updated%20<%20-30d
> > [2] https://issues.apache.org/jira/issues/?jql=project%
> 3Dzookeeper%20AND%20status%3D"Patch%20Available"%20AND%
> 20updated%20<%20-30d
> >
> > 2018-02-05 15:08 GMT+03:00 Pavel Tupitsyn :
> >> Fixed links:
> >>
> >> https://issues.apache.org/jira/issues/?jql=project%
> 3Dignite%20AND%20status%3D%22Patch%20Available%22%20AND%
> 20updated%20%3C%20-30d
> >>
> >> https://issues.apache.org/jira/issues/?jql=project%
> 3Dignite%20AND%20status%3D%22Patch%20Available%22%20AND%
> 20updated%20%3C%20-30d%20AND%20comment%20%20~%20currentUser()
> >>
> >> On Mon, Feb 5, 2018 at 3:05 PM, Dmitry Pavlov 
> wrote:
> >>
> >>> Hi Denis,
> >>>
> >>> There is an interesting aspect of this issue for me.
> >>>
> >>> How close is Ignite community in this metric (not reviewed/not merged
> >>> tickets) to other Apache communities? For example, to Cassandra.
> >>>
> >>> Sincerely,
> >>> Dmitriy Pavlov
> >>>
> >>>
> >>> пн, 5 февр. 2018 г. в 14:51, Pavel Tupitsyn :
> >>>
> >>> > Bumping the thread.
> >>> >
> >>> > Denis Magda, as a PMC chair, can you please look into this?
> >>> >
> >>> > On Tue, Jan 30, 2018 at 5:26 PM, Pavel Tupitsyn <
> ptupit...@apache.org>
> >>> > wrote:
> >>> >
> >>> > > Igniters,
> >>> > >
> >>> > > We have *almost 60* of PATCH AVAILABLE tickets that are inactive
> for
> >>> more
> >>> > > than a month:
> >>> > > https://issues.apache.org/jira/issues/?filter=12333285&;
> >>> > > jql=project%3Dignite%20AND%20status%3D%22Patch%
> 20Available%22%20AND%
> >>> > > 20updated%20%3C%20-30d
> >>> > >
> >>> > > (JQL: project=ignite AND status="Patch Available" AND updated <
> -30d)
> >>> > >
> >>> > >
> >>> > > Some of these tickets ask specific people for review without any
> >>> answer.
> >>> > > You can check if you have any pending reviews with this link:
> >>> > > https://issues.apache.org/jira/issues/?filter=12333285&;
> >>> > > jql=project%3Dignite%20AND%20status%3D%22Patch%
> 20Available%22%20AND%
> >>> > > 20updated%20%3C%20-30d%20AND%20comment%20%20~%20currentUser()
> >>> > >
> >>> > > (JQL: project=ignite AND status="Patch Available" AND updated <
> -30d
> >>> AND
> >>> > > comment  ~ currentUser())
> >>> > >
> >>> > >
> >>> > > I encourage everyone to walk through these tickets and see if you
> can
> >>> > help
> >>> > > with reviewing and merging some of these contributions.
> >>> > >
> >>> > > Thanks,
> >>> > > Pavel
> >>> > >
> >>> >
> >>>
>
>
>
> --
> Best Regards, Vyacheslav D.
>

Re: Abandoned Patch Available JIRA tickets

[Anton Vinogradov]

 Guys,

It looks like main problem is "no one from commiters interested in
featureXXX", so, featureXXX at status PA last XXX years.
Sounds like a bad joke, but true.

There 2 types of Issues in PA:

1) Created by commiter.
And I see only ~20% of such issues at list mentioned above.
So, that type of issues have less problems with merge and review.

2) Created and implemented by contributor.
~80% of PA issues.
I see no presentations and discussions for the most of such issues, and
that's the reason of interest lack.

It looks like the only one way to solve this problem is to agree that
"contributor always should present idea & solution and motivate community
to review and merge solution".

Thoughts?

On Mon, Feb 5, 2018 at 4:34 PM, Pavel Tupitsyn  wrote:

> Fix Version should not be a required field, uncheduled tickets are
> perfectly normal.
>
> On Mon, Feb 5, 2018 at 4:20 PM, Vyacheslav Daradur 
> wrote:
>
> > Hi Pavel,
> >
> > I've noticed that usually abandoned tickets are tickets without
> > specified "Fix Version", otherwise tickets should be viewed once at
> > least by release manager for moving to next release.
> >
> > Should we do "Fix Version" as required field?
> >
> >
> > On Mon, Feb 5, 2018 at 3:44 PM, Дмитрий Рябов 
> > wrote:
> > > Dmitry, there are 103 tickets for Cassandra [1] and 120 for Zookeeper
> > [2].
> > >
> > > [1] https://issues.apache.org/jira/browse/CASSANDRA-14132?
> > jql=project%3Dcassandra%20AND%20status%3D"Patch%20Available"
> > %20AND%20updated%20<%20-30d
> > > [2] https://issues.apache.org/jira/issues/?jql=project%
> > 3Dzookeeper%20AND%20status%3D"Patch%20Available"%20AND%
> > 20updated%20<%20-30d
> > >
> > > 2018-02-05 15:08 GMT+03:00 Pavel Tupitsyn :
> > >> Fixed links:
> > >>
> > >> https://issues.apache.org/jira/issues/?jql=project%
> > 3Dignite%20AND%20status%3D%22Patch%20Available%22%20AND%
> > 20updated%20%3C%20-30d
> > >>
> > >> https://issues.apache.org/jira/issues/?jql=project%
> > 3Dignite%20AND%20status%3D%22Patch%20Available%22%20AND%
> > 20updated%20%3C%20-30d%20AND%20comment%20%20~%20currentUser()
> > >>
> > >> On Mon, Feb 5, 2018 at 3:05 PM, Dmitry Pavlov 
> > wrote:
> > >>
> > >>> Hi Denis,
> > >>>
> > >>> There is an interesting aspect of this issue for me.
> > >>>
> > >>> How close is Ignite community in this metric (not reviewed/not merged
> > >>> tickets) to other Apache communities? For example, to Cassandra.
> > >>>
> > >>> Sincerely,
> > >>> Dmitriy Pavlov
> > >>>
> > >>>
> > >>> пн, 5 февр. 2018 г. в 14:51, Pavel Tupitsyn :
> > >>>
> > >>> > Bumping the thread.
> > >>> >
> > >>> > Denis Magda, as a PMC chair, can you please look into this?
> > >>> >
> > >>> > On Tue, Jan 30, 2018 at 5:26 PM, Pavel Tupitsyn <
> > ptupit...@apache.org>
> > >>> > wrote:
> > >>> >
> > >>> > > Igniters,
> > >>> > >
> > >>> > > We have *almost 60* of PATCH AVAILABLE tickets that are inactive
> > for
> > >>> more
> > >>> > > than a month:
> > >>> > > https://issues.apache.org/jira/issues/?filter=12333285&;
> > >>> > > jql=project%3Dignite%20AND%20status%3D%22Patch%
> > 20Available%22%20AND%
> > >>> > > 20updated%20%3C%20-30d
> > >>> > >
> > >>> > > (JQL: project=ignite AND status="Patch Available" AND updated <
> > -30d)
> > >>> > >
> > >>> > >
> > >>> > > Some of these tickets ask specific people for review without any
> > >>> answer.
> > >>> > > You can check if you have any pending reviews with this link:
> > >>> > > https://issues.apache.org/jira/issues/?filter=12333285&;
> > >>> > > jql=project%3Dignite%20AND%20status%3D%22Patch%
> > 20Available%22%20AND%
> > >>> > > 20updated%20%3C%20-30d%20AND%20comment%20%20~%20currentUser()
> > >>> > >
> > >>> > > (JQL: project=ignite AND status="Patch Available" AND updated <
> > -30d
> > >>> AND
> > >>> > > comment  ~ currentUser())
> > >>> > >
> > >>> > >
> > >>> > > I encourage everyone to walk through these tickets and see if you
> > can
> > >>> > help
> > >>> > > with reviewing and merging some of these contributions.
> > >>> > >
> > >>> > > Thanks,
> > >>> > > Pavel
> > >>> > >
> > >>> >
> > >>>
> >
> >
> >
> > --
> > Best Regards, Vyacheslav D.
> >
>

[jira] [Created] (IGNITE-7626) Unify code in test which clean up persistence directories

[Eduard Shangareev (JIRA)]

Eduard Shangareev created IGNITE-7626:
-

 Summary: Unify code in test which clean up persistence directories
 Key: IGNITE-7626
 URL: https://issues.apache.org/jira/browse/IGNITE-7626
 Project: Ignite
  Issue Type: Improvement
Reporter: Eduard Shangareev






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-7627) Introduce a new eviction policy to avoid eviction of Index pages

[Alexey Popov (JIRA)]

Alexey Popov created IGNITE-7627:


 Summary: Introduce a new eviction policy to avoid eviction of 
Index pages
 Key: IGNITE-7627
 URL: https://issues.apache.org/jira/browse/IGNITE-7627
 Project: Ignite
  Issue Type: Improvement
  Components: persistence
Reporter: Alexey Popov


It would be great if we can have a new type of eviction policy that does not 
affect index pages when persistence is on. So, the index pages always will be 
in-mem and that will improve an overall system performance for random reads 
with persistence is on.

For instance, MongoDB works that way. It always keeps all indexes in-mem.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Thin client: Java bindings

[Alexey Kukushkin]

We also need asynchronous versions of all the thin client APIs. I created 
JIRA-7623    to address
async APIs. Proposed design:
Async methods are named by appending "Async" to the corresponding
syncMethod, e.g. putAsync, clearAsync.
Async methods return java.util.concurrent.Future
Async methods are not cancellable (presently cancellation is not supported
by the thin client protocol)
Use thin client protocol's Request ID to map async requests and responses.



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

[GitHub] ignite pull request #3435: IGNITE-7317: SVM Examples

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/3435


---

Obsolete build projects on CI server

[vveider]

Hi, Igniters.


Found (and marked obsolete) several build project on our CI server:
* [Obsolete] Assembly and Release [1] - previous release project on Java 7.
* [Obsolete] Ignite JDK compatibility [2] - project that did not run for 2
years.
* [Obsolete] Ignite Release [3] - more previous release project (preseeding
[1]).
* [Obsolete] Ignite Tests For Code Coverage [4] - failed (for now)
experiment of code coverage metrics gathering.

I suggest we drop them for good and concentrate on stabilization of tests
for Java 8 / Java 9. Any objections?

Also, if any ideas will arise what can be improved on TeamCity - I'm all
ears!


[1] https://ci.ignite.apache.org/project.html?projectId=AssemblyAndRelease
[2]
https://ci.ignite.apache.org/project.html?projectId=IgniteJdkCompatibility
[3] https://ci.ignite.apache.org/project.html?projectId=IgniteRelease
[4]
https://ci.ignite.apache.org/project.html?projectId=IgniteTestsForCodeCoverage



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

Request for contributor permissions

[Vinokurov Pavel]

Hello Igniters!

My name is Pavel. I want to contribute to Apache Ignite and want to start
with fixing one of these tickets -
https://issues.apache.org/jira/projects/IGNITE/issues.
Any help on this  will be appreciated.

Thanks!
-- 

Regards

Pavel Vinokurov

Re: Request for contributor permissions

[Vinokurov Pavel]

My Jira id is pvinokurov

2018-02-05 19:26 GMT+03:00 Vinokurov Pavel :

> Hello Igniters!
>
> My name is Pavel. I want to contribute to Apache Ignite and want to start
> with fixing one of these tickets - https://issues.apache.org/
> jira/projects/IGNITE/issues.
> Any help on this  will be appreciated.
>
> Thanks!
> --
>
> Regards
>
> Pavel Vinokurov
>



-- 

Regards

Pavel Vinokurov

Re: Obsolete build projects on CI server

[Anton Vinogradov]

No objections.

5 февр. 2018 г. 18:43 пользователь "vveider"  написал:

> Hi, Igniters.
>
>
> Found (and marked obsolete) several build project on our CI server:
> * [Obsolete] Assembly and Release [1] - previous release project on Java 7.
> * [Obsolete] Ignite JDK compatibility [2] - project that did not run for 2
> years.
> * [Obsolete] Ignite Release [3] - more previous release project (preseeding
> [1]).
> * [Obsolete] Ignite Tests For Code Coverage [4] - failed (for now)
> experiment of code coverage metrics gathering.
>
> I suggest we drop them for good and concentrate on stabilization of tests
> for Java 8 / Java 9. Any objections?
>
> Also, if any ideas will arise what can be improved on TeamCity - I'm all
> ears!
>
>
> [1] https://ci.ignite.apache.org/project.html?projectId=AssemblyAndRelease
> [2]
> https://ci.ignite.apache.org/project.html?projectId=IgniteJdkCompatibility
> [3] https://ci.ignite.apache.org/project.html?projectId=IgniteRelease
> [4]
> https://ci.ignite.apache.org/project.html?projectId=
> IgniteTestsForCodeCoverage
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>

Re: Obsolete build projects on CI server

[Dmitry Pavlov]

Ok for me too. I've never heard about usages of these configs.

пн, 5 февр. 2018 г. в 20:25, Anton Vinogradov :

> No objections.
>
> 5 февр. 2018 г. 18:43 пользователь "vveider" 
> написал:
>
> > Hi, Igniters.
> >
> >
> > Found (and marked obsolete) several build project on our CI server:
> > * [Obsolete] Assembly and Release [1] - previous release project on Java
> 7.
> > * [Obsolete] Ignite JDK compatibility [2] - project that did not run for
> 2
> > years.
> > * [Obsolete] Ignite Release [3] - more previous release project
> (preseeding
> > [1]).
> > * [Obsolete] Ignite Tests For Code Coverage [4] - failed (for now)
> > experiment of code coverage metrics gathering.
> >
> > I suggest we drop them for good and concentrate on stabilization of tests
> > for Java 8 / Java 9. Any objections?
> >
> > Also, if any ideas will arise what can be improved on TeamCity - I'm all
> > ears!
> >
> >
> > [1]
> https://ci.ignite.apache.org/project.html?projectId=AssemblyAndRelease
> > [2]
> >
> https://ci.ignite.apache.org/project.html?projectId=IgniteJdkCompatibility
> > [3] https://ci.ignite.apache.org/project.html?projectId=IgniteRelease
> > [4]
> > https://ci.ignite.apache.org/project.html?projectId=
> > IgniteTestsForCodeCoverage
> >
> >
> >
> > --
> > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
> >
>

[GitHub] ignite pull request #3475: IGNITE-6341: PDS Direct IO 2, removing long runni...

[dspavlov]

GitHub user dspavlov opened a pull request:

https://github.com/apache/ignite/pull/3475

IGNITE-6341: PDS Direct IO 2, removing long running tests from suite



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-6341-tests

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3475.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3475


commit 470dad8363defae3c935af8d00c40afa35d1fb15
Author: dpavlov 
Date:   2018-02-05T17:46:02Z

IGNITE-6341: PDS Direct IO, removing long running tests from suite




---

[GitHub] ignite pull request #3476: IGNITE-7626

[EdShangGG]

GitHub user EdShangGG opened a pull request:

https://github.com/apache/ignite/pull/3476

IGNITE-7626



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-gg-13414

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3476.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3476


commit 4925ffc10ce8e8287980eaec38b587512568a302
Author: Alexey Goncharuk 
Date:   2018-01-17T12:26:03Z

IGNITE-7453 Use GridUnsafe.cleanDirectBuffer in PageSnapshot

commit bcd68a058683b2f17b7ac60471b6e7aab3e4f6de
Author: Pavel Tupitsyn 
Date:   2018-01-17T12:38:20Z

IGNITE-7301 .NET: Baseline topology

This closes #3352

commit 66b96316a7775ce8a6e2ff4475185d5929e4998b
Author: devozerov 
Date:   2018-01-17T12:54:17Z

Merge branch 'master' into ignite-2.4

commit 268481c1cf7fe57df24be130eb67c3e3a13afe01
Author: Alexey Goncharuk 
Date:   2018-01-17T13:50:34Z

IGNITE-7453 Use GridUnsafe.cleanDirectBuffer in WalStat

commit db0cd105719c8ae713b13b34d9dca0a8cd45d377
Author: Pavel Tupitsyn 
Date:   2018-01-17T14:05:25Z

IGNITE-6776 .NET: Thin client: Add SQL & LINQ example

This closes #3390

commit c214db879101aa5660e2a50b11cd20964c0bc114
Author: Andrey Gura 
Date:   2018-01-17T12:42:41Z

ignite-7450 FileWriteAheadLogManager always uses RandomAccessFileIOFactory 
now

commit 75c27d5e49d7458e46eb46e6f87a445c3f1320ea
Author: Alexey Kuznetsov 
Date:   2018-01-18T02:25:19Z

IGNITE-7274 Web Console: Support multiple statements on Queries screen.
(cherry picked from commit 1926783)

commit 36cc822935387b2150e690c29bc6992dca0563f7
Author: Dmitriy Shabalin 
Date:   2018-01-18T04:49:08Z

IGNITE-7306 Web Console: Fixed export data from tables.
(cherry picked from commit 1bb60ec)

commit d753298b4012894b56f5c9218325211cd84a21d5
Author: Peter Ivanov 
Date:   2018-01-18T06:18:53Z

IGNITE-7107 Apache Ignite RPM packages

* added changelog to package specification

This closes #3396

commit 63445893f1bc75dc9777184499f7eabc1d4e51b1
Author: Denis Mekhanikov 
Date:   2018-01-18T08:36:18Z

IGNITE-3935 Use PeerDeployAware for streamer transformer - Fixes #3378.

Signed-off-by: Alexey Goncharuk 

commit f3f9f2a24b23027cf0c835140322e41a788932ae
Author: Pavel Tupitsyn 
Date:   2018-01-18T09:05:12Z

IGNITE-7413 Fix SqlDmlExample

This closes #3389

commit 1daa7c41bf1460a4d9a2b0c26a7a317f2fca3fb7
Author: Alexey Kuznetsov 
Date:   2018-01-18T10:14:53Z

IGNITE-7461 UI tools: Actualized data storage configuration.
(cherry picked from commit 577e632)

commit cf0080210d24d9dd8b057f959446fac5f8a4ca01
Author: dpavlov 
Date:   2018-01-18T10:53:29Z

IGNITE-7380 Implemented pluggable Direct IO - Fixes #3226.

Signed-off-by: Alexey Goncharuk 

commit dd06d0bd7ef266bfbe156e858b312d1ac86e8982
Author: Pavel Tupitsyn 
Date:   2018-01-18T12:55:49Z

IGNITE-7465 .NET: Fix SqlDdlExample failure with standalone node

commit 57479ec564e1761716da3d5f9feb7a64c396a9f9
Author: Pavel Tupitsyn 
Date:   2018-01-18T13:45:54Z

.NET: Fix CacheLocalTest.TestTxDeadlockDetection

commit bd6be8a4653322905a3b63850c7e033ce3801ce5
Author: Pavel Tupitsyn 
Date:   2018-01-18T18:25:05Z

.NET: Thin client: Fix OP_BINARY_TYPE_GET schema passing format

commit 97564d160586d6d57d300937e6b8877994e35fc7
Author: rkondakov 
Date:   2018-01-19T08:24:51Z

IGNITE-6456: Ability to separately enable or disable JDBC, ODBC and thin 
client endpoints. This closes #3309.

commit d50274ca8875c9680c12e8786ac355a787ba95e0
Author: Yakov Zhdanov 
Date:   2018-01-18T17:57:17Z

Javadoc enhancements - added @see

commit cb2d3cf22388ab19fb2d34ae5bdfc8f1b608db75
Author: Dmitriy Govorukhin 
Date:   2018-01-18T14:14:26Z

IGNITE-7471 Use soft reference for checkpoint entry contents to avoid 
excessive memory usage

commit 3965923369870bb4e8e57e3332c1a1eb1e5f5ed3
Author: rkondakov 
Date:   2018-01-19T09:00:55Z

IGNITE-6772: SQL exception messages became more informative. This closes 
#3342.

commit ba68cb0fa87f776fcd0499d030c333f182611f41
Author: devozerov 
Date:   2018-01-19T09:03:52Z

Merge remote-tracking branch 'origin/ignite-2.4' into ignite-2.4

commit b54c0c8786bd74aa0abb208f537c29f0c4be4b1e
Author: tledkov-gridgain 
Date:   2018-01-19T09:09:34Z

IGNITE-7248: JDBC: fixed schema resolution for streaming mode. This closes 
#3384.

commit 2f5997788ccff265a088921210f561985f640517
Author: Dmitriy Govorukhin 
Date:   2018-01-19T11:46:38Z

IGNITE-7471 fix npe

commit 7adce10750704cc50cbf54fa7020aa3b20da87cb
Author: Oleg Ignatenko 
Date:   2018-01-19T11:59:01Z

IGNITE-7454 Wrong package in IgniteExamplesMLTestSuite

this closes #3393

(cherry picked from commit cb1233a)

commit 25e91b60694c589d8bf50c63a

[GitHub] ignite pull request #3477: IGNITE-7626 Unify code in test which clean up per...

[EdShangGG]

GitHub user EdShangGG opened a pull request:

https://github.com/apache/ignite/pull/3477

IGNITE-7626 Unify code in test which clean up persistence directories



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-7626

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3477.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3477


commit db61a16bb5713d60cc7af7b5d68c0d31fa66ace4
Author: EdShangGG 
Date:   2018-02-05T14:31:32Z

IGNITE-7626 Unify code in test which clean up persistence directories




---

What is the purpose of a binary schema and schema registry?

[John Wilson]

Hi,

When objects are marshaled, Ignites adds a schema (BinarySchema) to the
BinarySchemaRegistry. Moreover, the documentation says that an object can
have a few different schemas.

My question:

   1. What does it mean for an object to have multiple schemas? (e.g. for a
   simple person object Person obj = new Person())
   2. What is the purpose of the binary schema registry?

Thanks,

Re: Subject: new committer: Nikolay Izhikov

[Dmitriy Setrakyan]

Congrats, Nikolay, great work on Spark data frames!

D.

On Mon, Feb 5, 2018 at 1:21 AM, Anton Vinogradov  wrote:

> The Project Management Committee (PMC) for Apache Ignite
> has invited Nikolay Izhikov to become a committer and we are pleased
> to announce that he has accepted.
>
> Nikolay is an author of the Spark Data Frame API integration [1].
> From what I see, Nikolay one of the most strongest and active contributors
> for Spark integration.
>
> Also, Nikolay is one who made the last mile in Context switch
> implementation for optimistic transactions [2].
>
> [1] https://issues.apache.org/jira/browse/IGNITE-3084
> [2] https://issues.apache.org/jira/browse/IGNITE-5712
>

[jira] [Created] (IGNITE-7628) SqlQuery hangs indefinitely with additional not registered in baseline node.

[Stanilovsky Evgeny (JIRA)]

Stanilovsky Evgeny created IGNITE-7628:
--

 Summary: SqlQuery hangs indefinitely with additional not 
registered in baseline node.
 Key: IGNITE-7628
 URL: https://issues.apache.org/jira/browse/IGNITE-7628
 Project: Ignite
  Issue Type: Bug
  Components: general
Affects Versions: 2.4
Reporter: Stanilovsky Evgeny
Assignee: Dmitriy Govorukhin
 Attachments: 
IgniteChangingBaselineCacheQueryAdditionalNodeSelfTest.java

SqlQuery hangs indefinitely while additional node registered in topology but 
still not in baseline.

Reproducer attached.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-7629) NPE when Finished indexes rebuilding for cache

[Alexandr Kuramshin (JIRA)]

Alexandr Kuramshin created IGNITE-7629:
--

 Summary: NPE when Finished indexes rebuilding for cache
 Key: IGNITE-7629
 URL: https://issues.apache.org/jira/browse/IGNITE-7629
 Project: Ignite
  Issue Type: Bug
Affects Versions: 2.3
Reporter: Alexandr Kuramshin


Occurs after destroying cache while rebuilding indices in progress

{noformat}
Runtime error caught during grid runnable execution: GridWorker 
[name=index-rebuild-worker, igniteInstanceName=DPL_GRID%DplGridNodeName, 
finished=false, hashCode=1940633631, interrupted=false, 
runner=pub-#2054%DPL_GRID%DplGridNodeName%]
java.lang.NullPointerException: null
at 
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$11.apply(GridCacheDatabaseSharedManager.java:1163)
at 
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$11.apply(GridCacheDatabaseSharedManager.java:1159)
at 
org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:383)
at 
org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:347)
at 
org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:335)
at 
org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:495)
at 
org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:474)
at 
org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:462)
at 
org.apache.ignite.internal.util.future.GridCompoundFuture.apply(GridCompoundFuture.java:125)
at 
org.apache.ignite.internal.util.future.GridCompoundFuture.apply(GridCompoundFuture.java:45)
at 
org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:383)
at 
org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:347)
at 
org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:335)
at 
org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:495)
at 
org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:474)
at 
org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:462)
at 
org.apache.ignite.internal.processors.query.GridQueryProcessor$3.body(GridQueryProcessor.java:1678)
...
{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-7630) NPE in SchemaIndexCacheVisitorImpl.processKey()

[Alexandr Kuramshin (JIRA)]

Alexandr Kuramshin created IGNITE-7630:
--

 Summary: NPE in SchemaIndexCacheVisitorImpl.processKey()
 Key: IGNITE-7630
 URL: https://issues.apache.org/jira/browse/IGNITE-7630
 Project: Ignite
  Issue Type: Bug
Affects Versions: 2.3
Reporter: Alexandr Kuramshin


Occurs after destroying cache while rebuilding indices in progress

{noformat}
[Thread] parallel-idx-worker-GridDhtColocatedCache [...]
[Emitter] o.a.i.i.p.q.s.SchemaIndexCacheVisitorImpl$AsyncWorker
[Message]  Error during parallel index create/rebuild.
java.lang.NullPointerException: null
at 
org.apache.ignite.internal.processors.query.schema.SchemaIndexCacheVisitorImpl.processKey(SchemaIndexCacheVisitorImpl.java:246)
at 
org.apache.ignite.internal.processors.query.schema.SchemaIndexCacheVisitorImpl.processPartition(SchemaIndexCacheVisitorImpl.java:206)
at 
org.apache.ignite.internal.processors.query.schema.SchemaIndexCacheVisitorImpl.processPartitions(SchemaIndexCacheVisitorImpl.java:165)
at 
org.apache.ignite.internal.processors.query.schema.SchemaIndexCacheVisitorImpl.access$100(SchemaIndexCacheVisitorImpl.java:50)
at 
org.apache.ignite.internal.processors.query.schema.SchemaIndexCacheVisitorImpl$AsyncWorker.body(SchemaIndexCacheVisitorImpl.java:316)
at 
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at java.lang.Thread.run(Thread.java:745)
{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-7631) Failed to clear page memory with AssertionError: Release pinned page

[Alexandr Kuramshin (JIRA)]

Alexandr Kuramshin created IGNITE-7631:
--

 Summary: Failed to clear page memory with AssertionError: Release 
pinned page
 Key: IGNITE-7631
 URL: https://issues.apache.org/jira/browse/IGNITE-7631
 Project: Ignite
  Issue Type: Bug
  Components: cache
Affects Versions: 2.3
Reporter: Alexandr Kuramshin


The following scenario produces a problem:

# Cluster was started and activated.
# Snapshot has been restored.
# Rebuilding indexes in progress.
# Caches destroyed.
# Multiple NPE exceptions occurs.
# The following exception occurs:

{noformat}
Failed to clear page memory
org.apache.ignite.IgniteCheckedException: Compound exception for 
CountDownFuture.
at 
org.apache.ignite.internal.util.future.CountDownFuture.addError(CountDownFuture.java:72)
at 
org.apache.ignite.internal.util.future.CountDownFuture.onDone(CountDownFuture.java:46)
at 
org.apache.ignite.internal.util.future.CountDownFuture.onDone(CountDownFuture.java:28)
at 
org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:462)
at 
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl$ClearSegmentRunnable.run(PageMemoryImpl.java:2449)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Suppressed: java.lang.AssertionError: Release pinned page: FullPageId 
[pageId=000100f40007, effectivePageId=00f40007, grpId=321390040]
at 
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl$PagePool.releaseFreePage(PageMemoryImpl.java:1593)
at 
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl$PagePool.access$1900(PageMemoryImpl.java:1465)
at 
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl$ClearSegmentRunnable.run(PageMemoryImpl.java:2440)
... 3 common frames omitted
Suppressed: java.lang.AssertionError: Release pinned page: FullPageId 
[pageId=000200019986, effectivePageId=00019986, grpId=-1903385190]
at 
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl$PagePool.releaseFreePage(PageMemoryImpl.java:1593)
at 
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl$PagePool.access$1900(PageMemoryImpl.java:1465)
at 
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl$ClearSegmentRunnable.run(PageMemoryImpl.java:2440)
... 3 common frames omitted
Suppressed: java.lang.AssertionError: Release pinned page: FullPageId 
[pageId=0002c85c, effectivePageId=c85c, grpId=-1903385190]
at 
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl$PagePool.releaseFreePage(PageMemoryImpl.java:1593)
at 
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl$PagePool.access$1900(PageMemoryImpl.java:1465)
at 
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl$ClearSegmentRunnable.run(PageMemoryImpl.java:2440)
... 3 common frames omitted
Suppressed: java.lang.AssertionError: Release pinned page: FullPageId 
[pageId=000232da, effectivePageId=32da, grpId=321390040]
at 
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl$PagePool.releaseFreePage(PageMemoryImpl.java:1593)
at 
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl$PagePool.access$1900(PageMemoryImpl.java:1465)
at 
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl$ClearSegmentRunnable.run(PageMemoryImpl.java:2440)
... 3 common frames omitted
Suppressed: java.lang.AssertionError: Release pinned page: FullPageId 
[pageId=000200011d30, effectivePageId=00011d30, grpId=-1903385190]
at 
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl$PagePool.releaseFreePage(PageMemoryImpl.java:1593)
at 
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl$PagePool.access$1900(PageMemoryImpl.java:1465)
at 
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl$ClearSegmentRunnable.run(PageMemoryImpl.java:2440)
... 3 common frames omitted
Suppressed: java.lang.AssertionError: Release pinned page: FullPageId 
[pageId=0002d346, effectivePageId=d346, grpId=-1903385190]
at 
org.apache.ignite

[jira] [Created] (IGNITE-7632) NPE in IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.updateIgfsMetrics()

[Alexandr Kuramshin (JIRA)]

Alexandr Kuramshin created IGNITE-7632:
--

 Summary: NPE in 
IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.updateIgfsMetrics()
 Key: IGNITE-7632
 URL: https://issues.apache.org/jira/browse/IGNITE-7632
 Project: Ignite
  Issue Type: Bug
  Components: cache
Affects Versions: 2.3
Reporter: Alexandr Kuramshin


Occurs on destroying cache while rebuilding indices in progress

{noformat}
Partition eviction failed, this can cause grid hang.
java.lang.NullPointerException: null
at 
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.updateIgfsMetrics(IgniteCacheOffheapManagerImpl.java:1576)
at 
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.finishRemove(IgniteCacheOffheapManagerImpl.java:1403)
at 
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.remove(IgniteCacheOffheapManagerImpl.java:1368)
at 
org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.remove(GridCacheOffheapManager.java:1312)
at 
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.remove(IgniteCacheOffheapManagerImpl.java:368)
at 
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.removeValue(GridCacheMapEntry.java:3224)
at 
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheEntry.clearInternal(GridDhtCacheEntry.java:588)
at 
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLocalPartition.clearAll(GridDhtLocalPartition.java:895)
at 
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLocalPartition.tryEvict(GridDhtLocalPartition.java:753)
at 
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPreloader$3.call(GridDhtPreloader.java:593)
at 
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPreloader$3.call(GridDhtPreloader.java:580)
at 
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6639)
at 
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:967)
...
{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)