YARN parameters and default values

I see that our YARN parameters and default values do not quite align with Ignite 2.x Durable Memory: IGNITE_MEMORY_OVERHEAD_PER_NODE: The amount of memory allocated to handle JV

[GitHub] ignite pull request #3912: Ignite 2.4.5

GitHub user slukyano opened a pull request: https://github.com/apache/ignite/pull/3912 Ignite 2.4.5 You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite ignite-2.4.5 Alternatively you can review and apply these

Re: Apache Ignite 2.4+ Go language client

...forgot OP_RESOURCE_CLOSE that is implemented also. Thanks, Aleksandr Thanks, Aleksandr On 25 April 2018 at 00:06, Aleksandr Sokolovskii wrote: > Hello All, > > Besides the SQL driver I implemented the following operations also: > OP_CACHE_GET_NAMES > OP_CACHE_CREATE_WITH_NAME > OP_CACHE_GET

Re: Apache Ignite 2.4+ Go language client

Hello All, Besides the SQL driver I implemented the following operations also: OP_CACHE_GET_NAMES OP_CACHE_CREATE_WITH_NAME OP_CACHE_GET_OR_CREATE_WITH_NAME OP_CACHE_CREATE_WITH_CONFIGURATION OP_CACHE_GET_OR_CREATE_WITH_CONFIGURATION OP_CACHE_GET_CONFIGURATION OP_CACHE_DESTROY OP_CACHE_GET OP_CACH

Re: New definition for affinity node (issues with baseline)

On Wed, Apr 25, 2018 at 4:13 AM, Vladimir Ozerov wrote: > Right, as far as I understand we are not arguing on whether BLT is needed > or not. The main questions are how to properly deliver this feature to > users and how to deal with co-location issues between persistent and > non-persistent cach

Re: New definition for affinity node (issues with baseline)

Right, as far as I understand we are not arguing on whether BLT is needed or not. The main questions are how to properly deliver this feature to users and how to deal with co-location issues between persistent and non-persistent caches. Looks like change policies are the way to go for the first que

Re: Ticket review checklist

+1. Once again, I beg for "small refactoring permission" in a checklist. As of today, separate tickets for small refactorings has lowest priority, since they neither fix any flaw nor add new functionality. Also, the attempts to make issue-related code safer / cleaner / more readable in "real" pull

Re: New definition for affinity node (issues with baseline)

Well, this means that the concept of baseline is still needed because we must not reassign partitions immediately (note that this is not identical to rebalance delay!). The approach you describe is identical to baseline change policies and I have nothing against this, their implementation was plann

[jira] [Created] (IGNITE-8382) Problem with ignite-spring-data and Spring Boot 2

Patrice R created IGNITE-8382: - Summary: Problem with ignite-spring-data and Spring Boot 2 Key: IGNITE-8382 URL: https://issues.apache.org/jira/browse/IGNITE-8382 Project: Ignite Issue Type: Bug

Re: New definition for affinity node (issues with baseline)

Alex, CockroachDB is based on RAFT and is able to repair itself automatically [1] [2]. Their approach looks reasonable to me and is pretty much similar to MongoDB and Cassandra. In short, you distinguish between short-term and long-term failures. 1) First, you wait for small time window in hope th

[GitHub] ignite pull request #3670: IGNITE-7823

GitHub user xtern reopened a pull request: https://github.com/apache/ignite/pull/3670 IGNITE-7823 You can merge this pull request into a Git repository by running: $ git pull https://github.com/xtern/ignite IGNITE-7823 Alternatively you can review and apply these changes as t

[GitHub] ignite pull request #3670: Separate cache for non-collocated IgniteSet (TC r...

Github user xtern closed the pull request at: https://github.com/apache/ignite/pull/3670 ---

Re: New definition for affinity node (issues with baseline)

- for in-memory caches, affinity would calculate with SAT/BLAT on the first step and because of it collocation would work between in-memory and persistent caches; - on the next step, if there are offline nodes, we would spread their partitions among alive nodes. This would save us from data loss.

Re: New definition for affinity node (issues with baseline)

Igniters, I have introduced DAT in opposition to BLAT (SAT) because they reflect how Ignite works. But I actually have concerns about the necessity of such separation. DAT exists only because we don't want to lose any data in in-memory caches. But there are alternatives. Besides BLAT auto-chang

Re: New definition for affinity node (issues with baseline)

Vladimir, Automatic cluster membership changes may be implemented to grow the topology, but auto-shrinking topology is usually not possible because a process cannot distinguish between a node shutdown and network partitioning. If we want to deal with split-brain scenarios as a grown-up system, we

[GitHub] ignite pull request #3911: IGNITE-8358 Destroy partition inside evictor to p...

GitHub user Jokser opened a pull request: https://github.com/apache/ignite/pull/3911 IGNITE-8358 Destroy partition inside evictor to prevent possible deadlock You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite

Re: New definition for affinity node (issues with baseline)

Ivan, This reasoning sounds questionable to me. First, separate logic for in memory and persistent regions means that we loose collocation between persistent and non persistent caches. Second, “data is still on disk” assumption might be not valid if node has left due to disk crash, or when data is

[GitHub] ignite pull request #3910: IGNITE-7896 FilePageStore truncate now actually r...

GitHub user ivandasch opened a pull request: https://github.com/apache/ignite/pull/3910 IGNITE-7896 FilePageStore truncate now actually remove redundant partition page file. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ivandasc

Re: New definition for affinity node (issues with baseline)

Stan, I believe it was discussed at the design proposal thread: http://apache-ignite-developers.2346864.n4.nabble.com/Cluster-auto-activation-design-proposal-td20295.html The short answer: backup factor decreases if node leaves. In non-persistent mode we have to rebalance data ASAP - otherwis

Re: Ticket review checklist

Vladimir, I am not talking about massive/sophisticated refactoring. But I believe that ask to extract some methods should be OK to do without an extra ticket. A checklist shouldn't be necessarily a set of certain rules but also it could include suggestion and reminders. On Tue, Apr 24, 2018 at 6

Re: New definition for affinity node (issues with baseline)

+ for Vladimir's point - adding more complexity may (and likely will) be even more misleading. Can we take a step back and discuss why do we need to have different behavior for persistent and in-memory caches? Can we make in-memory caches honor baseline instead of special-casing them? Thanks, Sta

Re: New definition for affinity node (issues with baseline)

Ed, Agreed. Can we see proposed API changes? On Tue, Apr 24, 2018 at 6:39 PM, Eduard Shangareev < eduard.shangar...@gmail.com> wrote: > Vladimir, > > It will be fixed, But it is not user-list. > > We (developers) should decide ourselves how to go ahead with these > concepts. > > And I think that

Re: Ticket review checklist

Ed, Refactoring is a separate task. If you would like to rework exchange future - please do this in a ticket "Refactor exchange task", nobody would against this. This is just a matter of creating separate ticket and separate PR. If one have a time for refactoring, it should not be a problem for hi

Re: New definition for affinity node (issues with baseline)

Vladimir, It will be fixed, But it is not user-list. We (developers) should decide ourselves how to go ahead with these concepts. And I think that our old approach to describe BLAT is sophisticated and not clear (maybe, even error-prone). On Tue, Apr 24, 2018 at 6:28 PM, Vladimir Ozerov wrote:

Re: Orphaned, duplicate, and main-class tests!

Yakov, Agree. On Tue, Apr 24, 2018 at 6:11 AM, Yakov Zhdanov wrote: > Alexey Goncharuk, Vladimir Ozerov, what do you think about these tests? > > I believe they were created as a part of variuos optimization and profiling > activities. I also think we can remove them since nobody cares about th

Re: New definition for affinity node (issues with baseline)

Guys, As a user I definitely do not want to think about BLATs, SATs, DATs, whatsoever. I want to query data, iterate over data, send compute tasks to data. If certain node is outside of BLAT and do not have data, then this is not affinity node. Can we just fix affinity logic to take in count BLAT

[GitHub] ignite pull request #3881: IGNITE-8313 Add trace logs on exchange phases and...

Github user asfgit closed the pull request at: https://github.com/apache/ignite/pull/3881 ---

Re: New definition for affinity node (issues with baseline)

Eduard, Can you please summarize code changes that you are proposing? I agree that BLT is a bit misleading term and DAT/SAT make more sense. However, establishing a consensus on v2.4 Baseline Topology terminology took a long time and seems like you are going to cause a bit more perturbations.

Re: Orphaned, duplicate, and main-class tests!

I agree with Yakov here. If nobody responds here we can consider we have lazy consensus on removal of tests. I'm going to review PRs from Ilya. вт, 24 апр. 2018 г. в 6:11, Yakov Zhdanov : > Alexey Goncharuk, Vladimir Ozerov, what do you think about these tests? > > I believe they were created as

Re: Reconsider TTL expire mechanics in Ignite

I think, it would be more fair and simple to configure distributed expiration as flag in cache configuration. By the way, we still have to store ordered set of expirable entries on every node. Having https://issues.apache.org/jira/browse/IGNITE-5874 merged, we can do the following: if distribu

[GitHub] ignite pull request #3909: Fix tx hanging on node stop

GitHub user AMashenkov opened a pull request: https://github.com/apache/ignite/pull/3909 Fix tx hanging on node stop For test purposes. You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite ignite-gg-13317-1.8 Al

New definition for affinity node (issues with baseline)

Hi, Igniters, I want to raise a topic about our affinity node definition. After adding baseline (affinity) topology (BL(A)T) things start being complicated. Plenty of bugs appears: IGNITE-8173 ignite.getOrCreateCache(cacheConfig).iterator() method works incorrect for replicated cache in case if

[jira] [Created] (IGNITE-8381) testNodeSingletonDeploy in Basic 2 has high fail rate

Dmitriy Pavlov created IGNITE-8381: -- Summary: testNodeSingletonDeploy in Basic 2 has high fail rate Key: IGNITE-8381 URL: https://issues.apache.org/jira/browse/IGNITE-8381 Project: Ignite Is

I want to contribute to Apache Ignite

Hello Ignite Community! My name is Alexandr. I want to contribute to Apache Ignite and want to start with this issue - IGNITE-7883, my JIRA username "a-polyakov". Thanks!

Re: Reconsider TTL expire mechanics in Ignite

Alexey, Actually, there are 2 cases with readFromBackup=true which is used by default for Replicated caches: -user touch expired entry on backup node: we can just return null and keep entry as-is with hope primary will remove it. -user touch alive entry on backup node: TTL should be updated on pr

Re: Reconsider TTL expire mechanics in Ignite

Andrey, No, in this case the entry must not be evicted and kept as-is because only primary node can decide when an entry must be expired. The read in this case should return null, though. I understand that we can get non-monotonic reads, but this is always the case when readFromBackup is true. 20

Re: Reconsider TTL expire mechanics in Ignite

Alexey, What if user touch backup entry via readFromBackup=true? Should we start distributed operation ( e.g. TTL update or expiration) in that case? On Tue, Apr 24, 2018 at 5:02 PM, Alexey Goncharuk < alexey.goncha...@gmail.com> wrote: > Ivan, > > Agree about the use-case when we have a read-wr

Re: Reconsider TTL expire mechanics in Ignite

Ivan, Agree about the use-case when we have a read-write-through store. However, we allow to use Ignite in-memory caches even without 3rd party stores, in this case the same issue is still present. Maybe we can keep local expire for read-through caches and have strongly consistent expire for other

Re: Ticket review checklist

Igniters, I don't understand why you are so against refactoring. Code already smells like hell. Methods 200+ line is normal. Exchange future is asking to be separated on several one. Transaction code could understand few people. If we separate refactoring from development it would mean that no on

Re: Reconsider TTL expire mechanics in Ignite

Alexey, Distributed expire will result in serious performance overhead, mostly on network level. I think, the main use case of TTL are in-memory caches that accelerate access to slower third-party data source. In such case nothing is broken if data is missing; strong consistency guarantees are

[jira] [Created] (IGNITE-8380) Affinity node calculation doesn't take into account BLT

Eduard Shangareev created IGNITE-8380: - Summary: Affinity node calculation doesn't take into account BLT Key: IGNITE-8380 URL: https://issues.apache.org/jira/browse/IGNITE-8380 Project: Ignite

Re: Reconsider TTL expire mechanics in Ignite

Huge +1. On Tue, Apr 24, 2018 at 4:05 PM, Alexey Goncharuk < alexey.goncha...@gmail.com> wrote: > Igniters, > > We recently experienced some issues with TTL with enabled persistence, the > issues were related to persistence implementation details. However, when we > were adding tests to cover mor

[jira] [Created] (IGNITE-8379) Add maven-surefire-plugin support for PDS Compatibility tests

Peter Ivanov created IGNITE-8379: Summary: Add maven-surefire-plugin support for PDS Compatibility tests Key: IGNITE-8379 URL: https://issues.apache.org/jira/browse/IGNITE-8379 Project: Ignite

cache size() calculation for MVCC

Hi Igniters, I need your advice on a task at hand. Currently cache API size() is a constant time operation, since the number of entries is maintained as a separate counter. However, for MVCC-enabled cache there can be multiple versions of the same entry. In order to calculate the size we need to

[GitHub] ignite pull request #3885: IGNITE-8339 Do not log to WAL partition own durin...

Github user asfgit closed the pull request at: https://github.com/apache/ignite/pull/3885 ---

Reconsider TTL expire mechanics in Ignite

Igniters, We recently experienced some issues with TTL with enabled persistence, the issues were related to persistence implementation details. However, when we were adding tests to cover more cases, we found more failures, which, I think, reveal some fundamental issues with expire mechanism. In

Re: Service grid redesign

Hi, Denis M., I'd like to pick up a ticket from IEP-17 next week. Could you please advise a ticket to start? On Tue, Apr 24, 2018 at 11:47 AM, Dmitriy Setrakyan wrote: > On Tue, Apr 24, 2018, 3:59 PM Denis Mekhanikov > wrote: > >> Dmitriy, >> >> After the proposed changes are made the utility

[jira] [Created] (IGNITE-8378) Java crash upon node start after some restarts during failover test with 500 logical and 26 physical caches

Ksenia Rybakova created IGNITE-8378: --- Summary: Java crash upon node start after some restarts during failover test with 500 logical and 26 physical caches Key: IGNITE-8378 URL: https://issues.apache.org/jira/bro

[jira] [Created] (IGNITE-8377) Add cluster (de)activation LifecycleBean callbacks

Alexey Goncharuk created IGNITE-8377: Summary: Add cluster (de)activation LifecycleBean callbacks Key: IGNITE-8377 URL: https://issues.apache.org/jira/browse/IGNITE-8377 Project: Ignite I

[jira] [Created] (IGNITE-8376) Add cluster (de)activation events

Alexey Goncharuk created IGNITE-8376: Summary: Add cluster (de)activation events Key: IGNITE-8376 URL: https://issues.apache.org/jira/browse/IGNITE-8376 Project: Ignite Issue Type: Improv

[GitHub] ignite pull request #3908: IGNITE-8252 NPE is replaced with an IgniteExcepti...

GitHub user sergey-chugunov-1985 opened a pull request: https://github.com/apache/ignite/pull/3908 IGNITE-8252 NPE is replaced with an IgniteException with informative message You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridg

[jira] [Created] (IGNITE-8375) NPE due to race on cache stop and timeout handler execution.

Alexei Scherbakov created IGNITE-8375: - Summary: NPE due to race on cache stop and timeout handler execution. Key: IGNITE-8375 URL: https://issues.apache.org/jira/browse/IGNITE-8375 Project: Ignit

[jira] [Created] (IGNITE-8374) Test IgnitePdsCorruptedStoreTest.testCacheMetaCorruption hangs during node start

Aleksey Plekhanov created IGNITE-8374: - Summary: Test IgnitePdsCorruptedStoreTest.testCacheMetaCorruption hangs during node start Key: IGNITE-8374 URL: https://issues.apache.org/jira/browse/IGNITE-8374

Re: IGNITE-8167

Hi Pavel, In this case Run All PDS is ok, but as default I suggest to always use run-all. Hi Dmitriy G, could you please take a look to this change? Sincerely, Dmitriy Pavlov вт, 24 апр. 2018 г. в 10:56, Pavel Sapezhko : > Run-All? As mentioned in contribute guide I was only need to run tests

Re: Apache Ignite 2.4+ Go language client

Aleksandr, Great job! Do you have any plans on adding new features to your client? Pavel, There are also CacheGet and CachePut [1] operations, as far as I can see. [1] - https://github.com/amsokol/ignite-go-client/blob/master/binary/v1/client.go#L120 Best Regards, Igor On Tue, Apr 24, 2018 a

[jira] [Created] (IGNITE-8373) BinaryObjectException: Cannot find schema for object with compact footer during load test

Ksenia Rybakova created IGNITE-8373: --- Summary: BinaryObjectException: Cannot find schema for object with compact footer during load test Key: IGNITE-8373 URL: https://issues.apache.org/jira/browse/IGNITE-8373

Re: IGNITE-6827 - Review needed.

Hi all, I've implemented corresponded .NET api. Pavel, could you review my PR, please? https://issues.apache.org/jira/browse/IGNITE-8075 2018-04-10 21:06 GMT+03:00 Dmitry Pavlov : > Hi Pavel, > > thank you for bring up test questions. It seems my previous comments were > not taken into account

[GitHub] ignite pull request #3907: IGNITE-8372 ZookeeperClusterNode was made Externa...

GitHub user sergey-chugunov-1985 opened a pull request: https://github.com/apache/ignite/pull/3907 IGNITE-8372 ZookeeperClusterNode was made Externalizable You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite ig

[GitHub] ignite pull request #3906: IGNITE-8191 Hotfix (don't wait for transition)

GitHub user alex-plekhanov opened a pull request: https://github.com/apache/ignite/pull/3906 IGNITE-8191 Hotfix (don't wait for transition) You can merge this pull request into a Git repository by running: $ git pull https://github.com/alex-plekhanov/ignite ignite-8191-fix Al

[jira] [Created] (IGNITE-8372) Cluster metrics are reported incorrectly on joining node with ZK-based discovery

Sergey Chugunov created IGNITE-8372: --- Summary: Cluster metrics are reported incorrectly on joining node with ZK-based discovery Key: IGNITE-8372 URL: https://issues.apache.org/jira/browse/IGNITE-8372

[jira] [Created] (IGNITE-8371) MVCC TX: Force key request during rebalance may cause error on backups.

Roman Kondakov created IGNITE-8371: -- Summary: MVCC TX: Force key request during rebalance may cause error on backups. Key: IGNITE-8371 URL: https://issues.apache.org/jira/browse/IGNITE-8371 Project:

Re: Service grid redesign

On Tue, Apr 24, 2018, 3:59 PM Denis Mekhanikov wrote: > Dmitriy, > > After the proposed changes are made the utility cache won't be needed at > all. > I was rather talking about prioritization. In my view, first and foremost we must fix deployment before anything else. D.

[jira] [Created] (IGNITE-8370) Web console: split page-signin into three separate pages

Ilya Borisov created IGNITE-8370: Summary: Web console: split page-signin into three separate pages Key: IGNITE-8370 URL: https://issues.apache.org/jira/browse/IGNITE-8370 Project: Ignite Iss

Re: [GitHub] ignite pull request #3719: IGNITE-8048 merge query entities for dynamic cach...

Hi Manu, Activation/deactivation process does not trigger creation of new columns or indexes. They are only created either on cache create (dynamic or on node startup), or during DDL command execution. Vladimir. On Wed, Apr 18, 2018 at 6:57 PM, Manu wrote: > Hi, > > Sorry, I assumed that Query

Re: Service grid redesign

Dmitriy, After the proposed changes are made the utility cache won't be needed at all. I created the tickets for this IEP, you can find them on it's page: https://cwiki.apache.org/confluence/display/IGNITE/IEP-17%3A+Oil+Change+in+Service+Grid Or by label iep-17. The one for getting rid of the ut

Re: IGNITE-8167

Run-All? As mentioned in contribute guide I was only need to run tests that have been affected by my changes. So I used Persistent Data Store test suite. https://ci.ignite.apache.org/viewLog.html?buildId=1187554&tab=buildResultsDiv&buildTypeId=IgniteTests24Java8_RunAllPds On Mon, Apr 23, 2018 at 9

[jira] [Created] (IGNITE-8369) Zookeeper discovery SPI uses ConcurrentHashMap from netty

Alexey Goncharuk created IGNITE-8369: Summary: Zookeeper discovery SPI uses ConcurrentHashMap from netty Key: IGNITE-8369 URL: https://issues.apache.org/jira/browse/IGNITE-8369 Project: Ignite

Re: Apache Ignite 2.4+ Go language client

Any chance we can add key-value support as well? On Tue, Apr 24, 2018, 2:48 PM Pavel Tupitsyn wrote: > Hi Aleksandr, > > This is awesome, thank you! > > However, let's make it clear that this client supports SQL only, > and none of the other Thin Client protocol features. > > Pavel > > On Mon, A