Re: IGNITE-7285 Add default query timeout

2019-04-29 Thread Saikat Maitra
Hi Ivan, Yes, I checked this CacheQuery default value https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/query/CacheQuery.java#L200 Also, Andrew recommended the same in review feedback. https://github.com/apache/ignite/pull/6490#di

[MTCGA]: new failures in builds [3707580] needs to be handled

2019-04-29 Thread dpavlov . tasks
Hi Igniters, I've detected some new issue on TeamCity to be handled. You are more than welcomed to help. If your changes can lead to this failure(s): We're grateful that you were a volunteer to make the contribution to this project, but things change and you may no longer be able to finalize

Re: "Idle verify" to "Online verify"

2019-04-29 Thread Ivan Rakov
But how to keep this hash? I think, we can just adopt way of storing partition update counters. Update counters are: 1) Kept and updated in heap, see IgniteCacheOffheapManagerImpl.CacheDataStoreImpl#pCntr (accessed during regular cache operations, no page replacement latency issues) 2) Synchron

Re: IgniteDataFrame SparkSQL OR clause return incorrect result

2019-04-29 Thread Nikolay Izhikov
Hello, alex. Thanks for reporting this. 1. You can file a bug in Ignite Jira. 2. It would be even better if you write a simple self contained reproducer for your problem. 3. If you turnoff Ignite query optimization issue still reproducible? пн, 29 апр. 2019 г., 19:29 alexcwyu : > I am using Ign

IgniteDataFrame SparkSQL OR clause return incorrect result

2019-04-29 Thread alexcwyu
I am using IgniteDataFrame and using Spark SQL to query the dataframe. spark: 2.3.2 ignite: 2.7.0 I found a bug in SparkSQL while using Ignite. select count(*) from risk where val_date = '2019-04-26' and portf_id = 27315 -- correctly return 11 row select count(*) from risk w

Re: "Idle verify" to "Online verify"

2019-04-29 Thread Anton Vinogradov
Ivan, thanks for the analysis! >> With having pre-calculated partition hash value, we can automatically detect inconsistent partitions on every PME. Great idea, seems this covers all broken synс cases. It will check alive nodes in case the primary failed immediately and will check rejoining node

[jira] [Created] (IGNITE-11823) Client Heap Memory in Cluser

2019-04-29 Thread Venkatesh D J (JIRA)
Venkatesh D J created IGNITE-11823: -- Summary: Client Heap Memory in Cluser Key: IGNITE-11823 URL: https://issues.apache.org/jira/browse/IGNITE-11823 Project: Ignite Issue Type: Test

Re: "Idle verify" to "Online verify"

2019-04-29 Thread Ivan Rakov
Hi Anton, Thanks for sharing your ideas. I think your approach should work in general. I'll just share my concerns about possible issues that may come up. 1) Equality of update counters doesn't imply equality of partitions content under load. For every update, primary node generates update co

[jira] [Created] (IGNITE-11822) Wrong DistributedMetaStorage feature id

2019-04-29 Thread Ivan Bessonov (JIRA)
Ivan Bessonov created IGNITE-11822: -- Summary: Wrong DistributedMetaStorage feature id Key: IGNITE-11822 URL: https://issues.apache.org/jira/browse/IGNITE-11822 Project: Ignite Issue Type: Im

Re: Configuration for explicit plugins providing.

2019-04-29 Thread Anton Vinogradov
+1 to simplification. On Mon, Apr 29, 2019 at 3:32 PM Mikhail Petrov wrote: > Hi Igniters! > > Now plugin loading is based on presence of plugin class name in > META-INF/service/org.apache.ignite.plugin.PluginProvider file. I propose to > add special configuration in IgniteConfiguration for pass

Re: AI 3.0: writeSynchronizationMode re-thinking

2019-04-29 Thread Anton Vinogradov
Sergey, I'd like to continue the discussion since it closely linked to the problem I'm currently working on. 1) writeSynchronizationMode should not be a part of cache configuration, agree. This should be up to the user to decide how strong should be "update guarantee". So, I propose to have a spe

Configuration for explicit plugins providing.

2019-04-29 Thread Mikhail Petrov
Hi Igniters! Now plugin loading is based on presence of plugin class name in META-INF/service/org.apache.ignite.plugin.PluginProvider file. I propose to add special configuration in IgniteConfiguration for passing needed plugin instances explicitly. It provides ability to 1. specify plugins vi

[jira] [Created] (IGNITE-11821) Move rebalanceBatchSize and rebalanceBatchesPrefetchCnt to IgniteConfiguration level

2019-04-29 Thread Maxim Muzafarov (JIRA)
Maxim Muzafarov created IGNITE-11821: Summary: Move rebalanceBatchSize and rebalanceBatchesPrefetchCnt to IgniteConfiguration level Key: IGNITE-11821 URL: https://issues.apache.org/jira/browse/IGNITE-11821

[MTCGA]: new failures in builds [3704547] needs to be handled

2019-04-29 Thread dpavlov . tasks
Hi Igniters, I've detected some new issue on TeamCity to be handled. You are more than welcomed to help. If your changes can lead to this failure(s): We're grateful that you were a volunteer to make the contribution to this project, but things change and you may no longer be able to finalize

Re: Brainstorm: Make TC Run All faster

2019-04-29 Thread Nikolay Izhikov
Ivan, I thought by "faster" we all meant time between One push RunAll button and one get results. What else definition are possible here? В Пн, 29/04/2019 в 12:04 +0300, Павлухин Иван пишет: > Nikolay, > > > Why we should imagine it? > > Only in order to understand what do we mean by "faster".

"Idle verify" to "Online verify"

2019-04-29 Thread Anton Vinogradov
Igniters and especially Ivan Rakov, "Idle verify" [1] is a really cool tool, to make sure that cluster is consistent. 1) But it required to have operations paused during cluster check. At some clusters, this check requires hours (3-4 hours at cases I saw). I've checked the code of "idle verify" a

Re: Brainstorm: Make TC Run All faster

2019-04-29 Thread Павлухин Иван
Nikolay, > Why we should imagine it? Only in order to understand what do we mean by "faster". My question was: 1st approach will take less agent time in sum. 2nd approach will complete faster in wall clock time. And your main concern is "total agent time". Am I right? пн, 29 апр. 2019 г. в 12:0

Re: Brainstorm: Make TC Run All faster

2019-04-29 Thread Nikolay Izhikov
> Let's imagine that we have an infinite number of agents Why we should imagine it? We don't have infinite number of agents. And we have several concurrent Run All. В Пн, 29/04/2019 в 11:50 +0300, Павлухин Иван пишет: > Vyacheslav, > > I finally figured out that "faster" means "total agent tim

Re: Brainstorm: Make TC Run All faster

2019-04-29 Thread Павлухин Иван
Vyacheslav, I finally figured out that "faster" means "total agent time". Let's imagine that we have an infinite number of agents. And 2 approaches: 1. Uber "Build Apache Ignite" containing all checks. 2. Separate jobs for compilation, checkstyle and etc. 1st approach will take less agent time i

Re: [IEP-35] Monitoring & Profiling. Proof of concept

2019-04-29 Thread Nikolay Izhikov
Hello, Vyacheslav. Thanks for the feedback! > HttpExposer with Jetty's dependencies should be detached> from the core > module. Agreed. module hierarchy is the essence of the next steps. For now it just a proof of my ideas for Ignite monitoring we can discuss. > I like your approach with 'wrap

Re: Brainstorm: Make TC Run All faster

2019-04-29 Thread Vyacheslav Daradur
Hi, Ivan, We are in the thread "Make TC Run All faster", so the main benefit is to make TC faster :) Advantages: - 1 TC agent required instead of 4; - RunAll will be faster, in case of builds queue; Also, the "licenses" profile is included in the step of a release build. I believe check-style al

Re: Brainstorm: Make TC Run All faster

2019-04-29 Thread Павлухин Иван
Vyacheslav, Maxim, Can we once again outline what benefits aggregated "Build Apache Ignite" performing various checks has comparing to a modularized approach in which separate builds perform separate tasks? For example, modularized approach looks nice because it is similar to good practices in so

Re: IGNITE-7285 Add default query timeout

2019-04-29 Thread Павлухин Иван
Hi Saikat, It a compatibility with previous versions the reason for an indefinite timeout by default? сб, 27 апр. 2019 г. в 16:58, Saikat Maitra : > > Hi Alexey, Ivan, Andrew > > I think we can provide an option to configure defaultQueryOption at > IgniteConfiguration and set the default value =

[MTCGA]: new failures in builds [3703685] needs to be handled

2019-04-29 Thread dpavlov . tasks
Hi Igniters, I've detected some new issue on TeamCity to be handled. You are more than welcomed to help. If your changes can lead to this failure(s): We're grateful that you were a volunteer to make the contribution to this project, but things change and you may no longer be able to finalize