Re: Grid freezing

I figured this out at long last... The root cause of the problem was the Scan object's toString method: @Override public String toString() { return ReflectionToStringBuilder.reflectionToString(this); } It used the apache common-lang's RefelectionToStringBuilder and this jar

Re: Grid freezing

Is it possible that in dequeuePortionIds you will have the same keys(at least one the same key) for scanCache in different threads? If it's possible, I think it's pretty easy to face a deadlock here with _getCurrentMax(customerMaxCache, queryScan), while I suppose 2 different Scans could have the

Re: Grid freezing

The WARNING message above is not consistently showing up in the logs. Nodes are not going down either. What I have found is that running 1 server and 2 clients does NOT freeze, even under heavy loads. But having 2 ignite servers does freeze up. Two servers are unsuccessful either on separate box

Re: Grid freezing

Hi, guys. It looks like https://issues.apache.org/jira/browse/IGNITE-6313. On Fri, Oct 27, 2017 at 10:28 AM, Evgenii Zhuravlev < e.zhuravlev...@gmail.com> wrote: > In logs i see messages: > > WARNING: Job is being cancelled because master task node left grid (as there > is no one waiting for re

Re: Grid freezing

In logs i see messages: WARNING: Job is being cancelled because master task node left grid (as there is no one waiting for results, job will not be failed over) Most possible that it's a root cause of problem you've described earlier 2017-10-26 21:27 GMT+03:00 smurphy : > Here is an additional

Re: Grid freezing

Here is an additional log message that suggests that rolling back the transaction is not working. I do not know why the isolation level in this log is marked as READ_COMMITTED. All transactions are configured to be Optimistic and Serializable..: 2017-10-26 10:49:12,524 ERROR [dna-scan-engine 172.

Re: Grid freezing

One thing that I see in the logs looks to me like a NullPointerException on attempting to commit an optimistic/serializable transaction. My reading of this excpetion is that an optimistic lock conflict is correctly detected in prepare0 but that eventually the code incorrectly attempts to return th

Re: Grid freezing

log_1.txt log_2.txt log_3.txt I don't see any nodes shutting down until 9 minute

Re: Grid freezing

Looks like job was canceled. Are you sure that you didn't cancel this job directly or node on which this job running was not stopped? 2017-10-26 17:18 GMT+03:00 smurphy : > No - this is what the invocation looks like: > > DequeuePortionsCallable job = new > DequeuePortionsCallable(ignitePortionDe

Re: Grid freezing

No - this is what the invocation looks like: DequeuePortionsCallable job = new DequeuePortionsCallable(ignitePortionDequeuer); DequeuedPortionResponse response = _ignite.compute().call(job); -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Grid freezing

Is it possible that you run DequeuePortionsCallable with timeout? Evgenii 2017-10-26 6:40 GMT+03:00 smurphy : > I added a transaction timeout of 1 millis and a transation size of 100 > (transaction is Optimistic and Serializable)... > > I see no TransactionTimeoutExceptions, just the followi

Re: Grid freezing

I added a transaction timeout of 1 millis and a transation size of 100 (transaction is Optimistic and Serializable)... I see no TransactionTimeoutExceptions, just the following: CacheException: class org.apache.ignite.IgniteInterruptedException: Got interrupted while waiting for future to comp

Re: Grid freezing

Thanks Evgenii, I'll add that catch block and see if it sheds any light on the issue. The client's transaction configuration is set to Optimistic and Serializable and the transaction within the try with resources block is explicitly set to Optimistic and Serializable, which should preclude deadl

Re: Grid freezing

Looks like possible deadlock. Please add timeout for transactions and check it with deadlock detection, as described here: https://apacheignite.readme.io/docs/transactions#section-deadlock-detection Evgenii 2017-10-23 21:16 GMT+03:00 smurphy : > IgnitePortionDequeuer.java >

Re: Grid freezing

IgnitePortionDequeuer.java top.visor Hi Evgenii, See the attached top command and java file.. -- Sent from: http://apache-ignite-us

Re: Grid freezing

Hi! How many nodes do you have? How many CPUs? Could you provide a code of com.company.node.ignite.IgnitePortionDequeuer or at least a dequeuePortions method? Evgenii 2017-10-20 0:24 GMT+03:00 smurphy : > threaddump.tdump > t1317/threaddum

Grid freezing

threaddump.tdump I am using Ignite v2.1 and my code using Optimistic/Serializable transactions and is locking up. When it does, there are a lot of `WARNING: Found long running transaction` and `WARNING: Found long runni