[
https://issues.apache.org/jira/browse/IGNITE-2146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15058185#comment-15058185
]
Noam Liran commented on IGNITE-2146:
------------------------------------
Hi Semyon,
Glad to help! I think we might be hitting the second issue from time to time in
our cluster.. sometimes when nodes join/leave the topology, a deadlock that
involves some cache operations occurs. We haven't been able to pinpoint the
issue but this might be it.
We're looking forward to see this fixed in 1.5 as we already have some pending
issues that 1.5 will solve :)
Best Regards
Noam
> Service deployment using a cache-accessing-predicate in node filter sometimes
> causes a deadlock
> -----------------------------------------------------------------------------------------------
>
> Key: IGNITE-2146
> URL: https://issues.apache.org/jira/browse/IGNITE-2146
> Project: Ignite
> Issue Type: Bug
> Affects Versions: ignite-1.4, 1.5
> Environment: Mac OS X 10.11
> java version "1.8.0_66"
> Java(TM) SE Runtime Environment (build 1.8.0_66-b17)
> Java HotSpot(TM) 64-Bit Server VM (build 25.66-b17, mixed mode)
> Ignite 1.4.0 + Ignite-1.5.0-b1
> Reporter: Noam Liran
> Assignee: Semen Boikov
> Fix For: 1.5
>
> Attachments: stacktrace.txt
>
>
> We're deploying a service on a cluster group. The cluster group is filtered
> using a predicate. The predicate accesses a cache key to determine if it
> should return true for a node or not.
> If a new node joins the cluster while the predicate is running a deadlock is
> encountered: cache functions do not return and the node does not finish
> joining the cluster.
> I've created a sample program to reproduce this on 1.4.0 and also tried to
> reproduce it on 1.5.0-b1;
> On 1.4.0 the deadlock is usually reproduced, on 1.5.0-b1 it might take a few
> tries (I'm guessing it's a timing issue.. I tried to improve the odds of
> reproduction by using a countdown latch).
> Code:
> {code:title=Main.java}
> import org.apache.ignite.Ignite;
> import org.apache.ignite.IgniteCache;
> import org.apache.ignite.Ignition;
> import org.apache.ignite.cache.CacheAtomicWriteOrderMode;
> import org.apache.ignite.cache.CacheAtomicityMode;
> import org.apache.ignite.cache.CacheMode;
> import org.apache.ignite.cache.CacheWriteSynchronizationMode;
> import org.apache.ignite.cluster.ClusterGroup;
> import org.apache.ignite.configuration.CacheConfiguration;
> import org.apache.ignite.configuration.IgniteConfiguration;
> import org.apache.ignite.services.Service;
> import org.apache.ignite.services.ServiceContext;
> import java.util.concurrent.CountDownLatch;
> /**
> * Created by noliran on 08/12/2015.
> */
> public class Main {
> public static IgniteCache<String, Object> cache1;
> public static CountDownLatch latch = new CountDownLatch(1);
> public static CacheConfiguration<String, Object> CACHE_CONFIG = new
> CacheConfiguration<String, Object>()
> .setName("testCache")
> .setAtomicityMode(CacheAtomicityMode.ATOMIC)
> .setCacheMode(CacheMode.REPLICATED)
>
> .setWriteSynchronizationMode(CacheWriteSynchronizationMode.FULL_SYNC)
> .setAtomicWriteOrderMode(CacheAtomicWriteOrderMode.PRIMARY);
> public static void main(String[] args) throws InterruptedException {
> IgniteConfiguration igniteConfiguration = new IgniteConfiguration();
> Ignite ignite1 =
> Ignition.start(igniteConfiguration.setGridName("grid1"));
> System.out.println("Creating cache");
> cache1 = ignite1.getOrCreateCache(CACHE_CONFIG);
> ClusterGroup group = ignite1.cluster().forPredicate(node -> {
> System.out.println("predicate: starting");
> latch.countDown();
> try {
> System.out.println("predicate: before sleep");
> Thread.sleep(10_000);
> } catch (InterruptedException e) {
> e.printStackTrace();
> }
> System.out.println("predicate: before containsKey");
> boolean b1 = cache1.containsKey(node.id().toString());
> System.out.println("predicate: returning");
> return b1;
> });
> System.out.println("Deploying service with cache-based predicate");
> new Thread(() -> {
> ignite1.services(group).deployNodeSingleton("testService", new
> TestService());
> }).start();
> System.out.println("Service in deployment.");
> latch.await();
> System.out.println("Starting second Ignite instance..");
> Ignite ignite2 =
> Ignition.start(igniteConfiguration.setGridName("grid2"));
> System.out.println("Second Ignite instance started successfully!");
> // This isn't going to be printed.
> }
> public static class TestService implements Service
> {
> public void execute(ServiceContext ctx) throws Exception {
> System.out.println("execute()"); }
> public void init(ServiceContext ctx) throws Exception {
> System.out.println("init()"); }
> public void cancel(ServiceContext ctx) {
> System.out.println("cancel()"); }
> }
> }
> {code}
> Output:
> {noformat}
> [16:27:01] __________ ________________
> [16:27:01] / _/ ___/ |/ / _/_ __/ __/
> [16:27:01] _/ // (7 7 // / / / / _/
> [16:27:01] /___/\___/_/|_/___/ /_/ /___/
> [16:27:01]
> [16:27:01] ver. 1.5.0-b1#20151201-sha1:062d440c
> [16:27:01] 2015 Copyright(C) Apache Software Foundation
> [16:27:01]
> [16:27:01] Ignite documentation: http://ignite.apache.org
> [16:27:01]
> [16:27:01] Quiet mode.
> [16:27:01] ^-- To see **FULL** console log here add -DIGNITE_QUIET=false or
> "-v" to ignite.{sh|bat}
> [16:27:01]
> [16:27:01] OS: Mac OS X 10.11 x86_64
> [16:27:01] VM information: Java(TM) SE Runtime Environment 1.8.0_66-b17
> Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 25.66-b17
> [16:27:01] Initial heap size is 256MB (should be no less than 512MB, use
> -Xms512m -Xmx512m).
> [16:27:01] Configured plugins:
> [16:27:01] ^-- None
> [16:27:01]
> [16:27:01] Security status [authentication=off, tls/ssl=off]
> [16:27:03] To start Console Management & Monitoring run
> ignitevisorcmd.{sh|bat}
> [16:27:03]
> [16:27:03] Ignite node started OK (id=4821080c, grid=grid1)
> [16:27:03] Topology snapshot [ver=1, servers=1, clients=0, CPUs=4, heap=3.6GB]
> Creating cache
> Deploying service with cache-based predicate
> Service in deployment.
> predicate: starting
> predicate: before sleep
> Starting second Ignite instance..
> [16:27:03] __________ ________________
> [16:27:03] / _/ ___/ |/ / _/_ __/ __/
> [16:27:03] _/ // (7 7 // / / / / _/
> [16:27:03] /___/\___/_/|_/___/ /_/ /___/
> [16:27:03]
> [16:27:03] ver. 1.5.0-b1#20151201-sha1:062d440c
> [16:27:03] 2015 Copyright(C) Apache Software Foundation
> [16:27:03]
> [16:27:03] Ignite documentation: http://ignite.apache.org
> [16:27:03]
> [16:27:03] Quiet mode.
> [16:27:03] ^-- To see **FULL** console log here add -DIGNITE_QUIET=false or
> "-v" to ignite.{sh|bat}
> [16:27:03]
> [16:27:03] OS: Mac OS X 10.11 x86_64
> [16:27:03] VM information: Java(TM) SE Runtime Environment 1.8.0_66-b17
> Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 25.66-b17
> [16:27:03] Initial heap size is 256MB (should be no less than 512MB, use
> -Xms512m -Xmx512m).
> [16:27:03] Configured plugins:
> [16:27:03] ^-- None
> [16:27:03]
> [16:27:03] Security status [authentication=off, tls/ssl=off]
> [16:27:04] Topology snapshot [ver=2, servers=2, clients=0, CPUs=4, heap=3.6GB]
> predicate: before containsKey
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
