Issue with replicated cache

[Prasad Bhalerao]

Hi,

I am using ignite 2.6.0 version and the time out settings are as follows.

IgniteConfiguration cfg = new IgniteConfiguration();
cfg.setFailureDetectionTimeout(120000);
cfg.setNetworkTimeout(10000);
cfg.setClientFailureDetectionTimeout(120000);

I have 4 server nodes (n1,n2,n3,n4) and 6 client nodes. I am using a
replicated cache and cache configuration is as shown below.
As you can see write-through is false, read through is true and write
synchronization mode is FULL_SYNC.

I got an issue, a network entry was removed from network cache but some how
it was removed from only 3 server nodes (n2,n3,n4). I was able to see the
network entry on node n1 consistently for a day(when it was removed). So I
checked the logs for any errors/warnings but I could not find any.
I did not see any segmentation issue in logs, looked like cluster was in
healthy state.
When I checked the cache after 2 days, I could not find that entry. Cache
was in a state as it was supposed to be.  Servers were  not stopped and
restarted during this whole time.

Some how I am not able to reproduce this issue on dev env.

Is there any way to investigate/debug this issue? Can someone please advise?

private CacheConfiguration networkCacheCfg() {
  CacheConfiguration networkCacheCfg = new
CacheConfiguration<>(CacheName.NETWORK_CACHE.name());
  networkCacheCfg.setAtomicityMode(CacheAtomicityMode.ATOMIC);
  networkCacheCfg.setWriteThrough(false);
  networkCacheCfg.setReadThrough(true);
  networkCacheCfg.setRebalanceMode(CacheRebalanceMode.ASYNC);
  
networkCacheCfg.setWriteSynchronizationMode(CacheWriteSynchronizationMode.FULL_SYNC);
  networkCacheCfg.setBackups(this.backupCount);
  networkCacheCfg.setCacheMode(CacheMode.REPLICATED);
  Factory<NetworkDataCacheLoader> storeFactory =
FactoryBuilder.factoryOf(NetworkDataCacheLoader.class);
  networkCacheCfg.setCacheStoreFactory(storeFactory);
  networkCacheCfg.setIndexedTypes(DefaultDataAffinityKey.class,
NetworkData.class);
  networkCacheCfg.setSqlIndexMaxInlineSize(65);
  RendezvousAffinityFunction affinityFunction = new
RendezvousAffinityFunction();
  affinityFunction.setExcludeNeighbors(true);
  networkCacheCfg.setAffinity(affinityFunction);
  networkCacheCfg.setStatisticsEnabled(true);

  return networkCacheCfg;
}



Thanks,
PRasad