Hello All,
I am getting following exception when I try to load my system with large
amount of data.
My Setup Details:
1 locator, 3 cacheservers with 8g and all the regions are disk persistence
enabled. ( All this is running on single AWS cluster node )
Please give me some clues what I am missing here ?
[severe 2016/06/16 12:51:15.552 UTC S1 <Notification Handler> tid=0x40]
Uncaught exception in thread Thread[Notification
Handler,10,ResourceListenerInvokerThreadGroup]
com.gemstone.gemfire.distributed.DistributedSystemDisconnectedException:
DistributedSystem is shutting down, caused by
com.gemstone.gemfire.ForcedDisconnectException: Member isn't responding to
heartbeat requests
at
com.gemstone.gemfire.distributed.internal.membership.gms.mgr.GMSMembershipManager.directChannelSend(GMSMembershipManager.java:1719)
at
com.gemstone.gemfire.distributed.internal.membership.gms.mgr.GMSMembershipManager.send(GMSMembershipManager.java:1897)
at
com.gemstone.gemfire.distributed.internal.DistributionChannel.send(DistributionChannel.java:87)
at
com.gemstone.gemfire.distributed.internal.DistributionManager.sendOutgoing(DistributionManager.java:3427)
at
com.gemstone.gemfire.distributed.internal.DistributionManager.sendMessage(DistributionManager.java:3468)
at
com.gemstone.gemfire.distributed.internal.DistributionManager.putOutgoing(DistributionManager.java:1828)
at
com.gemstone.gemfire.internal.cache.control.ResourceAdvisor$ResourceProfileMessage.send(ResourceAdvisor.java:185)
at
com.gemstone.gemfire.internal.cache.control.ResourceAdvisor.updateRemoteProfile(ResourceAdvisor.java:448)
at
com.gemstone.gemfire.internal.cache.control.HeapMemoryMonitor.processLocalEvent(HeapMemoryMonitor.java:677)
at
com.gemstone.gemfire.internal.cache.control.HeapMemoryMonitor.updateStateAndSendEvent(HeapMemoryMonitor.java:485)
at
com.gemstone.gemfire.internal.cache.control.HeapMemoryMonitor.updateStateAndSendEvent(HeapMemoryMonitor.java:448)
at
com.gemstone.gemfire.internal.cache.control.HeapMemoryMonitor$2.run(HeapMemoryMonitor.java:718)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.gemstone.gemfire.ForcedDisconnectException: Member isn't
responding to heartbeat requests
at
com.gemstone.gemfire.distributed.internal.membership.gms.mgr.GMSMembershipManager.forceDisconnect(GMSMembershipManager.java:2551)
at
com.gemstone.gemfire.distributed.internal.membership.gms.membership.GMSJoinLeave.forceDisconnect(GMSJoinLeave.java:885)
at
com.gemstone.gemfire.distributed.internal.membership.gms.membership.GMSJoinLeave.processRemoveRequest(GMSJoinLeave.java:578)
at
com.gemstone.gemfire.distributed.internal.membership.gms.membership.GMSJoinLeave.processMessage(GMSJoinLeave.java:1540)
at
com.gemstone.gemfire.distributed.internal.membership.gms.messenger.JGroupsMessenger$JGroupsReceiver.receive(JGroupsMessenger.java:1061)
at org.jgroups.JChannel.invokeCallback(JChannel.java:816)
at org.jgroups.JChannel.up(JChannel.java:741)
at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1030)
at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
at org.jgroups.protocols.FlowControl.up(FlowControl.java:392)
at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1064)
at
org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:779)
at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426)
at
com.gemstone.gemfire.distributed.internal.membership.gms.messenger.StatRecorder.up(StatRecorder.java:69)
at
com.gemstone.gemfire.distributed.internal.membership.gms.messenger.AddressManager.up(AddressManager.java:74)
at org.jgroups.protocols.TP.passMessageUp(TP.java:1567)
at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1783)
at org.jgroups.util.DirectExecutor.execute(DirectExecutor.java:10)
at org.jgroups.protocols.TP.handleSingleMessage(TP.java:1695)
at org.jgroups.protocols.TP.receive(TP.java:1620)
at
com.gemstone.gemfire.distributed.internal.membership.gms.messenger.Transport.receive(Transport.java:158)
at org.jgroups.protocols.UDP$PacketReceiver.run(UDP.java:701)
... 1 more
