Hi,
We have an 8 Node 1.0.10 Cassandra cluster with Replication of 3, running
on customer location, they all should be in the same DC, but we are NOT
100% sure.
We are having problem with the our own Cassandra client, the log file of
the client shows many "socket timeout" and "socket closed" errors, and many
delays, in the Reads.
while debugging the issue we found the following error in the our Cassandra
client:
Caused by: UnavailableException()
Cassandra log file does not show errors..
What does that mean? any idea?
=========================================
[com.pdf.common.COMN_Log4jLogger] Failed to write Context Flow row to
column 20140317210716767_a3ccda262dbdd287a5e538ec95483a74_40:
UnavailableException()
at
org.apache.cassandra.thrift.Cassandra$batch_mutate_result.read(Cassandra.java:19487)
at
org.apache.cassandra.thrift.Cassandra$Client.recv_batch_mutate(Cassandra.java:1035)
at
org.apache.cassandra.thrift.Cassandra$Client.batch_mutate(Cassandra.java:1009)
at
com.pdf.datastore.cassandra.DSTO_CassandraClient.putLTStoreContextFlowRow(DSTO_CassandraClient.java:2673)
at
com.pdf.rtapi.RAPI_DataStore.putLTStoreContextFlowRow(RAPI_DataStore.java:1254)
at
com.pdf.rtapi.RAPI_InternalInterface.putLTStoreContextFlowRow(RAPI_InternalInterface.java:756)
at
com.pdf.common.rtapi.impl.RtApiFacadeImpl.putLTStoreContextFlowRow(RtApiFacadeImpl.java:340)
at
com.pdf.online.service.impl.OnlineServiceImpl.notifyDCContextFlow(OnlineServiceImpl.java:1499)
at
com.pdf.online.service.impl.OnlineServiceImpl.process(OnlineServiceImpl.java:602)
============================================
Different error..
2014-Mar-17 21:51:20,531 - INFO TID=Thread-6
(HornetQ-client-global-threads-4002470)
[com.pdf.online.service.impl.OnlineServiceImpl] Recipe name is:
P13_01_28NM_M5_002AS3_NOVA
2014-Mar-17 21:51:20,531 - INFO TID=Thread-6
(HornetQ-client-global-threads-4002470)
[com.pdf.online.service.impl.OnlineServiceImpl] Reading framed data (trace
+ context) for 'dcId' = 44140
2014-Mar-17 21:51:20,531 - INFO TID=Thread-6
(HornetQ-client-global-threads-4002470) [com.pdf.common.COMN_Log4jLogger]
RAPI_DataStore::readFramedData() -- Creating key base:
equipmentId_moduleId: POL1702_CleanerBrush1
2014-Mar-17 21:51:20,531 - INFO TID=Thread-6
(HornetQ-client-global-threads-4002470) [com.pdf.common.COMN_Log4jLogger]
RAPI_DataStore::readFramedData() -- Hashed key:
12169de1f273c1d38c893c0dfcbf10ff
2014-Mar-17 21:51:20,531 - INFO TID=Thread-6
(HornetQ-client-global-threads-4002470) [com.pdf.common.COMN_Log4jLogger]
RAPI_DataStore::readFramedData() -- DSTO_ContextKey =
12169de1f273c1d38c893c0dfcbf10ff:20131115070726207, dcId = 44140
2014-Mar-17 21:51:20,531 - INFO TID=Thread-6
(HornetQ-client-global-threads-4002470) [com.pdf.common.COMN_Log4jLogger]
Call checkConnection().
2014-Mar-17 21:51:20,531 - INFO TID=Thread-6
(HornetQ-client-global-threads-4002470) [com.pdf.common.COMN_Log4jLogger]
Call checkConnection().
2014-Mar-17 21:51:20,531 - INFO TID=Thread-6
(HornetQ-client-global-threads-4002470) [com.pdf.common.COMN_Log4jLogger]
UnavailableException : null: UnavailableException()
at
org.apache.cassandra.thrift.Cassandra$get_slice_result.read(Cassandra.java:7764)
at
org.apache.cassandra.thrift.Cassandra$Client.recv_get_slice(Cassandra.java:570)
at
org.apache.cassandra.thrift.Cassandra$Client.get_slice(Cassandra.java:542)
at
com.pdf.datastore.cassandra.DSTO_CassandraClient.readContextData(DSTO_CassandraClient.java:1894)
at
com.pdf.datastore.cassandra.DSTO_CassandraClient.readFramedData(DSTO_CassandraClient.java:1848)
at com.pdf.rtapi.RAPI_DataStore.readFramedData(RAPI_DataStore.java:720)
at
com.pdf.rtapi.RAPI_InternalInterface.readFramedData(RAPI_InternalInterface.java:741)
at
com.pdf.common.rtapi.impl.RtApiFacadeImpl.readFramedData(RtApiFacadeImpl.java:261)
at
com.pdf.online.service.impl.OnlineServiceImpl.readCassandraData(OnlineServiceImpl.java:2189)
at
com.pdf.online.service.impl.OnlineServiceImpl.process(OnlineServiceImpl.java:328)
at
com.pdf.online.mdb.OnlineEventDrivenBean.process(OnlineEventDrivenBean.java:302)
at
com.pdf.online.mdb.OnlineEventDrivenBean.onMessage(OnlineEventDrivenBean.java:211)
at sun.reflect.GeneratedMethodAccessor83.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.jboss.invocation.Invocation.performCall(Invocation.java:359)
at
org.jboss.ejb.MessageDrivenContainer$ContainerInterceptor.invoke(MessageDrivenContainer.java:495)
at
org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:158)
at
org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:63)
at
org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:121)
at
org.jboss.ejb.plugins.AbstractTxInterceptorBMT.invokeNext(AbstractTxInterceptorBMT.java:173)
at
org.jboss.ejb.plugins.MessageDrivenTxInterceptorBMT.invoke(MessageDrivenTxInterceptorBMT.java:47)
at
org.jboss.ejb.plugins.MessageDrivenInstanceInterceptor.invoke(MessageDrivenInstanceInterceptor.java:116)
at
org.jboss.ejb.plugins.RunAsSecurityInterceptor.invoke(RunAsSecurityInterceptor.java:109)
at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:205)
at
org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:138)
at
org.jboss.ejb.MessageDrivenContainer.internalInvoke(MessageDrivenContainer.java:402)
at org.jboss.ejb.Container.invoke(Container.java:960)
at sun.reflect.GeneratedMethodAccessor82.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
at
org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
at
org.jboss.invocation.local.LocalInvoker$MBeanServerAction.invoke(LocalInvoker.java:169)
at org.jboss.invocation.local.LocalInvoker.invoke(LocalInvoker.java:118)
at
org.jboss.invocation.InvokerInterceptor.invokeLocal(InvokerInterceptor.java:209)
at
org.jboss.invocation.InvokerInterceptor.invoke(InvokerInterceptor.java:195)
at
org.jboss.proxy.TransactionInterceptor.invoke(TransactionInterceptor.java:61)
at
org.jboss.ejb.plugins.inflow.MessageEndpointInterceptor.delivery(MessageEndpointInterceptor.java:249)
at
org.jboss.ejb.plugins.inflow.MessageEndpointInterceptor.invoke(MessageEndpointInterceptor.java:128)
at
org.jboss.proxy.ClientMethodInterceptor.invoke(ClientMethodInterceptor.java:74)
at org.jboss.proxy.ClientContainer.invoke(ClientContainer.java:100)
at $Proxy51.onMessage(Unknown Source)
at
org.hornetq.ra.inflow.HornetQMessageHandler.onMessage(HornetQMessageHandler.java:278)
at
org.hornetq.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:994)
at
org.hornetq.core.client.impl.ClientConsumerImpl.access$400(ClientConsumerImpl.java:48)
at
org.hornetq.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:1120)
at
org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
2014-Mar-17 21:51:20,562 - ERROR TID=Thread-6
(HornetQ-client-global-threads-4002470)
[com.pdf.online.service.impl.OnlineServiceImpl] Couldn't read data from
cassandra database using the parameters: key '2013-11-15T07:07:26.207',
equipment name: 'POL1702', module name 'CleanerBrush1'.
com.pdf.common.rtapi.exceptions.RtApiException: Could not read Framed data.
at
com.pdf.common.rtapi.impl.RtApiFacadeImpl.readFramedData(RtApiFacadeImpl.java:285)
at
com.pdf.online.service.impl.OnlineServiceImpl.readCassandraData(OnlineServiceImpl.java:2189)
at
com.pdf.online.service.impl.OnlineServiceImpl.process(OnlineServiceImpl.java:328)
at
com.pdf.online.mdb.OnlineEventDrivenBean.process(OnlineEventDrivenBean.java:302)
at
com.pdf.online.mdb.OnlineEventDrivenBean.onMessage(OnlineEventDrivenBean.java:211)
at sun.reflect.GeneratedMethodAccessor83.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.jboss.invocation.Invocation.performCall(Invocation.java:359)
at
org.jboss.ejb.MessageDrivenContainer$ContainerInterceptor.invoke(MessageDrivenContainer.java:495)
at
org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:158)
at
org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:63)
at
org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:121)
at
org.jboss.ejb.plugins.AbstractTxInterceptorBMT.invokeNext(AbstractTxInterceptorBMT.java:173)
at
org.jboss.ejb.plugins.MessageDrivenTxInterceptorBMT.invoke(MessageDrivenTxInterceptorBMT.java:47)
at
org.jboss.ejb.plugins.MessageDrivenInstanceInterceptor.invoke(MessageDrivenInstanceInterceptor.java:116)
at
org.jboss.ejb.plugins.RunAsSecurityInterceptor.invoke(RunAsSecurityInterceptor.java:109)
at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:205)
at
org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:138)
at
org.jboss.ejb.MessageDrivenContainer.internalInvoke(MessageDrivenContainer.java:402)
at org.jboss.ejb.Container.invoke(Container.java:960)
at sun.reflect.GeneratedMethodAccessor82.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
at
org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
at
org.jboss.invocation.local.LocalInvoker$MBeanServerAction.invoke(LocalInvoker.java:169)
at org.jboss.invocation.local.LocalInvoker.invoke(LocalInvoker.java:118)
at
org.jboss.invocation.InvokerInterceptor.invokeLocal(InvokerInterceptor.java:209)
at
org.jboss.invocation.InvokerInterceptor.invoke(InvokerInterceptor.java:195)
at
org.jboss.proxy.TransactionInterceptor.invoke(TransactionInterceptor.java:61)
at
org.jboss.ejb.plugins.inflow.MessageEndpointInterceptor.delivery(MessageEndpointInterceptor.java:249)
at
org.jboss.ejb.plugins.inflow.MessageEndpointInterceptor.invoke(MessageEndpointInterceptor.java:128)
at
org.jboss.proxy.ClientMethodInterceptor.invoke(ClientMethodInterceptor.java:74)
at org.jboss.proxy.ClientContainer.invoke(ClientContainer.java:100)
at $Proxy51.onMessage(Unknown Source)
at
org.hornetq.ra.inflow.HornetQMessageHandler.onMessage(HornetQMessageHandler.java:278)
at
org.hornetq.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:994)
at
org.hornetq.core.client.impl.ClientConsumerImpl.access$400(ClientConsumerImpl.java:48)
at
org.hornetq.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:1120)
at
org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: com.pdf.datastore.exception.DSTO_Exception: Failed to read
ContextData using key = 12169de1f273c1d38c893c0dfcbf10ff:20131115070726207,
DCID = 44140
at
com.pdf.datastore.cassandra.DSTO_CassandraClient.readContextData(DSTO_CassandraClient.java:1949)
at
com.pdf.datastore.cassandra.DSTO_CassandraClient.readFramedData(DSTO_CassandraClient.java:1848)
at com.pdf.rtapi.RAPI_DataStore.readFramedData(RAPI_DataStore.java:720)
at
com.pdf.rtapi.RAPI_InternalInterface.readFramedData(RAPI_InternalInterface.java:741)
at
com.pdf.common.rtapi.impl.RtApiFacadeImpl.readFramedData(RtApiFacadeImpl.java:261)
... 46 more
--
Alaa
--
*This message may contain confidential and privileged information. If it
has been sent to you in error, please reply to advise the sender of the
error and then immediately permanently delete it and all attachments to it
from your systems. If you are not the intended recipient, do not read,
copy, disclose or otherwise use this message or any attachments to it. The
sender disclaims any liability for such unauthorized use. PLEASE NOTE that
all incoming e-mails sent to PDF e-mail accounts will be archived and may
be scanned by us and/or by external service providers to detect and prevent
threats to our systems, investigate illegal or inappropriate behavior,
and/or eliminate unsolicited promotional e-mails ("spam"). If you have any
concerns about this process, please contact us at *
*legal.departm...@pdf.com* <legal.departm...@pdf.com>*.*
