Hi All,
I have setup 3 nodes Cassandra cluster(cassandra version is 0.7). In the
code base I am creating few column families using Hector client with
default consistency level and the replication factor of key space is 2.
While running the code I am getting following exception repeatedly on
client side. (this timeout comes when trying to find whether previously
written column families are exist or not)
[2012-02-20 11:40:52,324] INFO
{me.prettyprint.cassandra.hector.TimingLogger} - start[1329756052255]
time[69] tag[META_READ.success_]
[2012-02-20 11:40:52,403] INFO
{me.prettyprint.cassandra.hector.TimingLogger} - start[1329756052341]
time[62] tag[READ.fail_]
[2012-02-20 11:40:52,482] INFO
{me.prettyprint.cassandra.hector.TimingLogger} - start[1329756052405]
time[77] tag[READ.fail_]
[2012-02-20 11:40:52,932] INFO
{me.prettyprint.cassandra.hector.TimingLogger} - start[1329756052483]
time[449] tag[META_WRITE.success_]
[2012-02-20 11:40:52,935] INFO
{me.prettyprint.cassandra.hector.TimingLogger} - start[1329756052933]
time[2] tag[READ.fail_]
[2012-02-20 11:40:53,037] INFO
{me.prettyprint.cassandra.hector.TimingLogger} - start[1329756053035]
time[2] tag[READ.fail_]
[2012-02-20 11:41:03,173] ERROR
{me.prettyprint.cassandra.connection.HConnectionManager} - Could not
fullfill request on this host CassandraClient<192.168.0.101:9160-19>
[2012-02-20 11:41:03,176] ERROR
{me.prettyprint.cassandra.connection.HConnectionManager} - Exception:
me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutException()
at
me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:32)
at
me.prettyprint.cassandra.service.KeyspaceServiceImpl$10.execute(KeyspaceServiceImpl.java:397)
at
me.prettyprint.cassandra.service.KeyspaceServiceImpl$10.execute(KeyspaceServiceImpl.java:383)
at
me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:101)
at
me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:156)
at
me.prettyprint.cassandra.service.KeyspaceServiceImpl.operateWithFailover(KeyspaceServiceImpl.java:129)
at
me.prettyprint.cassandra.service.KeyspaceServiceImpl.multigetSlice(KeyspaceServiceImpl.java:401)
at
me.prettyprint.cassandra.model.thrift.ThriftMultigetSliceQuery$1.doInKeyspace(ThriftMultigetSliceQuery.java:67)
at
me.prettyprint.cassandra.model.thrift.ThriftMultigetSliceQuery$1.doInKeyspace(ThriftMultigetSliceQuery.java:59)
at
me.prettyprint.cassandra.model.KeyspaceOperationCallback.doInKeyspaceAndMeasure(KeyspaceOperationCallback.java:20)
at
me.prettyprint.cassandra.model.ExecutingKeyspace.doExecute(ExecutingKeyspace.java:72)
at
me.prettyprint.cassandra.model.thrift.ThriftMultigetSliceQuery.execute(ThriftMultigetSliceQuery.java:58)
at
org.wso2.carbon.bam.core.persistence.cassandra.CassandraDataStore.isTableExists(CassandraDataStore.java:509)
at
org.wso2.carbon.bam.core.persistence.cassandra.CassandraDataStore.createTable(CassandraDataStore.java:550)
at
org.wso2.carbon.bam.core.persistence.cassandra.CassandraDataStore.initialize(CassandraDataStore.java:125)
at
org.wso2.carbon.bam.core.persistence.cassandra.CassandraStoreFactory.getDataStore(CassandraStoreFactory.java:88)
at
org.wso2.carbon.bam.core.persistence.PersistenceManager.getDataStore(PersistenceManager.java:334)
at
org.wso2.carbon.bam.core.persistence.PersistenceManager.getDataStoreForTable(PersistenceManager.java:295)
at
org.wso2.carbon.bam.core.persistence.PersistenceManager.storeEvent(PersistenceManager.java:54)
at
org.wso2.carbon.bam.receiver.internal.QueueWorker.run(QueueWorker.java:63)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: TimedOutException()
at
org.apache.cassandra.thrift.Cassandra$multiget_slice_result.read(Cassandra.java:9628)
at
org.apache.cassandra.thrift.Cassandra$Client.recv_multiget_slice(Cassandra.java:636)
at
org.apache.cassandra.thrift.Cassandra$Client.multiget_slice(Cassandra.java:608)
at
me.prettyprint.cassandra.service.KeyspaceServiceImpl$10.execute(KeyspaceServiceImpl.java:388)
... 24 more
[2012-02-20 11:41:03,181] INFO
{me.prettyprint.cassandra.hector.TimingLogger} - start[1329756053139]
time[10042] tag[READ.fail_]
[2012-02-20 11:41:03,208] INFO
{me.prettyprint.cassandra.hector.TimingLogger} - start[1329756053139]
time[10068] tag[READ.success_]
[2012-02-20 11:41:03,239] INFO
{me.prettyprint.cassandra.hector.TimingLogger} - start[1329756063214]
time[25] tag[READ.success_]
[2012-02-20 11:41:03,288] INFO
{me.prettyprint.cassandra.hector.TimingLogger} - start[1329756063252]
time[36] tag[WRITE.success_]
[2012-02-20 11:41:03,322] INFO
{me.prettyprint.cassandra.hector.TimingLogger} - start[1329756063289]
time[33] tag[WRITE.success_]
[2012-02-20 11:41:03,325] INFO
{me.prettyprint.cassandra.hector.TimingLogger} - start[1329756063322]
time[3] tag[READ.fail_]
[2012-02-20 11:41:03,330] INFO
{me.prettyprint.cassandra.hector.TimingLogger} - start[1329756063326]
time[4] tag[READ.fail_]
[2012-02-20 11:41:03,629] INFO
{me.prettyprint.cassandra.hector.TimingLogger} - start[1329756063331]
time[298] tag[META_WRITE.success_]
[2012-02-20 11:41:03,632] INFO
{me.prettyprint.cassandra.hector.TimingLogger} - start[1329756063629]
time[3] tag[READ.fail_]
[2012-02-20 11:41:03,735] INFO
{me.prettyprint.cassandra.hector.TimingLogger} - start[1329756063732]
time[3] tag[READ.fail_]
[2012-02-20 11:41:13,842] ERROR
{me.prettyprint.cassandra.connection.HConnectionManager} - Could not
fullfill request on this host CassandraClient<192.168.0.100:9160-6>
[2012-02-20 11:41:13,842] ERROR
{me.prettyprint.cassandra.connection.HConnectionManager} - Exception:
me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutException()
at
me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:32)
at
me.prettyprint.cassandra.service.KeyspaceServiceImpl$10.execute(KeyspaceServiceImpl.java:397)
at
me.prettyprint.cassandra.service.KeyspaceServiceImpl$10.execute(KeyspaceServiceImpl.java:383)
at
me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:101)
at
me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:156)
at
me.prettyprint.cassandra.service.KeyspaceServiceImpl.operateWithFailover(KeyspaceServiceImpl.java:129)
at
me.prettyprint.cassandra.service.KeyspaceServiceImpl.multigetSlice(KeyspaceServiceImpl.java:401)
at
me.prettyprint.cassandra.model.thrift.ThriftMultigetSliceQuery$1.doInKeyspace(ThriftMultigetSliceQuery.java:67)
at
me.prettyprint.cassandra.model.thrift.ThriftMultigetSliceQuery$1.doInKeyspace(ThriftMultigetSliceQuery.java:59)
at
me.prettyprint.cassandra.model.KeyspaceOperationCallback.doInKeyspaceAndMeasure(KeyspaceOperationCallback.java:20)
at
me.prettyprint.cassandra.model.ExecutingKeyspace.doExecute(ExecutingKeyspace.java:72)
at
me.prettyprint.cassandra.model.thrift.ThriftMultigetSliceQuery.execute(ThriftMultigetSliceQuery.java:58)
at
org.wso2.carbon.bam.core.persistence.cassandra.CassandraDataStore.isTableExists(CassandraDataStore.java:509)
at
org.wso2.carbon.bam.core.persistence.cassandra.CassandraDataStore.createTable(CassandraDataStore.java:550)
at
org.wso2.carbon.bam.core.persistence.cassandra.CassandraDataStore.initialize(CassandraDataStore.java:153)
at
org.wso2.carbon.bam.core.persistence.cassandra.CassandraStoreFactory.getDataStore(CassandraStoreFactory.java:88)
at
org.wso2.carbon.bam.core.persistence.PersistenceManager.getDataStore(PersistenceManager.java:334)
at
org.wso2.carbon.bam.core.persistence.PersistenceManager.getDataStoreForTable(PersistenceManager.java:295)
at
org.wso2.carbon.bam.core.persistence.PersistenceManager.storeEvent(PersistenceManager.java:54)
at
org.wso2.carbon.bam.receiver.internal.QueueWorker.run(QueueWorker.java:63)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: TimedOutException()
at
org.apache.cassandra.thrift.Cassandra$multiget_slice_result.read(Cassandra.java:9628)
at
org.apache.cassandra.thrift.Cassandra$Client.recv_multiget_slice(Cassandra.java:636)
at
org.apache.cassandra.thrift.Cassandra$Client.multiget_slice(Cassandra.java:608)
at
me.prettyprint.cassandra.service.KeyspaceServiceImpl$10.execute(KeyspaceServiceImpl.java:388)
... 24 more
[2012-02-20 11:41:13,846] INFO
{me.prettyprint.cassandra.hector.TimingLogger} - start[1329756063836]
time[10010] tag[READ.fail_]
[2012-02-20 11:41:13,854] INFO
{me.prettyprint.cassandra.hector.TimingLogger} - start[1329756063836]
time[10018] tag[READ.success_]
[2012-02-20 11:41:13,861] INFO
{me.prettyprint.cassandra.hector.TimingLogger} - start[1329756073855]
time[6] tag[READ.success_]
[2012-02-20 11:41:13,867] INFO
{me.prettyprint.cassandra.hector.TimingLogger} - start[1329756073862]
time[5] tag[WRITE.success_]
[2012-02-20 11:41:13,905] INFO
{me.prettyprint.cassandra.hector.TimingLogger} - start[1329756073867]
time[38] tag[WRITE.success_]
[2012-02-20 11:41:13,908] INFO
{me.prettyprint.cassandra.hector.TimingLogger} - start[1329756073906]
time[2] tag[READ.fail_]
[2012-02-20 11:41:13,911] INFO
{me.prettyprint.cassandra.hector.TimingLogger} - start[1329756073909]
time[2] tag[READ.fail_]
[2012-02-20 11:41:14,835] INFO
{me.prettyprint.cassandra.hector.TimingLogger} - start[1329756073912]
time[923] tag[META_WRITE.success_]
[2012-02-20 11:41:14,838] INFO
{me.prettyprint.cassandra.hector.TimingLogger} - start[1329756074836]
time[2] tag[READ.fail_]
[2012-02-20 11:41:15,372] INFO
{me.prettyprint.cassandra.hector.TimingLogger} - start[1329756074939]
time[433] tag[READ.fail_]
[2012-02-20 11:41:25,480] ERROR
{me.prettyprint.cassandra.connection.HConnectionManager} - Could not
fullfill request on this host CassandraClient<192.168.0.102:9160-41>
[2012-02-20 11:41:25,481] ERROR
{me.prettyprint.cassandra.connection.HConnectionManager} - Exception:
me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutException()
at
me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:32)
at
me.prettyprint.cassandra.service.KeyspaceServiceImpl$10.execute(KeyspaceServiceImpl.java:397)
at
me.prettyprint.cassandra.service.KeyspaceServiceImpl$10.execute(KeyspaceServiceImpl.java:383)
at
me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:101)
at
me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:156)
at
me.prettyprint.cassandra.service.KeyspaceServiceImpl.operateWithFailover(KeyspaceServiceImpl.java:129)
at
me.prettyprint.cassandra.service.KeyspaceServiceImpl.multigetSlice(KeyspaceServiceImpl.java:401)
at
me.prettyprint.cassandra.model.thrift.ThriftMultigetSliceQuery$1.doInKeyspace(ThriftMultigetSliceQuery.java:67)
at
me.prettyprint.cassandra.model.thrift.ThriftMultigetSliceQuery$1.doInKeyspace(ThriftMultigetSliceQuery.java:59)
at
me.prettyprint.cassandra.model.KeyspaceOperationCallback.doInKeyspaceAndMeasure(KeyspaceOperationCallback.java:20)
at
me.prettyprint.cassandra.model.ExecutingKeyspace.doExecute(ExecutingKeyspace.java:72)
at
me.prettyprint.cassandra.model.thrift.ThriftMultigetSliceQuery.execute(ThriftMultigetSliceQuery.java:58)
at
org.wso2.carbon.bam.core.persistence.cassandra.CassandraDataStore.isTableExists(CassandraDataStore.java:509)
at
org.wso2.carbon.bam.core.persistence.cassandra.CassandraDataStore.createTable(CassandraDataStore.java:550)
at
org.wso2.carbon.bam.core.persistence.cassandra.CassandraDataStore.initialize(CassandraDataStore.java:180)
at
org.wso2.carbon.bam.core.persistence.cassandra.CassandraStoreFactory.getDataStore(CassandraStoreFactory.java:88)
at
org.wso2.carbon.bam.core.persistence.PersistenceManager.getDataStore(PersistenceManager.java:334)
at
org.wso2.carbon.bam.core.persistence.PersistenceManager.getDataStoreForTable(PersistenceManager.java:295)
at
org.wso2.carbon.bam.core.persistence.PersistenceManager.storeEvent(PersistenceManager.java:54)
at
org.wso2.carbon.bam.receiver.internal.QueueWorker.run(QueueWorker.java:63)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: TimedOutException()
at
org.apache.cassandra.thrift.Cassandra$multiget_slice_result.read(Cassandra.java:9628)
at
org.apache.cassandra.thrift.Cassandra$Client.recv_multiget_slice(Cassandra.java:636)
at
org.apache.cassandra.thrift.Cassandra$Client.multiget_slice(Cassandra.java:608)
at
me.prettyprint.cassandra.service.KeyspaceServiceImpl$10.execute(KeyspaceServiceImpl.java:388)
... 24 more
Also I can see only few column families has been created in cluster
nodes. In addition to that those created column families are not
identical across the cluster nodes (One node has
META_INFO<https://192.168.0.101:9448/carbon/cassandramgt/keyspace_dashboard.jsp?name=BAMKeySpace&loginStatus=true#>
column
family while other nodes doesn't have it ).
I can see the following types of exceptions printed in two cassandra nodes.
[2012-02-20 22:28:26,153] ERROR
{org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor} - Error in
ThreadPoolExecutor
java.lang.IllegalArgumentException: Unknown table/cf pair
(BAMKeySpace.META_INFO)
at org.apache.cassandra.db.Table.getColumnFamilyStore(Table.java:148)
at org.apache.cassandra.db.Table.getRow(Table.java:383)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:63)
at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:68)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:63)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[2012-02-20 22:28:26,156] ERROR
{org.apache.cassandra.service.AbstractCassandraDaemon} - Fatal exception
in thread Thread[ReadStage:106,5,main]
java.lang.IllegalArgumentException: Unknown table/cf pair
(BAMKeySpace.META_INFO)
at org.apache.cassandra.db.Table.getColumnFamilyStore(Table.java:148)
at org.apache.cassandra.db.Table.getRow(Table.java:383)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:63)
at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:68)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:63)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
What can be the issue?
Appreciate your help to resolve this issue.
Thanks,
Kasun.
