If you are starting out I recommend using at least 0.8, the memory management will make you life so much easier.
if the schema is not fully replicated solve that problem first. Using cassandra-cli and run the describe cluster command. If there are multiple schemas listed nuke it and start again (assuming you can nuke it.) Not sure about creating schema with Hector. But whatever the method be careful to only run one schema update at a time. Personally I prefer to create schema using the CLI or the CQL ( in 1.0). > Caused by: TimedOutException() > at > org.apache.cassandra.thrift.Cassandra$multiget_slice_result.read(Cassandra.java:9628) > at > org.apache.cassandra.thrift.Cassandra$Client.recv_multiget_slice(Cassandra.java:636) > at > org.apache.cassandra.thrift.Cassandra$Client.multiget_slice(Cassandra.java:608) > at > me.prettyprint.cassandra.service.KeyspaceServiceImpl$10.execute(KeyspaceServiceImpl.java:388) Happens because less than CL nodes replied before rpc_timeout. Check nodetool ring to see that nodes are up. Cheers ----------------- Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 21/02/2012, at 6:27 AM, Kasun Weranga wrote: > Hi All, > > I have setup 3 nodes Cassandra cluster(cassandra version is 0.7). In the > code base I am creating few column families using Hector client with default > consistency level and the replication factor of key space is 2. > > While running the code I am getting following exception repeatedly on client > side. (this timeout comes when trying to find whether previously written > column families are exist or not) > > [2012-02-20 11:40:52,324] INFO > {me.prettyprint.cassandra.hector.TimingLogger} - start[1329756052255] > time[69] tag[META_READ.success_] > [2012-02-20 11:40:52,403] INFO > {me.prettyprint.cassandra.hector.TimingLogger} - start[1329756052341] > time[62] tag[READ.fail_] > [2012-02-20 11:40:52,482] INFO > {me.prettyprint.cassandra.hector.TimingLogger} - start[1329756052405] > time[77] tag[READ.fail_] > [2012-02-20 11:40:52,932] INFO > {me.prettyprint.cassandra.hector.TimingLogger} - start[1329756052483] > time[449] tag[META_WRITE.success_] > [2012-02-20 11:40:52,935] INFO > {me.prettyprint.cassandra.hector.TimingLogger} - start[1329756052933] > time[2] tag[READ.fail_] > [2012-02-20 11:40:53,037] INFO > {me.prettyprint.cassandra.hector.TimingLogger} - start[1329756053035] > time[2] tag[READ.fail_] > [2012-02-20 11:41:03,173] ERROR > {me.prettyprint.cassandra.connection.HConnectionManager} - Could not > fullfill request on this host CassandraClient<192.168.0.101:9160-19> > [2012-02-20 11:41:03,176] ERROR > {me.prettyprint.cassandra.connection.HConnectionManager} - Exception: > me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutException() > at > me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:32) > at > me.prettyprint.cassandra.service.KeyspaceServiceImpl$10.execute(KeyspaceServiceImpl.java:397) > at > me.prettyprint.cassandra.service.KeyspaceServiceImpl$10.execute(KeyspaceServiceImpl.java:383) > at > me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:101) > at > me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:156) > at > me.prettyprint.cassandra.service.KeyspaceServiceImpl.operateWithFailover(KeyspaceServiceImpl.java:129) > at > me.prettyprint.cassandra.service.KeyspaceServiceImpl.multigetSlice(KeyspaceServiceImpl.java:401) > at > me.prettyprint.cassandra.model.thrift.ThriftMultigetSliceQuery$1.doInKeyspace(ThriftMultigetSliceQuery.java:67) > at > me.prettyprint.cassandra.model.thrift.ThriftMultigetSliceQuery$1.doInKeyspace(ThriftMultigetSliceQuery.java:59) > at > me.prettyprint.cassandra.model.KeyspaceOperationCallback.doInKeyspaceAndMeasure(KeyspaceOperationCallback.java:20) > at > me.prettyprint.cassandra.model.ExecutingKeyspace.doExecute(ExecutingKeyspace.java:72) > at > me.prettyprint.cassandra.model.thrift.ThriftMultigetSliceQuery.execute(ThriftMultigetSliceQuery.java:58) > at > org.wso2.carbon.bam.core.persistence.cassandra.CassandraDataStore.isTableExists(CassandraDataStore.java:509) > at > org.wso2.carbon.bam.core.persistence.cassandra.CassandraDataStore.createTable(CassandraDataStore.java:550) > at > org.wso2.carbon.bam.core.persistence.cassandra.CassandraDataStore.initialize(CassandraDataStore.java:125) > at > org.wso2.carbon.bam.core.persistence.cassandra.CassandraStoreFactory.getDataStore(CassandraStoreFactory.java:88) > at > org.wso2.carbon.bam.core.persistence.PersistenceManager.getDataStore(PersistenceManager.java:334) > at > org.wso2.carbon.bam.core.persistence.PersistenceManager.getDataStoreForTable(PersistenceManager.java:295) > at > org.wso2.carbon.bam.core.persistence.PersistenceManager.storeEvent(PersistenceManager.java:54) > at > org.wso2.carbon.bam.receiver.internal.QueueWorker.run(QueueWorker.java:63) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > Caused by: TimedOutException() > at > org.apache.cassandra.thrift.Cassandra$multiget_slice_result.read(Cassandra.java:9628) > at > org.apache.cassandra.thrift.Cassandra$Client.recv_multiget_slice(Cassandra.java:636) > at > org.apache.cassandra.thrift.Cassandra$Client.multiget_slice(Cassandra.java:608) > at > me.prettyprint.cassandra.service.KeyspaceServiceImpl$10.execute(KeyspaceServiceImpl.java:388) > ... 24 more > [2012-02-20 11:41:03,181] INFO > {me.prettyprint.cassandra.hector.TimingLogger} - start[1329756053139] > time[10042] tag[READ.fail_] > [2012-02-20 11:41:03,208] INFO > {me.prettyprint.cassandra.hector.TimingLogger} - start[1329756053139] > time[10068] tag[READ.success_] > [2012-02-20 11:41:03,239] INFO > {me.prettyprint.cassandra.hector.TimingLogger} - start[1329756063214] > time[25] tag[READ.success_] > [2012-02-20 11:41:03,288] INFO > {me.prettyprint.cassandra.hector.TimingLogger} - start[1329756063252] > time[36] tag[WRITE.success_] > [2012-02-20 11:41:03,322] INFO > {me.prettyprint.cassandra.hector.TimingLogger} - start[1329756063289] > time[33] tag[WRITE.success_] > [2012-02-20 11:41:03,325] INFO > {me.prettyprint.cassandra.hector.TimingLogger} - start[1329756063322] > time[3] tag[READ.fail_] > [2012-02-20 11:41:03,330] INFO > {me.prettyprint.cassandra.hector.TimingLogger} - start[1329756063326] > time[4] tag[READ.fail_] > [2012-02-20 11:41:03,629] INFO > {me.prettyprint.cassandra.hector.TimingLogger} - start[1329756063331] > time[298] tag[META_WRITE.success_] > [2012-02-20 11:41:03,632] INFO > {me.prettyprint.cassandra.hector.TimingLogger} - start[1329756063629] > time[3] tag[READ.fail_] > [2012-02-20 11:41:03,735] INFO > {me.prettyprint.cassandra.hector.TimingLogger} - start[1329756063732] > time[3] tag[READ.fail_] > [2012-02-20 11:41:13,842] ERROR > {me.prettyprint.cassandra.connection.HConnectionManager} - Could not > fullfill request on this host CassandraClient<192.168.0.100:9160-6> > [2012-02-20 11:41:13,842] ERROR > {me.prettyprint.cassandra.connection.HConnectionManager} - Exception: > me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutException() > at > me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:32) > at > me.prettyprint.cassandra.service.KeyspaceServiceImpl$10.execute(KeyspaceServiceImpl.java:397) > at > me.prettyprint.cassandra.service.KeyspaceServiceImpl$10.execute(KeyspaceServiceImpl.java:383) > at > me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:101) > at > me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:156) > at > me.prettyprint.cassandra.service.KeyspaceServiceImpl.operateWithFailover(KeyspaceServiceImpl.java:129) > at > me.prettyprint.cassandra.service.KeyspaceServiceImpl.multigetSlice(KeyspaceServiceImpl.java:401) > at > me.prettyprint.cassandra.model.thrift.ThriftMultigetSliceQuery$1.doInKeyspace(ThriftMultigetSliceQuery.java:67) > at > me.prettyprint.cassandra.model.thrift.ThriftMultigetSliceQuery$1.doInKeyspace(ThriftMultigetSliceQuery.java:59) > at > me.prettyprint.cassandra.model.KeyspaceOperationCallback.doInKeyspaceAndMeasure(KeyspaceOperationCallback.java:20) > at > me.prettyprint.cassandra.model.ExecutingKeyspace.doExecute(ExecutingKeyspace.java:72) > at > me.prettyprint.cassandra.model.thrift.ThriftMultigetSliceQuery.execute(ThriftMultigetSliceQuery.java:58) > at > org.wso2.carbon.bam.core.persistence.cassandra.CassandraDataStore.isTableExists(CassandraDataStore.java:509) > at > org.wso2.carbon.bam.core.persistence.cassandra.CassandraDataStore.createTable(CassandraDataStore.java:550) > at > org.wso2.carbon.bam.core.persistence.cassandra.CassandraDataStore.initialize(CassandraDataStore.java:153) > at > org.wso2.carbon.bam.core.persistence.cassandra.CassandraStoreFactory.getDataStore(CassandraStoreFactory.java:88) > at > org.wso2.carbon.bam.core.persistence.PersistenceManager.getDataStore(PersistenceManager.java:334) > at > org.wso2.carbon.bam.core.persistence.PersistenceManager.getDataStoreForTable(PersistenceManager.java:295) > at > org.wso2.carbon.bam.core.persistence.PersistenceManager.storeEvent(PersistenceManager.java:54) > at > org.wso2.carbon.bam.receiver.internal.QueueWorker.run(QueueWorker.java:63) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > Caused by: TimedOutException() > at > org.apache.cassandra.thrift.Cassandra$multiget_slice_result.read(Cassandra.java:9628) > at > org.apache.cassandra.thrift.Cassandra$Client.recv_multiget_slice(Cassandra.java:636) > at > org.apache.cassandra.thrift.Cassandra$Client.multiget_slice(Cassandra.java:608) > at > me.prettyprint.cassandra.service.KeyspaceServiceImpl$10.execute(KeyspaceServiceImpl.java:388) > ... 24 more > [2012-02-20 11:41:13,846] INFO > {me.prettyprint.cassandra.hector.TimingLogger} - start[1329756063836] > time[10010] tag[READ.fail_] > [2012-02-20 11:41:13,854] INFO > {me.prettyprint.cassandra.hector.TimingLogger} - start[1329756063836] > time[10018] tag[READ.success_] > [2012-02-20 11:41:13,861] INFO > {me.prettyprint.cassandra.hector.TimingLogger} - start[1329756073855] > time[6] tag[READ.success_] > [2012-02-20 11:41:13,867] INFO > {me.prettyprint.cassandra.hector.TimingLogger} - start[1329756073862] > time[5] tag[WRITE.success_] > [2012-02-20 11:41:13,905] INFO > {me.prettyprint.cassandra.hector.TimingLogger} - start[1329756073867] > time[38] tag[WRITE.success_] > [2012-02-20 11:41:13,908] INFO > {me.prettyprint.cassandra.hector.TimingLogger} - start[1329756073906] > time[2] tag[READ.fail_] > [2012-02-20 11:41:13,911] INFO > {me.prettyprint.cassandra.hector.TimingLogger} - start[1329756073909] > time[2] tag[READ.fail_] > [2012-02-20 11:41:14,835] INFO > {me.prettyprint.cassandra.hector.TimingLogger} - start[1329756073912] > time[923] tag[META_WRITE.success_] > [2012-02-20 11:41:14,838] INFO > {me.prettyprint.cassandra.hector.TimingLogger} - start[1329756074836] > time[2] tag[READ.fail_] > [2012-02-20 11:41:15,372] INFO > {me.prettyprint.cassandra.hector.TimingLogger} - start[1329756074939] > time[433] tag[READ.fail_] > [2012-02-20 11:41:25,480] ERROR > {me.prettyprint.cassandra.connection.HConnectionManager} - Could not > fullfill request on this host CassandraClient<192.168.0.102:9160-41> > [2012-02-20 11:41:25,481] ERROR > {me.prettyprint.cassandra.connection.HConnectionManager} - Exception: > me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutException() > at > me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:32) > at > me.prettyprint.cassandra.service.KeyspaceServiceImpl$10.execute(KeyspaceServiceImpl.java:397) > at > me.prettyprint.cassandra.service.KeyspaceServiceImpl$10.execute(KeyspaceServiceImpl.java:383) > at > me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:101) > at > me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:156) > at > me.prettyprint.cassandra.service.KeyspaceServiceImpl.operateWithFailover(KeyspaceServiceImpl.java:129) > at > me.prettyprint.cassandra.service.KeyspaceServiceImpl.multigetSlice(KeyspaceServiceImpl.java:401) > at > me.prettyprint.cassandra.model.thrift.ThriftMultigetSliceQuery$1.doInKeyspace(ThriftMultigetSliceQuery.java:67) > at > me.prettyprint.cassandra.model.thrift.ThriftMultigetSliceQuery$1.doInKeyspace(ThriftMultigetSliceQuery.java:59) > at > me.prettyprint.cassandra.model.KeyspaceOperationCallback.doInKeyspaceAndMeasure(KeyspaceOperationCallback.java:20) > at > me.prettyprint.cassandra.model.ExecutingKeyspace.doExecute(ExecutingKeyspace.java:72) > at > me.prettyprint.cassandra.model.thrift.ThriftMultigetSliceQuery.execute(ThriftMultigetSliceQuery.java:58) > at > org.wso2.carbon.bam.core.persistence.cassandra.CassandraDataStore.isTableExists(CassandraDataStore.java:509) > at > org.wso2.carbon.bam.core.persistence.cassandra.CassandraDataStore.createTable(CassandraDataStore.java:550) > at > org.wso2.carbon.bam.core.persistence.cassandra.CassandraDataStore.initialize(CassandraDataStore.java:180) > at > org.wso2.carbon.bam.core.persistence.cassandra.CassandraStoreFactory.getDataStore(CassandraStoreFactory.java:88) > at > org.wso2.carbon.bam.core.persistence.PersistenceManager.getDataStore(PersistenceManager.java:334) > at > org.wso2.carbon.bam.core.persistence.PersistenceManager.getDataStoreForTable(PersistenceManager.java:295) > at > org.wso2.carbon.bam.core.persistence.PersistenceManager.storeEvent(PersistenceManager.java:54) > at > org.wso2.carbon.bam.receiver.internal.QueueWorker.run(QueueWorker.java:63) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > Caused by: TimedOutException() > at > org.apache.cassandra.thrift.Cassandra$multiget_slice_result.read(Cassandra.java:9628) > at > org.apache.cassandra.thrift.Cassandra$Client.recv_multiget_slice(Cassandra.java:636) > at > org.apache.cassandra.thrift.Cassandra$Client.multiget_slice(Cassandra.java:608) > at > me.prettyprint.cassandra.service.KeyspaceServiceImpl$10.execute(KeyspaceServiceImpl.java:388) > ... 24 more > > > > Also I can see only few column families has been created in cluster nodes. > In addition to that those created column families are not identical across > the cluster nodes (One node has META_INFO column family while other nodes > doesn't have it ). > I can see the following types of exceptions printed in two cassandra nodes. > > [2012-02-20 22:28:26,153] ERROR > {org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor} - Error in > ThreadPoolExecutor > java.lang.IllegalArgumentException: Unknown table/cf pair > (BAMKeySpace.META_INFO) > at org.apache.cassandra.db.Table.getColumnFamilyStore(Table.java:148) > at org.apache.cassandra.db.Table.getRow(Table.java:383) > at > org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:63) > at > org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:68) > at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:63) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > [2012-02-20 22:28:26,156] ERROR > {org.apache.cassandra.service.AbstractCassandraDaemon} - Fatal exception in > thread Thread[ReadStage:106,5,main] > java.lang.IllegalArgumentException: Unknown table/cf pair > (BAMKeySpace.META_INFO) > at org.apache.cassandra.db.Table.getColumnFamilyStore(Table.java:148) > at org.apache.cassandra.db.Table.getRow(Table.java:383) > at > org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:63) > at > org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:68) > at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:63) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > > > What can be the issue? > > Appreciate your help to resolve this issue. > > > Thanks, > Kasun. > > > > >
