0.7.0-bet1 4 node clustered, i'd managed to get it into some sort of awful state (i think by accidently creating to many clients, it was also complaining about running out of file handles). Anyway I killed it all and restarted just one node, thought I would let it settle down then start the others. On the first node I got this.
(Sorry I cannot be more specific, was not paying too much attention it it all went bang)
I managed to get a couple of errors, one of which shutdown the server. Just checking before putting them into Jira, should I split them up?
ERROR [pool-1-thread-29] 2010-08-26 14:58:20,021 Cassandra.java (line
2651) Internal error processing get_slice
java.lang.IllegalStateException: replication factor (3) exceeds number of endpoints (1)
at org.apache.cassandra.locator.RackUnawareStrategy.calculateNaturalEndpoints(RackUnawareStrategy.java:57)
at org.apache.cassandra.locator.AbstractReplicationStrategy.getNaturalEndpoints(AbstractReplicationStrategy.java:88)
at org.apache.cassandra.service.StorageService.getNaturalEndpoints(StorageService.java:1289)
at org.apache.cassandra.service.StorageService.getNaturalEndpoints(StorageService.java:1277)
at org.apache.cassandra.service.StorageService.findSuitableEndpoint(StorageService.java:1323)
at org.apache.cassandra.service.StorageProxy.strongRead(StorageProxy.java:402)
at org.apache.cassandra.service.StorageProxy.readProtocol(StorageProxy.java:302)
at org.apache.cassandra.thrift.CassandraServer.readColumnFamily(CassandraServer.java:125)
at org.apache.cassandra.thrift.CassandraServer.getSlice(CassandraServer.java:231)
at org.apache.cassandra.thrift.CassandraServer.multigetSliceInternal(CassandraServer.java:309)
at org.apache.cassandra.thrift.CassandraServer.get_slice(CassandraServer.java:270)
at org.apache.cassandra.thrift.Cassandra$Processor$get_slice.process(Cassandra.java:2643)
at org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2499)
at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:167)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
java.lang.IllegalStateException: replication factor (3) exceeds number of endpoints (1)
at org.apache.cassandra.locator.RackUnawareStrategy.calculateNaturalEndpoints(RackUnawareStrategy.java:57)
at org.apache.cassandra.locator.AbstractReplicationStrategy.getNaturalEndpoints(AbstractReplicationStrategy.java:88)
at org.apache.cassandra.service.StorageService.getNaturalEndpoints(StorageService.java:1289)
at org.apache.cassandra.service.StorageService.getNaturalEndpoints(StorageService.java:1277)
at org.apache.cassandra.service.StorageService.findSuitableEndpoint(StorageService.java:1323)
at org.apache.cassandra.service.StorageProxy.strongRead(StorageProxy.java:402)
at org.apache.cassandra.service.StorageProxy.readProtocol(StorageProxy.java:302)
at org.apache.cassandra.thrift.CassandraServer.readColumnFamily(CassandraServer.java:125)
at org.apache.cassandra.thrift.CassandraServer.getSlice(CassandraServer.java:231)
at org.apache.cassandra.thrift.CassandraServer.multigetSliceInternal(CassandraServer.java:309)
at org.apache.cassandra.thrift.CassandraServer.get_slice(CassandraServer.java:270)
at org.apache.cassandra.thrift.Cassandra$Processor$get_slice.process(Cassandra.java:2643)
at org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2499)
at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:167)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
So I started the other 3, and two suffered the error below which caused the process to shutdown...
ERROR [main] 2010-08-26 14:59:22,315 AbstractCassandraDaemonjava
(line 107) Exception encountered during startup.
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError
at org.apache.cassandra.utils.FBUtilitieswaitOnFutures(FBUtilities.java:549)
at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:339)
at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:174)
at org.apache.cassandra.thrift.CassandraDaemon.setup(CassandraDaemon.java:120)
at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:90)
at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:224)
Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
at java.util.concurrent.FutureTask.get(FutureTask.java:83)
at org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:545)
... 5 more
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError
at org.apache.cassandra.db.commitlog.CommitLog.discardCompletedSegments(CommitLog.java:408)
at org.apache.cassandra.dbColumnFamilyStore$2.runMayThrow(ColumnFamilyStore.java:445)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
... 6 more
Caused by: java.util.concurrent.ExecutionException: java.lang.AssertionError
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
at java.util.concurrent.FutureTask.get(FutureTask.java:83)
at org.apache.cassandra.db.commitlog.CommitLog.discardCompletedSegments(CommitLog.java:400)
... 8 more
Caused by: java.lang.AssertionError
at org.apache.cassandra.db.commitlog.CommitLogHeader$CommitLogHeaderSerializer.serialize(CommitLogHeader.java:157)
at org.apache.cassandra.db.commitlog.CommitLogHeader.writeCommitLogHeader(CommitLogHeader.java:124)
at org.apache.cassandra.db.commitlog.CommitLogSegment.writeHeader(CommitLogSegment.java:70)
at org.apache.cassandra.db.commitlog.CommitLog.discardCompletedSegmentsInternal(CommitLog.java:450)
at org.apache.cassandra.db.commitlog.CommitLog.access$300(CommitLog.java:75)
at org.apache.cassandra.db.commitlog.CommitLog$6.call(CommitLog.java:394)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at org.apache.cassandra.db.commitlog.PeriodicCommitLogExecutorService$1.runMayThrow(PeriodicCommitLogExecutorService.java:52)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
... 1 more
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError
at org.apache.cassandra.utils.FBUtilitieswaitOnFutures(FBUtilities.java:549)
at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:339)
at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:174)
at org.apache.cassandra.thrift.CassandraDaemon.setup(CassandraDaemon.java:120)
at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:90)
at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:224)
Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
at java.util.concurrent.FutureTask.get(FutureTask.java:83)
at org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:545)
... 5 more
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError
at org.apache.cassandra.db.commitlog.CommitLog.discardCompletedSegments(CommitLog.java:408)
at org.apache.cassandra.dbColumnFamilyStore$2.runMayThrow(ColumnFamilyStore.java:445)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
... 6 more
Caused by: java.util.concurrent.ExecutionException: java.lang.AssertionError
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
at java.util.concurrent.FutureTask.get(FutureTask.java:83)
at org.apache.cassandra.db.commitlog.CommitLog.discardCompletedSegments(CommitLog.java:400)
... 8 more
Caused by: java.lang.AssertionError
at org.apache.cassandra.db.commitlog.CommitLogHeader$CommitLogHeaderSerializer.serialize(CommitLogHeader.java:157)
at org.apache.cassandra.db.commitlog.CommitLogHeader.writeCommitLogHeader(CommitLogHeader.java:124)
at org.apache.cassandra.db.commitlog.CommitLogSegment.writeHeader(CommitLogSegment.java:70)
at org.apache.cassandra.db.commitlog.CommitLog.discardCompletedSegmentsInternal(CommitLog.java:450)
at org.apache.cassandra.db.commitlog.CommitLog.access$300(CommitLog.java:75)
at org.apache.cassandra.db.commitlog.CommitLog$6.call(CommitLog.java:394)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at org.apache.cassandra.db.commitlog.PeriodicCommitLogExecutorService$1.runMayThrow(PeriodicCommitLogExecutorService.java:52)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
... 1 more
Aaron
