yes, starting the others made things a bit happier.
Any thoughts on the assertion error that cause the startup to fail ? I've seen it a couple of times.
Seems to be from this line in CommitLogHeader.java
157: assert clHeader.cfDirtiedAt.size() <= clHeader.cfCount;
Thanks
Aaron
On 26 Aug, 2010,at 03:25 PM, Jonathan Ellis <jbel...@gmail.com> wrote:
the one node you restarted thinks it's the only node in the cluster.
starting the others will fix that.
On Wed, Aug 25, 2010 at 10:10 PM, Aaron Morton <aa...@thelastpickle.com> wrote:
> 0.7.0-bet1 4 node clustered, i'd managed to get it into some sort of awful
> state (i think by accidently creating to many clients, it was also
> complaining about running out of file handles). Anyway I killed it all and
> restarted just one node, thought I would let it settle down then start the
> others. On the first node I got this.
> (Sorry I cannot be more specific, was not paying too much attention it it
> all went bang)
> I managed to get a couple of errors, one of which shutdown the server. Just
> checking before putting them into Jira, should I split them up?
> ERROR [pool-1-thread-29] 2010-08-26 14:58:20,021 Cassandra.java (line 2651)
> Internal error processing get_slice
> java.lang.IllegalStateException: replication factor (3) exceeds number of
> endpoints (1)
> at
> org.apache.cassandra.locator.RackUnawareStrategy.calculateNaturalEndpoints(RackUnawareStrategy.java:57)
> at
> org.apache.cassandra.locator.AbstractReplicationStrategy.getNaturalEndpoints(AbstractReplicationStrategy.java:88)
> at
> org.apache.cassandra.service.StorageService.getNaturalEndpoints(StorageService.java:1289)
> at
> org.apache.cassandra.service.StorageService.getNaturalEndpoints(StorageService.java:1277)
> at
> org.apache.cassandra.service.StorageService.findSuitableEndpoint(StorageService.java:1323)
> at
> org.apache.cassandra.service.StorageProxy.strongRead(StorageProxy.java:402)
> at
> org.apache.cassandra.service.StorageProxy.readProtocol(StorageProxy.java:302)
> at
> org.apache.cassandra.thrift.CassandraServer.readColumnFamily(CassandraServer.java:125)
> at
> org.apache.cassandra.thrift.CassandraServer.getSlice(CassandraServer.java:231)
> at
> org.apache.cassandra.thriftCassandraServer.multigetSliceInternal(CassandraServer.java:309)
> at
> org.apache.cassandra.thrift.CassandraServer.get_slice(CassandraServer.java:270)
> at
> org.apache.cassandra.thrift.Cassandra$Processor$get_slice.process(Cassandra.java:2643)
> at
> org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2499)
> at
> org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:167)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
>
> So I started the other 3, and two suffered the error below which caused the
> process to shutdown...
> ERROR [main] 2010-08-26 14:59:22,315 AbstractCassandraDaemonjava (line 107)
> Exception encountered during startup.
> java.lang.RuntimeException: java.util.concurrentExecutionException:
> java.lang.RuntimeException: java.lang.RuntimeException:
> java.util.concurrent.ExecutionException: java.lang.AssertionError
> at
> org.apache.cassandra.utils.FBUtilitieswaitOnFutures(FBUtilitiesjava:549)
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:339)
> at
> orgapache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:174)
> at
> org.apache.cassandra.thrift.CassandraDaemon.setup(CassandraDaemon.java:120)
> at
> org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:90)
> at
> org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:224)
> Caused by: java.util.concurrent.ExecutionException:
> java.lang.RuntimeException: java.lang.RuntimeException:
> java.util.concurrent.ExecutionException: java.lang.AssertionError
> at
> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
> at java.util.concurrent.FutureTask.get(FutureTask.java:83)
> at
> org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:545)
> ... 5 more
> Caused by: java.lang.RuntimeException: java.lang.RuntimeException:
> java.util.concurrent.ExecutionException: java.lang.AssertionError
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at
> java.util.concurrentFutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.RuntimeException:
> java.util.concurrent.ExecutionException: java.lang.AssertionError
> at
> org.apache.cassandra.db.commitlog.CommitLog.discardCompletedSegments(CommitLog.java:408)
> at
> org.apache.cassandra.dbColumnFamilyStore$2.runMayThrow(ColumnFamilyStore.java:445)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> .. 6 more
> Caused by: java.util.concurrent.ExecutionException: java.lang.AssertionError
> at
> java.util.concurrent.FutureTask$Sync.innerGet(FutureTaskjava:222)
> at java.util.concurrent.FutureTask.get(FutureTask.java:83)
> at
> org.apache.cassandra.db.commitlog.CommitLog.discardCompletedSegments(CommitLog.java:400)
> ... 8 more
> Caused by: java.lang.AssertionError
> at
> org.apache.cassandra.db.commitlog.CommitLogHeader$CommitLogHeaderSerializer.serialize(CommitLogHeader.java:157)
> at
> org.apache.cassandra.db.commitlog.CommitLogHeader.writeCommitLogHeader(CommitLogHeader.java:124)
> at
> org.apache.cassandra.db.commitlog.CommitLogSegment.writeHeader(CommitLogSegment.java:70)
> at
> org.apache.cassandra.db.commitlog.CommitLog.discardCompletedSegmentsInternal(CommitLog.java:450)
> at
> org.apache.cassandra.db.commitlog.CommitLog.access$300(CommitLog.java:75)
> at
> org.apache.cassandra.db.commitlog.CommitLog$6.call(CommitLog.java:394)
> at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at
> org.apache.cassandra.db.commitlog.PeriodicCommitLogExecutorService$1.runMayThrow(PeriodicCommitLogExecutorService.java:52)
> at
> orgapache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> ... 1 more
>
> Aaron
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com
