> java.io.IOError: org.apache.cassandra.db.UnserializableColumnFamilyException: > Couldn't find cfId=-387130991 Schema may have diverged between nodes.
use cassandra-cli and run describe cluster; to see how many schema versions you have. Cheers ----------------- Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 22/03/2012, at 6:27 AM, Daning Wang wrote: > and we are on 0.8.6. > > > > On Wed, Mar 21, 2012 at 10:24 AM, Daning Wang <dan...@netseer.com> wrote: > Hi All, > > > We got lots of Exception in the log, and later the server crashed. any idea > what is happening and how to fix it? > > ERROR [RequestResponseStage:4] 2012-03-21 04:16:30,482 > AbstractCassandraDaemon.java (line 139) Fatal exception in thread > Thread[RequestResponseStage:4,5,main] > java.io.IOError: java.io.EOFException > at > org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:71) > at > org.apache.cassandra.service.ReadCallback.response(ReadCallback.java:125) > at > org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:49) > at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:722) > Caused by: java.io.EOFException > at java.io.DataInputStream.readFully(DataInputStream.java:197) > at > org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:104) > at > org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:82) > at > org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:64) > ... 6 more > ERROR [RequestResponseStage:2] 2012-03-21 04:16:30,480 > AbstractCassandraDaemon.java (line 139) Fatal exception in thread > Thread[RequestResponseStage:2,5,main] > java.io.IOError: org.apache.cassandra.db.UnserializableColumnFamilyException: > Couldn't find cfId=-387130991 > at > org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:71) > at > org.apache.cassandra.service.AsyncRepairCallback.response(AsyncRepairCallback.java:47) > at > org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:49) > at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:722) > Caused by: org.apache.cassandra.db.UnserializableColumnFamilyException: > Couldn't find cfId=-387130991 > at > org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:123) > at org.apache.cassandra.db.RowSerializer.deserialize(Row.java:69) > at > org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:113) > at > org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:82) > at > org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:64 > > > > The is the exception before server crashes. > > > ERROR [ReadRepairStage:299] 2012-03-21 05:02:53,808 > AbstractCassandraDaemon.java (line 139) Fatal exception in thread > Thread[ReadRepairStage:299,5,main] > java.lang.RuntimeException: java.util.concurrent.RejectedExecutionException: > ThreadPoolExecutor has shut down > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:722) > Caused by: java.util.concurrent.RejectedExecutionException: > ThreadPoolExecutor has shut down > at > org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:60) > at > java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:816) > at > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1337) > at > org.apache.cassandra.net.MessagingService.receive(MessagingService.java:490) > at > org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:388) > at > org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:346) > at > org.apache.cassandra.service.RowRepairResolver.maybeScheduleRepairs(RowRepairResolver.java:121) > at > org.apache.cassandra.service.RowRepairResolver.resolve(RowRepairResolver.java:85) > at > org.apache.cassandra.service.AsyncRepairCallback$1.runMayThrow(AsyncRepairCallback.java:54) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) > ... 3 more > > Thank you in advance, > > Daning >
