yes, you should create a ticket On Wed, Aug 25, 2010 at 2:52 AM, Aaron Morton <aa...@thelastpickle.com> wrote: > I'm running the v0.7-beta1 in a 4 nodes cluster and just doing some simple > testing. One of the nodes had been down (machine off, unclean shutdown) for > an hour or so not sure how many writes were going on, when I bought it back > up this message appears in the other 3 nodes... > > INFO [GOSSIP_STAGE:1] 2010-08-25 19:29:51,199 Gossiper.java (line 584) Node > /192.168.34.27 has restarted, now UP again > INFO [HINTED-HANDOFF-POOL:1] 2010-08-25 19:29:51,200 > HintedHandOffManager.java (line 191) Started hinted handoff for endpoint > /192.168.34.27 > INFO [GOSSIP_STAGE:1] 2010-08-25 19:29:51,201 StorageService.java (line > 636) Node /192.168.34.27 state jump to normal > INFO [GOSSIP_STAGE:1] 2010-08-25 19:29:51,201 StorageService.java (line > 643) Will not change my token ownership to /192.168.34.27 > ERROR [HINTED-HANDOFF-POOL:1] 2010-08-25 19:29:51,640 CassandraDaemon.java > (line 82) Uncaught exception in thread Thread[HINTED-HANDOFF-POOL:1,5,main] > java.util.concurrent.ExecutionException: java.lang.RuntimeException: > java.util.NoSuchElementException > at > java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) > at java.util.concurrent.FutureTask.get(FutureTask.java:83) > at > org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:87) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:619) > Caused by: java.lang.RuntimeException: java.util.NoSuchElementException > at > orgapache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > at > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > ... 2 more > Caused by: java.util.NoSuchElementException > at > java.util.concurrent.ConcurrentSkipListMap.lastKey(ConcurrentSkipListMap.java:1981) > at > java.util.concurrent.ConcurrentSkipListMap$KeySet.last(ConcurrentSkipListMap.java:2331) > at > org.apache.cassandra.db.HintedHandOffManager.sendMessage(HintedHandOffManager.java:121) > at > org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:218) > at > org.apache.cassandra.db.HintedHandOffManager.access$000(HintedHandOffManager.java:78) > at > org.apache.cassandra.db.HintedHandOffManager$1.runMayThrow(HintedHandOffManager.java:296) not > sure how many writes were going on > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) > ... 6 more > > On the machine that was off (34.27) there are no errors in the logs, and > here are the entries for around the same time... > INFO [main] 2010-08-25 19:29:50,679 CommitLog.java (line 340) Recovery > complete > INFO [main] 2010-08-25 19:29:50,769 CommitLog.java (line 180) Log replay > complete > INFO [main] 2010-08-25 19:29:50,797 StorageService.java (line 342) > Cassandra version: 0.7.0-beta1-SNAPSHOT > INFO [main] 2010-08-25 19:29:50,797 StorageService.java (line 343) Thrift > API version: 10.0.0 > INFO [main] 2010-08-25 19:29:50,813 SystemTable.java (line 240) Saved Token > found: 85070591730234615865843651857942052864 > INFO [main] 2010-08-25 19:29:50,813 SystemTable.java (line 257) Saved > ClusterName found: FOO > INFO [main] 2010-08-25 19:29:50,813 SystemTable.java (line 272) Saved > partitioner not found. Using org.apache.cassandra.dht.RandomPartitioner > INFO [main] 2010-08-25 19:29:50,814 ColumnFamilyStore.java (line 422) > switching in a fresh Memtable for LocationInfo at > CommitLogContext(file='/local1/junkbox/cassandra/commitlog/CommitLog-12827213897 > 70.log', position=41336) > INFO [main] 2010-08-25 19:29:50,814 ColumnFamilyStore.java (line 706) > Enqueuing flush of memtable-locationi...@916236367(95 bytes, 2 operations) > INFO [FLUSH-WRITER-POOL:1] 2010-08-25 19:29:50,815 Memtable.java (line 150) > Writing memtable-locationi...@916236367(95 bytes, 2 operations) > INFO [FLUSH-WRITER-POOL:1] 2010-08-25 19:29:50,873 Memtable.java (line 157) > Completed flushing > /local1/junkbox/cassandra/data/system/LocationInfo-e-6-Data.db > INFO [main] 2010-08-25 19:29:50,917 StorageService.java (line 374) Starting > up server gossip > INFO [main] 2010-08-25 19:29:51,093 ColumnFamilyStore.java (line 1239) > Loaded 0 rows into the Super2 cache > INFO [main] 2010-08-25 19:29:51,170 CassandraDaemon.java (line 153) Binding > thrift service to /0.0.0.0:9160 > INFO [main] 2010-08-25 19:29:51,174 CassandraDaemon.java (line 167) Using > TFramedTransport with a max frame size of 15728640 bytes. > INFO [GOSSIP_STAGE:1] 2010-08-25 19:29:51,198 Gossiper.java (line 578) Node > /192.168.34.28 is now part of the cluster > INFO [GOSSIP_STAGE:1] 2010-08-25 19:29:51,199 Gossiper.java (line 578) Node > /192.168.34.29 is now part of the cluster > INFO [GOSSIP_STAGE:1] 2010-08-25 19:29:51,199 Gossiper.java (line 578) Node > /192.168.34.26 is now part of the cluster > INFO [main] 2010-08-25 19:29:51,204 CassandraDaemon.java (line 208) > Listening for thrift clients... > INFO [main] 2010-08-25 19:29:51,210 Mx4jTool.java (line 73) Will not load > MX4J, mx4j-tools.jar is not in the classpath > INFO [HINTED-HANDOFF-POOL:1] 2010-08-25 19:29:51,417 > HintedHandOffManager.java (line 191) Started hinted handoff for endpoint > /192.168.34.28 > INFO [GOSSIP_STAGE:1] 2010-08-25 19:29:51,417 Gossiper.java (line 570) > InetAddress /192.168.34.28 is now UP > INFO [HINTED-HANDOFF-POOL:1] 2010-08-25 19:29:51,418 > HintedHandOffManager.java (line 247) Finished hinted handoff of 0 rows to > endpoint /192.168.34.28 > INFO [HINTED-HANDOFF-POOL:1] 2010-08-25 19:29:51,855 > HintedHandOffManager.java (line 191) Started hinted handoff for endpoint > /192.168.34.29 > INFO [GOSSIP_STAGE:1] 2010-08-25 19:29:51,855 Gossiper.java (line 570) > InetAddress /192.168.34.29 is now UP > INFO [HINTED-HANDOFF-POOL:1] 2010-08-25 19:29:51,860 > HintedHandOffManager.java (line 247) Finished hinted handoff of 0 rows to > endpoint /192.168.34.29 > INFO [HINTED-HANDOFF-POOL:1] 2010-08-25 19:29:52,930 > HintedHandOffManager.java (line 191) Started hinted handoff for endpoint > /192.168.34.26 > INFO [GOSSIP_STAGE:1] 2010-08-25 19:29:52,930 Gossiper.java (line 570) > InetAddress /192.168.34.26 is now UP > INFO [HINTED-HANDOFF-POOL:1] 2010-08-25 19:29:52,930 > HintedHandOffManager.java (line 247) Finished hinted handoff of 0 rows to > endpoint /192.168.34.26 > I ran a repair on all the nodes and this was all that they each logged > INFO [manual-repair-fe7c5abb-bb0a-4415-aa75-0d72ba4e7f1b] 2010-08-25 > 19:49:24,194 AntiEntropyService.java (line 803) Waiting for repair requests > to: [] > > Cluster seems fine. Should I create a Jira ? > thanks > Aaron >
-- Jonathan Ellis Project Chair, Apache Cassandra co-founder of Riptano, the source for professional Cassandra support http://riptano.com
