Hi Aaron Removing NodeInfo did do the trick, thanks. I see the ticket is already resolved, good news.
Thanks for the help. On Fri, Sep 7, 2012 at 12:26 AM, aaron morton <aa...@thelastpickle.com>wrote: > This is a problem… > > [default@system] list NodeIdInfo ; >> Using default limit of 100 >> ... >> >> ------------------- >> RowKey: 43757272656e744c6f63616c >> => (column=01efa5d0-e133-11e1-0000-51be601cd0ff, value=0a1020d2, >> timestamp=1344414498989) >> => (column=92109b80-ea0a-11e1-0000-51be601cd0af, value=0a1020d2, >> timestamp=1345386691897) >> > > There is an assertion that we should only have one column for the > "CurrentLocal" row. The exception is occurring in a static class member > initialisation and so is getting lost. > > I'm not sure how two ended up there. > > Deleting the NodeIdInfo CF SSTables should fix it. > > I created https://issues.apache.org/jira/browse/CASSANDRA-4626 can you > please add more information there if you can and/or watch the ticket incase > there are other questions. > > Thanks > > > ----------------- > Aaron Morton > Freelance Developer > @aaronmorton > http://www.thelastpickle.com > > On 5/09/2012, at 10:18 PM, Thomas van Neerijnen <t...@bossastudios.com> > wrote: > > forgot to answer your first question. I see this: > INFO 14:31:31,896 No saved local node id, using newly generated: > 92109b80-ea0a-11e1-0000-51be601cd0af > > > On Wed, Sep 5, 2012 at 8:41 AM, Thomas van Neerijnen <t...@bossastudios.com > > wrote: > >> Thanks for the help Aaron. >> I've checked NodeIdInfo and LocationInfo as below. >> What am I looking at? I'm guessing the first row in NodeIdInfo represents >> the ring with the node ids, but the second row perhaps dead nodes with old >> schemas? That's a total guess, I'd be very interested to know what it and >> the LocationInfo are. >> If there's anything else you'd like me to check let me know, otherwise >> I'll attempt your workaround later today. >> >> [default@system] list NodeIdInfo ; >> Using default limit of 100 >> ------------------- >> RowKey: 4c6f63616c >> => (column=b10552c0-ea0f-11e0-0000-cb1f02ccbcff, value=0a1020d2, >> timestamp=1317241393645) >> => (column=e64fc8f0-595b-11e1-0000-51be601cd0d7, value=0a1020d2, >> timestamp=1329478703871) >> => (column=732d4690-a596-11e1-0000-51be601cd09f, value=0a1020d2, >> timestamp=1337860139385) >> => (column=bffd9d40-aa45-11e1-0000-51be601cd0fe, value=0a1020d2, >> timestamp=1338375234836) >> => (column=01efa5d0-e133-11e1-0000-51be601cd0ff, value=0a1020d2, >> timestamp=1344414498989) >> => (column=92109b80-ea0a-11e1-0000-51be601cd0af, value=0a1020d2, >> timestamp=1345386691897) >> ------------------- >> RowKey: 43757272656e744c6f63616c >> => (column=01efa5d0-e133-11e1-0000-51be601cd0ff, value=0a1020d2, >> timestamp=1344414498989) >> => (column=92109b80-ea0a-11e1-0000-51be601cd0af, value=0a1020d2, >> timestamp=1345386691897) >> >> 2 Rows Returned. >> Elapsed time: 128 msec(s). >> [default@system] list LocationInfo ; >> Using default limit of 100 >> ------------------- >> RowKey: 52696e67 >> => (column=00, value=0a1080d2, timestamp=1346666104900) >> => (column=04a7128b6c83505dcd618720f92028f4, value=0a1020b7, >> timestamp=1332360971660) >> => (column=09249249249249249249249249249249, value=0a1080cd, >> timestamp=1341136002862) >> => (column=12492492492492492492492492492492, value=0a1020d3, >> timestamp=1341135999465) >> => (column=15555555555555000000000000000000, value=0a1060d3, >> timestamp=1346666104671) >> => (column=15555555555555555555555555555555, value=0a1020d3, >> timestamp=1344530188382) >> => (column=1b6db6db6db6db6db6db6db6db6db6db, value=0a1020b1, >> timestamp=1341135997643) >> => (column=1c71c71c71c71bffffffffffffffffff, value=0a1080d2, >> timestamp=1317241889689) >> => (column=24924924924924924924924924924924, value=0a1060d3, >> timestamp=1341135996555) >> => (column=2aaaaaaaaaaaa9ffffffffffffffffff, value=0a1020d3, >> timestamp=1317241534292) >> => (column=2aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa, value=0a1060d3, >> timestamp=1344530187539) >> => (column=38e38e38e38e37ffffffffffffffffff, value=0a1060d3, >> timestamp=1317241257569) >> => (column=38e38e38e38e38e38e38e38e38e38e38, value=0a1060d3, >> timestamp=1343136501647) >> => (column=393170e0207a17d8519f0c1bfe325d51, value=0a1020d3, >> timestamp=1345381375120) >> => (column=3fffffffffffffffffffffffffffffff, value=0a1080d3, >> timestamp=1346666104939) >> => (column=471c71c71c71c71c71c71c71c71c71c6, value=0a1080d3, >> timestamp=1343133153701) >> => (column=471c71c71c71c7ffffffffffffffffff, value=0a1080d3, >> timestamp=1317241786636) >> => (column=49249249249249249249249249249249, value=0a1080d3, >> timestamp=1341136002693) >> => (column=52492492492492492492492492492492, value=0a106010, >> timestamp=1341136002626) >> => (column=55555555555553ffffffffffffffffff, value=0a1020d4, >> timestamp=1328473688357) >> => (column=55555555555555555555555555555554, value=0a1060d4, >> timestamp=1346666104910) >> => (column=5b6db6db6db6db6db6db6db6db6db6da, value=0a1060d4, >> timestamp=1332389784945) >> => (column=5b6db6db6db6db6db6db6db6db6db6db, value=0a1060d4, >> timestamp=1341136001027) >> => (column=638e38e38e38e38e38e38e38e38e38e2, value=0a1060d4, >> timestamp=1343125208462) >> => (column=638e38e38e38e3ffffffffffffffffff, value=0a1060d4, >> timestamp=1317241257577) >> => (column=6aaaaaaaaaaaac000000000000000000, value=0a1020d3, >> timestamp=1346666104789) >> ------------------- >> RowKey: 4c >> => (column=436c75737465724e616d65, >> value=4d6f6e737465724d696e642050726f6420436c7573746572, >> timestamp=1317241251097000) >> => (column=47656e65726174696f6e, value=50447e78, >> timestamp=1346666104152000) >> => (column=50617274696f6e6572, >> value=6f72672e6170616368652e63617373616e6472612e6468742e52616e646f6d506172746974696f6e6572, >> timestamp=1317241251097000) >> => (column=546f6b656e, value=2aaaaaaaaaaaaa000000000000000000, >> timestamp=1346666104214) >> ------------------- >> RowKey: 436f6f6b696573 >> => >> (column=48696e7473207075726765642061732070617274206f6620757067726164696e672066726f6d20302e362e7820746f20302e37, >> value=6f68207965732c20697420746865792077657265207075726765642e, >> timestamp=1317241251249) >> => (column=5072652d312e302068696e747320707572676564, >> value=6f68207965732c2074686579207765726520707572676564, >> timestamp=1326274339337) >> ------------------- >> RowKey: 426f6f747374726170 >> => (column=42, value=01, timestamp=1346666104213) >> >> 4 Rows Returned. >> Elapsed time: 34 msec(s). >> >> >> On Wed, Sep 5, 2012 at 2:42 AM, aaron morton <aa...@thelastpickle.com>wrote: >> >>> Hmmm, this looks like an error in ctor for NodeId$LocalNodeIdHistory. >>> Are there any other ERROR log messages? >>> >>> Do you see either of these two messages in the log: >>> "No saved local node id, using newly generated: {}" >>> or >>> "Saved local node id: {}" >>> >>> >>> Can you use cassandra-cli / cqlsh to print the contents of >>> the NodeIdInfo and LocationInfo CF's from system ? >>> >>> I *think* a work around may be to: >>> >>> * stop the node >>> * remove LocationInfo and NodeInfo cfs. >>> * restart >>> >>> Note this will read the token from the yaml file again, so make sure >>> it's right. >>> >>> cheers >>> >>> ----------------- >>> Aaron Morton >>> Freelance Developer >>> @aaronmorton >>> http://www.thelastpickle.com >>> >>> On 4/09/2012, at 9:51 PM, Thomas van Neerijnen <t...@bossastudios.com> >>> wrote: >>> >>> Hi >>> >>> I have a single node in a 6 node Cassandra 1.0.11 cluster that seems to >>> have a single column family in a weird state. >>> >>> Repairs, upgradesstables, anything that touches this CF crashes. >>> I've drained the node, removed every file for this CF from said node, >>> removed the commit log, started it up and as soon as data is written to >>> this CF on this node I'm in the same situation again. Anyone have any >>> suggestions for how to fix this? >>> I'm tempted to remove the node and re-add it but I was hoping for >>> something a little less disruptive. >>> >>> $ nodetool -h localhost upgradesstables Player PlayerCounters >>> Error occured while upgrading the sstables for keyspace Player >>> java.util.concurrent.ExecutionException: java.lang.NoClassDefFoundError: >>> Could not initialize class org.apache.cassandra.utils.NodeId$LocalIds >>> at >>> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) >>> at java.util.concurrent.FutureTask.get(FutureTask.java:83) >>> at >>> org.apache.cassandra.db.compaction.CompactionManager.performAllSSTableOperation(CompactionManager.java:219) >>> at >>> org.apache.cassandra.db.compaction.CompactionManager.performSSTableRewrite(CompactionManager.java:235) >>> at >>> org.apache.cassandra.db.ColumnFamilyStore.sstablesRewrite(ColumnFamilyStore.java:999) >>> at >>> org.apache.cassandra.service.StorageService.upgradeSSTables(StorageService.java:1652) >>> at sun.reflect.GeneratedMethodAccessor59.invoke(Unknown Source) >>> at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>> at java.lang.reflect.Method.invoke(Method.java:597) >>> at >>> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93) >>> at >>> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27) >>> at >>> com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208) >>> at >>> com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120) >>> at >>> com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262) >>> at >>> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836) >>> at >>> com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761) >>> at >>> javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427) >>> at >>> javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72) >>> at >>> javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265) >>> at >>> javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360) >>> at >>> javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788) >>> at sun.reflect.GeneratedMethodAccessor58.invoke(Unknown Source) >>> at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>> at java.lang.reflect.Method.invoke(Method.java:597) >>> at >>> sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:303) >>> at sun.rmi.transport.Transport$1.run(Transport.java:159) >>> at java.security.AccessController.doPrivileged(Native Method) >>> at sun.rmi.transport.Transport.serviceCall(Transport.java:155) >>> at >>> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535) >>> at >>> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790) >>> at >>> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >>> at java.lang.Thread.run(Thread.java:662) >>> Caused by: java.lang.NoClassDefFoundError: Could not initialize class >>> org.apache.cassandra.utils.NodeId$LocalIds >>> at org.apache.cassandra.utils.NodeId.localIds(NodeId.java:49) >>> at >>> org.apache.cassandra.utils.NodeId.getOldLocalNodeIds(NodeId.java:79) >>> at >>> org.apache.cassandra.db.CounterColumn.computeOldShardMerger(CounterColumn.java:251) >>> at >>> org.apache.cassandra.db.CounterColumn.mergeAndRemoveOldShards(CounterColumn.java:297) >>> at >>> org.apache.cassandra.db.CounterColumn.mergeAndRemoveOldShards(CounterColumn.java:271) >>> at >>> org.apache.cassandra.db.compaction.PrecompactedRow.removeDeletedAndOldShards(PrecompactedRow.java:81) >>> at >>> org.apache.cassandra.db.compaction.PrecompactedRow.<init>(PrecompactedRow.java:97) >>> at >>> org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:137) >>> at >>> org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:97) >>> at >>> org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:82) >>> at >>> org.apache.cassandra.utils.MergeIterator$OneToOne.computeNext(MergeIterator.java:207) >>> at >>> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140) >>> at >>> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135) >>> at >>> com.google.common.collect.Iterators$7.computeNext(Iterators.java:614) >>> at >>> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140) >>> at >>> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135) >>> at >>> org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:141) >>> at >>> org.apache.cassandra.db.compaction.CompactionManager$4.perform(CompactionManager.java:247) >>> at >>> org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:198) >>> at >>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) >>> at java.util.concurrent.FutureTask.run(FutureTask.java:138) >>> ... 3 more >>> $ tail /var/log/cassandra/system.log >>> INFO [CompactionExecutor:852] 2012-09-04 09:45:37,021 >>> CompactionTask.java (line 112) Compacting >>> [SSTableReader(path='/var/lib/cassandra/data/Player/PlayerCounters-hd-1-Data.db')] >>> ERROR [CompactionExecutor:852] 2012-09-04 09:45:37,030 >>> AbstractCassandraDaemon.java (line 139) Fatal exception in thread >>> Thread[CompactionExecutor:852,1,RMI Runtime] >>> java.lang.NoClassDefFoundError: Could not initialize class >>> org.apache.cassandra.utils.NodeId$LocalIds >>> at org.apache.cassandra.utils.NodeId.localIds(NodeId.java:49) >>> at >>> org.apache.cassandra.utils.NodeId.getOldLocalNodeIds(NodeId.java:79) >>> at >>> org.apache.cassandra.db.CounterColumn.computeOldShardMerger(CounterColumn.java:251) >>> at >>> org.apache.cassandra.db.CounterColumn.mergeAndRemoveOldShards(CounterColumn.java:297) >>> at >>> org.apache.cassandra.db.CounterColumn.mergeAndRemoveOldShards(CounterColumn.java:271) >>> at >>> org.apache.cassandra.db.compaction.PrecompactedRow.removeDeletedAndOldShards(PrecompactedRow.java:81) >>> at >>> org.apache.cassandra.db.compaction.PrecompactedRow.<init>(PrecompactedRow.java:97) >>> at >>> org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:137) >>> at >>> org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:97) >>> at >>> org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:82) >>> at >>> org.apache.cassandra.utils.MergeIterator$OneToOne.computeNext(MergeIterator.java:207) >>> at >>> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140) >>> at >>> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135) >>> at >>> com.google.common.collect.Iterators$7.computeNext(Iterators.java:614) >>> at >>> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140) >>> at >>> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135) >>> at >>> org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:141) >>> at >>> org.apache.cassandra.db.compaction.CompactionManager$4.perform(CompactionManager.java:247) >>> at >>> org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:198) >>> at >>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) >>> at java.util.concurrent.FutureTask.run(FutureTask.java:138) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >>> at java.lang.Thread.run(Thread.java:662) >>> >>> >>> >> > >
