*face palm* You are totally right. I built from the wrong branch. I am so sorry. But at least you got yourself a development bug to figure out. :) The specific commit I built from was this: f3e0aa683f3f310678d62ba8345fe33633b709e0.
On Fri, Mar 8, 2013 at 9:14 PM, Yuki Morishita <[email protected]> wrote: > Are you sure you are using 1.2.2? > Because LegacyLeveledManifest is from unreleased development version. > > On Friday, March 8, 2013 at 11:02 PM, Arya Goudarzi wrote: > > Hi, > > I am exercising the rolling upgrade from 1.1.6 to 1.2.2. When I upgraded > to 1.2.2 on the first node, during startup I got this exception: > > ERROR [main] 2013-03-09 04:24:30,771 CassandraDaemon.java (line 213) > Could not migrate old leveled manifest. Move away the .json file in the > data directory > java.io.EOFException > at java.io.DataInputStream.readInt(DataInputStream.java:375) > at > org.apache.cassandra.utils.EstimatedHistogram$EstimatedHistogramSerializer.deserialize(EstimatedHistogram.java:265) > at > org.apache.cassandra.io.sstable.SSTableMetadata$SSTableMetadataSerializer.deserialize(SSTableMetadata.java:365) > at > org.apache.cassandra.io.sstable.SSTableMetadata$SSTableMetadataSerializer.deserialize(SSTableMetadata.java:351) > at > org.apache.cassandra.db.compaction.LegacyLeveledManifest.migrateManifests(LegacyLeveledManifest.java:100) > at > org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:209) > at > org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:391) > at > org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:434) > > > This is when it is trying to migrate LCS I believe. I removed the Json > files from data directories: > > data/ $ for i in `find . | grep 'json' | awk '{print $1}'`; do rm -rf $i; > done > > Then during the second attempt at restart, I got the following exception: > > ERROR [main] 2013-03-09 04:24:30,771 CassandraDaemon.java (line 213) > Could not migrate old leveled manifest. Move away the .json file in the > data directory > java.io.EOFException > at java.io.DataInputStream.readInt(DataInputStream.java:375) > at > org.apache.cassandra.utils.EstimatedHistogram$EstimatedHistogramSerializer.deserialize(EstimatedHistogram.java:265) > at > org.apache.cassandra.io.sstable.SSTableMetadata$SSTableMetadataSerializer.deserialize(SSTableMetadata.java:365) > at > org.apache.cassandra.io.sstable.SSTableMetadata$SSTableMetadataSerializer.deserialize(SSTableMetadata.java:351) > at > org.apache.cassandra.db.compaction.LegacyLeveledManifest.migrateManifests(LegacyLeveledManifest.java:100) > at > org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:209) > at > org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:391) > at > org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:434) > > OK. I seems it created snapshots prior to migration step. So it is safe to > remove those, right? > > data/ $ for i in `find . | grep 'pre-sstablemetamigration' | awk '{print > $1}'`; do rm -rf $i; done > > Now startup again, but I see bunch of corrupt sstable logs messages: > > ERROR [SSTableBatchOpen:1] 2013-03-09 04:55:39,826 SSTableReader.java > (line 242) Corrupt sstable > /var/lib/cassandra/data/keyspace_production/UniqueIndexes/keyspace_production-UniqueIndexes-hf-98318=[Filter.db, > Data.db, CompressionInfo.db, Statistics.db, Index.db]; skipped > java.io.EOFException > at java.io.DataInputStream.readInt(DataInputStream.java:375) > at > org.apache.cassandra.utils.BloomFilterSerializer.deserialize(BloomFilterSerializer.java:45) > at > org.apache.cassandra.utils.Murmur2BloomFilter$Murmur2BloomFilterSerializer.deserialize(Murmur2BloomFilter.java:40) > at > org.apache.cassandra.utils.FilterFactory.deserialize(FilterFactory.java:71) > at > org.apache.cassandra.io.sstable.SSTableReader.loadBloomFilter(SSTableReader.java:334) > at > org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:199) > at > org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:149) > at > org.apache.cassandra.io.sstable.SSTableReader$1.run(SSTableReader.java:238) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) > at java.lang.Thread.run(Thread.java:662) > > This is worrisome. How should I deal with this situation? scrub maybe? > Should I open a bug? > > Cheers, > Arya > > >
