I did a test upgrade first and ran the scrub as part of that process to make sure everything was working.
Cheers ----------------- Aaron Morton Freelance Cassandra Developer @aaronmorton http://www.thelastpickle.com On 11 Aug 2011, at 21:47, Zhu Han wrote: > > > On Wed, Aug 10, 2011 at 5:24 PM, aaron morton <aa...@thelastpickle.com> wrote: > I remember seeing this once before upgrading a system from 0.6 to 0.7 on a > Ubuntu EC2 (non data stax build) with EBS disks. I did the same thing and > just assumed it was an EBS or 0.6 bug. From memory after the upgrade that > node had some serialisation issues with a few SSTables that I had to > eventually take off the node. > > I would recommend running nodetool scrub *before* the upgrade to ensure the > files can be read. > > But "nodetool scrub" is provided by 0.7.3 or later. How can we run scrub > before upgrade? > > > Cheers > > ----------------- > Aaron Morton > Freelance Cassandra Developer > @aaronmorton > http://www.thelastpickle.com > > On 10 Aug 2011, at 13:09, Jonathan Ellis wrote: > > > What OS was this? > > > > On Tue, Aug 9, 2011 at 6:32 PM, ian douglas <i...@armorgames.com> wrote: > >> Thanks to 'driftx' in IRC, adding this line to my cassandra.yaml file seems > >> to have fixed the problem: > >> > >> disk_access_mode: standard > >> > >> > >> > >> > >> On 08/09/2011 04:06 PM, ian douglas wrote: > >> > >> updated stack trace after fixing the corrupted GameSave2 tables: > >> > >> > >> > >> INFO 16:05:20,314 Logging initialized > >> INFO 16:05:20,330 Heap size: 6290407424/6291456000 > >> INFO 16:05:20,336 JNA not found. Native methods will be disabled. > >> INFO 16:05:20,344 Loading settings from > >> file:/data/apache-cassandra-0.8.1/conf/cassandra.yaml > >> INFO 16:05:20,474 DiskAccessMode 'auto' determined to be mmap, > >> indexAccessMode is mmap > >> INFO 16:05:20,627 Global memtable threshold is enabled at 2000MB > >> INFO 16:05:20,628 Removing compacted SSTable files (see > >> http://wiki.apache.org/cassandra/MemtableSSTable) > >> INFO 16:05:20,633 Removing compacted SSTable files (see > >> http://wiki.apache.org/cassandra/MemtableSSTable) > >> INFO 16:05:20,634 Removing compacted SSTable files (see > >> http://wiki.apache.org/cassandra/MemtableSSTable) > >> INFO 16:05:20,635 Removing compacted SSTable files (see > >> http://wiki.apache.org/cassandra/MemtableSSTable) > >> INFO 16:05:20,636 Removing compacted SSTable files (see > >> http://wiki.apache.org/cassandra/MemtableSSTable) > >> INFO 16:05:20,637 Removing compacted SSTable files (see > >> http://wiki.apache.org/cassandra/MemtableSSTable) > >> INFO 16:05:20,737 Opening /data/cassandra8/data/system/Schema-g-13 > >> INFO 16:05:20,766 Opening /data/cassandra8/data/system/Migrations-g-13 > >> INFO 16:05:20,770 Opening /data/cassandra8/data/system/LocationInfo-g-2 > >> INFO 16:05:20,772 Opening /data/cassandra8/data/system/LocationInfo-g-1 > >> INFO 16:05:20,821 Loading schema version > >> 8fbed060-c2ae-11e0-0000-242d50cf1fff > >> INFO 16:05:21,028 Removing compacted SSTable files (see > >> http://wiki.apache.org/cassandra/MemtableSSTable) > >> INFO 16:05:21,030 Removing compacted SSTable files (see > >> http://wiki.apache.org/cassandra/MemtableSSTable) > >> INFO 16:05:21,032 Removing compacted SSTable files (see > >> http://wiki.apache.org/cassandra/MemtableSSTable) > >> INFO 16:05:21,033 Removing compacted SSTable files (see > >> http://wiki.apache.org/cassandra/MemtableSSTable) > >> INFO 16:05:21,035 Removing compacted SSTable files (see > >> http://wiki.apache.org/cassandra/MemtableSSTable) > >> INFO 16:05:21,039 Removing compacted SSTable files (see > >> http://wiki.apache.org/cassandra/MemtableSSTable) > >> INFO 16:05:21,040 Removing compacted SSTable files (see > >> http://wiki.apache.org/cassandra/MemtableSSTable) > >> INFO 16:05:21,044 Removing compacted SSTable files (see > >> http://wiki.apache.org/cassandra/MemtableSSTable) > >> INFO 16:05:21,045 Removing compacted SSTable files (see > >> http://wiki.apache.org/cassandra/MemtableSSTable) > >> INFO 16:05:21,047 Removing compacted SSTable files (see > >> http://wiki.apache.org/cassandra/MemtableSSTable) > >> INFO 16:05:21,047 Removing compacted SSTable files (see > >> http://wiki.apache.org/cassandra/MemtableSSTable) > >> INFO 16:05:21,048 Removing compacted SSTable files (see > >> http://wiki.apache.org/cassandra/MemtableSSTable) > >> INFO 16:05:21,049 Removing compacted SSTable files (see > >> http://wiki.apache.org/cassandra/MemtableSSTable) > >> INFO 16:05:21,049 Removing compacted SSTable files (see > >> http://wiki.apache.org/cassandra/MemtableSSTable) > >> INFO 16:05:21,050 Removing compacted SSTable files (see > >> http://wiki.apache.org/cassandra/MemtableSSTable) > >> INFO 16:05:21,054 Opening /data/cassandra8/data/armorgames/Feeds2-1777 > >> INFO 16:05:21,774 Opening /data/cassandra8/data/armorgames/Status-764 > >> INFO 16:05:21,800 Opening /data/cassandra8/data/armorgames/Comments-763 > >> INFO 16:05:21,816 Opening > >> /data/cassandra8/data/armorgames/UserGameshareData-8483 > >> INFO 16:05:31,645 Opening /data/cassandra8/data/armorgames/GameSave2-7309 > >> INFO 16:05:33,235 Opening /data/cassandra8/data/armorgames/GameSave2-7391 > >> INFO 16:05:33,240 Opening /data/cassandra8/data/armorgames/GameSave2-7390 > >> INFO 16:05:33,306 Opening /data/cassandra8/data/armorgames/UserData-9055 > >> INFO 16:05:38,058 Opening /data/cassandra8/data/armorgames/Feeds2-1777 > >> ERROR 16:05:38,071 Exception encountered during startup. > >> java.io.IOError: java.io.IOException: Channel not open for writing - cannot > >> extend file to required size > >> at > >> org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.createSegments(MmappedSegmentedFile.java:170) > >> at > >> org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.complete(MmappedSegmentedFile.java:147) > >> at > >> org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:340) > >> at > >> org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:194) > >> at > >> org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:280) > >> at > >> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:455) > >> at > >> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:436) > >> at org.apache.cassandra.db.Table.initCf(Table.java:369) > >> at org.apache.cassandra.db.Table.<init>(Table.java:306) > >> at org.apache.cassandra.db.Table.open(Table.java:111) > >> at > >> org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:161) > >> at > >> org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:315) > >> at > >> org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:80) > >> Caused by: java.io.IOException: Channel not open for writing - cannot > >> extend > >> file to required size > >> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:768) > >> at > >> org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.createSegments(MmappedSegmentedFile.java:162) > >> ... 12 more > >> Exception encountered during startup. > >> java.io.IOError: java.io.IOException: Channel not open for writing - cannot > >> extend file to required size > >> at > >> org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.createSegments(MmappedSegmentedFile.java:170) > >> at > >> org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.complete(MmappedSegmentedFile.java:147) > >> at > >> org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:340) > >> at > >> org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:194) > >> at > >> org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:280) > >> at > >> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:455) > >> at > >> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:436) > >> at org.apache.cassandra.db.Table.initCf(Table.java:369) > >> at org.apache.cassandra.db.Table.<init>(Table.java:306) > >> at org.apache.cassandra.db.Table.open(Table.java:111) > >> at > >> org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:161) > >> at > >> org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:315) > >> at > >> org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:80) > >> Caused by: java.io.IOException: Channel not open for writing - cannot > >> extend > >> file to required size > >> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:768) > >> at > >> org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.createSegments(MmappedSegmentedFile.java:162) > >> ... 12 more > >> > >> > >> > > > > > > > > -- > > Jonathan Ellis > > Project Chair, Apache Cassandra > > co-founder of DataStax, the source for professional Cassandra support > > http://www.datastax.com > >
