eeek, HW errors. I would guess (thats all it is) that an IO error may have stopped the schema from migrating.
Stop cassandra on that node and copy the files off as best you can. I would then try a node replacement First remove the failed new node with nodetool decomission or removetoken. You are now down to one server. Copy the yaml file from the old machine (with IO errors) to a new one. To make things potentially less complicated bump the initial token slightly (e.g. add 1 to it) so the new node will not be exactly replacing the old one. Now start the new node. The other node will notice the schema is out of date and send it across. Once all the CF's are added and the schema's match stop the new node, copy the SSTable data from the old node to the new one and restart it. There are other ways to do this, this is the simplest though. With the new node in place the ring should now show the IO error node as down, the new node with a token very close to the IO error node and the one other node. You can now remove the IO error node with decomission or removetoken. Now run a repair from the new node. At any stage the rollback plan is to simply turn the IO error node back on. Hope that helps. ----------------- Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 16/01/2012, at 10:41 PM, Alexis Lauthier wrote: > The I/O errors are caused by disk failure. Syslog contains some of those > things: > > Jan 16 09:53:24 --- kernel: [7065781.460804] sd 4:0:0:0: [sda] Add. Sense: > Unrecovered read error > Jan 16 09:53:24 --- kernel: [7065781.460810] sd 4:0:0:0: [sda] CDB: Read(10): > 28 00 11 cf 60 70 00 00 08 00 > Jan 16 09:53:24 --- kernel: [7065781.460820] end_request: I/O error, dev sda, > sector 298803312 > > > Scrub failed: > > <a lot of lines saying it is scrubbing happily up to this point> > INFO [CompactionExecutor:5818] 2012-01-16 09:45:20,650 > CompactionManager.java (line 477) Scrubbing > SSTableReader(path='/home/cassprod/data/ptprod/UrlInfo-hb-1326-Data.db') > ERROR [CompactionExecutor:5818] 2012-01-16 09:47:51,531 PrecompactedRow.java > (line 119) Skipping row > DecoratedKey(Token(bytes[01f9332e566a3a8d5a1cc17e530ae46e]), > 01f9332e566a3a8d5a1cc17e530ae46e) in > /home/cassprod/data/ptprod/UrlInfo-hb-1326-Data.db > java.io.IOException: (/home/cassprod/data/ptprod/UrlInfo-hb-1326-Data.db) > failed to read 13705 bytes from offset 3193541. > at > org.apache.cassandra.io.compress.CompressedRandomAccessReader.decompressChunk(CompressedRandomAccessReader.java:87) > at > org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:75) > at > org.apache.cassandra.io.util.RandomAccessReader.read(RandomAccessReader.java:302) > at java.io.RandomAccessFile.readFully(RandomAccessFile.java:397) > at java.io.RandomAccessFile.readFully(RandomAccessFile.java:377) > at > org.apache.cassandra.utils.BytesReadTracker.readFully(BytesReadTracker.java:95) > at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:392) > at > org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:354) > at > org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:120) > at > org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:37) > at > org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:147) > at > org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:232) > at > org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:115) > at > org.apache.cassandra.db.compaction.PrecompactedRow.<init>(PrecompactedRow.java:102) > at > org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:133) > at > org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:139) > at > org.apache.cassandra.db.compaction.CompactionManager.scrubOne(CompactionManager.java:565) > at > org.apache.cassandra.db.compaction.CompactionManager.doScrub(CompactionManager.java:472) > at > org.apache.cassandra.db.compaction.CompactionManager.access$300(CompactionManager.java:63) > at > org.apache.cassandra.db.compaction.CompactionManager$3.call(CompactionManager.java:224) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > WARN [CompactionExecutor:5818] 2012-01-16 09:47:51,531 > CompactionManager.java (line 581) Non-fatal error reading row (stacktrace > follows) > java.lang.NullPointerException > WARN [CompactionExecutor:5818] 2012-01-16 09:47:51,532 > CompactionManager.java (line 623) Row at 14740167 is unreadable; skipping to > next > ERROR [CompactionExecutor:5818] 2012-01-16 09:53:24,395 > AbstractCassandraDaemon.java (line 133) Fatal exception in thread > Thread[CompactionExecutor:5818,1,RMI Runtime] > java.io.IOException: (/home/cassprod/data/ptprod/UrlInfo-hb-1326-Data.db) > failed to read 13705 bytes from offset 3193541. > at > org.apache.cassandra.io.compress.CompressedRandomAccessReader.decompressChunk(CompressedRandomAccessReader.java:87) > at > org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:75) > at > org.apache.cassandra.io.util.RandomAccessReader.seek(RandomAccessReader.java:259) > at > org.apache.cassandra.db.compaction.CompactionManager.scrubOne(CompactionManager.java:625) > at > org.apache.cassandra.db.compaction.CompactionManager.doScrub(CompactionManager.java:472) > at > org.apache.cassandra.db.compaction.CompactionManager.access$300(CompactionManager.java:63) > at > org.apache.cassandra.db.compaction.CompactionManager$3.call(CompactionManager.java:224) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > > > The same kind of "failed to read" IOExceptions have been routinely logged for > 13 days now. > > The best idea I can come up with now is: decommission the failing node, then > add the new node. And hope the schema will fully replicate. This will leave > me with only one node for a time, and I'm not sure it will play nice with > replication_factor=2. > This feels a lot like jumping out of a plane with an untested parachute. So > any other ideas ? > > Thanks, > Alexis Lauthier > > > > De : aaron morton <aa...@thelastpickle.com> > À : user@cassandra.apache.org > Envoyé le : Lundi 16 Janvier 2012 1h05 > Objet : Re: Compressed families not created on new node > > Without knowing what the IOErrors are I would do the following: > > > nodetool scrub to fix any on disk errors, this will also take a snapshot you > can use for rollback. > nodetool repair to ensure data is consistent. > > Hope that helps. > > ----------------- > Aaron Morton > Freelance Developer > @aaronmorton > http://www.thelastpickle.com > > On 16/01/2012, at 7:53 AM, Alexis Lauthier wrote: > >> I see only one line "JOINING: sleeping 30000 ms for pending range setup". >> >> Before that, I have a lot of migration INFO messages, only for the >> uncompressed families. >> >> I have currently killed the new node, so a describe cluster shows only one >> schema version on the first two nodes. >> >> [default@unknown] describe cluster; >> Cluster Information: >> Snitch: org.apache.cassandra.locator.SimpleSnitch >> Partitioner: org.apache.cassandra.dht.ByteOrderedPartitioner >> Schema versions: >> b42595d0-2247-11e1-0000-a0e9ff9ab7bf: [---.36, ---.35] >> >> UNREACHABLE: [---.56] >> >> >> Also, on one of the old nodes, I have a lot of I/O errors on the data files >> for some (but not all) of the compressed families. It began a few days ago. >> All "nodetool repair" calls have been blocking since then. >> >> Any ideas on how I can get the data on the new node, before the old one dies? >> >> >> Thanks, >> Alexis Lauthier >> >> >> De : aaron morton <aaron@> >> À : user@cassandra.apache.org >> Envoyé le : Dimanche 15 Janvier 2012 19h17 >> Objet : Re: Compressed families not created on new node >> >> Sounds like the schema has not fully migrated to the new node. It is applied >> to the joining node one change at a time. A quick scan of the changes file >> does not find anything fixed after 1.0.3 >> >> You can check schema versions in the CLI using the describe cluster command. >> >> Check for errors in the logs with Migration in the text. >> >> Are you seeing this line a lot in the log ? >>> INFO [main] 2012-01-13 14:55:00,493 StorageService.java (line 616) >>> JOINING: sleeping 30000 ms for pending range setup >> >> cheers >> >> ----------------- >> Aaron Morton >> Freelance Developer >> @aaronmorton >> http://www.thelastpickle.com >> >> On 14/01/2012, at 4:20 AM, Alexis Lauthier wrote: >> >>> I'm using Cassandra 1.0.3 on a 2 nodes cluster. My schema (with >>> replication_factor=2) contains both compressed (with >>> sstable_compression=DeflateCompressor) and uncompressed column families. >>> >>> When bootstrapping a third node, the uncompressed families are created on >>> the new node as expected, but the compressed families are not. Only the >>> uncompressed families appear in a "show schema", and the new node data size >>> is small, which is consistent with the big compressed data not being there. >>> >>> I'm seeing frequent exceptions in the log : >>> >>> INFO [main] 2012-01-13 14:55:00,493 StorageService.java (line 616) >>> JOINING: sleeping 30000 ms for pending range setup >>> ERROR [MutationStage:1] 2012-01-13 14:55:01,511 RowMutationVerbHandler.java >>> (line 65) Error in row mutation >>> org.apache.cassandra.db.UnserializableColumnFamilyException: Couldn't find >>> cfId=1008 >>> at >>> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:129) >>> at >>> org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:401) >>> at >>> org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:409) >>> at org.apache.cassandra.db.RowMutation.fromBytes(RowMutation.java:357) >>> at >>> org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:46) >>> at >>> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) >>> at java.lang.Thread.run(Thread.java:722) >>> >>> >>> After a few minutes, The column family names are shown instead of their ids >>> ("UrlText" is one of the compressed families) : >>> >>> ERROR [ReadStage:46] 2012-01-13 14:59:33,924 AbstractCassandraDaemon.java >>> (line 133) Fatal exception in thread Thread[ReadStage:46,5,main] >>> java.lang.IllegalArgumentException: Unknown ColumnFamily UrlText in >>> keyspace ptprod >>> at org.apache.cassandra.config.Schema.getComparator(Schema.java:226) >>> at >>> org.apache.cassandra.db.ColumnFamily.getComparatorFor(ColumnFamily.java:300) >>> at >>> org.apache.cassandra.db.ReadCommand.getComparator(ReadCommand.java:92) >>> at >>> org.apache.cassandra.db.SliceByNamesReadCommand.<init>(SliceByNamesReadCommand.java:44) >>> at >>> org.apache.cassandra.db.SliceByNamesReadCommandSerializer.deserialize(SliceByNamesReadCommand.java:106) >>> at >>> org.apache.cassandra.db.SliceByNamesReadCommandSerializer.deserialize(SliceByNamesReadCommand.java:74) >>> at >>> org.apache.cassandra.db.ReadCommandSerializer.deserialize(ReadCommand.java:132) >>> at >>> org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:51) >>> at >>> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) >>> at java.lang.Thread.run(Thread.java:722) >>> >>> >>> How can I get the compressed families on the new node ? >>> >>> Thanks, >>> Alexis Lauthier >> >> >> > > >
