I see only one line "JOINING: sleeping 30000 ms for pending range setup".
Before that, I have a lot of migration INFO messages, only for the uncompressed families. I have currently killed the new node, so a describe cluster shows only one schema version on the first two nodes. [default@unknown] describe cluster; Cluster Information: Snitch: org.apache.cassandra.locator.SimpleSnitch Partitioner: org.apache.cassandra.dht.ByteOrderedPartitioner Schema versions: b42595d0-2247-11e1-0000-a0e9ff9ab7bf: [---.36, ---.35] UNREACHABLE: [---.56] Also, on one of the old nodes, I have a lot of I/O errors on the data files for some (but not all) of the compressed families. It began a few days ago. All "nodetool repair" calls have been blocking since then. Any ideas on how I can get the data on the new node, before the old one dies? Thanks, Alexis Lauthier ________________________________ De : aaron morton <aaron@> À : user@cassandra.apache.org Envoyé le : Dimanche 15 Janvier 2012 19h17 Objet : Re: Compressed families not created on new node Sounds like the schema has not fully migrated to the new node. It is applied to the joining node one change at a time. A quick scan of the changes file does not find anything fixed after 1.0.3 You can check schema versions in the CLI using the describe cluster command. Check for errors in the logs with Migration in the text. Are you seeing this line a lot in the log ? INFO [main] 2012-01-13 14:55:00,493 StorageService.java (line 616) JOINING: sleeping 30000 ms for pending range setup > cheers ----------------- Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 14/01/2012, at 4:20 AM, Alexis Lauthier wrote: I'm using Cassandra 1.0.3 on a 2 nodes cluster. My schema (with replication_factor=2) contains both compressed (with sstable_compression=DeflateCompressor) and uncompressed column families. > > > >When bootstrapping a third node, the uncompressed families are created on the >new node as expected, but the compressed families are not. Only the >uncompressed families appear in a "show schema", and the new node data size is >small, which is consistent with the big compressed data not being there. > > >I'm seeing frequent exceptions in the log : > > > INFO [main] 2012-01-13 14:55:00,493 StorageService.java (line 616) JOINING: >sleeping 30000 ms for pending range setup >ERROR [MutationStage:1] 2012-01-13 14:55:01,511 RowMutationVerbHandler.java >(line 65) Error in row mutation >org.apache.cassandra.db.UnserializableColumnFamilyException: Couldn't find >cfId=1008 > at >org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:129) > at >org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:401) > at >org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:409) > at org.apache.cassandra.db.RowMutation.fromBytes(RowMutation.java:357) > at >org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:46) > at >org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59) > at >java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:722) > > > > > >After a few minutes, The column family names are shown instead of their ids >("UrlText" is one of the compressed families) : > > > >ERROR [ReadStage:46] 2012-01-13 14:59:33,924 AbstractCassandraDaemon.java >(line 133) Fatal exception in thread Thread[ReadStage:46,5,main] >java.lang.IllegalArgumentException: Unknown ColumnFamily UrlText in keyspace >ptprod > at org.apache.cassandra.config.Schema.getComparator(Schema.java:226) > at >org.apache.cassandra.db.ColumnFamily.getComparatorFor(ColumnFamily.java:300) > at org.apache.cassandra.db.ReadCommand.getComparator(ReadCommand.java:92) > at org.apache.cassandra.db.SliceByNamesReadCommand.<init>(SliceByNamesReadCommand.java:44) > at >org.apache.cassandra.db.SliceByNamesReadCommandSerializer.deserialize(SliceByNamesReadCommand.java:106) > at >org.apache.cassandra.db.SliceByNamesReadCommandSerializer.deserialize(SliceByNamesReadCommand.java:74) > at >org.apache.cassandra.db.ReadCommandSerializer.deserialize(ReadCommand.java:132) > at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:51) > at >org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59) > at >java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at >java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:722) > > > > > >How can I get the compressed families on the new node ? > > >Thanks, >Alexis Lauthier >
