Repair corrupt SSTable from power outage?
I have a 25 node cluster and we lost power on one of the racks last night and now 6 of our nodes will not start up and we are getting the following error: INFO [main] 2015-10-01 10:19:22,111 CassandraDaemon.java:122 - JMX is enabled to receive remote connections on port: 7199 INFO [main] 2015-10-01 10:19:22,124 CacheService.java:111 - Initializing key cache with capacity of 100 MBs. INFO [main] 2015-10-01 10:19:22,129 CacheService.java:133 - Initializing row cache with capacity of 0 MBs INFO [main] 2015-10-01 10:19:22,133 CacheService.java:150 - Initializing counter cache with capacity of 50 MBs INFO [main] 2015-10-01 10:19:22,135 CacheService.java:161 - Scheduling counter cache save to every 7200 seconds (going to save all keys). INFO [main] 2015-10-01 10:19:22,211 ColumnFamilyStore.java:363 - Initializing system.sstable_activity INFO [SSTableBatchOpen:1] 2015-10-01 10:19:22,639 SSTableReader.java:478 - Opening /mnt/cassandra/data/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/system-sstable_activity-ka-45 (805 bytes) ERROR [SSTableBatchOpen:1] 2015-10-01 10:19:22,657 FileUtils.java:447 - Exiting forcefully due to file system exception on startup, disk failure policy "stop" org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.EOFException at org.apache.cassandra.io.compress.CompressionMetadata.(CompressionMetadata.java:131) ~[apache-cassandra-2.1.9.jar:2.1.9] at org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:85) ~[apache-cassandra-2.1.9.jar:2.1.9] at org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.metadata(CompressedSegmentedFile.java:79) ~[apache-cassandra-2.1.9.jar:2.1.9] at org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:72) ~[apache-cassandra-2.1.9.jar:2.1.9] at org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:168) ~[apache-cassandra-2.1.9.jar:2.1.9] at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:752) ~[apache-cassandra-2.1.9.jar:2.1.9] at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:703) ~[apache-cassandra-2.1.9.jar:2.1.9] at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:491) ~[apache-cassandra-2.1.9.jar:2.1.9] at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:387) ~[apache-cassandra-2.1.9.jar:2.1.9] at org.apache.cassandra.io.sstable.SSTableReader$4.run(SSTableReader.java:534) ~[apache-cassandra-2.1.9.jar:2.1.9] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_60] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_60] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_60] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_60] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60] Caused by: java.io.EOFException: null at java.io.DataInputStream.readUnsignedShort(DataInputStream.java:340) ~[na:1.8.0_60] at java.io.DataInputStream.readUTF(DataInputStream.java:589) ~[na:1.8.0_60] at java.io.DataInputStream.readUTF(DataInputStream.java:564) ~[na:1.8.0_60] at org.apache.cassandra.io.compress.CompressionMetadata.(CompressionMetadata.java:106) ~[apache-cassandra-2.1.9.jar:2.1.9] ... 14 common frames omitted I found some people recommending scrubbing the sstable so I attempted that and got the following error: bin/sstablescrub system sstable_activity -v ERROR 17:26:03 Exiting forcefully due to file system exception on startup, disk failure policy "stop" org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.EOFException at org.apache.cassandra.io.compress.CompressionMetadata.(CompressionMetadata.java:131) ~[apache-cassandra-2.1.9.jar:2.1.9] at org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:85) ~[apache-cassandra-2.1.9.jar:2.1.9] at org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.metadata(CompressedSegmentedFile.java:79) ~[apache-cassandra-2.1.9.jar:2.1.9] at org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:72) ~[apache-cassandra-2.1.9.jar:2.1.9] at org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:168) ~[apache-cassandra-2.1.9.jar:2.1.9] at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:752) ~[apache-cassandra-2.1.9.jar:2.1.9] at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:703) ~[apache-cassandra-2.1.9.jar:2.1.9] at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:491) ~[apache-cassandra-2.1.9.jar:2.1.9] at org.apache.cassandra.io.sstable.SSTable
Re: Repair corrupt SSTable from power outage?
I'm running 2.1.9 On Fri, Oct 2, 2015 at 8:40 AM, Tyler Hobbs wrote: > What version of Cassandra are you running? It sounds like the disk > failure policy is incorrectly being applied to scrub, which kind of defeats > the purpose of it. I'd recommend opening a ticket ( > https://issues.apache.org/jira/browse/CASSANDRA) with the information you > posted here. > > On Fri, Oct 2, 2015 at 3:01 AM, George Sigletos > wrote: > >> I'm also facing problems regarding corrupt sstables and also couldn't run >> sstablescrub successfully. >> >> I restarted my nodes with disk failure policy "best_effort", then I run >> the "nodetool scrub " >> >> Once done I removed the corrupt tables manually and started repair >> >> >> On Thu, Oct 1, 2015 at 7:27 PM, John Anderson wrote: >> >>> I have a 25 node cluster and we lost power on one of the racks last >>> night and now 6 of our nodes will not start up and we are getting the >>> following error: >>> >>> INFO [main] 2015-10-01 10:19:22,111 CassandraDaemon.java:122 - JMX is >>> enabled to receive remote connections on port: 7199 >>> INFO [main] 2015-10-01 10:19:22,124 CacheService.java:111 - >>> Initializing key cache with capacity of 100 MBs. >>> INFO [main] 2015-10-01 10:19:22,129 CacheService.java:133 - >>> Initializing row cache with capacity of 0 MBs >>> INFO [main] 2015-10-01 10:19:22,133 CacheService.java:150 - >>> Initializing counter cache with capacity of 50 MBs >>> INFO [main] 2015-10-01 10:19:22,135 CacheService.java:161 - Scheduling >>> counter cache save to every 7200 seconds (going to save all keys). >>> INFO [main] 2015-10-01 10:19:22,211 ColumnFamilyStore.java:363 - >>> Initializing system.sstable_activity >>> INFO [SSTableBatchOpen:1] 2015-10-01 10:19:22,639 >>> SSTableReader.java:478 - Opening >>> /mnt/cassandra/data/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/system-sstable_activity-ka-45 >>> (805 bytes) >>> ERROR [SSTableBatchOpen:1] 2015-10-01 10:19:22,657 FileUtils.java:447 - >>> Exiting forcefully due to file system exception on startup, disk failure >>> policy "stop" >>> org.apache.cassandra.io.sstable.CorruptSSTableException: >>> java.io.EOFException >>> at >>> org.apache.cassandra.io.compress.CompressionMetadata.(CompressionMetadata.java:131) >>> ~[apache-cassandra-2.1.9.jar:2.1.9] >>> at >>> org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:85) >>> ~[apache-cassandra-2.1.9.jar:2.1.9] >>> at >>> org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.metadata(CompressedSegmentedFile.java:79) >>> ~[apache-cassandra-2.1.9.jar:2.1.9] >>> at >>> org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:72) >>> ~[apache-cassandra-2.1.9.jar:2.1.9] >>> at >>> org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:168) >>> ~[apache-cassandra-2.1.9.jar:2.1.9] >>> at >>> org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:752) >>> ~[apache-cassandra-2.1.9.jar:2.1.9] >>> at >>> org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:703) >>> ~[apache-cassandra-2.1.9.jar:2.1.9] >>> at >>> org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:491) >>> ~[apache-cassandra-2.1.9.jar:2.1.9] >>> at >>> org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:387) >>> ~[apache-cassandra-2.1.9.jar:2.1.9] >>> at >>> org.apache.cassandra.io.sstable.SSTableReader$4.run(SSTableReader.java:534) >>> ~[apache-cassandra-2.1.9.jar:2.1.9] >>> at >>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >>> [na:1.8.0_60] >>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>> [na:1.8.0_60] >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >>> [na:1.8.0_60] >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >>> [na:1.8.0_60] >>> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60] >>> Caused by: java.io.EOFException: null >>> at >>> java.io.DataInputStream.readUns
Reading hundreds of thousands of rows at once?
Hey, I'm looking at querying around 500,000 rows that I need to pull into a Pandas data frame for processing. Currently testing this on a single cassandra node it takes around 21 seconds: https://gist.github.com/sontek/4ca95f5c5aa539663eaf I tried introducing multiprocessing so I could use 4 processes at a time to query this and I got it down to 14 seconds: https://gist.github.com/sontek/542f13307ef9679c0094 Although shaving off 7 seconds is great it still isn't really where I would like to be in regards to performance, for this many rows I'd really like to get down to a max of 1-2 seconds query time. What types of optimization's can I make to improve the read performance when querying a large set of data? Will this timing speed up linearly as I add more nodes? This is what the schema looks like currently: https://gist.github.com/sontek/d6fa3fc1b6d085ad3fa4 I'm not tied to the current schema at all, its mostly just a replication of what we have in SQL Server. I'm more interested in what things I can change to make querying it faster. Thanks, John
Data Modeling help for representing a survey form.
I have an existing system in postgres that I would like to move to cassandra. The system is for building registration forms for conferences. For example, you might want to build a registration form (or survey) that has a bunch of questions on it. An overview of this system I whiteboarded here: http://paste2.org/JeHP1tV0 What I'm trying to figure out is how this data should be structured in a de-normalized way? Basic queries would be: 1. Give me all surveys for an account 2. Give me all questions for a survey 3. Give me all responses for a survey 4. Give me all responses for a specific question 5. Compare responses for question "What is your favorite color" with people who answered question "What is your gender". i.e a crosstab of males/females and the colors they like. 6. Give me a time series of how many people responded to a question per hour The reason I would like to get it on cassandra is because currently at peak times this is an extremely write heavy application since people are registering for a conference that launched or filling out a new survey, so everyone comes in all at once. Also, if anyone is in the bay area and wants to discuss cassandra data modeling over some beers, let me know! Thanks, John
