The analysis on https://issues.apache.org/jira/browse/CASSANDRA-5727 will be of interest.
As of 1.2.9 160mb is the new default for LCS. On Wed, Sep 18, 2013 at 3:35 PM, Jayadev Jayaraman <jdisal...@gmail.com>wrote: > m1.xlarge ( total ephemeral volume size 1.7TB ) is the most widely used > node configuration for Cassandra on EC2 according to this DataStax > presentation : http://www.slideshare.net/mattdennis/cassandra-on-ec2 . > > That said, I'm going with 40 for sstable_size_in_mb. My logic is as > follows : > > we load around 10+ GB in updates daily using SSTable loader, and with a 24 > node cluster, that is about 500 MB per node per day. We have around 5 large > column families and so the amount of newly flushed unleveled SSTable data > per node per column-family per day is around 100 MB on average. Compaction > triggers once every 2 days at least ( when unleveled data per CF exceeds > 40*4 = 160 MB ) so there should be about 32 unleveled SSTables per > column-family per node on average, a manageable number. If I reduce > sstable_size_in_mb to 5 MB, I will have 8 times as many SSTable files on > disk (even though all of them will be leveled immediately after flushing) > and I don't know how that impacts I/O performance / number of file-sockets > kept open for serving reads. > > > On Wed, Sep 18, 2013 at 3:38 PM, Hiller, Dean <dean.hil...@nrel.gov>wrote: > >> Sorry, bad bad typo…..300G is what I meant. >> >> Cassandra heavily advises to stay under 1T per node or you run into big >> troubles and most people stay under 500G per node. >> >> Later, >> Dean >> >> From: Jayadev Jayaraman <jdisal...@gmail.com<mailto:jdisal...@gmail.com>> >> Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" < >> user@cassandra.apache.org<mailto:user@cassandra.apache.org>> >> Date: Wednesday, September 18, 2013 1:30 PM >> To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" < >> user@cassandra.apache.org<mailto:user@cassandra.apache.org>> >> Subject: Re: What is the ideal value for sstable_size_in_mb when using >> LeveledCompactionStrategy ? >> >> Thanks for the quick reply. We've already upped the ulimit as high as our >> Linux distro allows us to ( around 1.8 million ). >> >> I have a follow-up question. I see that the size of individual nodes in >> your use case is quite massive. Does the safe number vary widely based on >> differences in underlying hardware, or would you say from experience that >> something around 50M for medium to large datasets ( with upped >> file-descriptor limits ) is safe for most medium-sized (1 - 5 TB per node) >> to high-end (hundreds of TB) hardware ? >> >> >> On Wed, Sep 18, 2013 at 3:15 PM, Hiller, Dean <dean.hil...@nrel.gov >> <mailto:dean.hil...@nrel.gov>> wrote: >> 1. Always in cassandra up your file descriptor limits on linux and even >> in 0.7 that was the recommendation so cassandra could open tons of files >> 2. We use 50M for our LCS with no performance issues. We had it 10M on >> our previous with no issues but a huge amount of files of course with our >> 300T per node. >> >> Dean >> >> From: Jayadev Jayaraman <jdisal...@gmail.com<mailto:jdisal...@gmail.com >> ><mailto:jdisal...@gmail.com<mailto:jdisal...@gmail.com>>> >> Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org >> ><mailto:user@cassandra.apache.org<mailto:user@cassandra.apache.org>>" < >> user@cassandra.apache.org<mailto:user@cassandra.apache.org><mailto: >> user@cassandra.apache.org<mailto:user@cassandra.apache.org>>> >> Date: Wednesday, September 18, 2013 1:02 PM >> To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org><mailto: >> user@cassandra.apache.org<mailto:user@cassandra.apache.org>>" < >> user@cassandra.apache.org<mailto:user@cassandra.apache.org><mailto: >> user@cassandra.apache.org<mailto:user@cassandra.apache.org>>> >> Subject: What is the ideal value for sstable_size_in_mb when using >> LeveledCompactionStrategy ? >> >> We have set up a 24 node (m1.xlarge nodes, 1.7 TB per node) cassandra >> cluster on Amazon EC2 : >> >> version=1.2.9 >> replication factor = 2 >> snitch=EC2Snitch >> placement_strategy=NetworkTopologyStrategy (with 12 nodes each in 2 >> availability zones) >> >> Background on our use-case : >> >> We plan on using hadoop with sstableloader to load 10GB+ of analytics >> data per day ( 100 million+ row keys, 5 or so columns per day on average.) >> . We have chosen LeveledCompactionStrategy in the hope that it constrains >> the number of SSTables that are read in order to retrieve a >> sliced-predicate for a row. We don't want too many file-sockets ( > 1000) >> open to SSTables by the Cassandra JVM as this has caused us network / >> unreachability issues before. We faced this when we were on cassandra 0.8.9 >> and we were using SizeTieredCompactionStrategy and in order to mitigate >> this, we ran minor compaction daily and major compaction semi-regularly to >> ensure as few SSTable files as possible on disk. >> >> >> >> >> >> If we use LeveledCompactionStrategy with a small value for >> sstable_size_in_mb ( default = 5 MB ) , wouldn't that result in a very >> large number of SSTable files on disk ? How does that affect the number of >> file-sockets open (reading the docs, I get the impression that the number >> of SSTable seeks per query is reduced by a large margin) ? But if we use a >> larger value for sstable_size_in_mb, say around 200 MB, there will be 800 >> MB of small uncompacted SSTables on disk per column-family to which there >> will inevitably be file-sockets open. >> >> All in all, can someone help us figure out what we should set the value >> of sstable_size_in_mb to ? I figure it's not a very good idea to set it to >> a larger value but I don't know how things perform if we set it to a small >> value. Do we have to run major compaction regularly in this case too ? >> >> Thanks >> Jayadev >> >> >> >> >
