> each node has 256G of memory, 24x1T drives, 2x Xeon CPU I don't have first hand experience running Cassandra on such massive hardware, but it strikes me that these machines are dramatically oversized to be good candidates for Cassandra (though I wonder how many cores are in those CPUs; I'm guessing closer to 18 than 2 based on the other hardware).
A larger cluster of smaller hardware would be a much better shape for Cassandra. Or several clusters of smaller hardware since you're running multiple instances on this hardware - best practices have one instance per host no matter the hardware size. On Thu, Feb 12, 2015 at 12:36 AM, Jiri Horky <ho...@avast.com> wrote: > Hi Chris, > > On 02/09/2015 04:22 PM, Chris Lohfink wrote: > > - number of tombstones - how can I reliably find it out? > https://github.com/spotify/cassandra-opstools > https://github.com/cloudian/support-tools > > thanks. > > > If not getting much compression it may be worth trying to disable it, it > may contribute but its very unlikely that its the cause of the gc pressure > itself. > > 7000 sstables but STCS? Sounds like compactions couldn't keep up. Do > you have a lot of pending compactions (nodetool)? You may want to increase > your compaction throughput (nodetool) to see if you can catch up a little, > it would cause a lot of heap overhead to do reads with that many. May even > need to take more drastic measures if it cant catch back up. > > I am sorry, I was wrong. We actually do use LCS (the switch was done > recently). There are almost none pending compaction. We have increased the > size sstable to 768M, so it should help as as well. > > > May also be good to check `nodetool cfstats` for very wide partitions. > > There are basically none, this is fine. > > It seems that the problem really comes from having so much data in so many > sstables, so org.apache.cassandra.io.compress.CompressedRandomAccessReader > classes consumes more memory than 0.75*HEAP_SIZE, which triggers the CMS > over and over. > > We have turned off the compression and so far, the situation seems to be > fine. > > Cheers > Jirka H. > > > > Theres a good chance if under load and you have over 8gb heap your GCs > could use tuning. The bigger the nodes the more manual tweaking it will > require to get the most out of them > https://issues.apache.org/jira/browse/CASSANDRA-8150 also has some ideas. > > Chris > > On Mon, Feb 9, 2015 at 2:00 AM, Jiri Horky <ho...@avast.com> wrote: > >> Hi all, >> >> thank you all for the info. >> >> To answer the questions: >> - we have 2 DCs with 5 nodes in each, each node has 256G of memory, >> 24x1T drives, 2x Xeon CPU - there are multiple cassandra instances running >> for different project. The node itself is powerful enough. >> - there 2 keyspaces, one with 3 replicas per DC, one with 1 replica per >> DC (because of amount of data and because it serves more or less like a >> cache) >> - there are about 4k/s Request-response, 3k/s Read and 2k/s Mutation >> requests - numbers are sum of all nodes >> - we us STCS (LCS would be quite IO have for this amount of data) >> - number of tombstones - how can I reliably find it out? >> - the biggest CF (3.6T per node) has 7000 sstables >> >> Now, I understand that the best practice for Cassandra is to run "with >> the minimum size of heap which is enough" which for this case we thought is >> about 12G - there is always 8G consumbed by the SSTable readers. Also, I >> though that high number of tombstones create pressure in the new space >> (which can then cause pressure in old space as well), but this is not what >> we are seeing. We see continuous GC activity in Old generation only. >> >> Also, I noticed that the biggest CF has Compression factor of 0.99 which >> basically means that the data come compressed already. Do you think that >> turning off the compression should help with memory consumption? >> >> Also, I think that tuning CMSInitiatingOccupancyFraction=75 might help >> here, as it seems that 8G is something that Cassandra needs for bookkeeping >> this amount of data and that this was sligtly above the 75% limit which >> triggered the CMS again and again. >> >> I will definitely have a look at the presentation. >> >> Regards >> Jiri Horky >> >> >> On 02/08/2015 10:32 PM, Mark Reddy wrote: >> >> Hey Jiri, >> >> While I don't have any experience running 4TB nodes (yet), I would >> recommend taking a look at a presentation by Arron Morton on large nodes: >> http://planetcassandra.org/blog/cassandra-community-webinar-videoslides-large-nodes-with-cassandra-by-aaron-morton/ >> to see if you can glean anything from that. >> >> I would note that at the start of his talk he mentions that in version >> 1.2 we can now talk about nodes around 1 - 3 TB in size, so if you are >> storing anything more than that you are getting into very specialised use >> cases. >> >> If you could provide us with some more information about your cluster >> setup (No. of CFs, read/write patterns, do you delete / update often, etc.) >> that may help in getting you to a better place. >> >> >> Regards, >> Mark >> >> On 8 February 2015 at 21:10, Kevin Burton <bur...@spinn3r.com> wrote: >> >>> Do you have a lot of individual tables? Or lots of small compactions? >>> >>> I think the general consensus is that (at least for Cassandra), 8GB >>> heaps are ideal. >>> >>> If you have lots of small tables it’s a known anti-pattern (I believe) >>> because the Cassandra internals could do a better job on handling the in >>> memory metadata representation. >>> >>> I think this has been improved in 2.0 and 2.1 though so the fact that >>> you’re on 1.2.18 could exasperate the issue. You might want to consider an >>> upgrade (though that has its own issues as well). >>> >>> On Sun, Feb 8, 2015 at 12:44 PM, Jiri Horky <ho...@avast.com> wrote: >>> >>>> Hi all, >>>> >>>> we are seeing quite high GC pressure (in old space by CMS GC Algorithm) >>>> on a node with 4TB of data. It runs C* 1.2.18 with 12G of heap memory >>>> (2G for new space). The node runs fine for couple of days when the GC >>>> activity starts to raise and reaches about 15% of the C* activity which >>>> causes dropped messages and other problems. >>>> >>>> Taking a look at heap dump, there is about 8G used by SSTableReader >>>> classes in >>>> org.apache.cassandra.io.compress.CompressedRandomAccessReader. >>>> >>>> Is this something expected and we have just reached the limit of how >>>> many data a single Cassandra instance can handle or it is possible to >>>> tune it better? >>>> >>>> Regards >>>> Jiri Horky >>>> >>> >>> >>> >>> -- >>> Founder/CEO Spinn3r.com >>> Location: *San Francisco, CA* >>> blog: http://burtonator.wordpress.com >>> … or check out my Google+ profile >>> <https://plus.google.com/102718274791889610666/posts> >>> <http://spinn3r.com> >>> >> >> >> > >
