There's another thread going on right now in this list about compactions not happening when they seemingly should. Tyler Hobbs postulates a bug and workaround for it, so maybe try that out, and if that fixes anything for you, certainly let him know. The bug Tyler postulates on is triggered when you have a write-heavy with zero read workload, and if you're testing data loading, maybe you're triggering that.
Also, it's probably a long shot, but make sure that your SSTable counts haven't gone down since you last looked, if you're load testing your cluster and throwing big bursts of writes at it, you can temporarily fall behind on compaction (getting a large number of sstables), and if you stop writes, you can catch back up again as a result - maybe you looked at table counts during loading, and looked at compactionstats during a quiet time. On Fri, Jan 16, 2015 at 12:37 AM, Jan Kesten <j.kes...@enercast.de> wrote: > Hi Eric and all, > > I almost expected this kind answer. I did a nodetool compactionstats > already to see if those sstables are beeing compacted, but on all nodes > there are 0 outstanding compactions (right now in the morning, not running > any tests on this cluster). > > The reported read latency is about 1-3ms and on nodes which have many > sstables (new highscore are ~18k sstables). The 99% percentile is about > 30-40 micros and a cell count of about 80-90 (if I got the docs right these > are the number of sstables accessed, that changed from 2.0 to 2.1 I think > as I see this only on testing cluster). > > I looks to me that compactions were not triggered. I tried a nodetool > compact on one node overnight - but that crashed the entire node. > > Roland > > Am 15.01.2015 um 19:14 schrieb Eric Stevens: > > Yes, many sstables can have a huge negative impact read performance, and > will also create memory pressure on that node. > > There are a lot of things which can produce this effect, and it strongly > also suggests you're falling behind on compaction in general (check > nodetool compactionstats, you should have <5 outstanding/pending, > preferably 0-1). To see whether and how much it is impacting your read > performance, check nodetool cfstats <keyspace.table> and nodetool > cfhistograms <keyspace> <table>. > > > On Thu, Jan 15, 2015 at 2:11 AM, Roland Etzenhammer < > r.etzenham...@t-online.de> wrote: > >> Hi, >> >> I'm testing around with cassandra fair a bit, using 2.1.2 which I know >> has some major issues,but it is a test environment. After some bulk >> loading, testing with incremental repairs and running out of heap once I >> found that now I have a quit large number of sstables which are really >> small: >> >> <1k 0 0,0% >> <10k 2780 76,8% >> <100k 3392 93,7% >> <1000k 3461 95,6% >> <10000k 3471 95,9% >> <100000k 3517 97,1% >> <1000000k 3596 99,3% >> all 3621 100,0% >> >> 76,8% of all sstables in this particular column familiy are smaller that >> 10kB, 93.7% are smaller then 100kB. >> >> Just for my understanding - does that impact performance? And is there >> any way to reduce the number of sstables? A full run of nodetool compact is >> running for a really long time (more than 1day). >> >> Thanks for any input, >> Roland >> > > > -- > i.A. Jan Kesten Systemadministration enercast GmbH Friedrich - Ebert - > Straße 104 D–34119 Kassel Tel.: +49 561 / 4739664-0 Fax: > (+49)561/4739664-9 mailto: j.kes...@enercast.de http://www.enercast.de AG > Kassel HRB 15471 Thomas Landgraf Geschäftsführer t.landg...@enercast.de > Tel.: (+49)561/4739664-0 FAX: -9 Mobil: (+49)172/6565087 enercast GmbH > Friedrich-Ebert-Str. 104 D-34119 Kassel HRB15471 http://www.enercast.de > Online-Prognosen für erneuerbare Energien Geschäftsführung: Thomas Landgraf > (CEO), Bernd Kratz (CTO), Philipp Rinder (CSO) Diese E-Mail und etwaige > Anhänge können vertrauliche und/oder rechtlich geschützte Informationen > enthalten. Falls Sie nicht der angegebene Empfänger sind oder falls diese > E-Mail irrtümlich an Sie adressiert wurde, benachrichtigen Sie uns bitte > sofort durch Antwort-E-Mail und löschen Sie diese E-Mail nebst etwaigen > Anlagen von Ihrem System. Ebenso dürfen Sie diese E-Mail oder ihre Anlagen > nicht kopieren oder an Dritte weitergeben. Vielen Dank. This e-mail and any > attachment may contain confidential and/or privileged information. If you > are not the named addressee or if this transmission has been addressed to > you in error, please notify us immediately by reply e-mail and then delete > this e-mail and any attachment from your system. Please understand that you > must not copy this e-mail or any attachment or disclose the contents to any > other person. Thank you for your cooperation. >
