Re: Regular dropped READ messages

2017-06-06 Thread Vincent Rischmann
Thanks Alexander for the help, lots of good info in there. I'll try to switch back to CMS and see how it fares. On Tue, Jun 6, 2017, at 05:06 PM, Alexander Dejanovski wrote: > Hi Vincent, > > it is very clear, thanks for all the info. > > I would not stick with G1 in your case, as it requires

Re: Regular dropped READ messages

2017-06-06 Thread Alexander Dejanovski
Hi Vincent, it is very clear, thanks for all the info. I would not stick with G1 in your case, as it requires much more heap to perform correctly (>24GB). CMS/ParNew should be much more efficient here and I would go with some settings I usually apply on big workloads : 16GB heap / 6GB new gen / M

Re: Regular dropped READ messages

2017-06-06 Thread Vincent Rischmann
Hi Alexander. Yeah, the minor GCs I see are usually around 300ms but sometimes jumping to 1s or even more. Hardware specs are: - 8 core CPUs - 32 GB of RAM - 4 SSDs in hardware Raid 0, around 3TB of space per node GC settings:-Xmx12G -Xms12G -XX:+UseG1GC - XX:G1RSetUpdatingPauseTimePer

Re: Regular dropped READ messages

2017-06-06 Thread Alexander Dejanovski
Hi Vincent, dropped messages are indeed common in case of long GC pauses. Having 4s to 6s pauses is not normal and is the sign of an unhealthy cluster. Minor GCs are usually faster but you can have long ones too. If you can share your hardware specs along with your current GC settings (CMS or G1,

Regular dropped READ messages

2017-06-06 Thread Vincent Rischmann
Hi, we have a cluster of 11 nodes running Cassandra 2.2.9 where we regularly get READ messages dropped: > READ messages were dropped in last 5000 ms: 974 for internal timeout > and 0 for cross node timeout Looking at the logs, some are logged at the same time as Old Gen GCs. These GCs all take aro