In our 2.1 cluster, I find that hints handoff is using a lot of memory on our proxy nodes, when delivering hints to a data node that was dead for 3+ hours (our hints window is 3 hours). It makes the young gen GC time as long as 2 secs.
I'm using 64G max heap size, and 4G young gen size. I'm considering increasing the young gen size, I'm wondering any improvements to hints handoff in newer version? some logs: 2016-09-20_05:04:30.67928 INFO 05:04:30 [Service Thread]: ParNew GC in 2200ms. CMS Old Gen: 49112623080 -> 50052787792; Par Eden Space: 2863398912 -> 0; mem usage: 2016-09-20T01:25:23.522+0000 Process summary process cpu=0.00% application cpu=140.36% (user=132.28% sys=8.08%) other: cpu=-140.36% heap allocation rate 1500mb/s [001343] user=90.42% sys= 0.01% alloc= 1439mb/s - HintedHandoff:3 [290234] user=18.85% sys= 5.14% alloc= 40mb/s - CompactionExecutor:176226 [064445] user=17.57% sys= 1.09% alloc= 10mb/s - RMI TCP Connection(148376)-127.0.0.1 [290439] user= 3.20% sys= 1.10% alloc= 4607kb/s - RMI TCP Connection(148380)-127.0.0.1 [000016] user= 0.75% sys=-0.03% alloc= 2321kb/s - ScheduledTasks:1 [000149] user= 0.32% sys=-0.04% alloc= 1516kb/s - GossipStage:1 [001370] user= 0.11% sys= 0.04% alloc= 420kb/s - GossipTasks:1 Thanks Dikang.
