Hello
Some times i have very long GC pauses:
Total time for which application threads were stopped: 0.0303150 seconds
2011-03-17T13:19:56.476+0300: 33295.671: [GC 33295.671: [ParNew:
678855K->20708K(737280K), 0.0271230 secs] 1457643K->806795K(4112384K),
0.027305
0 secs] [Times: user=0.33 sys=0.00, real=0.03 secs]
Total time for which application threads were stopped: 0.0291820 seconds
2011-03-17T13:20:32.962+0300: 33332.157: [GC 33332.157: [ParNew:
676068K->23527K(737280K), 0.0302180 secs] 1462155K->817599K(4112384K),
0.030402
0 secs] [Times: user=0.31 sys=0.00, real=0.03 secs]
Total time for which application threads were stopped: 0.1270270 seconds
2011-03-17T13:21:11.908+0300: 33371.103: [GC 33371.103: [ParNew:
678887K->21564K(737280K), 0.0268160 secs] 1472959K->823191K(4112384K),
0.027011
0 secs] [Times: user=0.28 sys=0.00, real=0.03 secs]
Total time for which application threads were stopped: 0.0293330 seconds
2011-03-17T13:21:50.482+0300: 33409.677: [GC 33409.677: [ParNew:
676924K->21115K(737280K), 0.0281720 secs] 1478551K->829900K(4112384K),
0.028363
0 secs] [Times: user=0.27 sys=0.00, real=0.03 secs]
Total time for which application threads were stopped: 0.0339610 seconds
2011-03-17T13:22:32.849+0300: 33452.044: [GC 33452.044: [ParNew:
676475K->25948K(737280K), 0.0317600 secs] 1485260K->842061K(4112384K),
0.031952
0 secs] [Times: user=0.22 sys=0.00, real=0.03 secs]
Total time for which application threads were stopped: 0.0344430 seconds
2011-03-17T13:23:14.924+0300: 33494.119: [GC 33494.119: [ParNew:
681308K->25087K(737280K), 0.0282600 secs] 1497421K->848300K(4112384K),
0.028436
0 secs] [Times: user=0.32 sys=0.00, real=0.03 secs]
Total time for which application threads were stopped: 0.0309160 seconds
2011-03-17T13:23:57.192+0300: 33536.387: [GC 33536.387: [ParNew:
680447K->24805K(737280K), 0.0299910 secs] 1503660K->855829K(4112384K),
0.030167
0 secs] [Times: user=0.29 sys=0.01, real=0.03 secs]
Total time for which application threads were stopped: 0.0324200 seconds
2011-03-17T13:24:01.553+0300: 33540.748: [GC 33540.749: [ParNew:
680165K->31886K(737280K), 0.0495620 secs] 1511189K->936503K(4112384K),
0.049742
0 secs] [Times: user=0.57 sys=0.00, real=0.05 secs]
Total time for which application threads were stopped: 0.0507030 seconds
2011-03-17T13:37:56.009+0300: 34375.204: [GC 34375.204: [ParNew:
687246K->28727K(737280K), 0.0244720 secs] 1591863K->942459K(4112384K),
0.024690
0 secs] [Times: user=0.18 sys=0.00, real=0.02 secs]
Total time for which application threads were stopped: 806.7442720 seconds
Total time for which application threads were stopped: 0.0006590 seconds
Total time for which application threads were stopped: 0.0004360 seconds
Total time for which application threads were stopped: 0.0004630 seconds
Total time for which application threads were stopped: 0.0008120 seconds
2011-03-17T13:37:59.018+0300: 34378.213: [GC 34378.213: [ParNew:
676678K->21640K(737280K), 0.0137740 secs] 1590410K->949991K(4112384K),
0.013961
0 secs] [Times: user=0.13 sys=0.02, real=0.01 secs]
Total time for which application threads were stopped: 0.0145920 seconds
Total time for which application threads were stopped: 0.1036080 seconds
Total time for which application threads were stopped: 0.0585600 seconds
Total time for which application threads were stopped: 0.0600550 seconds
Total time for which application threads were stopped: 0.0008560 seconds
Total time for which application threads were stopped: 0.0006770 seconds
Total time for which application threads were stopped: 0.0005910 seconds
Total time for which application threads were stopped: 0.0351330 seconds
Total time for which application threads were stopped: 0.0329020 seconds
Total time for which application threads were stopped: 0.0728490 seconds
Total time for which application threads were stopped: 0.0480990 seconds
Total time for which application threads were stopped: 0.0804250 seconds
2011-03-17T13:38:04.394+0300: 34383.589: [GC 34383.589: [ParNew:
677000K->8375K(737280K), 0.0218310 secs] 1605351K->944271K(4112384K),
0.0220300
secs]
I have follow nodetoll cfstats on hung node:
Keyspace: fishdom_tuenti
Read Count: 4970999
Read Latency: 1.0267005945887335 ms.
Write Count: 1441619
Write Latency: 0.013146585887117193 ms.
Pending Tasks: 0
Column Family: decor
SSTable count: 3
Space used (live): 1296203532
Space used (total): 1302520037
Memtable Columns Count: 1066
Memtable Data Size: 121742
Memtable Switch Count: 11
Read Count: 108125
Read Latency: 2.809 ms.
Write Count: 11261
Write Latency: 0.006 ms.
Pending Tasks: 0
Key cache capacity: 300000
Key cache size: 46470
Key cache hit rate: 0.40384615384615385
Row cache: disabled
Compacted row minimum size: 36
Compacted row maximum size: 73457
Compacted row mean size: 958
Column Family: adopt
SSTable count: 1
Space used (live): 3285283
Space used (total): 6585649
Memtable Columns Count: 14
Memtable Data Size: 322
Memtable Switch Count: 19
Read Count: 208
Read Latency: 0.009 ms.
Write Count: 248
Write Latency: 0.005 ms.
Pending Tasks: 0
Key cache capacity: 200000
Key cache size: 171
Key cache hit rate: NaN
Row cache: disabled
Compacted row minimum size: 87
Compacted row maximum size: 103
Compacted row mean size: 103
Column Family: user_store
SSTable count: 0
Space used (live): 0
Space used (total): 0
Memtable Columns Count: 0
Memtable Data Size: 0
Memtable Switch Count: 0
Read Count: 0
Read Latency: NaN ms.
Write Count: 0
Write Latency: NaN ms.
Pending Tasks: 0
Key cache capacity: 200000
Key cache size: 0
Key cache hit rate: NaN
Row cache: disabled
Compacted row minimum size: 0
Compacted row maximum size: 0
Compacted row mean size: 0
Column Family: gifts
SSTable count: 1
Space used (live): 8767135
Space used (total): 17581781
Memtable Columns Count: 9
Memtable Data Size: 5993
Memtable Switch Count: 11
Read Count: 230
Read Latency: NaN ms.
Write Count: 306
Write Latency: 0.004 ms.
Pending Tasks: 0
Key cache capacity: 200000
Key cache size: 186
Key cache hit rate: NaN
Row cache: disabled
Compacted row minimum size: 125
Compacted row maximum size: 14237
Compacted row mean size: 498
Column Family: user_awards
SSTable count: 4
Space used (live): 285142885
Space used (total): 285142885
Memtable Columns Count: 399
Memtable Data Size: 21265
Memtable Switch Count: 11
Read Count: 602884
Read Latency: 0.075 ms.
Write Count: 3741
Write Latency: 0.003 ms.
Pending Tasks: 0
Key cache capacity: 200000
Key cache size: 9692
Key cache hit rate: 0.9929742388758782
Row cache: disabled
Compacted row minimum size: 104
Compacted row maximum size: 770
Compacted row mean size: 262
Column Family: aquarium
SSTable count: 4
Space used (live): 945412589
Space used (total): 970196634
Memtable Columns Count: 10973
Memtable Data Size: 1805059
Memtable Switch Count: 11
Read Count: 737198
Read Latency: 0.419 ms.
Write Count: 259785
Write Latency: 0.011 ms.
Pending Tasks: 0
Key cache capacity: 200000
Key cache size: 68960
Key cache hit rate: 0.9497041420118343
Row cache: disabled
Compacted row minimum size: 87
Compacted row maximum size: 1916
Compacted row mean size: 539
Column Family: user_messages
SSTable count: 1
Space used (live): 4648148
Space used (total): 9311688
Memtable Columns Count: 23
Memtable Data Size: 2525
Memtable Switch Count: 19
Read Count: 8782
Read Latency: 0.012 ms.
Write Count: 246
Write Latency: 0.012 ms.
Pending Tasks: 0
Key cache capacity: 200000
Key cache size: 2707
Key cache hit rate: NaN
Row cache: disabled
Compacted row minimum size: 180
Compacted row maximum size: 9887
Compacted row mean size: 380
Column Family: user_stat
SSTable count: 4
Space used (live): 872814042
Space used (total): 872814042
Memtable Columns Count: 36852
Memtable Data Size: 902102
Memtable Switch Count: 11
Read Count: 1663028
Read Latency: 0.446 ms.
Write Count: 406925
Write Latency: 0.010 ms.
Pending Tasks: 0
Key cache capacity: 400000
Key cache size: 78611
Key cache hit rate: 0.9583333333333334
Row cache: disabled
Compacted row minimum size: 51
Compacted row maximum size: 770
Compacted row mean size: 516
Column Family: user_stat_friend
SSTable count: 4
Space used (live): 614446488
Space used (total): 614446488
Memtable Columns Count: 9288
Memtable Data Size: 250169
Memtable Switch Count: 11
Read Count: 237287
Read Latency: 1.147 ms.
Write Count: 116691
Write Latency: 0.006 ms.
Pending Tasks: 0
Key cache capacity: 200000
Key cache size: 39410
Key cache hit rate: 0.8387096774193549
Row cache: disabled
Compacted row minimum size: 61
Compacted row maximum size: 258
Compacted row mean size: 199
Column Family: aquarium_friend
SSTable count: 4
Space used (live): 7135867112
Space used (total): 7145863363
Memtable Columns Count: 6061
Memtable Data Size: 618699
Memtable Switch Count: 11
Read Count: 333994
Read Latency: 1.358 ms.
Write Count: 168818
Write Latency: 0.010 ms.
Pending Tasks: 0
Key cache capacity: 200000
Key cache size: 47973
Key cache hit rate: 0.8808664259927798
Row cache: disabled
Compacted row minimum size: 104
Compacted row maximum size: 924
Compacted row mean size: 304
Column Family: aquarium_messages
SSTable count: 1
Space used (live): 5949016
Space used (total): 11912616
Memtable Columns Count: 11
Memtable Data Size: 1355
Memtable Switch Count: 19
Read Count: 26441
Read Latency: 0.010 ms.
Write Count: 279
Write Latency: 0.003 ms.
Pending Tasks: 0
Key cache capacity: 200000
Key cache size: 4095
Key cache hit rate: NaN
Row cache: disabled
Compacted row minimum size: 180
Compacted row maximum size: 14237
Compacted row mean size: 382
Column Family: user_params
SSTable count: 0
Space used (live): 0
Space used (total): 0
Memtable Columns Count: 0
Memtable Data Size: 0
Memtable Switch Count: 0
Read Count: 0
Read Latency: NaN ms.
Write Count: 0
Write Latency: NaN ms.
Pending Tasks: 0
Key cache capacity: 200000
Key cache size: 0
Key cache hit rate: NaN
Row cache: disabled
Compacted row minimum size: 0
Compacted row maximum size: 0
Compacted row mean size: 0
Column Family: gems
SSTable count: 4
Space used (live): 582170315
Space used (total): 582170315
Memtable Columns Count: 8713
Memtable Data Size: 285047
Memtable Switch Count: 11
Read Count: 97307
Read Latency: 0.733 ms.
Write Count: 35400
Write Latency: 0.009 ms.
Pending Tasks: 0
Key cache capacity: 200000
Key cache size: 21120
Key cache hit rate: 0.8691588785046729
Row cache: disabled
Compacted row minimum size: 36
Compacted row maximum size: 2759
Compacted row mean size: 691
Column Family: adopt_accept
SSTable count: 1
Space used (live): 3531249
Space used (total): 7077144
Memtable Columns Count: 15
Memtable Data Size: 433
Memtable Switch Count: 19
Read Count: 271
Read Latency: 0.011 ms.
Write Count: 249
Write Latency: 0.004 ms.
Pending Tasks: 0
Key cache capacity: 200000
Key cache size: 325
Key cache hit rate: NaN
Row cache: disabled
Compacted row minimum size: 87
Compacted row maximum size: 179
Compacted row mean size: 109
Column Family: gems_friend
SSTable count: 5
Space used (live): 1620588533
Space used (total): 1620588533
Memtable Columns Count: 21441
Memtable Data Size: 577929
Memtable Switch Count: 11
Read Count: 265787
Read Latency: 0.974 ms.
Write Count: 135310
Write Latency: 0.011 ms.
Pending Tasks: 0
Key cache capacity: 200000
Key cache size: 49702
Key cache hit rate: 0.8641304347826086
Row cache: disabled
Compacted row minimum size: 36
Compacted row maximum size: 1916
Compacted row mean size: 536
Column Family: fish
SSTable count: 5
Space used (live): 3424896182
Space used (total): 3543066763
Memtable Columns Count: 88706
Memtable Data Size: 10660109
Memtable Switch Count: 11
Read Count: 889507
Read Latency: 0.718 ms.
Write Count: 302385
Write Latency: 0.024 ms.
Pending Tasks: 0
Key cache capacity: 400000
Key cache size: 90802
Key cache hit rate: 0.9509594882729211
Row cache: disabled
Compacted row minimum size: 51
Compacted row maximum size: 51012
Compacted row mean size: 1890
----------------
