Thanks for the pointers! I edited jvm.options in
$CASSANDRA_HOME/conf/jvm.options to increase -Xms and -Xmx to 1536M.
The result is the same.
And in $CASSANDRA_HOME/logs/system.log, grep GC system.log produces this
(when jvm.options had not been changed):
INFO [Service Thread] 2015-12-18 15:26:31,668 GCInspector.java:284 -
ConcurrentMarkSweep GC in 296ms. CMS Old Gen: 18133664 -> 15589256;
Code Cache: 5650880 -> 8122304; Compressed Class Space: 2530064 ->
3345624; Metaspace: 21314000 -> 28040984; Par Eden Space: 7019256 ->
164070848;
INFO [Service Thread] 2015-12-18 15:48:39,736 GCInspector.java:284 -
ConcurrentMarkSweep GC in 379ms. CMS Old Gen: 649257416 -> 84190176;
Code Cache: 20772224 -> 20726848; Par Eden Space: 2191408 -> 52356736;
Par Survivor Space: 2378448 -> 2346840
INFO [Service Thread] 2015-12-18 15:58:35,118 GCInspector.java:284 -
ConcurrentMarkSweep GC in 406ms. CMS Old Gen: 648847808 -> 86954856;
Code Cache: 21182080 -> 21188032; Par Eden Space: 1815696 -> 71525744;
Par Survivor Space: 2388648 -> 2364696
INFO [Service Thread] 2015-12-18 16:13:45,821 GCInspector.java:284 -
ConcurrentMarkSweep GC in 211ms. CMS Old Gen: 648343768 -> 73135720;
Par Eden Space: 3224880 -> 7957464; Par Survivor Space: 2379912 -> 2414520
INFO [Service Thread] 2015-12-18 16:32:46,419 GCInspector.java:284 -
ConcurrentMarkSweep GC in 387ms. CMS Old Gen: 648476072 -> 68888832;
Par Eden Space: 2006624 -> 64263360; Par Survivor Space: 2403792 -> 2387664
INFO [Service Thread] 2015-12-18 16:42:38,648 GCInspector.java:284 -
ConcurrentMarkSweep GC in 365ms. CMS Old Gen: 649126336 -> 137359384;
Code Cache: 22972224 -> 22979840; Metaspace: 41374464 -> 41375104; Par
Eden Space: 4286080 -> 154449480; Par Survivor Space: 1575440 -> 2310768
INFO [Service Thread] 2015-12-18 16:51:57,538 GCInspector.java:284 -
ConcurrentMarkSweep GC in 322ms. CMS Old Gen: 648338928 -> 79783856;
Par Eden Space: 2058968 -> 56931312; Par Survivor Space: 2342760 -> 2400336
INFO [Service Thread] 2015-12-18 17:02:49,543 GCInspector.java:284 -
ConcurrentMarkSweep GC in 212ms. CMS Old Gen: 648702008 -> 122954344;
Par Eden Space: 3269032 -> 61433328; Par Survivor Space: 2395824 -> 3448760
INFO [Service Thread] 2015-12-18 17:11:54,090 GCInspector.java:284 -
ConcurrentMarkSweep GC in 306ms. CMS Old Gen: 648748576 -> 70965096;
Par Eden Space: 2174840 -> 27074432; Par Survivor Space: 2365992 -> 2373984
INFO [Service Thread] 2015-12-18 17:22:28,949 GCInspector.java:284 -
ConcurrentMarkSweep GC in 350ms. CMS Old Gen: 648243024 -> 90897272;
Par Eden Space: 2150168 -> 43487192; Par Survivor Space: 2401872 -> 2410728
After modifying jvm.options to increase -Xms & -Xmx (to 1536M):
INFO [Service Thread] 2015-12-21 11:39:24,918 GCInspector.java:284 -
ConcurrentMarkSweep GC in 342ms. CMS Old Gen: 18579136 -> 16305144;
Code Cache: 8600128 -> 10898752; Compressed Class Space: 3431288 ->
3761496; Metaspace: 29551832 -> 33307352; Par Eden Space: 4822000 ->
94853272;
INFO [Service Thread] 2015-12-21 11:39:30,710 GCInspector.java:284 -
ParNew GC in 206ms. CMS Old Gen: 22932208 -> 41454520; Par Eden Space:
167772160 -> 0; Par Survivor Space: 13144872 -> 20971520
INFO [Service Thread] 2015-12-21 13:08:14,922 GCInspector.java:284 -
ConcurrentMarkSweep GC in 468ms. CMS Old Gen: 21418016 -> 16146528;
Code Cache: 11693888 -> 11744704; Compressed Class Space: 4331224 ->
4344192; Metaspace: 37191144 -> 37249960; Par Eden Space: 146089224 ->
148476848;
INFO [Service Thread] 2015-12-21 13:08:53,068 GCInspector.java:284 -
ParNew GC in 216ms. CMS Old Gen: 16146528 -> 26858568; Par Eden Space:
167772160 -> 0;
Earlier the node had OpenJDK 8. For today's tests I installed and used
Oracle Java 8.
Do the above messages provide any clue? Or any debug logging I can
enable to progress further?
Thanks,
Dinesh.
On 12/18/2015 9:56 PM, Tyler Hobbs wrote:
On Fri, Dec 18, 2015 at 9:17 AM, DuyHai Doan <doanduy...@gmail.com
<mailto:doanduy...@gmail.com>> wrote:
Cassandra will perform a full table scan and fetch all the data in
memory to apply the aggregate function.
Just to clarify for others on the list: when executing aggregation
functions, Cassandra /will/ use paging internally, so at most one page
worth of data will be held in memory at a time. However, if your
aggregation function retains a large amount of data, this may
contribute to heap pressure.
--
Tyler Hobbs
DataStax <http://datastax.com/>
