One possibility might be CASSANDRA-11206 (Support large partitions on the 3.0 sstable format), which reduces heap usage for other operations (like repair, compactions) as well. You can verify that by setting column_index_cache_size_in_kb in c.yaml to a really high value like 10000000 - if you see the same behaviour in 3.7 with that setting, there’s not much you can do except upgrading to 3.7 as that change went into 3.6 and not into 3.0.x.
— Robert Stupp @snazy > On 20 Jun 2016, at 18:13, Bhuvan Rawal <bhu1ra...@gmail.com> wrote: > > Hi All, > > We are running Cassandra 3.0.3 on Production with Max Heap Size of 8GB. There > has been a consistent issue with nodetool repair for a while and we have > tried issuing it with multiple options --pr, --local as well, sometimes node > went down with Out of Memory error and at times nodes did stopped connecting > any connection, even jmx nodetool commands. > > On trying with same data on 3.7 Repair Ran successfully without encountering > any of the above mentioned issues. I then tried increasing heap to 16GB on > 3.0.3 and repair ran successfully. > > I then analyzed memory usage during nodetool repair for 3.0.3(16GB heap) vs > 3.7 (8GB Heap) and 3.0.3 occupied 11-14 GB at all times, whereas 3.7 spiked > between 1-4.5 GB while repair runs. As they ran on same dataset and > unrepaired data with full repair. > > We would like to know if it is a known bug that was fixed post 3.0.3 and > there could be a possible way by which we can run repair on 3.0.3 without > increasing heap size as for all other activities 8GB works for us. > > PFA the visualvm snapshots. > > <Screenshot from 2016-06-20 21:06:09.png> > 3.0.3 VisualVM Snapshot, consistent heap usage of greater than 12 GB. > > > <Screenshot from 2016-06-20 21:05:57.png> > 3.7 VisualVM Snapshot, 8GB Max Heap and max heap usage till about 5GB. > > Thanks & Regards, > Bhuvan Rawal > > > PS: In case if the snapshots are not visible, they can be viewed from the > following links: > 3.0.3: > https://s31.postimg.org/4e7ifsjaz/Screenshot_from_2016_06_20_21_06_09.png > <https://s31.postimg.org/4e7ifsjaz/Screenshot_from_2016_06_20_21_06_09.png> > 3.7: > https://s31.postimg.org/xak32s9m3/Screenshot_from_2016_06_20_21_05_57.png > <https://s31.postimg.org/xak32s9m3/Screenshot_from_2016_06_20_21_05_57.png>
