Roughly how much data do you have per node? Sent from my iPhone
On Feb 20, 2013, at 10:49 AM, "Hiller, Dean" <dean.hil...@nrel.gov> wrote: > I took this jmap dump of cassandra(in production). Before I restarted the > whole production cluster, I had some nodes running compaction and it looked > like all memory had been consumed(kind of like cassandra is not clearing out > the caches or memtables fast enough). I am trying to still debug compaction > causes slowness on the cluster since all cassandra.yaml files are pretty much > the defaults with size tiered compaction. > > The weird thing is I dump and get a 5.4G heap.bin file and load that into > ecipse who tells me total is 142.8MB….what???? So low????, top was showing > 1.9G at the time(and I took this top snapshot later(2 hours after)… (how is > eclipse profile telling me the jmap showed 142.8MB in use instead of 1.9G in > use?) > > Tasks: 398 total, 1 running, 397 sleeping, 0 stopped, 0 zombie > Cpu(s): 2.8%us, 0.5%sy, 0.0%ni, 96.5%id, 0.1%wa, 0.0%hi, 0.1%si, 0.0%st > Mem: 32854680k total, 31910708k used, 943972k free, 89776k buffers > Swap: 33554424k total, 18288k used, 33536136k free, 23428596k cached > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 20909 cassandr 20 0 64.1g 9.2g 2.1g S 75.7 29.4 182:37.92 java > 22455 cassandr 20 0 15288 1340 824 R 3.9 0.0 0:00.02 top > > It almost seems like cassandra is not being good about memory management here > as we slowly get into a situation where compaction is run which takes out our > memory(configured for 8G). I can easily go higher than 8G on these systems > as I have 32gig each node, but there was docs that said 8G is better for GC. > Has anyone else taken a jmap dump of cassandra? > > Thanks, > Dean >
