it could be the "really big row during compaction" limitation, then, too.
On Tue, Mar 16, 2010 at 3:04 PM, B. Todd Burruss <bburr...@real.com> wrote: > the row/key cache is set to 10%, but memory usage stays good until an > anticompaction, hinted handoff, etc starts. (of course maybe i simply don't > pay attention to memory until something bad happens) > > doesn't sound like anyone else is having trouble, so i'll keep review my > settings for cache, keep testing and try to get something more concrete. > > Jonathan Ellis wrote: >> >> it's almost certainly GC storming due to memory pressure. (matching >> the thread dump / the threads using the CPU in top will confirm this.) >> reducing your cache sizes might be the best option since you already >> have a 44GB heap. >> >> On Tue, Mar 16, 2010 at 12:17 PM, B. Todd Burruss <bburr...@real.com> >> wrote: >> >>> >>> thx, i'll try that next time, already restarted node .. but i will say >>> the >>> exact thing happened on another node as well. >>> >>> Jonathan Ellis wrote: >>> >>>> >>>> You can still get a thread list w/ jstack, though. >>>> >>>> On Tue, Mar 16, 2010 at 11:46 AM, Gary Dusbabek <gdusba...@gmail.com> >>>> wrote: >>>> >>>> >>>>> >>>>> On Tue, Mar 16, 2010 at 11:39, B. Todd Burruss <bburr...@real.com> >>>>> wrote: >>>>> >>>>> >>>>>> >>>>>> any other ideas on how to troubleshoot? i have tried kill -3 >>>>>> <java_pid> >>>>>> in >>>>>> the past but don't know where cassandra writes the console out. i'll >>>>>> look >>>>>> at scripts. >>>>>> >>>>>> >>>>>> >>>>> >>>>> I have a sneaking suspicion that unless you're running with '-f' that >>>>> the thread dump goes into the ether. The only logical place to look >>>>> would be the log, which you've probably already done. >>>>> >>>>> Gary. >>>>> >>>>> >>>>> >
