I've started seeing this issue as well. Running 0.6.2.
One interesting thing I happened upon, I explicitly called the GC via
jconsole and the heap dropped completely fixing the issue. When you
explicitly call System.gc() it does a full sweep. I'm wondering if this
issue is to do with the GC fla
We've also seen something like this. Will soon investigate and try
again with 0.6.2
On Wed, Jun 2, 2010 at 20:27, Paul Brown wrote:
>
> FWIW, I'm seeing similar issues on a cluster. Three nodes, Cassandra 0.6.1,
> SUN JDK 1.6.0_b20. I will try to get some heap dumps to see what's building
> u
FWIW, I'm seeing similar issues on a cluster. Three nodes, Cassandra 0.6.1,
SUN JDK 1.6.0_b20. I will try to get some heap dumps to see what's building up.
I've seen this sort of issue in systems that make heavy use of
java.util.concurrent queues/executors, e.g.:
http://bugs.sun.com/bugdatab
Hmm...heap usage slowly growing over time...the picture doesnt look like the
problems we were running into, and I dont think they should be related to
mmap-ing your data (since that is not counted in heap mem usage).
Kyusik Chung
On May 29, 2010, at 10:16 AM, James Golick wrote:
> Well, it's b
I'm also seeing the same thing. I need to do more log-staring to understand
what's up... incompatible with holiday weekends ;-). tpstats are all 0.
Ian
On Sat, May 29, 2010 at 5:54 PM, Jonathan Ellis wrote:
> Did you manually invoke a GC? It doesn't look like you're using
> enough of the hea
Did you manually invoke a GC? It doesn't look like you're using
enough of the heap for it to care much on its own.
On Sat, May 29, 2010 at 2:58 PM, James Golick wrote:
> Well, it's been a few days on 0.6.2 and the new jvm and the behaviour looks
> to be about the same:
> http://skitch.com/jamesg
Well, it's been a few days on 0.6.2 and the new jvm and the behaviour looks
to be about the same:
http://skitch.com/jamesgolick/df46f/munin-fetlife.com-cassandra0.fetlife.com-cassandra-memory
There's only one cache turned on, and it's a row cache, but the sizes of the
rows are identical and it's
> I tried setting the IO mode to standard, but it seemed to be a little slower
> and couldn't get the machine to come back online with adequate read
> performance, so I set it back. I'll have to write a solid cache warming
> script if I'm going to try that again.
What cache are you talking about?
Hi Philip,
I think they chose to go with OpenJDK bc Sun's is not open source.
Here's what we did on ubuntu 10.04 (if youre using a different debian distro,
you can prob do something very similar):
# this install gives us the convenient add-apt-repository command
sudo apt-get install python-soft
d Cassandra 0.6.1 fixed a lot of memory problems I ran into.
>
> *From:* James Golick [mailto:jamesgol...@gmail.com]
> *Sent:* Thursday, May 27, 2010 9:05 AM
> *To:* user@cassandra.apache.org
> *Subject:* Re: Continuously increasing RAM usage
>
> When I say unresponsive, I mean
apache.org
> Subject: Re: Continuously increasing RAM usage
>
> When I say unresponsive, I mean that latency becomes very high. Swap is
> turned off, but before I turned it on, it used to swap heavily at this point.
>
> Cassandra version is 0.6.0 Beta1
>
> [cassandr
cause memory problems. Sun jvm, u19 (or higher) and
Cassandra 0.6.1 fixed a lot of memory problems I ran into.
From: James Golick [mailto:jamesgol...@gmail.com]
Sent: Thursday, May 27, 2010 9:05 AM
To: user@cassandra.apache.org
Subject: Re: Continuously increasing RAM usage
When I say unresponsive, I
When I say unresponsive, I mean that latency becomes very high. Swap is
turned off, but before I turned it on, it used to swap heavily at this
point.
Cassandra version is 0.6.0 Beta1
[cassandra1 ~]# java -version
java version "1.6.0"
OpenJDK Runtime Environment (build 1.6.0-b09)
OpenJDK 64-Bit S
On 5/26/10 11:32 PM, James Golick wrote:
We're seeing RAM usage continually climb until eventually, cassandra
becomes unresponsive.
Given the handful of bugs related to memory bloat in specific versions
of Cassandra combined with specific versions of JVMs, that information
may be relevant to yo
What else are you seeing that correlates with "unresponsive?"
lots of pending tasks in stage queues?
high cpu in a single thread?
swapping?
(I sent a github pull request, but I've updated
http://github.com/jbellis/cassandra-munin-plugins with a lot more
metrics to monitor.)
On Thu, May 27, 201
A lot of folks have reported this issue, and there are a few JIRAs related
to it. Post the output of nodetool tpstats. Also, are there lots of GCs in
the system.log? If so, are they something besides ParNew?
Ian
On Thu, May 27, 2010 at 2:32 AM, James Golick wrote:
> We're seeing RAM usage co
We're seeing RAM usage continually climb until eventually, cassandra becomes
unresponsive.
The JVM isn't OOM'ing. It has only committed 14/24GB of memory. So, I am
assuming that the memory usage is related to mmap'd IO. Fair assumption?
I tried setting the IO mode to standard, but it seemed to be
17 matches
Mail list logo
