Cassandra crashed during major compaction

After a bulk load of writes to existing partition keys (with a higher timestamp), I wanted to free disk space, suspecting that rows will be in the highest levels and it would take a time until they were compacted. I've started a major compaction, and the disk usage went from ~30% to ~40% (as expect

Fwd: Re: Cassandra crashed with no log

: Sun, 22 Jul 2018 10:43:38 +0430 Subject : Re: Cassandra crashed with no log Forwarded message Anything in non-Cassandra logs? Dmesg? --  Jeff Jirsa On Jul 21, 2018, at 11:07 PM, onmstester onmstester wrote: Cassandra in one of my nodes, crashed without any error/warning

Re: Cassandra crashed with no log

Anything in non-Cassandra logs? Dmesg? -- Jeff Jirsa > On Jul 21, 2018, at 11:07 PM, onmstester onmstester > wrote: > > Cassandra in one of my nodes, crashed without any error/warning in > system/gc/debug log. All jmx metrics is being monitored, last fetched values > for heap usage is 50%

Cassandra crashed with no log

Cassandra in one of my nodes, crashed without any error/warning in system/gc/debug log. All jmx metrics is being monitored, last fetched values for heap usage is 50% and for cpu usage is 20%. How can i find the cause of crash? Sent using Zoho Mail

Re: Cassandra crashed with OOM, and the system.log and debug.log doesn't match.

Thanks for your explanation! > It's taking a full minute to sync your memtable to disk. This is either indication that your disk is broken, or your JVM is pausing for GC. The disk is ok, the long time JVM pausing happens many times, I didn't disable the paging file in windows, may be that's the re

Re: Cassandra crashed with OOM, and the system.log and debug.log doesn't match.

On 2017-07-10 02:07 (-0700), 张强 wrote: > Hi experts, I've a single cassandra 3.11.0 node working with kairosdb (a > time series database), after running 4 days with stable workload, the > database client start to get "request errors", but there are not a lot of > error or warning messages i

Re: Cassandra crashed with OOM, and the system.log and debug.log doesn't match.

Thanks for your reply! There are 3 column families, they are created by kairosdb, one column family takes almost all the workload. I didn't tune the heap size, so by default it'll be 3GB. I have monitored the cpu and memory usage, the cpu usage is about 30% in average, and the available memory is a

Re: Cassandra crashed with OOM, and the system.log and debug.log doesn't match.

Hi, *How many column families are there? What is the heap size?* You can turn off logs for statusLogger.java and gc to optimize heap usage. Can you also monitor cpu usage and memory usage? IMO, in your case memory is the bottle-neck. Thanks!! On Mon, Jul 10, 2017 at 5:07 PM, 张强 wrote: > Hi

Re: cassandra crashed while repairing, leave node size X3

got it, thanks! On Tue, Sep 20, 2011 at 12:27 AM, Peter Schuller < peter.schul...@infidyne.com> wrote: > > In my tests I have seen repair sometimes take a lot of space (2-3 times), > > cleanup did not clean it, the only way I could clean that was using major > > compaction. > > https://issues.apa

Re: cassandra crashed while repairing, leave node size X3

> In my tests I have seen repair sometimes take a lot of space (2-3 times), > cleanup did not clean it, the only way I could clean that was using major > compaction. https://issues.apache.org/jira/browse/CASSANDRA-2816 (follow links to other jiras) https://issues.apache.org/jira/browse/CASSANDRA-2

Re: cassandra crashed while repairing, leave node size X3

I am using 0.7.4 too. and would waiting for 0.8.6 stable to release because of CASSANDRA-3166. did you already using 0.8.6 in production? 2011/9/19 Jonas Borgström > On 09/19/2011 04:26 AM, Anand Somani wrote: > > In my tests I have seen repair sometimes take a lot of space (2-3 > > times), cle

Re: cassandra crashed while repairing, leave node size X3

On 09/19/2011 04:26 AM, Anand Somani wrote: > In my tests I have seen repair sometimes take a lot of space (2-3 > times), cleanup did not clean it, the only way I could clean that was > using major compaction. Do you remember with what version you saw these problems? I've had the same problems wi

Re: cassandra crashed while repairing, leave node size X3

so does major compaction actually "clean it" or "merge it", I am afraid it give me a single large file On Mon, Sep 19, 2011 at 10:26 AM, Anand Somani wrote: > In my tests I have seen repair sometimes take a lot of space (2-3 times), > cleanup did not clean it, the only way I could clean that

Re: cassandra crashed while repairing, leave node size X3

In my tests I have seen repair sometimes take a lot of space (2-3 times), cleanup did not clean it, the only way I could clean that was using major compaction. On Sun, Sep 18, 2011 at 6:51 PM, Yan Chunlu wrote: > while doing repair on node3, the "Load" keep increasing, suddenly cassandra > has e

cassandra crashed while repairing, leave node size X3

while doing repair on node3, the "Load" keep increasing, suddenly cassandra has encountered OOM, and the "Load" stopped at 140GB, after cassandra came back, I tried node cleanup but it seems not working does node repair generate many temp sstables? how to get rid of them? thanks! Address

Re: Cassandra crashed - possible JMX threads leak

Depending on finalize() is really not want you want todo, so I think the API change would be preferable. Bye, Norman 2010/10/26 Bill Au : > I would be happy to submit a patch but is it a bit more trickier than simply > calling JMXConenctor.close().  NodeProbe's use of the JMXConnector is not > e

Re: Cassandra crashed - possible JMX threads leak

I would be happy to submit a patch but is it a bit more trickier than simply calling JMXConenctor.close(). NodeProbe's use of the JMXConnector is not exposed in its API The JMX connection is created in NodeProbe's constructor. Without changing the API, the only place to call close() would be in

Re: Cassandra crashed - possible JMX threads leak

Is the fix as simple as calling close() then? Can you submit a patch for that? On Fri, Oct 22, 2010 at 2:49 PM, Bill Au wrote: > Not with the nodeprobe or nodetool command because the JVM these two > commands spawn has a very short life span. > > I am using a webapp to monitor my cassandra clust

Re: Cassandra crashed - possible JMX threads leak

Not with the nodeprobe or nodetool command because the JVM these two commands spawn has a very short life span. I am using a webapp to monitor my cassandra cluster. It pretty much uses the same code as NodeCmd class. For each incoming request, it creates an NodeProbe object and use it to get get

Re: Cassandra crashed - possible JMX threads leak

Ah, missed that. Thanks Aaron. On Wed, Oct 20, 2010 at 11:49 PM, Aaron Morton wrote: > Sounds like the problem discussed here > http://wiki.apache.org/cassandra/Operations?highlight=(allocate)|(memory) > > >

Re: Cassandra crashed - possible JMX threads leak

Sounds like the problem discussed here http://wiki.apache.org/cassandra/Operations?highlight=(allocate)|(memory)If you have the JNA jar it should work http://www.mail-archive.com/user@cassandra.apache.org/msg06284.htmlhttp://www.riptano.com/blog/whats-new-cassandra-065AaronOn 21 Oct, 2010,at 06:29

Re: Cassandra crashed - possible JMX threads leak

I have a cluster of 3 0.7 beta 2 nodes (built today from the latest trunk) running on Large, EBS-backed, x64 EC2 instances; RF=3. I attempted to write somewhere near 500,000 records every 15 minutes from a total of 5 different computers (using Pelops and multi-threading). Though my network blew

Re: Cassandra crashed - possible JMX threads leak

can you reproduce this by, say, running nodeprobe ring in a bash while loop? On Wed, Oct 20, 2010 at 3:09 PM, Bill Au wrote: > One of my Cassandra server crashed with the following: > > ERROR [ACCEPT-xxx.xxx.xxx/nnn.nnn.nnn.nnn] 2010-10-19 00:25:10,419 > CassandraDaemon.java (line 82) Uncaught ex

Cassandra crashed - possible JMX threads leak

One of my Cassandra server crashed with the following: ERROR [ACCEPT-xxx.xxx.xxx/nnn.nnn.nnn.nnn] 2010-10-19 00:25:10,419 CassandraDaemon.java (line 82) Uncaught exception in thread Thread[ACCEPT-xxx.xxx.xxx/nnn.nnn.nnn.nnn,5,main] java.lang.OutOfMemoryError: unable to create new native thread

Re: Ran into an issue where Cassandra Crashed when running out of heap space

On Tue, Jul 20, 2010 at 1:28 PM, Peter Schuller wrote: >> Attaching Jconsole shows that there is a growth of memory and weird >> spikes. Unfortunately I did not take a screen shot of the growth of >> the spike over time. I'll do that when it occurs again. > > Note that expected behavior for CMS is

Re: Ran into an issue where Cassandra Crashed when running out of heap space

> Attaching Jconsole shows that there is a growth of memory and weird > spikes. Unfortunately I did not take a screen shot of the growth of > the spike over time. I'll do that when it occurs again. Note that expected behavior for CMS is to have lots of small ups and downs as a result of young gene

Re: Ran into an issue where Cassandra Crashed when running out of heap space

The storage structure is rather simple. For every 1 key there is 1 column and a timestamp for that column. We don't enable pulling a huge amount of data and all other nodes are up servicing the same request. I suspect there may be another problem with Memory management inside Cassandra. Attac

Re: Ran into an issue where Cassandra Crashed when running out of heap space

On Tue, Jul 20, 2010 at 9:09 PM, Peter Schuller wrote: >> CassandraDaemon.java (line 83) Uncaught exception in thread >> Thread[pool-1-thread-37895,5,main] >> java.lang.OutOfMemoryError: Java heap space >>     at >> org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.jav

Re: Ran into an issue where Cassandra Crashed when running out of heap space

> CassandraDaemon.java (line 83) Uncaught exception in thread > Thread[pool-1-thread-37895,5,main] > java.lang.OutOfMemoryError: Java heap space >     at > org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:296) >     at > org.apache.thrift.protocol.TBinaryProt

Ran into an issue where Cassandra Crashed when running out of heap space

 INFO [HINTED-HANDOFF-POOL:1] 2010-07-20 15:10:43,721 HintedHandOffManager.java (line 210) Finished hinted handoff of 0 rows to endpoint /10.129.28.23 ERROR [pool-1-thread-37895] 2010-07-20 15:10:51,622 CassandraDaemon.java (line 83) Uncaught exception in thread Thread[pool-1-thread-37895,5,main] j

Re: cassandra crashed

These look more like symptoms of a crash, than causes of it. I.e., these are the kind of messages you would see if you kill -9 a node that is in the middle of replying to another. On Fri, Jun 11, 2010 at 6:47 AM, hive13 Wong wrote: > One of our cassandra nodes suddenly crashed, then the other 2.

cassandra crashed

One of our cassandra nodes suddenly crashed, then the other 2... Exceptions found in the system.log are attached below. Any ideas? Does it mean that we've got some bad data running around in the cluster? Many thanks The exeption on the node that crashed first was like ERROR [RESPONSE-STAGE:669] 20