Re: Mutation Stage does not finish

2014-09-11 Thread Eduardo Cusa
yes we have a huge amount insert that can be repeated, now we are working in a new data model On Thu, Sep 11, 2014 at 2:54 PM, Robert Coli wrote: > On Thu, Sep 11, 2014 at 10:34 AM, Eduardo Cusa < > eduardo.c...@usmediaconsulting.com> wrote: > >> I deleted commit logs, restarted cassandra and fi

Re: Mutation Stage does not finish

2014-09-11 Thread Robert Coli
On Thu, Sep 11, 2014 at 10:34 AM, Eduardo Cusa < eduardo.c...@usmediaconsulting.com> wrote: > I deleted commit logs, restarted cassandra and finally the node is up. > Do you have some crazy workload where you do a huge amount of delete or something? Replaying a commitlog should not take longer th

Re: Mutation Stage does not finish

2014-09-11 Thread Eduardo Cusa
Robert/Elliot. I deleted commit logs, restarted cassandra and finally the node is up. Thanks for helps! Regards. Eduardo On Thu, Sep 11, 2014 at 12:08 PM, Eduardo Cusa < eduardo.c...@usmediaconsulting.com> wrote: > Hello, > > The jstack output can be seen in : http://pastebin.com/LXnN

Re: Mutation Stage does not finish

2014-09-11 Thread Eduardo Cusa
Hello, The jstack output can be seen in : http://pastebin.com/LXnNyY3U. I run the tpstats today and always get the same output: Pool NameActive Pending Completed Blocked All time blocked ReadStage 0 0 0 0

Re: Mutation Stage does not finish

2014-09-10 Thread Benedict Elliott Smith
Could you post the results of jstack on the process somewhere? On Thu, Sep 11, 2014 at 7:07 AM, Robert Coli wrote: > On Wed, Sep 10, 2014 at 1:53 PM, Eduardo Cusa < > eduardo.c...@usmediaconsulting.com> wrote: > >> No, is still running the Mutation Stage. >> > > If you're sure that it is not re

Re: Mutation Stage does not finish

2014-09-10 Thread Robert Coli
On Wed, Sep 10, 2014 at 1:53 PM, Eduardo Cusa < eduardo.c...@usmediaconsulting.com> wrote: > No, is still running the Mutation Stage. > If you're sure that it is not receiving Hinted Handoff, then the only mutations in question can be from the replay of the commit log. The commit log should take

Re: Mutation Stage does not finish

2014-09-10 Thread Eduardo Cusa
No, is still running the Mutation Stage. On Wed, Sep 10, 2014 at 5:38 PM, Robert Coli wrote: > On Wed, Sep 10, 2014 at 12:16 PM, Eduardo Cusa < > eduardo.c...@usmediaconsulting.com> wrote: > >> Yes, I restarted the node becaouse the write latency was 2500 ms, when >> usually is 5 ms. >> > > And

Re: Mutation Stage does not finish

2014-09-10 Thread Robert Coli
On Wed, Sep 10, 2014 at 12:16 PM, Eduardo Cusa < eduardo.c...@usmediaconsulting.com> wrote: > Yes, I restarted the node becaouse the write latency was 2500 ms, when > usually is 5 ms. > And did that help? =Rob

Re: Mutation Stage does not finish

2014-09-10 Thread Eduardo Cusa
Yes, I restarted the node becaouse the write latency was 2500 ms, when usually is 5 ms. On Wed, Sep 10, 2014 at 4:05 PM, Robert Coli wrote: > On Wed, Sep 10, 2014 at 12:03 PM, Eduardo Cusa < > eduardo.c...@usmediaconsulting.com> wrote: > >> Yes, the tpstats is printing. The Opcenter show the no

Re: Mutation Stage does not finish

2014-09-10 Thread Eduardo Cusa
Yes, the tpstats is printing. The Opcenter show the node down. Also OpCenter show the following status: Status: Unresponsive - Starting Gossip:Down Thrift:Down Native Transport: Down Pending Tasks: 0 On Wed, Sep 10, 2014 at 3:54 PM, Robert Coli wrote: > On Wed, Sep 10, 2014 at 11:38 AM

Re: Mutation Stage does not finish

2014-09-10 Thread Robert Coli
On Wed, Sep 10, 2014 at 12:03 PM, Eduardo Cusa < eduardo.c...@usmediaconsulting.com> wrote: > Yes, the tpstats is printing. The Opcenter show the node down. > Have you recently restarted it or anything? If not, try doing so? =Rob

Re: Mutation Stage does not finish

2014-09-10 Thread Robert Coli
On Wed, Sep 10, 2014 at 11:38 AM, Eduardo Cusa < eduardo.c...@usmediaconsulting.com> wrote: > Actually the node is *down*. > The node can't be that "down" if it's printing tpstats... https://issues.apache.org/jira/browse/CASSANDRA-4162 ? =Rob

Mutation Stage does not finish

2014-09-10 Thread Eduardo Cusa
Hello, I have a node that is in MutationStage for the last 5 hours. Actually the node is *down*. The pendings task go from 776 to 110 and then to 964. There are some way to finish this stage? The last heavy write workload was 5 days ago. Pool NameActive Pending Com