Re: New node block in autobootstrap

2016-09-28 Thread Alain RODRIGUEZ
> > Forgot to set replication for new data center :( I was feeling like it could be it :-). From the other thread: > It should be ran from DC3 servers, after altering keyspace to add > keyspaces to the new datacenter. Is this the way you're doing it? > >- Are all the nodes using the same ve

Re: New node block in autobootstrap

2016-09-28 Thread techpyaasa .
Forgot to set replication for new data center :( On Wed, Sep 28, 2016 at 11:33 PM, Jonathan Haddad wrote: > What was the reason? > > On Wed, Sep 28, 2016 at 9:58 AM techpyaasa . wrote: > >> Very sorry...I got the reason for this issue.. >> Please ignore. >> >> >> On Wed, Sep 28, 2016 at 10:14 P

Re: New node block in autobootstrap

2016-09-28 Thread Jonathan Haddad
What was the reason? On Wed, Sep 28, 2016 at 9:58 AM techpyaasa . wrote: > Very sorry...I got the reason for this issue.. > Please ignore. > > > On Wed, Sep 28, 2016 at 10:14 PM, techpyaasa . > wrote: > >> @Paulo >> >> We have done changes as you said >> net.ipv4.tcp_keepalive_time=60 >> net.ip

Re: New node block in autobootstrap

2016-09-28 Thread techpyaasa .
Very sorry...I got the reason for this issue.. Please ignore. On Wed, Sep 28, 2016 at 10:14 PM, techpyaasa . wrote: > @Paulo > > We have done changes as you said > net.ipv4.tcp_keepalive_time=60 > net.ipv4.tcp_keepalive_probes=3 > net.ipv4.tcp_keepalive_intvl=10 > > and increased streaming_sock

Re: New node block in autobootstrap

2016-09-28 Thread techpyaasa .
@Paulo We have done changes as you said net.ipv4.tcp_keepalive_time=60 net.ipv4.tcp_keepalive_probes=3 net.ipv4.tcp_keepalive_intvl=10 and increased streaming_socket_timeout_in_ms to 48 hours , "phi_convict_threshold : 9". And once again recommissioned new data center (DC3) , ran " nodetool reb

Re: New node block in autobootstrap

2016-09-27 Thread laxmikanth sadula
Ok... Thanks for the reply... I'm going to retry nodetool rebuild with following changes as you said net.ipv4.tcp_keepalive_time=60 net.ipv4.tcp_keepalive_probes=3 net.ipv4.tcp_keepalive_intvl=10 Hope this changes would be enough on the new node where I'm running 'nodetool rebuild' and hope NOT r

Re: New node block in autobootstrap

2016-09-27 Thread Paulo Motta
Yeah this is likely to be caused by idle connections being shut down, so you may need to update your tcp_keepalive* and/or network/firewall settings. 2016-09-27 15:29 GMT-03:00 laxmikanth sadula : > Hi paul, > > Thanks for the reply... > > I'm getting following streaming exceptions during nodetoo

Re: New node block in autobootstrap

2016-09-27 Thread laxmikanth sadula
Hi paul, Thanks for the reply... I'm getting following streaming exceptions during nodetool rebuild in c*-2.0.17 *04:24:49,759 StreamSession.java (line 461) [Stream #5e1b7f40-8496-11e6-8847-1b88665e430d] Streaming error occurred* *java.io.IOException: Connection timed out* *at sun.nio.ch.Fil

Re: New node block in autobootstrap

2016-09-27 Thread Paulo Motta
What type of streaming timeout are you getting? Do you have a stack trace? What version are you in? See more information about tuning tcp_keepalive* here: https://docs.datastax.com/en/cassandra/2.0/cassandra/troubleshooting/trblshootIdleFirewall.html 2016-09-27 14:07 GMT-03:00 laxmikanth sadula :

Re: New node block in autobootstrap

2016-09-27 Thread laxmikanth sadula
@Paulo Motta Even we are facing Streaming timeout exceptions during 'nodetool rebuild' , I set streaming_socket_timeout_in_ms to 8640 (24 hours) as suggested in datastax blog - https://support.datastax.com/hc/en-us/articles/206502913- FAQ-How-to-reduce-the-impact-of-streaming-errors-or-failur

Re: New node block in autobootstrap

2016-08-16 Thread Jérôme Mainaud
Hello Paul, Thank you for your reply. The version is 2.2.6. I received the logs today and can confirm three streams failed after timeout. We will try to resume the bootstrap as you recommended. I didn't use -Dreplace_address for two reasons: 1. Because someone tried to reset the node someway

Re: New node block in autobootstrap

2016-08-15 Thread Paulo Motta
What version are you in? This seems like a typical case were there was a problem with streaming (hanging, etc), do you have access to the logs? Maybe look for streaming errors? Typically streaming errors are related to timeouts, so you should review your cassandra streaming_socket_timeout_in_ms and

New node block in autobootstrap

2016-08-15 Thread Jérôme Mainaud
Hello, A client of mime have problems when adding a node in the cluster. After 4 days, the node is still in joining mode, it doesn't have the same level of load than the other and there seems to be no streaming from and to the new node. This node has a history. 1. At the begin, it was in a se