rebuild constantly fails, 3.11
Hi, it seems I'm not able to add add 3 node dc to a 3 node dc. After starting the rebuild on a new node, nodetool netstats show it will receive 1200 files from node-1 and 5000 from node-2. The stream from node-1 completes but the stream from node-2 allways fails, after sending ca 4000 files. After restarting the rebuild it again starts to send the 5000 files. The whole cluster is connected via one switch only , no firewall between, the networks shows no errors. The machines have 8 cores, 32GB RAM and two 1TB discs as raid0. the logs show no errors. The size of the data is ca 1TB. Any help is really welcome, cheers Michael The error is: Cassandra has shutdown. error: null -- StackTrace -- java.io.EOFException at java.io.DataInputStream.readByte(DataInputStream.java:267) at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:222) at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:161) at com.sun.jmx.remote.internal.PRef.invoke(Unknown Source) at javax.management.remote.rmi.RMIConnectionImpl_Stub.invoke(Unknown Source) at javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.invoke(RMIConnector.java:1020) at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:298) at com.sun.proxy.$Proxy7.rebuild(Unknown Source) at org.apache.cassandra.tools.NodeProbe.rebuild(NodeProbe.java:1190) at org.apache.cassandra.tools.nodetool.Rebuild.execute(Rebuild.java:58) at org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:254) at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:168) - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
Re: rebuild constantly fails, 3.11
If the error is reproducible can you upload the logs to a gist from the same time period as when the error occurs?
RE: rebuild constantly fails, 3.11
Is there any chance you've set streaming_socket_timeout_in_ms parameter set too low on failing node? -Original Message- From: Micha [mailto:mich...@fantasymail.de] Sent: Tuesday, August 08, 2017 3:01 AM To: user@cassandra.apache.org; d...@cassandra.apache.org Subject: rebuild constantly fails, 3.11 Hi, it seems I'm not able to add add 3 node dc to a 3 node dc. After starting the rebuild on a new node, nodetool netstats show it will receive 1200 files from node-1 and 5000 from node-2. The stream from node-1 completes but the stream from node-2 allways fails, after sending ca 4000 files. After restarting the rebuild it again starts to send the 5000 files. The whole cluster is connected via one switch only , no firewall between, the networks shows no errors. The machines have 8 cores, 32GB RAM and two 1TB discs as raid0. the logs show no errors. The size of the data is ca 1TB. Any help is really welcome, cheers Michael The error is: Cassandra has shutdown. error: null -- StackTrace -- java.io.EOFException at java.io.DataInputStream.readByte(DataInputStream.java:267) at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:222) at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:161) at com.sun.jmx.remote.internal.PRef.invoke(Unknown Source) at javax.management.remote.rmi.RMIConnectionImpl_Stub.invoke(Unknown Source) at javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.invoke(RMIConnector.java:1020) at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:298) at com.sun.proxy.$Proxy7.rebuild(Unknown Source) at org.apache.cassandra.tools.NodeProbe.rebuild(NodeProbe.java:1190) at org.apache.cassandra.tools.nodetool.Rebuild.execute(Rebuild.java:58) at org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:254) at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:168) - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
Re: rebuild constantly fails, 3.11
no, I have left it at the default value of 24hours. I've read about adjusting phi_convict_threshold, but I haven't done this yet as the network is stable. maybe I set this to 10. On 08.08.2017 15:24, ZAIDI, ASAD A wrote: > Is there any chance you've set streaming_socket_timeout_in_ms parameter set > too low on failing node? > > - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
RE: rebuild constantly fails, 3.11
Without exact failure text, it is really hard to guess what may be going-on - can you please share logfile excerpt detailing the failure error so we can have better idea of the nature failure. Adjusting phi_convict_threshold may yet be another shot in the dark when we don’t know what is causing the failure and network is supposedly stable. ~Asad -Original Message- From: Micha [mailto:mich...@fantasymail.de] Sent: Tuesday, August 08, 2017 8:35 AM To: user@cassandra.apache.org; ZAIDI, ASAD A ; user@cassandra.apache.org Subject: Re: rebuild constantly fails, 3.11 no, I have left it at the default value of 24hours. I've read about adjusting phi_convict_threshold, but I haven't done this yet as the network is stable. maybe I set this to 10. On 08.08.2017 15:24, ZAIDI, ASAD A wrote: > Is there any chance you've set streaming_socket_timeout_in_ms parameter set > too low on failing node? > > - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
Re: rebuild constantly fails, 3.11
The logs didn't show an error. I have started it again with higher log level allthough errors should be logged despite the log level. If it breaks again I share the log with the possible error in it. The only error output I got was on the console: Cassandra has shutdown. error: null -- StackTrace -- java.io.EOFException at java.io.DataInputStream.readByte(DataInputStream.java:267) at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:222) at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:161) at com.sun.jmx.remote.internal.PRef.invoke(Unknown Source) at javax.management.remote.rmi.RMIConnectionImpl_Stub.invoke(Unknown Source) at javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.invoke(RMIConnector.java:1020) at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:298) at com.sun.proxy.$Proxy7.rebuild(Unknown Source) at org.apache.cassandra.tools.NodeProbe.rebuild(NodeProbe.java:1190) at org.apache.cassandra.tools.nodetool.Rebuild.execute(Rebuild.java:58) at org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:254) at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:168) On 08.08.2017 17:03, ZAIDI, ASAD A wrote: > Without exact failure text, it is really hard to guess what may be going-on - > can you please share logfile excerpt detailing the failure error so we can > have better idea of the nature failure. > Adjusting phi_convict_threshold may yet be another shot in the dark when we > don’t know what is causing the failure and network is supposedly stable. > > ~Asad > > > > -Original Message- > From: Micha [mailto:mich...@fantasymail.de] > Sent: Tuesday, August 08, 2017 8:35 AM > To: user@cassandra.apache.org; ZAIDI, ASAD A ; > user@cassandra.apache.org > Subject: Re: rebuild constantly fails, 3.11 > > no, I have left it at the default value of 24hours. > > I've read about adjusting phi_convict_threshold, but I haven't done this yet > as the network is stable. maybe I set this to 10. > > > On 08.08.2017 15:24, ZAIDI, ASAD A wrote: >> Is there any chance you've set streaming_socket_timeout_in_ms parameter set >> too low on failing node? >> >> > > > - > To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org > For additional commands, e-mail: user-h...@cassandra.apache.org > - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
optimal value for native_transport_max_threads
Dear All, any suggestion for optimal value for native_transport_max_threads? as per https://issues.apache.org/jira/browse/CASSANDRA-11363,max_queued_native_transport_requests=4096,how about native_transport_max_threads? Thanks, Peng Xiao
JOB | Permanent Database Engineer (the Netherlands)
Hello, I'm working with an employer that is looking to hire someone to become their lead on Apache Cassandra in the Netherlands. Consequently I had hoped that some members of this mailing list may like to discuss further off-list using "JamesBTobin (at) Gmail (dot) com". Kind regards, James - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
