Hi all, Need urgent help. We have one Physical Data Center of 5 nodes with 1 TB data on each (Location: Dallas). Currently we are using Cassandra ver 3.0.9. Now we are Adding one more Data Center of 5 nodes(Location GCP-US) and have joined it to the existing one.
While running nodetool rebuild command, we are getting following error : On GCP node (where we ran rebuild command) : > ERROR [STREAM-IN-/192.x.x.x] 2021-10-05 15:56:52,246 > StreamSession.java:639 - [Stream #66646d30-25a2-11ec-903b-774f88efe725] > Remote peer 192.x.x.x failed stream session. > INFO [STREAM-IN-/192.x.x.x] 2021-10-05 15:56:52,266 > StreamResultFuture.java:183 - [Stream > #66646d30-25a2-11ec-903b-774f88efe725] Session with /192.x.x.x is complete On DL source node : > INFO [STREAM-IN-/34.x.x.x] 2021-10-05 15:55:53,785 > StreamResultFuture.java:183 - [Stream > #66646d30-25a2-11ec-903b-774f88efe725] Session with /34.x.x.x is complete > ERROR [STREAM-OUT-/34.x.x.x] 2021-10-05 15:55:53,785 > StreamSession.java:534 - [Stream #66646d30-25a2-11ec-903b-774f88efe725] > Streaming error occurred > java.lang.RuntimeException: Transfer of file > /var/lib/cassandra/data/clickstream/glusr_usr_paid_url_mv-3c49c392b35511e9bd0a8f42dfb09617/mc-45676-big-Data.db > already completed or aborted (perhaps session failed?). > at > org.apache.cassandra.streaming.messages.OutgoingFileMessage.startTransfer(OutgoingFileMessage.java:120) > ~[apache-cassandra-3.0.9.jar:3.0.9] > at > org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:50) > ~[apache-cassandra-3.0.9.jar:3.0.9] > at > org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:42) > ~[apache-cassandra-3.0.9.jar:3.0.9] > at > org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:48) > ~[apache-cassandra-3.0.9.jar:3.0.9] > at > org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:387) > ~[apache-cassandra-3.0.9.jar:3.0.9] > at > org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:367) > ~[apache-cassandra-3.0.9.jar:3.0.9] > at java.lang.Thread.run(Thread.java:748) [na:1.8.0_192] > WARN [STREAM-IN-/34.x.x.x] 2021-10-05 15:55:53,786 > StreamResultFuture.java:210 - [Stream > #66646d30-25a2-11ec-903b-774f88efe725] Stream failed Before starting this rebuild, we have made the following changes: 1. Set setstreamthroughput to 600 Mb/sec 2. Set setinterdcstreamthroughput to 600 Mb/sec 3. streaming_socket_timeout_in_ms is 24 hrs 4. Disabled autocompaction on GCP node as this was heavily utilising CPU resource FYI, GCP rebuild process starts with data streaming from 3 nodes, and all fails one by one after streaming for a few hours. Please help out how to correct this issue. Is there any other way to rebuild such big data. We have a few tables with 200 - 400GB of data and some smaller tables. Also, we have Mviews in our environment Regards, Ashish Gupta
