Hi , Try to adjust phi_convict_threshold and see if that helps. When we did migration from on prim to AWS, this was one of the factor to consider.
Thanks On Tue, Oct 5, 2021 at 4:00 AM MyWorld <timeplus.1...@gmail.com> wrote: > Hi all, > > Need urgent help. > We have one Physical Data Center of 5 nodes with 1 TB data on each > (Location: Dallas). Currently we are using Cassandra ver 3.0.9. Now we are > Adding one more Data Center of 5 nodes(Location GCP-US) and have joined it > to the existing one. > > While running nodetool rebuild command, we are getting following error : > On GCP node (where we ran rebuild command) : > >> ERROR [STREAM-IN-/192.x.x.x] 2021-10-05 15:56:52,246 >> StreamSession.java:639 - [Stream #66646d30-25a2-11ec-903b-774f88efe725] >> Remote peer 192.x.x.x failed stream session. >> INFO [STREAM-IN-/192.x.x.x] 2021-10-05 15:56:52,266 >> StreamResultFuture.java:183 - [Stream >> #66646d30-25a2-11ec-903b-774f88efe725] Session with /192.x.x.x is complete > > > On DL source node : > >> INFO [STREAM-IN-/34.x.x.x] 2021-10-05 15:55:53,785 >> StreamResultFuture.java:183 - [Stream >> #66646d30-25a2-11ec-903b-774f88efe725] Session with /34.x.x.x is complete >> ERROR [STREAM-OUT-/34.x.x.x] 2021-10-05 15:55:53,785 >> StreamSession.java:534 - [Stream #66646d30-25a2-11ec-903b-774f88efe725] >> Streaming error occurred >> java.lang.RuntimeException: Transfer of file >> /var/lib/cassandra/data/clickstream/glusr_usr_paid_url_mv-3c49c392b35511e9bd0a8f42dfb09617/mc-45676-big-Data.db >> already completed or aborted (perhaps session failed?). >> at >> org.apache.cassandra.streaming.messages.OutgoingFileMessage.startTransfer(OutgoingFileMessage.java:120) >> ~[apache-cassandra-3.0.9.jar:3.0.9] >> at >> org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:50) >> ~[apache-cassandra-3.0.9.jar:3.0.9] >> at >> org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:42) >> ~[apache-cassandra-3.0.9.jar:3.0.9] >> at >> org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:48) >> ~[apache-cassandra-3.0.9.jar:3.0.9] >> at >> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:387) >> ~[apache-cassandra-3.0.9.jar:3.0.9] >> at >> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:367) >> ~[apache-cassandra-3.0.9.jar:3.0.9] >> at java.lang.Thread.run(Thread.java:748) [na:1.8.0_192] >> WARN [STREAM-IN-/34.x.x.x] 2021-10-05 15:55:53,786 >> StreamResultFuture.java:210 - [Stream >> #66646d30-25a2-11ec-903b-774f88efe725] Stream failed > > > Before starting this rebuild, we have made the following changes: > 1. Set setstreamthroughput to 600 Mb/sec > 2. Set setinterdcstreamthroughput to 600 Mb/sec > 3. streaming_socket_timeout_in_ms is 24 hrs > 4. Disabled autocompaction on GCP node as this was heavily utilising CPU > resource > > FYI, GCP rebuild process starts with data streaming from 3 nodes, and all > fails one by one after streaming for a few hours. > Please help out how to correct this issue. > Is there any other way to rebuild such big data. > We have a few tables with 200 - 400GB of data and some smaller tables. > Also, we have Mviews in our environment > > Regards, > Ashish Gupta > >
