None of these throttling are helpful for streaming if you have even a 150-200 Mbit/s bandwidth which is affordable in any cloud. Tweaking network tcp memory, window size etc does not help, the bottleneck is not the network. These are my findings on how streaming is limited in C* 3.0.*
1) Streaming of the particular range which needs to be steamed to the new node is limited with one 1 thread and no tweaking of cpu affinity etc helps, probably the powerfull computing VM will help 2) Disabling compression internode_compression and disabling compression per table in our case helps a bit 3) When streaming has been dropped there is no resume available for the streaming range so it will start from the beginning One of the options could be to create snapshots of sstables on the source node and just copy all sstable snapshots to new node and then run repair, data is ~5TB, RF3 ? How is it possible at all to stream data fast to a new node/nodes ? Vitali. On Wed, Sep 12, 2018 at 5:02 PM Surbhi Gupta <surbhi.gupt...@gmail.com> wrote: > Increase 3 throughput > Compaction throughput > Stream throughput > Interdcstream throughput (if rebuilding from another DC) > > Make all of the above to 0 and see if there is any improvement and later > set the value if u can’t leave these values to 0. > > On Wed, Sep 12, 2018 at 5:42 AM Vitali Dyachuk <vdjat...@gmail.com> wrote: > >> Hi, >> I'm currently streaming data with nodetool rebuild on 2 nodes, each node >> is streaming from different location. The problem is that it takes ~7 days >> to stream 4Tb of data to 1 node, the speed on each side is ~150Mbit/s so >> it should take around >> ~2,5 days . Although there are resources on the destnodes and in the >> source regions. >> I've increased stream throughput, but its only affects outbound >> connections. >> Tested with iperf the bandwidth is 600Mibt/s from both sides. Last week >> i've changed the CS from ST to LC because of huge sstables and compaction >> of them is still ongoing. >> How does rebuild command works ? Does it calculate the range then request >> the needed sstables from that node and start streaming ? How is it possible >> to speed up the streaming ? >> >> Vitali. >> >
