Hi David, Are you sure you ran the repair entirely (9 days + repair logs ok on opscenter server) before adding the 10th node ? This is important to avoid potential data loss ! Did you set auto_bootstrap to true on this 10th node ?
C*heers, Alain 2015-06-29 14:54 GMT+02:00 David CHARBONNIER <david.charbonn...@rgsystem.com >: > Hi, > > > > We’re using Cassandra 2.0.8.39 through Datastax Enterprise 4.5.1 with a 9 > nodes cluster. > > We need to add a few new nodes to the cluster but we’re experiencing an > issue we don’t know how to solve. > > Here is exactly what we did : > > - We had 8 nodes and need to add a few ones > > - We tried to add 9th node but stream stucked a very long time > and bootstrap never finish (related to streaming_socket_timeout_in_ms > default value in cassandra.yaml) > > - We ran a solution given by a Datastax’s architect : restart > the node with auto_bootstrap set to false and run a repair > > - After this issue, we ran into pathing the default > configuration on all our nodes to avoid this problem and made a rolling > restart of the cluster > > - Then, we tried adding a 10th node but it receives stream from > only one node (node2). > > > > Here is the logs on this problematic node (node10) : > > INFO [main] 2015-06-26 15:25:59,490 StreamResultFuture.java (line 87) > [Stream #a5226b30-1c17-11e5-a58b-e35f08264ca1] Executing streaming plan for > Bootstrap > > INFO [main] 2015-06-26 15:25:59,490 StreamResultFuture.java (line 91) > [Stream #a5226b30-1c17-11e5-a58b-e35f08264ca1] Beginning stream session > with /node6 > > INFO [main] 2015-06-26 15:25:59,491 StreamResultFuture.java (line 91) > [Stream #a5226b30-1c17-11e5-a58b-e35f08264ca1] Beginning stream session > with /node5 > > INFO [main] 2015-06-26 15:25:59,492 StreamResultFuture.java (line 91) > [Stream #a5226b30-1c17-11e5-a58b-e35f08264ca1] Beginning stream session > with /node4 > > INFO [main] 2015-06-26 15:25:59,493 StreamResultFuture.java (line 91) > [Stream #a5226b30-1c17-11e5-a58b-e35f08264ca1] Beginning stream session > with /node3 > > INFO [main] 2015-06-26 15:25:59,493 StreamResultFuture.java (line 91) > [Stream #a5226b30-1c17-11e5-a58b-e35f08264ca1] Beginning stream session > with /node9 > > INFO [main] 2015-06-26 15:25:59,493 StreamResultFuture.java (line 91) > [Stream #a5226b30-1c17-11e5-a58b-e35f08264ca1] Beginning stream session > with /node8 > > INFO [main] 2015-06-26 15:25:59,493 StreamResultFuture.java (line 91) > [Stream #a5226b30-1c17-11e5-a58b-e35f08264ca1] Beginning stream session > with /node7 > > INFO [main] 2015-06-26 15:25:59,494 StreamResultFuture.java (line 91) > [Stream #a5226b30-1c17-11e5-a58b-e35f08264ca1] Beginning stream session > with /node1 > > INFO [main] 2015-06-26 15:25:59,494 StreamResultFuture.java (line 91) > [Stream #a5226b30-1c17-11e5-a58b-e35f08264ca1] Beginning stream session > with /node2 > > INFO [STREAM-IN-/node6] 2015-06-26 15:25:59,515 StreamResultFuture.java > (line 186) [Stream #a5226b30-1c17-11e5-a58b-e35f08264ca1] Session with > /node6 is complete > > INFO [STREAM-IN-/node4] 2015-06-26 15:25:59,516 StreamResultFuture.java > (line 186) [Stream #a5226b30-1c17-11e5-a58b-e35f08264ca1] Session with > /node4 is complete > > INFO [STREAM-IN-/node5] 2015-06-26 15:25:59,517 StreamResultFuture.java > (line 186) [Stream #a5226b30-1c17-11e5-a58b-e35f08264ca1] Session with > /node5 is complete > > INFO [STREAM-IN-/node3] 2015-06-26 15:25:59,527 StreamResultFuture.java > (line 186) [Stream #a5226b30-1c17-11e5-a58b-e35f08264ca1] Session with > /node3 is complete > > INFO [STREAM-IN-/node1] 2015-06-26 15:25:59,528 StreamResultFuture.java > (line 186) [Stream #a5226b30-1c17-11e5-a58b-e35f08264ca1] Session with > /node1 is complete > > INFO [STREAM-IN-/node8] 2015-06-26 15:25:59,530 StreamResultFuture.java > (line 186) [Stream #a5226b30-1c17-11e5-a58b-e35f08264ca1] Session with > /node8 is complete > > INFO [STREAM-IN-/node7] 2015-06-26 15:25:59,531 StreamResultFuture.java > (line 186) [Stream #a5226b30-1c17-11e5-a58b-e35f08264ca1] Session with > /node7 is complete > > INFO [STREAM-IN-/node9] 2015-06-26 15:25:59,533 StreamResultFuture.java > (line 186) [Stream #a5226b30-1c17-11e5-a58b-e35f08264ca1] Session with > /node9 is complete > > INFO [STREAM-IN-/node2] 2015-06-26 15:26:04,874 StreamResultFuture.java > (line 173) [Stream #a5226b30-1c17-11e5-a58b-e35f08264ca1] Prepare > completed. Receiving 171 files(14844054090 bytes), sending 0 files(0 bytes) > > > > On the other nodes (not node2 which streams data), there is an error > telling that node10 has no hostID. > > > > Did you ran into this issue or do you have any idea on how to solve this ? > > > > Thank you for your help. > > > > Best regards, > > > > *David CHARBONNIER* > > Sysadmin > > T : +33 411 934 200 > > david.charbonn...@rgsystem.com > > ZAC Aéroport > > 125 Impasse Adam Smith > > 34470 Pérols - France > > *www.rgsystem.com* <http://www.rgsystem.com/> > > > > > > >
