I think you want to run `nodetool bootstrap resume` ( https://cassandra.apache.org/doc/latest/tools/nodetool/bootstrap.html) to pick up where it last left off. Sorry for the late reply.
On Thu, May 7, 2020 at 2:22 PM Surbhi Gupta <surbhi.gupt...@gmail.com> wrote: > So after failed bootstrapped , if we start cassandra again on the new node > , will it resume bootstrap or will it start over? > > On Thu, 7 May 2020 at 13:32, Adam Scott <adam.c.sc...@gmail.com> wrote: > >> I recommend it on all nodes. This will eliminate that as a source of >> trouble further on down the road. >> >> >> On Thu, May 7, 2020 at 1:30 PM Surbhi Gupta <surbhi.gupt...@gmail.com> >> wrote: >> >>> streaming_socket_timeout_in_ms is 24 hour. >>> So tcp settings should be changed on the new bootstrap node or on all >>> nodes ? >>> >>> >>> On Thu, 7 May 2020 at 13:23, Adam Scott <adam.c.sc...@gmail.com> wrote: >>> >>>> >>>> *edit >>>> /etc/sysctl.confnet.ipv4.tcp_keepalive_time=60 >>>> net.ipv4.tcp_keepalive_probes=3net.ipv4.tcp_keepalive_intvl=10* >>>> then run sysctl -p to cause the kernel to reload the settings >>>> >>>> 5 minutes (300) seconds is probably too long. >>>> >>>> On Thu, May 7, 2020 at 1:09 PM Surbhi Gupta <surbhi.gupt...@gmail.com> >>>> wrote: >>>> >>>>> [root@abc cassandra]# cat /proc/sys/net/ipv4/tcp_keepalive_time >>>>> >>>>> 300 >>>>> >>>>> [root@abc cassandra]# cat /proc/sys/net/ipv4/tcp_keepalive_intvl >>>>> >>>>> 30 >>>>> >>>>> [root@abc cassandra]# cat /proc/sys/net/ipv4/tcp_keepalive_probes >>>>> >>>>> 9 >>>>> >>>>> On Thu, 7 May 2020 at 12:32, Adam Scott <adam.c.sc...@gmail.com> >>>>> wrote: >>>>> >>>>>> Maybe a firewall killing a connection? >>>>>> >>>>>> What does the following show? >>>>>> cat /proc/sys/net/ipv4/tcp_keepalive_time >>>>>> cat /proc/sys/net/ipv4/tcp_keepalive_intvl >>>>>> cat /proc/sys/net/ipv4/tcp_keepalive_probes >>>>>> >>>>>> On Thu, May 7, 2020 at 10:31 AM Surbhi Gupta < >>>>>> surbhi.gupt...@gmail.com> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> We are trying to expand a datacenter and trying to add nodes but >>>>>>> when node is bootstrapping , it goes half way through and then fail with >>>>>>> below error, We have increased stremthroughput from 200 to 400 when we >>>>>>> were >>>>>>> trying for the 2nd time but still it failed. We are on 3.11.0 , using >>>>>>> G1GC >>>>>>> with 31GB heap. >>>>>>> >>>>>>> ERROR [MessagingService-Incoming-/10.X.X.X] 2020-05-07 09:42:38,933 >>>>>>> CassandraDaemon.java:228 - Exception in thread >>>>>>> Thread[MessagingService-Incoming-/10.X.X.X,main] >>>>>>> >>>>>>> java.io.IOError: java.io.EOFException: Stream ended prematurely >>>>>>> >>>>>>> at >>>>>>> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:227) >>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0] >>>>>>> >>>>>>> at >>>>>>> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:215) >>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0] >>>>>>> >>>>>>> at >>>>>>> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) >>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0] >>>>>>> >>>>>>> at >>>>>>> org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize30(PartitionUpdate.java:839) >>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0] >>>>>>> >>>>>>> at >>>>>>> org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize(PartitionUpdate.java:814) >>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0] >>>>>>> >>>>>>> at >>>>>>> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:425) >>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0] >>>>>>> >>>>>>> at >>>>>>> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:434) >>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0] >>>>>>> >>>>>>> at >>>>>>> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:371) >>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0] >>>>>>> >>>>>>> at >>>>>>> org.apache.cassandra.net.MessageIn.read(MessageIn.java:123) >>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0] >>>>>>> >>>>>>> at >>>>>>> org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:192) >>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0] >>>>>>> >>>>>>> at >>>>>>> org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:180) >>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0] >>>>>>> >>>>>>> at >>>>>>> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:94) >>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0] >>>>>>> >>>>>>> Caused by: java.io.EOFException: Stream ended prematurely >>>>>>> >>>>>>> at >>>>>>> net.jpountz.lz4.LZ4BlockInputStream.readFully(LZ4BlockInputStream.java:218) >>>>>>> ~[lz4-1.3.0.jar:na] >>>>>>> >>>>>>> at >>>>>>> net.jpountz.lz4.LZ4BlockInputStream.refill(LZ4BlockInputStream.java:150) >>>>>>> ~[lz4-1.3.0.jar:na] >>>>>>> >>>>>>> at >>>>>>> net.jpountz.lz4.LZ4BlockInputStream.read(LZ4BlockInputStream.java:117) >>>>>>> ~[lz4-1.3.0.jar:na] >>>>>>> >>>>>>> at >>>>>>> java.io.DataInputStream.readFully(DataInputStream.java:195) >>>>>>> ~[na:1.8.0_242] >>>>>>> >>>>>>> at >>>>>>> java.io.DataInputStream.readFully(DataInputStream.java:169) >>>>>>> ~[na:1.8.0_242] >>>>>>> >>>>>>> at >>>>>>> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:402) >>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0] >>>>>>> >>>>>>> at >>>>>>> org.apache.cassandra.db.marshal.AbstractType.readValue(AbstractType.java:437) >>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0] >>>>>>> >>>>>>> at >>>>>>> org.apache.cassandra.db.rows.Cell$Serializer.deserialize(Cell.java:245) >>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0] >>>>>>> >>>>>>> at >>>>>>> org.apache.cassandra.db.rows.UnfilteredSerializer.readComplexColumn(UnfilteredSerializer.java:665) >>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0] >>>>>>> >>>>>>> at >>>>>>> org.apache.cassandra.db.rows.UnfilteredSerializer.lambda$deserializeRowBody$1(UnfilteredSerializer.java:606) >>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0] >>>>>>> >>>>>>> at >>>>>>> org.apache.cassandra.utils.btree.BTree.applyForwards(BTree.java:1242) >>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0] >>>>>>> >>>>>>> at >>>>>>> org.apache.cassandra.utils.btree.BTree.apply(BTree.java:1197) >>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0] >>>>>>> >>>>>>> at org.apache.cassandra.db.Columns.apply(Columns.java:377) >>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0] >>>>>>> >>>>>>> at >>>>>>> org.apache.cassandra.db.rows.UnfilteredSerializer.deserializeRowBody(UnfilteredSerializer.java:600) >>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0] >>>>>>> >>>>>>> at >>>>>>> org.apache.cassandra.db.rows.UnfilteredSerializer.deserializeOne(UnfilteredSerializer.java:475) >>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0] >>>>>>> >>>>>>> at >>>>>>> org.apache.cassandra.db.rows.UnfilteredSerializer.deserialize(UnfilteredSerializer.java:431) >>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0] >>>>>>> >>>>>>> at >>>>>>> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:222) >>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0] >>>>>>> >>>>>>> ... 11 common frames omitted >>>>>>> >>>>>>> Thanks >>>>>>> Surbhi >>>>>>> >>>>>>
