heck if [streaming_socket_timeout_in_ms ] setting in Cassandra.yaml file if that sufficient enough before streaming is interrupted ? ~Asad
From: Surbhi Gupta [mailto:surbhi.gupt...@gmail.com] Sent: Thursday, May 7, 2020 3:09 PM To: user@cassandra.apache.org Subject: Re: Bootstraping is failing [root@abc cassandra]# cat /proc/sys/net/ipv4/tcp_keepalive_time 300 [root@abc cassandra]# cat /proc/sys/net/ipv4/tcp_keepalive_intvl 30 [root@abc cassandra]# cat /proc/sys/net/ipv4/tcp_keepalive_probes 9 On Thu, 7 May 2020 at 12:32, Adam Scott <adam.c.sc...@gmail.com<mailto:adam.c.sc...@gmail.com>> wrote: Maybe a firewall killing a connection? What does the following show? cat /proc/sys/net/ipv4/tcp_keepalive_time cat /proc/sys/net/ipv4/tcp_keepalive_intvl cat /proc/sys/net/ipv4/tcp_keepalive_probes On Thu, May 7, 2020 at 10:31 AM Surbhi Gupta <surbhi.gupt...@gmail.com<mailto:surbhi.gupt...@gmail.com>> wrote: Hi, We are trying to expand a datacenter and trying to add nodes but when node is bootstrapping , it goes half way through and then fail with below error, We have increased stremthroughput from 200 to 400 when we were trying for the 2nd time but still it failed. We are on 3.11.0 , using G1GC with 31GB heap. ERROR [MessagingService-Incoming-/10.X.X.X] 2020-05-07 09:42:38,933 CassandraDaemon.java:228 - Exception in thread Thread[MessagingService-Incoming-/10.X.X.X,main] java.io.IOError: java.io.EOFException: Stream ended prematurely at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:227) ~[apache-cassandra-3.11.0.jar:3.11.0] at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:215) ~[apache-cassandra-3.11.0.jar:3.11.0] at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.11.0.jar:3.11.0] at org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize30(PartitionUpdate.java:839) ~[apache-cassandra-3.11.0.jar:3.11.0] at org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize(PartitionUpdate.java:814) ~[apache-cassandra-3.11.0.jar:3.11.0] at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:425) ~[apache-cassandra-3.11.0.jar:3.11.0] at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:434) ~[apache-cassandra-3.11.0.jar:3.11.0] at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:371) ~[apache-cassandra-3.11.0.jar:3.11.0] at org.apache.cassandra.net.MessageIn.read(MessageIn.java:123) ~[apache-cassandra-3.11.0.jar:3.11.0] at org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:192) ~[apache-cassandra-3.11.0.jar:3.11.0] at org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:180) ~[apache-cassandra-3.11.0.jar:3.11.0] at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:94) ~[apache-cassandra-3.11.0.jar:3.11.0] Caused by: java.io.EOFException: Stream ended prematurely at net.jpountz.lz4.LZ4BlockInputStream.readFully(LZ4BlockInputStream.java:218) ~[lz4-1.3.0.jar:na] at net.jpountz.lz4.LZ4BlockInputStream.refill(LZ4BlockInputStream.java:150) ~[lz4-1.3.0.jar:na] at net.jpountz.lz4.LZ4BlockInputStream.read(LZ4BlockInputStream.java:117) ~[lz4-1.3.0.jar:na] at java.io.DataInputStream.readFully(DataInputStream.java:195) ~[na:1.8.0_242] at java.io.DataInputStream.readFully(DataInputStream.java:169) ~[na:1.8.0_242] at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:402) ~[apache-cassandra-3.11.0.jar:3.11.0] at org.apache.cassandra.db.marshal.AbstractType.readValue(AbstractType.java:437) ~[apache-cassandra-3.11.0.jar:3.11.0] at org.apache.cassandra.db.rows.Cell$Serializer.deserialize(Cell.java:245) ~[apache-cassandra-3.11.0.jar:3.11.0] at org.apache.cassandra.db.rows.UnfilteredSerializer.readComplexColumn(UnfilteredSerializer.java:665) ~[apache-cassandra-3.11.0.jar:3.11.0] at org.apache.cassandra.db.rows.UnfilteredSerializer.lambda$deserializeRowBody$1(UnfilteredSerializer.java:606) ~[apache-cassandra-3.11.0.jar:3.11.0] at org.apache.cassandra.utils.btree.BTree.applyForwards(BTree.java:1242) ~[apache-cassandra-3.11.0.jar:3.11.0] at org.apache.cassandra.utils.btree.BTree.apply(BTree.java:1197) ~[apache-cassandra-3.11.0.jar:3.11.0] at org.apache.cassandra.db.Columns.apply(Columns.java:377) ~[apache-cassandra-3.11.0.jar:3.11.0] at org.apache.cassandra.db.rows.UnfilteredSerializer.deserializeRowBody(UnfilteredSerializer.java:600) ~[apache-cassandra-3.11.0.jar:3.11.0] at org.apache.cassandra.db.rows.UnfilteredSerializer.deserializeOne(UnfilteredSerializer.java:475) ~[apache-cassandra-3.11.0.jar:3.11.0] at org.apache.cassandra.db.rows.UnfilteredSerializer.deserialize(UnfilteredSerializer.java:431) ~[apache-cassandra-3.11.0.jar:3.11.0] at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:222) ~[apache-cassandra-3.11.0.jar:3.11.0] ... 11 common frames omitted Thanks Surbhi
