I don't know what logging is available driver side, I'd probably be writing a shell script to ping all three servers to see if I can prove there's a network problem outside of cassandra first.
On Fri, Sep 29, 2017 at 8:19 AM, Jonathan Baynes < [email protected]> wrote: > Thank you Jeff, you are a credit to this community.. very helpful > > > > Last question Is there any logging I can turn on (especially with the > cassandra drive 2.5.0) that would assist in gaining more of an insight as > to what’s going on? If I’m getting any failed connections at the cassandra > layer? > > > > Thanks > > J > > > > *From:* Jeff Jirsa [mailto:[email protected]] > *Sent:* 29 September 2017 16:07 > > *To:* cassandra > *Subject:* Re: Issue with New Production Cluster > > > > The failure detector is seeing updates every 2.1-2.5 seconds, which it > will ignore because it's over the 2 second default failure detector > interval. > > > > If there's no load, there's no reason it shouldn't be seeing it far more > frequently. > > > > If you're not seeing any signs of GC pauses (like GCInspector log lines), > then it seems like you've got a network interface flapping - maybe STP is > triggering on a port, or a bad cable/interface, or something along those > lines. > > > > > > > > > > > > On Fri, Sep 29, 2017 at 7:32 AM, Jonathan Baynes < > [email protected]> wrote: > > Hi Jeff, > > > > This is version 3.0.11. Being run on Oracle Red Hat Linux. > > > > If I retry immediately it fails, leave it for 20 minutes, like I have just > now and retry it and it has worked. (?!?!) > > Ive checked all the logs (system and Debug ) and in the logs I have this: > > > > DEBUG [GossipStage:1] 2017-09-29 15:28:04,390 FailureDetector.java:456 - > Ignoring interval time of 2102011879 <(210)%20201-1879> for /10.172.181.62 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.181.62&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=F0Xvn8VrY2v_GJioteQGl2Xl9NrlI4T532IaGBwQL-U&e=> > > INFO [SharedPool-Worker-1] 2017-09-29 15:28:07,382 Message.java:615 - > Unexpected exception during request; channel = [id: 0x963c334d, L:/ > 10.172.117.61:9042 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.117.61-3A9042&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=N1yysCzQbMJfPonp7tTVzQ5V6P_NGmCRCIGYb-JJS0w&e=> > - R:/10.172.117.21:51853 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.117.21-3A51853&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=NwQs5pv53MTu7RXO4tUrDSVZAtsEdXTsObmf9GWbNKE&e=> > ] > > io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() > failed: Connection reset by peerat io.netty.channel.unix. > FileDescriptor.readAddress(...)(Unknown Source) > ~[netty-all-4.0.44.Final.jar:4.0.44.Final] > > DEBUG [GossipStage:1] 2017-09-29 15:28:07,389 FailureDetector.java:456 - > Ignoring interval time of 2085147992 <(208)%20514-7992> for /10.172.117.63 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.117.63&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=kcNQ4izrbn-35U5vWnv5cBKG6XfN3nmgkKozyHEfaWA&e=> > > INFO [SharedPool-Worker-2] 2017-09-29 15:28:07,400 Message.java:615 - > Unexpected exception during request; channel = [id: 0xa7633234, L:/ > 10.172.117.61:9042 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.117.61-3A9042&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=N1yysCzQbMJfPonp7tTVzQ5V6P_NGmCRCIGYb-JJS0w&e=> > ! R:/10.172.117.21:51818 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.117.21-3A51818&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=Zd17m05nbPXla0hA2GWhNKHRRo-N_nFqOfrN2_c6meY&e=> > ] > > io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() > failed: Connection reset by peerat io.netty.channel.unix. > FileDescriptor.readAddress(...)(Unknown Source) > ~[netty-all-4.0.44.Final.jar:4.0.44.Final] > > INFO [SharedPool-Worker-1] 2017-09-29 15:28:07,418 Message.java:615 - > Unexpected exception during request; channel = [id: 0x8c685c2c, L:/ > 10.172.117.61:9042 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.117.61-3A9042&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=N1yysCzQbMJfPonp7tTVzQ5V6P_NGmCRCIGYb-JJS0w&e=> > ! R:/10.172.117.21:51838 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.117.21-3A51838&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=P-JajFY4WPK7GbEnPsK2RAoRmzPCztQnE8RT5atNnhY&e=> > ] > > io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() > failed: Connection reset by peerat io.netty.channel.unix. > FileDescriptor.readAddress(...)(Unknown Source) > ~[netty-all-4.0.44.Final.jar:4.0.44.Final] > > INFO [HANDSHAKE-/10.172.181.62 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.181.62&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=F0Xvn8VrY2v_GJioteQGl2Xl9NrlI4T532IaGBwQL-U&e=>] > 2017-09-29 15:28:10,140 OutboundTcpConnection.java:523 - Handshaking > version with /10.172.181.62 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.181.62&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=F0Xvn8VrY2v_GJioteQGl2Xl9NrlI4T532IaGBwQL-U&e=> > > DEBUG [GossipStage:1] 2017-09-29 15:28:12,288 FailureDetector.java:456 - > Ignoring interval time of 2153356364 <(215)%20335-6364> for /10.172.117.63 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.117.63&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=kcNQ4izrbn-35U5vWnv5cBKG6XfN3nmgkKozyHEfaWA&e=> > > DEBUG [GossipStage:1] 2017-09-29 15:28:15,289 FailureDetector.java:456 - > Ignoring interval time of 2000150143 for /10.172.181.62 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.181.62&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=F0Xvn8VrY2v_GJioteQGl2Xl9NrlI4T532IaGBwQL-U&e=> > > DEBUG [GossipStage:1] 2017-09-29 15:28:16,305 FailureDetector.java:456 - > Ignoring interval time of 2790916098 for /10.172.117.62 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.117.62&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=S4jvfMu8gpqeNX_WcphxcT67g_pCf41sOwHuiM4LWRg&e=> > > DEBUG [GossipStage:1] 2017-09-29 15:28:17,289 FailureDetector.java:456 - > Ignoring interval time of 2000209617 for /10.172.181.62 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.181.62&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=F0Xvn8VrY2v_GJioteQGl2Xl9NrlI4T532IaGBwQL-U&e=> > > DEBUG [GossipStage:1] 2017-09-29 15:28:18,391 FailureDetector.java:456 - > Ignoring interval time of 2085345347 <(208)%20534-5347> for /10.172.181.63 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.181.63&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=C4UhWVbSaGt0q4hKIko0f7TmUwYlgXU39lZ3xvAZrRc&e=> > > DEBUG [GossipStage:1] 2017-09-29 15:28:19,391 FailureDetector.java:456 - > Ignoring interval time of 2102054326 <(210)%20205-4326> for /10.172.181.62 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.181.62&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=F0Xvn8VrY2v_GJioteQGl2Xl9NrlI4T532IaGBwQL-U&e=> > > DEBUG [GossipStage:1] 2017-09-29 15:28:20,391 FailureDetector.java:456 - > Ignoring interval time of 2000754004 for /10.172.181.61 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.181.61&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=-lefWC2Mo3ksHhmGfeD7BgS4uynKiolxK6_cLxaySOM&e=> > > DEBUG [GossipStage:1] 2017-09-29 15:28:20,391 FailureDetector.java:456 - > Ignoring interval time of 2000787648 for /10.172.181.63 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.181.63&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=C4UhWVbSaGt0q4hKIko0f7TmUwYlgXU39lZ3xvAZrRc&e=> > > DEBUG [GossipStage:1] 2017-09-29 15:28:23,391 FailureDetector.java:456 - > Ignoring interval time of 2254752613 <(225)%20475-2613> for /10.172.117.63 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.117.63&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=kcNQ4izrbn-35U5vWnv5cBKG6XfN3nmgkKozyHEfaWA&e=> > > DEBUG [GossipStage:1] 2017-09-29 15:28:23,392 FailureDetector.java:456 - > Ignoring interval time of 2086206045 <(208)%20620-6045> for /10.172.181.62 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.181.62&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=F0Xvn8VrY2v_GJioteQGl2Xl9NrlI4T532IaGBwQL-U&e=> > > DEBUG [GossipStage:1] 2017-09-29 15:28:23,392 FailureDetector.java:456 - > Ignoring interval time of 2086239333 <(208)%20623-9333> for /10.172.181.63 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.181.63&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=C4UhWVbSaGt0q4hKIko0f7TmUwYlgXU39lZ3xvAZrRc&e=> > > DEBUG [GossipStage:1] 2017-09-29 15:28:26,392 FailureDetector.java:456 - > Ignoring interval time of 2000751862 for /10.172.181.62 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.181.62&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=F0Xvn8VrY2v_GJioteQGl2Xl9NrlI4T532IaGBwQL-U&e=> > > DEBUG [GossipStage:1] 2017-09-29 15:28:26,516 FailureDetector.java:456 - > Ignoring interval time of 2124952635 <(212)%20495-2635> for /10.172.181.63 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.181.63&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=C4UhWVbSaGt0q4hKIko0f7TmUwYlgXU39lZ3xvAZrRc&e=> > > DEBUG [GossipStage:1] 2017-09-29 15:28:29,392 FailureDetector.java:456 - > Ignoring interval time of 2254658974 <(225)%20465-8974> for /10.172.117.63 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.117.63&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=kcNQ4izrbn-35U5vWnv5cBKG6XfN3nmgkKozyHEfaWA&e=> > > DEBUG [GossipStage:1] 2017-09-29 15:28:30,306 FailureDetector.java:456 - > Ignoring interval time of 2000326847 for /10.172.181.63 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.181.63&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=C4UhWVbSaGt0q4hKIko0f7TmUwYlgXU39lZ3xvAZrRc&e=> > > DEBUG [GossipStage:1] 2017-09-29 15:28:32,393 FailureDetector.java:456 - > Ignoring interval time of 2086445998 <(208)%20644-5998> for /10.172.181.62 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.181.62&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=F0Xvn8VrY2v_GJioteQGl2Xl9NrlI4T532IaGBwQL-U&e=> > > DEBUG [GossipStage:1] 2017-09-29 15:28:32,393 FailureDetector.java:456 - > Ignoring interval time of 2086481727 <(208)%20648-1727> for /10.172.181.63 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.181.63&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=C4UhWVbSaGt0q4hKIko0f7TmUwYlgXU39lZ3xvAZrRc&e=> > > DEBUG [GossipStage:1] 2017-09-29 15:28:35,290 FailureDetector.java:456 - > Ignoring interval time of 2363746015 <(236)%20374-6015> for /10.172.117.62 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.117.62&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=S4jvfMu8gpqeNX_WcphxcT67g_pCf41sOwHuiM4LWRg&e=> > > DEBUG [GossipStage:1] 2017-09-29 15:28:35,290 FailureDetector.java:456 - > Ignoring interval time of 2151813348 for /10.172.117.63 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.117.63&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=kcNQ4izrbn-35U5vWnv5cBKG6XfN3nmgkKozyHEfaWA&e=> > > DEBUG [GossipStage:1] 2017-09-29 15:28:37,393 FailureDetector.java:456 - > Ignoring interval time of 2103432914 <(210)%20343-2914> for /10.172.181.62 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.181.62&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=F0Xvn8VrY2v_GJioteQGl2Xl9NrlI4T532IaGBwQL-U&e=> > > DEBUG [GossipStage:1] 2017-09-29 15:28:40,394 FailureDetector.java:456 - > Ignoring interval time of 2255155834 <(225)%20515-5834> for /10.172.117.63 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.117.63&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=kcNQ4izrbn-35U5vWnv5cBKG6XfN3nmgkKozyHEfaWA&e=> > > DEBUG [GossipStage:1] 2017-09-29 15:28:41,394 FailureDetector.java:456 - > Ignoring interval time of 2086966679 <(208)%20696-6679> for /10.172.117.62 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.117.62&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=S4jvfMu8gpqeNX_WcphxcT67g_pCf41sOwHuiM4LWRg&e=> > > DEBUG [GossipStage:1] 2017-09-29 15:28:41,394 FailureDetector.java:456 - > Ignoring interval time of 2087037617 <(208)%20703-7617> for /10.172.181.63 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.181.63&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=C4UhWVbSaGt0q4hKIko0f7TmUwYlgXU39lZ3xvAZrRc&e=> > > INFO [HANDSHAKE-/10.172.181.61 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.181.61&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=-lefWC2Mo3ksHhmGfeD7BgS4uynKiolxK6_cLxaySOM&e=>] > 2017-09-29 15:28:47,132 OutboundTcpConnection.java:523 - Handshaking > version with /10.172.181.61 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.181.61&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=-lefWC2Mo3ksHhmGfeD7BgS4uynKiolxK6_cLxaySOM&e=> > > > > > > The Handshake @15:28 is about the same time as I just tried it and it > working. So it has an inability to handshake with the nodes?? > > > > Any suggestions would be great, I have to go to the Network team with > this, but I’m not sure where to start… > > > > *From:* Jeff Jirsa [mailto:[email protected]] > *Sent:* 29 September 2017 14:54 > *To:* cassandra > *Subject:* Re: Issue with New Production Cluster > > > > What version? > > If you retry immediately, does it reconnect? > > Anything in the logs? > > > > What you describe is atypical - timeouts on queries can (and will) happen > occasionally under load, but timeout on connect is atypical. Any sign of > networking issues/slowness/dns problems/etc? > > > > > > On Fri, Sep 29, 2017 at 6:43 AM, Jonathan Baynes < > [email protected]> wrote: > > Hi Community, > > > > I have a 6 node ring, covering 2 DC’s, the ring isn’t being used yet and > we are just in the connectivity and testing phase. So the boxes are NOT > under any load. > > > > I’ve gone to connect to CQLSH this afternoon and I’ve had this returned: > > > > cqlsh xx.xxx.xxx.xx -u cassandra -p cassandra > > Connection error: ('Unable to connect to any servers', { xx.xxx.xxx.xx’: > OperationTimedOut('errors=Timed out creating connection (5 seconds), > last_host=None',)}) > > > > Status of the service is “Running” > > Nodetool status is UP for all nodes. > > Tested telnet on all ports and they are all up and connecting. > > Checked the PID is up for cassandra (it is) > > > > Any idea how I can debug the cause of this further? > > > > > > Thanks > > J > > > > > > > > *Jonathan Baynes* > > DBA > Tradeweb Europe Limited > > Moor Place • 1 Fore Street Avenue > <https://urldefense.proofpoint.com/v2/url?u=https-3A__maps.google.com_-3Fq-3D1-2BFore-2BStreet-2BAvenue-25C2-25A0-25C2-25A0-25E2-2580-25A2-25C2-25A0-25C2-25A0London-2BEC2Y-2B9DT-26entry-3Dgmail-26source-3Dg&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=ynRqj6nQ2Rc0PoXhoOv_0HAaz8G8xuGO8wNbuHB-3pU&s=46cq-66-QUqnj-taNMRFYe7YK3eNez6ogHJCv014X9w&e=> > • > <https://urldefense.proofpoint.com/v2/url?u=https-3A__maps.google.com_-3Fq-3D1-250D-2BFore-2BStreet-2BAvenue-25C2-25A0-25C2-25A0-25E2-2580-25A2-25C2-25A0-25C2-25A0London-250D-2BEC2Y-2B9DT-2B-253Chttps-3A__urldefense.proofpoint.com_v2_url-3Fu-253Dhttps-2D3A-5F-5Fmaps.google.com-5F-2D3Fq-2D3D1-2D2BFore-2D2BStreet-2D2BAvenue-2D25C2-2D25A0-2D25C2-2D25A0-2D25E2-2D2580-2D25A2-2D25C2-2D25A0-2D25C2-2D25A0London-2D2BEC2Y-2D2B9DT-2D26entry-2D3Dgmail-2D26source-2D3Dg-2526d-253DDwMFaQ-2526c-253DsA0VaJZJFLZREu2pbPeqjXHJ-2DWd9NNzgHW3gpUOLSSk-2526r-253DCNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-2DdiQ8-2526m-253DynRqj6nQ2Rc0PoXhoOv-5F0HAaz8G8xuGO8wNbuHB-2D3pU-2526s-253D46cq-2D66-2DQUqnj-2DtaNMRFYe7YK3eNez6ogHJCv014X9w-2526e-253D-253E-26entry-3Dgmail-26source-3Dg&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=a46k9sWzI3NKA2ZlZh8o78iGyheFPgXJFCmwuhYy-Qo&e=> > London EC2Y 9DT > <https://urldefense.proofpoint.com/v2/url?u=https-3A__maps.google.com_-3Fq-3D1-2BFore-2BStreet-2BAvenue-25C2-25A0-25C2-25A0-25E2-2580-25A2-25C2-25A0-25C2-25A0London-2BEC2Y-2B9DT-26entry-3Dgmail-26source-3Dg&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=ynRqj6nQ2Rc0PoXhoOv_0HAaz8G8xuGO8wNbuHB-3pU&s=46cq-66-QUqnj-taNMRFYe7YK3eNez6ogHJCv014X9w&e=> > P +44 (0)20 77760988 <+44%2020%207776%200988> • > <https://maps.google.com/?q=1%0D+Fore+Street+Avenue%C2%A0%C2%A0%E2%80%A2+%3Chttps://urldefense.proofpoint.com/v2/url?u%3Dhttps-3A__maps.google.com_-3Fq-3D1-250D-2BFore-2BStreet-2BAvenue-25C2-25A0-25C2-25A0-25E2-2580-25A2-25C2-25A0-25C2-25A0London-250D-2BEC2Y-2B9DT-2B-253Chttps-3A__urldefense.proofpoint.com_v2_url-3Fu-253Dhttps-2D3A-5F-5Fmaps.google.com-5F-2D3Fq-2D3D1-2D2BFore-2D2BStreet-2D2BAvenue-2D25C2-2D25A0-2D25C2-2D25A0-2D25E2-2D2580-2D25A2-2D25C2-2D25A0-2D25C2-2D25A0London-2D2BEC2Y-2D2B9DT-2D26entry-2D3Dgmail-2D26source-2D3Dg-2526d-253DDwMFaQ-2526c-253DsA0VaJZJFLZREu2pbPeqjXHJ-2DWd9NNzgHW3gpUOLSSk-2526r-253DCNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-2DdiQ8-2526m-253DynRqj6nQ2Rc0PoXhoOv-5F0HAaz8G8xuGO8wNbuHB-2D3pU-2526s-253D46cq-2D66-2DQUqnj-2DtaNMRFYe7YK3eNez6ogHJCv014X9w-2526e-253D-253E-26entry-3Dgmail-26source-3Dg%26d%3DDwMFaQ%26c%3DsA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk%26r%3DCNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8%26m%3DLHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA%26s%3Da46k9sWzI3NKA2ZlZh8o78iGyheFPgXJFCmwuhYy-Qo%26e%3D%3E%C2%A0%C2%A0London%0D+EC2Y+9DT+%3Chttps://urldefense.proofpoint.com/v2/url?u%3Dhttps-3A__maps.google.com_-3Fq-3D1-2BFore-2BStreet-2BAvenue-25C2-25A0-25C2-25A0-25E2-2580-25A2-25C2-25A0-25C2-25A0London-2BEC2Y-2B9DT-26entry-3Dgmail-26source-3Dg%26d%3DDwMFaQ%26c%3DsA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk%26r%3DCNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8%26m%3DynRqj6nQ2Rc0PoXhoOv_0HAaz8G8xuGO8wNbuHB-3pU%26s%3D46cq-66-QUqnj-taNMRFYe7YK3eNez6ogHJCv014X9w%26e%3D%3E&entry=gmail&source=g> > F > +44 (0)20 7776 3201 <+44%2020%207776%203201> • M +44 (0)xxxx xxxxxx > > [email protected] > > > > [image: cid:[email protected]] <http://www.tradeweb.com/> > follow us: *[image: cid:[email protected]]* > <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.linkedin.com_company_tradeweb-3Ftrk-3Dtop-5Fnav-5Fhome&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=ynRqj6nQ2Rc0PoXhoOv_0HAaz8G8xuGO8wNbuHB-3pU&s=saD3ZbF7LhTn3IHCHWSzhGp8Ns6SIRO0s0KXCnEbqLU&e=> > [image: cid:[email protected]] > <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.twitter.com_Tradeweb&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=ynRqj6nQ2Rc0PoXhoOv_0HAaz8G8xuGO8wNbuHB-3pU&s=Duw_ud1x7HIX3h-0UEtKorCrlKn8C5tUoH6tkrvXDy0&e=> > > ————————————————————————————————————————— > > A leading marketplace <http://www.tradeweb.com/About-Us/Awards/> for > electronic fixed income, derivatives and ETF trading > > > > ________________________________________________________________________ > > This e-mail may contain confidential and/or privileged information. If you > are not the intended recipient (or have received this e-mail in error) > please notify the sender immediately and destroy it. Any unauthorized > copying, disclosure or distribution of the material in this e-mail is > strictly forbidden. Tradeweb reserves the right to monitor all e-mail > communications through its networks. If you do not wish to receive > marketing emails about our products / services, please let us know by > contacting us, either by email at [email protected] or by writing to > us at the registered office of Tradeweb in the UK, which is: Tradeweb > Europe Limited (company number 3912826), 1 Fore Street Avenue London EC2Y > 9DT > <https://urldefense.proofpoint.com/v2/url?u=https-3A__maps.google.com_-3Fq-3Dnumber-2B3912826-29-2C-2B1-2BFore-2BStreet-2BAvenue-2BLondon-2BEC2Y-2B9DT-26entry-3Dgmail-26source-3Dg&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=ynRqj6nQ2Rc0PoXhoOv_0HAaz8G8xuGO8wNbuHB-3pU&s=VSn9NuhaGsP_VCpKoBZFV2hz_46iJ3VhVk5rYBKHeN8&e=>. > To see our privacy policy, visit our website @ www.tradeweb.com. > > > > ________________________________________________________________________ > > This e-mail may contain confidential and/or privileged information. If you > are not the intended recipient (or have received this e-mail in error) > please notify the sender immediately and destroy it. Any unauthorized > copying, disclosure or distribution of the material in this e-mail is > strictly forbidden. Tradeweb reserves the right to monitor all e-mail > communications through its networks. If you do not wish to receive > marketing emails about our products / services, please let us know by > contacting us, either by email at [email protected] or by writing to > us at the registered office of Tradeweb in the UK, which is: Tradeweb > Europe Limited (company number 3912826), 1 Fore Street Avenue London EC2Y > 9DT > <https://urldefense.proofpoint.com/v2/url?u=https-3A__maps.google.com_-3Fq-3Dnumber-2B3912826-29-2C-2B1-2BFore-2BStreet-2BAvenue-2BLondon-2BEC2Y-2B9DT-26entry-3Dgmail-26source-3Dg&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=m5TdlyiKgTyluGY2ssC5qtvdFJw9hAnGpfbDAKEHzxA&e=>. > To see our privacy policy, visit our website @ www.tradeweb.com. > > > > ________________________________________________________________________ > > This e-mail may contain confidential and/or privileged information. If you > are not the intended recipient (or have received this e-mail in error) > please notify the sender immediately and destroy it. Any unauthorized > copying, disclosure or distribution of the material in this e-mail is > strictly forbidden. Tradeweb reserves the right to monitor all e-mail > communications through its networks. If you do not wish to receive > marketing emails about our products / services, please let us know by > contacting us, either by email at [email protected] or by writing to > us at the registered office of Tradeweb in the UK, which is: Tradeweb > Europe Limited (company number 3912826), 1 Fore Street Avenue London EC2Y > 9DT > <https://maps.google.com/?q=number+3912826),+1+Fore+Street+Avenue+London+EC2Y+9DT&entry=gmail&source=g>. > To see our privacy policy, visit our website @ www.tradeweb.com. >
