Hi Jeff, This is version 3.0.11. Being run on Oracle Red Hat Linux.
If I retry immediately it fails, leave it for 20 minutes, like I have just now and retry it and it has worked. (?!?!) Ive checked all the logs (system and Debug ) and in the logs I have this: DEBUG [GossipStage:1] 2017-09-29 15:28:04,390 FailureDetector.java:456 - Ignoring interval time of 2102011879 for /10.172.181.62 INFO [SharedPool-Worker-1] 2017-09-29 15:28:07,382 Message.java:615 - Unexpected exception during request; channel = [id: 0x963c334d, L:/10.172.117.61:9042 - R:/10.172.117.21:51853] io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed: Connection reset by peerat io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown Source) ~[netty-all-4.0.44.Final.jar:4.0.44.Final] DEBUG [GossipStage:1] 2017-09-29 15:28:07,389 FailureDetector.java:456 - Ignoring interval time of 2085147992 for /10.172.117.63 INFO [SharedPool-Worker-2] 2017-09-29 15:28:07,400 Message.java:615 - Unexpected exception during request; channel = [id: 0xa7633234, L:/10.172.117.61:9042 ! R:/10.172.117.21:51818] io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed: Connection reset by peerat io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown Source) ~[netty-all-4.0.44.Final.jar:4.0.44.Final] INFO [SharedPool-Worker-1] 2017-09-29 15:28:07,418 Message.java:615 - Unexpected exception during request; channel = [id: 0x8c685c2c, L:/10.172.117.61:9042 ! R:/10.172.117.21:51838] io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed: Connection reset by peerat io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown Source) ~[netty-all-4.0.44.Final.jar:4.0.44.Final] INFO [HANDSHAKE-/10.172.181.62] 2017-09-29 15:28:10,140 OutboundTcpConnection.java:523 - Handshaking version with /10.172.181.62 DEBUG [GossipStage:1] 2017-09-29 15:28:12,288 FailureDetector.java:456 - Ignoring interval time of 2153356364 for /10.172.117.63 DEBUG [GossipStage:1] 2017-09-29 15:28:15,289 FailureDetector.java:456 - Ignoring interval time of 2000150143 for /10.172.181.62 DEBUG [GossipStage:1] 2017-09-29 15:28:16,305 FailureDetector.java:456 - Ignoring interval time of 2790916098 for /10.172.117.62 DEBUG [GossipStage:1] 2017-09-29 15:28:17,289 FailureDetector.java:456 - Ignoring interval time of 2000209617 for /10.172.181.62 DEBUG [GossipStage:1] 2017-09-29 15:28:18,391 FailureDetector.java:456 - Ignoring interval time of 2085345347 for /10.172.181.63 DEBUG [GossipStage:1] 2017-09-29 15:28:19,391 FailureDetector.java:456 - Ignoring interval time of 2102054326 for /10.172.181.62 DEBUG [GossipStage:1] 2017-09-29 15:28:20,391 FailureDetector.java:456 - Ignoring interval time of 2000754004 for /10.172.181.61 DEBUG [GossipStage:1] 2017-09-29 15:28:20,391 FailureDetector.java:456 - Ignoring interval time of 2000787648 for /10.172.181.63 DEBUG [GossipStage:1] 2017-09-29 15:28:23,391 FailureDetector.java:456 - Ignoring interval time of 2254752613 for /10.172.117.63 DEBUG [GossipStage:1] 2017-09-29 15:28:23,392 FailureDetector.java:456 - Ignoring interval time of 2086206045 for /10.172.181.62 DEBUG [GossipStage:1] 2017-09-29 15:28:23,392 FailureDetector.java:456 - Ignoring interval time of 2086239333 for /10.172.181.63 DEBUG [GossipStage:1] 2017-09-29 15:28:26,392 FailureDetector.java:456 - Ignoring interval time of 2000751862 for /10.172.181.62 DEBUG [GossipStage:1] 2017-09-29 15:28:26,516 FailureDetector.java:456 - Ignoring interval time of 2124952635 for /10.172.181.63 DEBUG [GossipStage:1] 2017-09-29 15:28:29,392 FailureDetector.java:456 - Ignoring interval time of 2254658974 for /10.172.117.63 DEBUG [GossipStage:1] 2017-09-29 15:28:30,306 FailureDetector.java:456 - Ignoring interval time of 2000326847 for /10.172.181.63 DEBUG [GossipStage:1] 2017-09-29 15:28:32,393 FailureDetector.java:456 - Ignoring interval time of 2086445998 for /10.172.181.62 DEBUG [GossipStage:1] 2017-09-29 15:28:32,393 FailureDetector.java:456 - Ignoring interval time of 2086481727 for /10.172.181.63 DEBUG [GossipStage:1] 2017-09-29 15:28:35,290 FailureDetector.java:456 - Ignoring interval time of 2363746015 for /10.172.117.62 DEBUG [GossipStage:1] 2017-09-29 15:28:35,290 FailureDetector.java:456 - Ignoring interval time of 2151813348 for /10.172.117.63 DEBUG [GossipStage:1] 2017-09-29 15:28:37,393 FailureDetector.java:456 - Ignoring interval time of 2103432914 for /10.172.181.62 DEBUG [GossipStage:1] 2017-09-29 15:28:40,394 FailureDetector.java:456 - Ignoring interval time of 2255155834 for /10.172.117.63 DEBUG [GossipStage:1] 2017-09-29 15:28:41,394 FailureDetector.java:456 - Ignoring interval time of 2086966679 for /10.172.117.62 DEBUG [GossipStage:1] 2017-09-29 15:28:41,394 FailureDetector.java:456 - Ignoring interval time of 2087037617 for /10.172.181.63 INFO [HANDSHAKE-/10.172.181.61] 2017-09-29 15:28:47,132 OutboundTcpConnection.java:523 - Handshaking version with /10.172.181.61 The Handshake @15:28 is about the same time as I just tried it and it working. So it has an inability to handshake with the nodes?? Any suggestions would be great, I have to go to the Network team with this, but I’m not sure where to start… From: Jeff Jirsa [mailto:[email protected]] Sent: 29 September 2017 14:54 To: cassandra Subject: Re: Issue with New Production Cluster What version? If you retry immediately, does it reconnect? Anything in the logs? What you describe is atypical - timeouts on queries can (and will) happen occasionally under load, but timeout on connect is atypical. Any sign of networking issues/slowness/dns problems/etc? On Fri, Sep 29, 2017 at 6:43 AM, Jonathan Baynes <[email protected]<mailto:[email protected]>> wrote: Hi Community, I have a 6 node ring, covering 2 DC’s, the ring isn’t being used yet and we are just in the connectivity and testing phase. So the boxes are NOT under any load. I’ve gone to connect to CQLSH this afternoon and I’ve had this returned: cqlsh xx.xxx.xxx.xx -u cassandra -p cassandra Connection error: ('Unable to connect to any servers', { xx.xxx.xxx.xx’: OperationTimedOut('errors=Timed out creating connection (5 seconds), last_host=None',)}) Status of the service is “Running” Nodetool status is UP for all nodes. Tested telnet on all ports and they are all up and connecting. Checked the PID is up for cassandra (it is) Any idea how I can debug the cause of this further? Thanks J Jonathan Baynes DBA Tradeweb Europe Limited Moor Place • 1 Fore Street Avenue<https://urldefense.proofpoint.com/v2/url?u=https-3A__maps.google.com_-3Fq-3D1-2BFore-2BStreet-2BAvenue-25C2-25A0-25C2-25A0-25E2-2580-25A2-25C2-25A0-25C2-25A0London-2BEC2Y-2B9DT-26entry-3Dgmail-26source-3Dg&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=ynRqj6nQ2Rc0PoXhoOv_0HAaz8G8xuGO8wNbuHB-3pU&s=46cq-66-QUqnj-taNMRFYe7YK3eNez6ogHJCv014X9w&e=> • London EC2Y 9DT<https://urldefense.proofpoint.com/v2/url?u=https-3A__maps.google.com_-3Fq-3D1-2BFore-2BStreet-2BAvenue-25C2-25A0-25C2-25A0-25E2-2580-25A2-25C2-25A0-25C2-25A0London-2BEC2Y-2B9DT-26entry-3Dgmail-26source-3Dg&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=ynRqj6nQ2Rc0PoXhoOv_0HAaz8G8xuGO8wNbuHB-3pU&s=46cq-66-QUqnj-taNMRFYe7YK3eNez6ogHJCv014X9w&e=> P +44 (0)20 77760988<tel:+44%2020%207776%200988> • F +44 (0)20 7776 3201<tel:+44%2020%207776%203201> • M +44 (0)xxxx xxxxxx [email protected]<mailto:[email protected]> [cid:[email protected]]<http://www.tradeweb.com/> follow us: [cid:[email protected]] <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.linkedin.com_company_tradeweb-3Ftrk-3Dtop-5Fnav-5Fhome&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=ynRqj6nQ2Rc0PoXhoOv_0HAaz8G8xuGO8wNbuHB-3pU&s=saD3ZbF7LhTn3IHCHWSzhGp8Ns6SIRO0s0KXCnEbqLU&e=> [cid:[email protected]] <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.twitter.com_Tradeweb&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=ynRqj6nQ2Rc0PoXhoOv_0HAaz8G8xuGO8wNbuHB-3pU&s=Duw_ud1x7HIX3h-0UEtKorCrlKn8C5tUoH6tkrvXDy0&e=> ————————————————————————————————————————— A leading marketplace<http://www.tradeweb.com/About-Us/Awards/> for electronic fixed income, derivatives and ETF trading ________________________________________________________________________ This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy it. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. Tradeweb reserves the right to monitor all e-mail communications through its networks. If you do not wish to receive marketing emails about our products / services, please let us know by contacting us, either by email at [email protected]<mailto:[email protected]> or by writing to us at the registered office of Tradeweb in the UK, which is: Tradeweb Europe Limited (company number 3912826), 1 Fore Street Avenue London EC2Y 9DT<https://urldefense.proofpoint.com/v2/url?u=https-3A__maps.google.com_-3Fq-3Dnumber-2B3912826-29-2C-2B1-2BFore-2BStreet-2BAvenue-2BLondon-2BEC2Y-2B9DT-26entry-3Dgmail-26source-3Dg&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=ynRqj6nQ2Rc0PoXhoOv_0HAaz8G8xuGO8wNbuHB-3pU&s=VSn9NuhaGsP_VCpKoBZFV2hz_46iJ3VhVk5rYBKHeN8&e=>. To see our privacy policy, visit our website @ www.tradeweb.com<http://www.tradeweb.com>. ________________________________________________________________________ This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy it. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. Tradeweb reserves the right to monitor all e-mail communications through its networks. If you do not wish to receive marketing emails about our products / services, please let us know by contacting us, either by email at [email protected] or by writing to us at the registered office of Tradeweb in the UK, which is: Tradeweb Europe Limited (company number 3912826), 1 Fore Street Avenue London EC2Y 9DT. To see our privacy policy, visit our website @ www.tradeweb.com.
