Are you using any of the Tuning Policies (https://docs.datastax.com/en/developer/java-driver/2.0/common/drivers/reference/tuningPolicies_c.html)? It could be that you are hitting some peak load and the driver is not retrying hosts once they are marked “down.”
Sean Durity – Lead Cassandra Admin Big DATA Team For support, create a JIRA<https://portal.homedepot.com/sites/bigdata/Shared%20Documents/Jira%20Hadoop%20Support%20Workflow.pdf> From: Bo Finnerup Madsen [mailto:bo.gunder...@gmail.com] Sent: Tuesday, March 15, 2016 5:24 AM To: user@cassandra.apache.org Subject: Experiencing strange disconnect issue Hi, We are currently trying to convert an existing java web application to use cassandra, and while most of it works great :) we have a "small" issue. After some time, we all connectivity seems to be lost and we get the following errors: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.61.70.107:9042<http://10.61.70.107:9042> (com.datastax.driver.core.exceptions.TransportException: [/10.61.70.107<http://10.61.70.107>] Connection has been closed), /10.61.70.108:9042<http://10.61.70.108:9042> (com.datastax.driver.core.exceptions.TransportException: [/10.61.70.108<http://10.61.70.108>] Connection has been closed)) com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.61.70.107:9042<http://10.61.70.107:9042> (com.datastax.driver.core.exceptions.DriverException: Timeout while trying to acquire available connection (you may want to increase the driver number of per-host connections)), /10.61.70.108:9042<http://10.61.70.108:9042> (com.datastax.driver.core.exceptions.TransportException: [/10.61.70.108<http://10.61.70.108>] Connection has been closed), /10.61.70.110:9042<http://10.61.70.110:9042> (com.datastax.driver.core.exceptions.TransportException: [/10.61.70.110<http://10.61.70.110>] Connection has been closed)) The errors persists, and the application needs to be restarted to recover. At application startup we create a cluster and a session which we reuse through out the application as pr. the documentation. We don't specify any other options when connecting than the IP's of the three servers. We are running cassandra 3.0.3 tar ball in EC2 in a cluster of three machines. The connections are made using v3.0.0 java driver. I have uploaded the configuration and logs from our cassandra cluster here: https://gist.github.com/anonymous/452e736b401317b5b38d The issue happend at 00:44:46. I would greatly appreciate any ideas as to what we are doing wrong to experience this? :) Thank you in advance! Yours sincerely, Bo Madsen ________________________________ The information in this Internet Email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this Email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. When addressed to our clients any opinions or advice contained in this Email are subject to the terms and conditions expressed in any applicable governing The Home Depot terms of business or client engagement letter. The Home Depot disclaims all responsibility and liability for the accuracy and content of this attachment and for any damages or losses arising from any inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other items of a destructive nature, which may be contained in this attachment and shall not be liable for direct, indirect, consequential or special damages in connection with this e-mail message or its attachment.
