We get the below error in app logs when it's trying to hit DC which doesn't get traffic in multi DC cluster. But other DC where all traffic hits seems to be responding without any issues. App read/write requests are timing out. Hector client version is 1.0.5. On cassandra side , we are getting socket timeouts. Configs are all same in both DC and on app side as well?
We expanded this cluster by adding 3 nodes to 2 DC . It has 6 nodes in each DC. App did not recognize new nodes until it was explicitly defined in property file. Since then we are seeing the issue/error below. This DC acts like failover DC and this is only place where get socket timeout errors in cassandra layer in all nodes. Do we need to change any config ? Will increasing this value thrift_framed_transport_size_in_mb: 15 help? Tried enabling assandra.useSocketKeepalive=true, timeout still occurs. cassandraThriftSocketTimeout=0 in app property 2014-02-06 10:38:57,936 -0800 [Hector.me.prettyprint.cassandra.connection.NodeAutoDiscoverService-1] ERROR me.prettyprint.cassandra.connection.client.HThriftClient - Could not flush transport (to be expected if the pool is shutting down) in close for client: CassandraClient<:9160-10220> org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pipe at org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:147) at org.apache.thrift.transport.TFramedTransport.flush(TFramedTransport.java:156) at me.prettyprint.cassandra.connection.client.HThriftClient.close(HThriftClient.java:98) at me.prettyprint.cassandra.connection.client.HThriftClient.close(HThriftClient.java:26) at me.prettyprint.cassandra.connection.HConnectionManager.closeClient(HConnectionManager.java:323) at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:272) at me.prettyprint.cassandra.service.AbstractCluster.describeKeyspaces(AbstractCluster.java:136) at me.prettyprint.cassandra.connection.NodeAutoDiscoverService.discoverNodes(NodeAutoDiscoverService.java:97) at me.prettyprint.cassandra.connection.NodeAutoDiscoverService.doAddNodes(NodeAutoDiscoverService.java:68) at me.prettyprint.cassandra.connection.NodeAutoDiscoverService$QueryRing.run(NodeAutoDiscoverService.java:59) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) Caused by: java.net.SocketException: Broken pipe at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109) at java.net.SocketOutputStream.write(SocketOutputStream.java:153) at org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:145)
