I’ve resolved the issue. These hosts were initially created in a different VLAN. There was a holdover entry in /etc/hosts pointing to the old IP Addr for this host.
So, the return back to the client never happened because it was sending the connection to the old IP to a host that did not exist. Thanks for looking! James From: James Lovato <james.lov...@iacapps.com> Reply-To: "user@cassandra.apache.org" <user@cassandra.apache.org> Date: Thursday, January 25, 2018 at 10:44 AM To: "user@cassandra.apache.org" <user@cassandra.apache.org> Subject: Re: Nodetool hanging - Cassandra 3.10, Oracle Java 1.8.0_131 It starts up, I can telnet to the port. Also the log states so: ^C[root@dfprdbcdlpcs5 logs]# grep JMX cassandra.log INFO [main] 2018-01-25 10:52:54,176 JMXServerUtils.java:249 - Configured JMX server at: service:jmx:rmi://0.0.0.0/jndi/rmi://0.0.0.0:7199/jmxrmi INFO [main] 2018-01-25 10:52:54,379 StartupChecks.java:164 - JMX is enabled to receive remote connections on port: 7199 INFO [main] 2018-01-25 10:55:25,751 JMXServerUtils.java:249 - Configured JMX server at: service:jmx:rmi://0.0.0.0/jndi/rmi://0.0.0.0:7199/jmxrmi INFO [main] 2018-01-25 10:55:25,957 StartupChecks.java:164 - JMX is enabled to receive remote connections on port: 7199 From: Roger Brown <roger.br...@perfectsearchcorp.com> Reply-To: "user@cassandra.apache.org" <user@cassandra.apache.org> Date: Thursday, January 25, 2018 at 9:51 AM To: "user@cassandra.apache.org" <user@cassandra.apache.org> Subject: Re: Nodetool hanging - Cassandra 3.10, Oracle Java 1.8.0_131 My first thought is restart cassandra and monitor its log to make sure it starts up. On Thu, Jan 25, 2018 at 9:15 AM Jeff Jirsa <jji...@gmail.com<mailto:jji...@gmail.com>> wrote: Can you telnet to the 7199 port? -- Jeff Jirsa On Jan 25, 2018, at 8:00 AM, James Lovato <james.lov...@iacapps.com<mailto:james.lov...@iacapps.com>> wrote: I have a 15 node cluster, 5 in each of 3 dcs. One host just recently started hanging doing any `nodetool` requests. I’ve enabled remote JMX on all these hosts and it was working fine until this week. No other changes have been made. All the Cassandra-env.sh files are the same among the nodes, and only seeds and listen_address are different for Cassandra.yaml files. Nothing in debug log about why the Connection timed out. This is what I get running nodetool from the same host: [root@host5 conf]# nodetool status nodetool: Failed to connect to 'xx.xx.xx.xx:7199' - ConnectException: 'Connection timed out (Connection timed out)'. Any thoughts or how to log nodetool’s actions? Or see why it’s not accepting the request? Telnet to the port from localhost as well as another node work fine. Thanks, James
