I figured this one out. As it turns out, the nodes that I couldn't connect to, had the hostname set to 127.0.1.1. The listen IP is *not* that IP.
Thanks for the logging tip, it helped track it down. On Thu, Sep 3, 2015 at 10:43 PM, Jason Lewis <jle...@packetnexus.com> wrote: > After enabling that option, I'm seeing errors like this on the node I > can't connect to. > > > Sep 04, 2015 2:35:48 AM sun.rmi.server.UnicastServerRef logCallException > FINE: RMI TCP Connection(4)-127.0.0.1: [127.0.0.1] exception: > javax.management.InstanceNotFoundException: > org.apache.cassandra.metrics:type=ColumnFamily,keyspace=ks,scope=cf,name=PendingTasks > at > com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1095) > at > com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:643) > at > com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678) > at > javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1448) > at > javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76) > at > javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1312) > at > javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1404) > at > javax.management.remote.rmi.RMIConnectionImpl.getAttribute(RMIConnectionImpl.java:641) > at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:497) > at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:323) > at sun.rmi.transport.Transport$1.run(Transport.java:200) > at sun.rmi.transport.Transport$1.run(Transport.java:197) > at java.security.AccessController.doPrivileged(Native Method) > at sun.rmi.transport.Transport.serviceCall(Transport.java:196) > at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568) > at > sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826) > at > sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$79(TCPTransport.java:683) > at java.security.AccessController.doPrivileged(Native Method) > at > sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > > On Wed, Aug 26, 2015 at 2:05 PM, Nate McCall <n...@thelastpickle.com> wrote: >>> LOCAL_JMX=no >>> >>> if [ "$LOCAL_JMX" = "yes" ]; then >>> JVM_OPTS="$JVM_OPTS -Dcassandra.jmx.local.port=$JMX_PORT >>> -XX:+DisableExplicitGC" >>> else >>> JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.port=$JMX_PORT" >>> JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.rmi.port=$JMX_PORT" >>> JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.ssl=false" >>> JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.authenticate=false" >>> JVM_OPTS="$JVM_OPTS >>> >>> -Dcom.sun.management.jmxremote.password.file=/etc/cassandra/jmxremote.password" >>> fi >>> >> >> Retry with the following option added to your JVM_OPTS: >> java.rmi.server.logCalls=true >> >> This should produce some more information about what is going on. >> >> >> >> >> -- >> ----------------- >> Nate McCall >> Austin, TX >> @zznate >> >> Co-Founder & Sr. Technical Consultant >> Apache Cassandra Consulting >> http://www.thelastpickle.com
