Thanks for the reply, Michael! Yes, I did followed the upgrade nodes.
I am running Ubuntu Ubuntu 14.04.2 LTS on all and kernel 3.13.0-57-generic on all. I have 4 machines: .31, .32, .33 and .34. If I run nodetool status from .34 I now see all the others as DN the same happens if I log in from the others: DN 192.168.1.31 ? 256 ? 1f8000f5-026c-42c7-8189-cf19fbede566 RAC1 DN 192.168.1.32 ? 256 ? 12478d45-3d5e-418b-a0dc-dba6d4307af3 RAC1 DN 192.168.1.33 ? 256 ? 994172b3-cd36-4558-a4b8-054cfac027f3 RAC1 UN 192.168.1.34 1.7 MB 256 ? b66be1f3-bb4a-49bd-9835-5c8ee2a71e5c RAC1 If I do a netstat -atn from .34 I get: tcp 0 0 127.0.1.1:53 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN tcp 0 0 192.168.1.34:7001 0.0.0.0:* LISTEN tcp 0 0 127.0.0.1:7199 0.0.0.0:* LISTEN tcp 0 0 192.168.1.34:9160 0.0.0.0:* LISTEN tcp 0 0 127.0.0.1:59441 0.0.0.0:* LISTEN tcp 0 0 192.168.1.34:52951 192.168.1.31:7001 ESTABLISHED On the logs I now have the following errors (/var/log/syslog.log): WARN [MessagingService-Outgoing-/192.168.1.31] 2015-07-22 17:29:48,764 SSLFactory.java:163 - Filtering out TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA as it isnt supported by the socket ERROR [MessagingService-Outgoing-/192.168.1.31] 2015-07-22 17:29:48,764 OutboundTcpConnection.java:229 - error processing a message intended for / 192.168.1.31 java.lang.NullPointerException: null at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:213) ~[guava-16.0.jar:na] at org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.<init>(BufferedDataOutputStreamPlus.java:74) ~[apache-cassandra-2.2.0.jar:2.2.0] at org.apache.cassandra.net.OutboundTcpConnection.connect(OutboundTcpConnection.java:404) ~[apache-cassandra-2.2.0.jar:2.2.0] at org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:218) ~[apache-cassandra-2.2.0.jar:2.2.0] ERROR [MessagingService-Outgoing-/192.168.1.31] 2015-07-22 17:29:48,764 OutboundTcpConnection.java:316 - error writing to /192.168.1.31 java.lang.NullPointerException: null at org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:323) [apache-cassandra-2.2.0.jar:2.2.0] at org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:285) [apache-cassandra-2.2.0.jar:2.2.0] at org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:219) [apache-cassandra-2.2.0.jar:2.2.0] WARN [MessagingService-Outgoing-/192.168.1.33] 2015-07-22 17:29:49,764 SSLFactory.java:163 - Filtering out TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA as it isnt supported by the socket WARN [MessagingService-Outgoing-/192.168.1.31] 2015-07-22 17:29:49,764 SSLFactory.java:163 - Filtering out TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA as it isnt supported by the socket ERROR [MessagingService-Outgoing-/192.168.1.33] 2015-07-22 17:29:49,764 OutboundTcpConnection.java:229 - error processing a message intended for / 192.168.1.33 java.lang.NullPointerException: null at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:213) ~[guava-16.0.jar:na] at org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.<init>(BufferedDataOutputStreamPlus.java:74) ~[apache-cassandra-2.2.0.jar:2.2.0] at org.apache.cassandra.net.OutboundTcpConnection.connect(OutboundTcpConnection.java:404) ~[apache-cassandra-2.2.0.jar:2.2.0] at org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:218) ~[apache-cassandra-2.2.0.jar:2.2.0] ERROR [MessagingService-Outgoing-/192.168.1.31] 2015-07-22 17:29:49,764 OutboundTcpConnection.java:229 - error processing a message intended for / 192.168.1.31 java.lang.NullPointerException: null at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:213) ~[guava-16.0.jar:na] at org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.<init>(BufferedDataOutputStreamPlus.java:74) ~[apache-cassandra-2.2.0.jar:2.2.0] at org.apache.cassandra.net.OutboundTcpConnection.connect(OutboundTcpConnection.java:404) ~[apache-cassandra-2.2.0.jar:2.2.0] at org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:218) ~[apache-cassandra-2.2.0.jar:2.2.0] ERROR [MessagingService-Outgoing-/192.168.1.31] 2015-07-22 17:29:50,763 OutboundTcpConnection.java:316 - error writing to /192.168.1.31 java.lang.NullPointerException: null at org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:323) [apache-cassandra-2.2.0.jar:2.2.0] at org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:285) [apache-cassandra-2.2.0.jar:2.2.0] at org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:219) [apache-cassandra-2.2.0.jar:2.2.0] WARN [MessagingService-Outgoing-/192.168.1.31] 2015-07-22 17:29:51,766 SSLFactory.java:163 - Filtering out TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA as it isnt supported by the socket ERROR [MessagingService-Outgoing-/192.168.1.31] 2015-07-22 17:29:51,767 OutboundTcpConnection.java:229 - error processing a message intended for / 192.168.1.31 java.lang.NullPointerException: null at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:213) ~[guava-16.0.jar:na] at org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.<init>(BufferedDataOutputStreamPlus.java:74) ~[apache-cassandra-2.2.0.jar:2.2.0] at org.apache.cassandra.net.OutboundTcpConnection.connect(OutboundTcpConnection.java:404) ~[apache-cassandra-2.2.0.jar:2.2.0] at org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:218) ~[apache-cassandra-2.2.0.jar:2.2.0] ERROR [MessagingService-Outgoing-/192.168.1.33] 2015-07-22 17:29:52,764 OutboundTcpConnection.java:316 - error writing to /192.168.1.33 java.lang.NullPointerException: null at org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:323) [apache-cassandra-2.2.0.jar:2.2.0] at org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:285) [apache-cassandra-2.2.0.jar:2.2.0] at org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:219) [apache-cassandra-2.2.0.jar:2.2.0] ERROR [MessagingService-Outgoing-/192.168.1.31] 2015-07-22 17:29:52,764 OutboundTcpConnection.java:316 - error writing to /192.168.1.31 java.lang.NullPointerException: null at org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:323) [apache-cassandra-2.2.0.jar:2.2.0] at org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:285) [apache-cassandra-2.2.0.jar:2.2.0] at org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:219) [apache-cassandra-2.2.0.jar:2.2.0] WARN [MessagingService-Outgoing-/192.168.1.31] 2015-07-22 17:29:53,767 SSLFactory.java:163 - Filtering out TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA as it isnt supported by the socket ERROR [MessagingService-Outgoing-/192.168.1.31] 2015-07-22 17:29:53,767 OutboundTcpConnection.java:229 - error processing a message intended for / 192.168.1.31 java.lang.NullPointerException: null at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:213) ~[guava-16.0.jar:na] at org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.<init>(BufferedDataOutputStreamPlus.java:74) ~[apache-cassandra-2.2.0.jar:2.2.0] at org.apache.cassandra.net.OutboundTcpConnection.connect(OutboundTcpConnection.java:404) ~[apache-cassandra-2.2.0.jar:2.2.0] at org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:218) ~[apache-cassandra-2.2.0.jar:2.2.0] On my cassandra.yaml I have the end point snitch: endpoint_snitch: GossipingPropertyFileSnitch On the cassandra-rackdc.properties # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # These properties are used with GossipingPropertyFileSnitch and will # indicate the rack and dc for this node dc=DC1 rack=RAC1 # Add a suffix to a datacenter name. Used by the Ec2Snitch and Ec2MultiRegionSnitch # to append a string to the EC2 region name. #dc_suffix= # Uncomment the following line to make this snitch prefer the internal ip when possible, as the Ec2MultiRegionSnitch does. # prefer_local=true On my cassandra-topology.properties # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # Cassandra Node IP=Data Center:Rack 192.168.1.31=DC1:RAC1 192.168.1.32=DC1:RAC1 192.168.1.33=DC1:RAC1 192.168.1.34=DC1:RAC1 # default for unknown nodes default=DC1:RAC1 On Wed, Jul 22, 2015 at 5:23 PM, Michael Shuler <mich...@pbandjelly.org> wrote: > On 07/22/2015 04:45 PM, Carlos Scheidecker wrote: > >> I have a 4 node Cassandra system running on 4 Ubuntu boxes. After >> updating to Cassandra 2.2.0 and keeping the same cassandra.yaml file, >> the nodes cannot see each other. >> > > What version did you upgrade from? > > Usually, when upgrading, it is probably a good idea to start with the > default cassandra.yaml from the new version (2.2.0 in your case) and edit > the necessary items from your old version; i.e. num_tokens, initial_token, > listen_address, broadcast_address, etc. You are perhaps missing some sort > of default setting that 2.2.0 is looking for? > > When I do a nodetool status it only reports as being up the machine >> where I had issue the command. >> >> In other words, all the machines cannot communicate to each other any >> longer. Nodetool status behave the same on each machine. >> >> I am trying to debug that, hopefully only something on the configuration >> that has changed. >> >> Any ideas? >> > > Anything helpful in the system.log on each of your nodes? > > Did you follow all the upgrade notes from your previous release to 2.2.0? > > https://github.com/apache/cassandra/blob/cassandra-2.2.0/NEWS.txt > > -- > Kind regards, > Michael >
