I agreed Michael. I was generating stuff for it again, Looks like they had the SSL stack changed. I came from 2.1.6 to 2.2.0. Thanks.
On Wed, Jul 22, 2015 at 5:45 PM, Michael Shuler <mich...@pbandjelly.org> wrote: > What version of Cassandra did you upgrade to 2.2.0 *from*? > > This would help with looking at config differences, changelogs, etc. > > It seems you have some pretty clear SSL connection errors, according to > the logs, which at least helps with seeing why the nodes can't talk to each > other. I'm not terribly familiar with using SSL with Cassandra, but it > seems clear that you have an incorrect server_encryption_options: > cipher_suites: configuration. > > -- > Kind regards, > Michael > > On 07/22/2015 06:33 PM, Carlos Scheidecker wrote: > >> Thanks for the reply, Michael! >> >> Yes, I did followed the upgrade nodes. >> >> I am running Ubuntu Ubuntu 14.04.2 LTS on all and >> kernel 3.13.0-57-generic on all. >> >> I have 4 machines: .31, .32, .33 and .34. If I run nodetool status from >> .34 I now see all the others as DN the same happens if I log in from the >> others: >> >> DN 192.168.1.31 ? 256 ? >> 1f8000f5-026c-42c7-8189-cf19fbede566 RAC1 >> DN 192.168.1.32 ? 256 ? >> 12478d45-3d5e-418b-a0dc-dba6d4307af3 RAC1 >> DN 192.168.1.33 ? 256 ? >> 994172b3-cd36-4558-a4b8-054cfac027f3 RAC1 >> UN 192.168.1.34 1.7 MB 256 ? >> b66be1f3-bb4a-49bd-9835-5c8ee2a71e5c RAC1 >> >> If I do a netstat -atn from .34 I get: >> >> tcp 0 0 127.0.1.1:53 <http://127.0.1.1:53> >> 0.0.0.0:* LISTEN >> tcp 0 0 0.0.0.0:22 <http://0.0.0.0:22> >> 0.0.0.0:* LISTEN >> tcp 0 0 127.0.0.1:631 <http://127.0.0.1:631> >> 0.0.0.0:* LISTEN >> tcp 0 0 192.168.1.34:7001 <http://192.168.1.34:7001> >> 0.0.0.0:* LISTEN >> tcp 0 0 127.0.0.1:7199 <http://127.0.0.1:7199> >> 0.0.0.0:* LISTEN >> tcp 0 0 192.168.1.34:9160 <http://192.168.1.34:9160> >> 0.0.0.0:* LISTEN >> tcp 0 0 127.0.0.1:59441 <http://127.0.0.1:59441> >> 0.0.0.0:* LISTEN >> tcp 0 0 192.168.1.34:52951 <http://192.168.1.34:52951> >> 192.168.1.31:7001 <http://192.168.1.31:7001> ESTABLISHED >> >> On the logs I now have the following errors (/var/log/syslog.log): >> >> WARN [MessagingService-Outgoing-/192.168.1.31 <http://192.168.1.31>] >> 2015-07-22 17:29:48,764 SSLFactory.java:163 - Filtering out >> >> TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA >> as it isnt supported by the socket >> ERROR [MessagingService-Outgoing-/192.168.1.31 <http://192.168.1.31>] >> 2015-07-22 17:29:48,764 OutboundTcpConnection.java:229 - error >> processing a message intended for /192.168.1.31 <http://192.168.1.31> >> java.lang.NullPointerException: null >> at >> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:213) >> ~[guava-16.0.jar:na] >> at >> >> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.<init>(BufferedDataOutputStreamPlus.java:74) >> ~[apache-cassandra-2.2.0.jar:2.2.0] >> at >> org.apache.cassandra.net >> .OutboundTcpConnection.connect(OutboundTcpConnection.java:404) >> ~[apache-cassandra-2.2.0.jar:2.2.0] >> at >> org.apache.cassandra.net >> .OutboundTcpConnection.run(OutboundTcpConnection.java:218) >> ~[apache-cassandra-2.2.0.jar:2.2.0] >> ERROR [MessagingService-Outgoing-/192.168.1.31 <http://192.168.1.31>] >> 2015-07-22 17:29:48,764 OutboundTcpConnection.java:316 - error writing >> to /192.168.1.31 <http://192.168.1.31> >> java.lang.NullPointerException: null >> at >> org.apache.cassandra.net >> .OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:323) >> [apache-cassandra-2.2.0.jar:2.2.0] >> at >> org.apache.cassandra.net >> .OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:285) >> [apache-cassandra-2.2.0.jar:2.2.0] >> at >> org.apache.cassandra.net >> .OutboundTcpConnection.run(OutboundTcpConnection.java:219) >> [apache-cassandra-2.2.0.jar:2.2.0] >> WARN [MessagingService-Outgoing-/192.168.1.33 <http://192.168.1.33>] >> 2015-07-22 17:29:49,764 SSLFactory.java:163 - Filtering out >> >> TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA >> as it isnt supported by the socket >> WARN [MessagingService-Outgoing-/192.168.1.31 <http://192.168.1.31>] >> 2015-07-22 17:29:49,764 SSLFactory.java:163 - Filtering out >> >> TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA >> as it isnt supported by the socket >> ERROR [MessagingService-Outgoing-/192.168.1.33 <http://192.168.1.33>] >> 2015-07-22 17:29:49,764 OutboundTcpConnection.java:229 - error >> processing a message intended for /192.168.1.33 <http://192.168.1.33> >> java.lang.NullPointerException: null >> at >> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:213) >> ~[guava-16.0.jar:na] >> at >> >> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.<init>(BufferedDataOutputStreamPlus.java:74) >> ~[apache-cassandra-2.2.0.jar:2.2.0] >> at >> org.apache.cassandra.net >> .OutboundTcpConnection.connect(OutboundTcpConnection.java:404) >> ~[apache-cassandra-2.2.0.jar:2.2.0] >> at >> org.apache.cassandra.net >> .OutboundTcpConnection.run(OutboundTcpConnection.java:218) >> ~[apache-cassandra-2.2.0.jar:2.2.0] >> ERROR [MessagingService-Outgoing-/192.168.1.31 <http://192.168.1.31>] >> 2015-07-22 17:29:49,764 OutboundTcpConnection.java:229 - error >> processing a message intended for /192.168.1.31 <http://192.168.1.31> >> java.lang.NullPointerException: null >> at >> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:213) >> ~[guava-16.0.jar:na] >> at >> >> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.<init>(BufferedDataOutputStreamPlus.java:74) >> ~[apache-cassandra-2.2.0.jar:2.2.0] >> at >> org.apache.cassandra.net >> .OutboundTcpConnection.connect(OutboundTcpConnection.java:404) >> ~[apache-cassandra-2.2.0.jar:2.2.0] >> at >> org.apache.cassandra.net >> .OutboundTcpConnection.run(OutboundTcpConnection.java:218) >> ~[apache-cassandra-2.2.0.jar:2.2.0] >> ERROR [MessagingService-Outgoing-/192.168.1.31 <http://192.168.1.31>] >> 2015-07-22 17:29:50,763 OutboundTcpConnection.java:316 - error writing >> to /192.168.1.31 <http://192.168.1.31> >> java.lang.NullPointerException: null >> at >> org.apache.cassandra.net >> .OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:323) >> [apache-cassandra-2.2.0.jar:2.2.0] >> at >> org.apache.cassandra.net >> .OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:285) >> [apache-cassandra-2.2.0.jar:2.2.0] >> at >> org.apache.cassandra.net >> .OutboundTcpConnection.run(OutboundTcpConnection.java:219) >> [apache-cassandra-2.2.0.jar:2.2.0] >> WARN [MessagingService-Outgoing-/192.168.1.31 <http://192.168.1.31>] >> 2015-07-22 17:29:51,766 SSLFactory.java:163 - Filtering out >> >> TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA >> as it isnt supported by the socket >> ERROR [MessagingService-Outgoing-/192.168.1.31 <http://192.168.1.31>] >> 2015-07-22 17:29:51,767 OutboundTcpConnection.java:229 - error >> processing a message intended for /192.168.1.31 <http://192.168.1.31> >> java.lang.NullPointerException: null >> at >> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:213) >> ~[guava-16.0.jar:na] >> at >> >> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.<init>(BufferedDataOutputStreamPlus.java:74) >> ~[apache-cassandra-2.2.0.jar:2.2.0] >> at >> org.apache.cassandra.net >> .OutboundTcpConnection.connect(OutboundTcpConnection.java:404) >> ~[apache-cassandra-2.2.0.jar:2.2.0] >> at >> org.apache.cassandra.net >> .OutboundTcpConnection.run(OutboundTcpConnection.java:218) >> ~[apache-cassandra-2.2.0.jar:2.2.0] >> ERROR [MessagingService-Outgoing-/192.168.1.33 <http://192.168.1.33>] >> 2015-07-22 17:29:52,764 OutboundTcpConnection.java:316 - error writing >> to /192.168.1.33 <http://192.168.1.33> >> java.lang.NullPointerException: null >> at >> org.apache.cassandra.net >> .OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:323) >> [apache-cassandra-2.2.0.jar:2.2.0] >> at >> org.apache.cassandra.net >> .OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:285) >> [apache-cassandra-2.2.0.jar:2.2.0] >> at >> org.apache.cassandra.net >> .OutboundTcpConnection.run(OutboundTcpConnection.java:219) >> [apache-cassandra-2.2.0.jar:2.2.0] >> ERROR [MessagingService-Outgoing-/192.168.1.31 <http://192.168.1.31>] >> 2015-07-22 17:29:52,764 OutboundTcpConnection.java:316 - error writing >> to /192.168.1.31 <http://192.168.1.31> >> java.lang.NullPointerException: null >> at >> org.apache.cassandra.net >> .OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:323) >> [apache-cassandra-2.2.0.jar:2.2.0] >> at >> org.apache.cassandra.net >> .OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:285) >> [apache-cassandra-2.2.0.jar:2.2.0] >> at >> org.apache.cassandra.net >> .OutboundTcpConnection.run(OutboundTcpConnection.java:219) >> [apache-cassandra-2.2.0.jar:2.2.0] >> WARN [MessagingService-Outgoing-/192.168.1.31 <http://192.168.1.31>] >> 2015-07-22 17:29:53,767 SSLFactory.java:163 - Filtering out >> >> TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA >> as it isnt supported by the socket >> ERROR [MessagingService-Outgoing-/192.168.1.31 <http://192.168.1.31>] >> 2015-07-22 17:29:53,767 OutboundTcpConnection.java:229 - error >> processing a message intended for /192.168.1.31 <http://192.168.1.31> >> java.lang.NullPointerException: null >> at >> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:213) >> ~[guava-16.0.jar:na] >> at >> >> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.<init>(BufferedDataOutputStreamPlus.java:74) >> ~[apache-cassandra-2.2.0.jar:2.2.0] >> at >> org.apache.cassandra.net >> .OutboundTcpConnection.connect(OutboundTcpConnection.java:404) >> ~[apache-cassandra-2.2.0.jar:2.2.0] >> at >> org.apache.cassandra.net >> .OutboundTcpConnection.run(OutboundTcpConnection.java:218) >> >> ~[apache-cassandra-2.2.0.jar:2.2.0] >> >> >> On my cassandra.yaml I have the end point snitch: >> >> endpoint_snitch: GossipingPropertyFileSnitch >> >> On the cassandra-rackdc.properties >> # Licensed to the Apache Software Foundation (ASF) under one >> # or more contributor license agreements. See the NOTICE file >> # distributed with this work for additional information >> # regarding copyright ownership. The ASF licenses this file >> # to you under the Apache License, Version 2.0 (the >> # "License"); you may not use this file except in compliance >> # with the License. You may obtain a copy of the License at >> # >> # http://www.apache.org/licenses/LICENSE-2.0 >> # >> # Unless required by applicable law or agreed to in writing, software >> # distributed under the License is distributed on an "AS IS" BASIS, >> # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. >> # See the License for the specific language governing permissions and >> # limitations under the License. >> >> # These properties are used with GossipingPropertyFileSnitch and will >> # indicate the rack and dc for this node >> dc=DC1 >> rack=RAC1 >> >> # Add a suffix to a datacenter name. Used by the Ec2Snitch and >> Ec2MultiRegionSnitch >> # to append a string to the EC2 region name. >> #dc_suffix= >> >> # Uncomment the following line to make this snitch prefer the internal >> ip when possible, as the Ec2MultiRegionSnitch does. >> # prefer_local=true >> >> On my cassandra-topology.properties >> >> # Licensed to the Apache Software Foundation (ASF) under one >> # or more contributor license agreements. See the NOTICE file >> # distributed with this work for additional information >> # regarding copyright ownership. The ASF licenses this file >> # to you under the Apache License, Version 2.0 (the >> # "License"); you may not use this file except in compliance >> # with the License. You may obtain a copy of the License at >> # >> # http://www.apache.org/licenses/LICENSE-2.0 >> # >> # Unless required by applicable law or agreed to in writing, software >> # distributed under the License is distributed on an "AS IS" BASIS, >> # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. >> # See the License for the specific language governing permissions and >> # limitations under the License. >> >> # Cassandra Node IP=Data Center:Rack >> 192.168.1.31=DC1:RAC1 >> 192.168.1.32=DC1:RAC1 >> 192.168.1.33=DC1:RAC1 >> 192.168.1.34=DC1:RAC1 >> >> # default for unknown nodes >> default=DC1:RAC1 >> >> >> >> >> >> On Wed, Jul 22, 2015 at 5:23 PM, Michael Shuler <mich...@pbandjelly.org >> <mailto:mich...@pbandjelly.org>> wrote: >> >> On 07/22/2015 04:45 PM, Carlos Scheidecker wrote: >> >> I have a 4 node Cassandra system running on 4 Ubuntu boxes. After >> updating to Cassandra 2.2.0 and keeping the same cassandra.yaml >> file, >> the nodes cannot see each other. >> >> >> What version did you upgrade from? >> >> Usually, when upgrading, it is probably a good idea to start with >> the default cassandra.yaml from the new version (2.2.0 in your case) >> and edit the necessary items from your old version; i.e. num_tokens, >> initial_token, listen_address, broadcast_address, etc. You are >> perhaps missing some sort of default setting that 2.2.0 is looking >> for? >> >> When I do a nodetool status it only reports as being up the >> machine >> where I had issue the command. >> >> In other words, all the machines cannot communicate to each >> other any >> longer. Nodetool status behave the same on each machine. >> >> I am trying to debug that, hopefully only something on the >> configuration >> that has changed. >> >> Any ideas? >> >> >> Anything helpful in the system.log on each of your nodes? >> >> Did you follow all the upgrade notes from your previous release to >> 2.2.0? >> >> https://github.com/apache/cassandra/blob/cassandra-2.2.0/NEWS.txt >> >> -- >> Kind regards, >> Michael >> >> >> >
