What version of Cassandra did you upgrade to 2.2.0 *from*?
This would help with looking at config differences, changelogs, etc.
It seems you have some pretty clear SSL connection errors, according to
the logs, which at least helps with seeing why the nodes can't talk to
each other. I'm not terribly familiar with using SSL with Cassandra, but
it seems clear that you have an incorrect server_encryption_options:
cipher_suites: configuration.
--
Kind regards,
Michael
On 07/22/2015 06:33 PM, Carlos Scheidecker wrote:
Thanks for the reply, Michael!
Yes, I did followed the upgrade nodes.
I am running Ubuntu Ubuntu 14.04.2 LTS on all and
kernel 3.13.0-57-generic on all.
I have 4 machines: .31, .32, .33 and .34. If I run nodetool status from
.34 I now see all the others as DN the same happens if I log in from the
others:
DN 192.168.1.31 ? 256 ?
1f8000f5-026c-42c7-8189-cf19fbede566 RAC1
DN 192.168.1.32 ? 256 ?
12478d45-3d5e-418b-a0dc-dba6d4307af3 RAC1
DN 192.168.1.33 ? 256 ?
994172b3-cd36-4558-a4b8-054cfac027f3 RAC1
UN 192.168.1.34 1.7 MB 256 ?
b66be1f3-bb4a-49bd-9835-5c8ee2a71e5c RAC1
If I do a netstat -atn from .34 I get:
tcp 0 0 127.0.1.1:53 <http://127.0.1.1:53>
0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:22 <http://0.0.0.0:22>
0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:631 <http://127.0.0.1:631>
0.0.0.0:* LISTEN
tcp 0 0 192.168.1.34:7001 <http://192.168.1.34:7001>
0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:7199 <http://127.0.0.1:7199>
0.0.0.0:* LISTEN
tcp 0 0 192.168.1.34:9160 <http://192.168.1.34:9160>
0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:59441 <http://127.0.0.1:59441>
0.0.0.0:* LISTEN
tcp 0 0 192.168.1.34:52951 <http://192.168.1.34:52951>
192.168.1.31:7001 <http://192.168.1.31:7001> ESTABLISHED
On the logs I now have the following errors (/var/log/syslog.log):
WARN [MessagingService-Outgoing-/192.168.1.31 <http://192.168.1.31>]
2015-07-22 17:29:48,764 SSLFactory.java:163 - Filtering out
TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
as it isnt supported by the socket
ERROR [MessagingService-Outgoing-/192.168.1.31 <http://192.168.1.31>]
2015-07-22 17:29:48,764 OutboundTcpConnection.java:229 - error
processing a message intended for /192.168.1.31 <http://192.168.1.31>
java.lang.NullPointerException: null
at
com.google.common.base.Preconditions.checkNotNull(Preconditions.java:213)
~[guava-16.0.jar:na]
at
org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.<init>(BufferedDataOutputStreamPlus.java:74)
~[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.net.OutboundTcpConnection.connect(OutboundTcpConnection.java:404)
~[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:218)
~[apache-cassandra-2.2.0.jar:2.2.0]
ERROR [MessagingService-Outgoing-/192.168.1.31 <http://192.168.1.31>]
2015-07-22 17:29:48,764 OutboundTcpConnection.java:316 - error writing
to /192.168.1.31 <http://192.168.1.31>
java.lang.NullPointerException: null
at
org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:323)
[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:285)
[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:219)
[apache-cassandra-2.2.0.jar:2.2.0]
WARN [MessagingService-Outgoing-/192.168.1.33 <http://192.168.1.33>]
2015-07-22 17:29:49,764 SSLFactory.java:163 - Filtering out
TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
as it isnt supported by the socket
WARN [MessagingService-Outgoing-/192.168.1.31 <http://192.168.1.31>]
2015-07-22 17:29:49,764 SSLFactory.java:163 - Filtering out
TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
as it isnt supported by the socket
ERROR [MessagingService-Outgoing-/192.168.1.33 <http://192.168.1.33>]
2015-07-22 17:29:49,764 OutboundTcpConnection.java:229 - error
processing a message intended for /192.168.1.33 <http://192.168.1.33>
java.lang.NullPointerException: null
at
com.google.common.base.Preconditions.checkNotNull(Preconditions.java:213)
~[guava-16.0.jar:na]
at
org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.<init>(BufferedDataOutputStreamPlus.java:74)
~[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.net.OutboundTcpConnection.connect(OutboundTcpConnection.java:404)
~[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:218)
~[apache-cassandra-2.2.0.jar:2.2.0]
ERROR [MessagingService-Outgoing-/192.168.1.31 <http://192.168.1.31>]
2015-07-22 17:29:49,764 OutboundTcpConnection.java:229 - error
processing a message intended for /192.168.1.31 <http://192.168.1.31>
java.lang.NullPointerException: null
at
com.google.common.base.Preconditions.checkNotNull(Preconditions.java:213)
~[guava-16.0.jar:na]
at
org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.<init>(BufferedDataOutputStreamPlus.java:74)
~[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.net.OutboundTcpConnection.connect(OutboundTcpConnection.java:404)
~[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:218)
~[apache-cassandra-2.2.0.jar:2.2.0]
ERROR [MessagingService-Outgoing-/192.168.1.31 <http://192.168.1.31>]
2015-07-22 17:29:50,763 OutboundTcpConnection.java:316 - error writing
to /192.168.1.31 <http://192.168.1.31>
java.lang.NullPointerException: null
at
org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:323)
[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:285)
[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:219)
[apache-cassandra-2.2.0.jar:2.2.0]
WARN [MessagingService-Outgoing-/192.168.1.31 <http://192.168.1.31>]
2015-07-22 17:29:51,766 SSLFactory.java:163 - Filtering out
TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
as it isnt supported by the socket
ERROR [MessagingService-Outgoing-/192.168.1.31 <http://192.168.1.31>]
2015-07-22 17:29:51,767 OutboundTcpConnection.java:229 - error
processing a message intended for /192.168.1.31 <http://192.168.1.31>
java.lang.NullPointerException: null
at
com.google.common.base.Preconditions.checkNotNull(Preconditions.java:213)
~[guava-16.0.jar:na]
at
org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.<init>(BufferedDataOutputStreamPlus.java:74)
~[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.net.OutboundTcpConnection.connect(OutboundTcpConnection.java:404)
~[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:218)
~[apache-cassandra-2.2.0.jar:2.2.0]
ERROR [MessagingService-Outgoing-/192.168.1.33 <http://192.168.1.33>]
2015-07-22 17:29:52,764 OutboundTcpConnection.java:316 - error writing
to /192.168.1.33 <http://192.168.1.33>
java.lang.NullPointerException: null
at
org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:323)
[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:285)
[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:219)
[apache-cassandra-2.2.0.jar:2.2.0]
ERROR [MessagingService-Outgoing-/192.168.1.31 <http://192.168.1.31>]
2015-07-22 17:29:52,764 OutboundTcpConnection.java:316 - error writing
to /192.168.1.31 <http://192.168.1.31>
java.lang.NullPointerException: null
at
org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:323)
[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:285)
[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:219)
[apache-cassandra-2.2.0.jar:2.2.0]
WARN [MessagingService-Outgoing-/192.168.1.31 <http://192.168.1.31>]
2015-07-22 17:29:53,767 SSLFactory.java:163 - Filtering out
TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
as it isnt supported by the socket
ERROR [MessagingService-Outgoing-/192.168.1.31 <http://192.168.1.31>]
2015-07-22 17:29:53,767 OutboundTcpConnection.java:229 - error
processing a message intended for /192.168.1.31 <http://192.168.1.31>
java.lang.NullPointerException: null
at
com.google.common.base.Preconditions.checkNotNull(Preconditions.java:213)
~[guava-16.0.jar:na]
at
org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.<init>(BufferedDataOutputStreamPlus.java:74)
~[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.net.OutboundTcpConnection.connect(OutboundTcpConnection.java:404)
~[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:218)
~[apache-cassandra-2.2.0.jar:2.2.0]
On my cassandra.yaml I have the end point snitch:
endpoint_snitch: GossipingPropertyFileSnitch
On the cassandra-rackdc.properties
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# These properties are used with GossipingPropertyFileSnitch and will
# indicate the rack and dc for this node
dc=DC1
rack=RAC1
# Add a suffix to a datacenter name. Used by the Ec2Snitch and
Ec2MultiRegionSnitch
# to append a string to the EC2 region name.
#dc_suffix=
# Uncomment the following line to make this snitch prefer the internal
ip when possible, as the Ec2MultiRegionSnitch does.
# prefer_local=true
On my cassandra-topology.properties
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Cassandra Node IP=Data Center:Rack
192.168.1.31=DC1:RAC1
192.168.1.32=DC1:RAC1
192.168.1.33=DC1:RAC1
192.168.1.34=DC1:RAC1
# default for unknown nodes
default=DC1:RAC1
On Wed, Jul 22, 2015 at 5:23 PM, Michael Shuler <mich...@pbandjelly.org
<mailto:mich...@pbandjelly.org>> wrote:
On 07/22/2015 04:45 PM, Carlos Scheidecker wrote:
I have a 4 node Cassandra system running on 4 Ubuntu boxes. After
updating to Cassandra 2.2.0 and keeping the same cassandra.yaml
file,
the nodes cannot see each other.
What version did you upgrade from?
Usually, when upgrading, it is probably a good idea to start with
the default cassandra.yaml from the new version (2.2.0 in your case)
and edit the necessary items from your old version; i.e. num_tokens,
initial_token, listen_address, broadcast_address, etc. You are
perhaps missing some sort of default setting that 2.2.0 is looking for?
When I do a nodetool status it only reports as being up the machine
where I had issue the command.
In other words, all the machines cannot communicate to each
other any
longer. Nodetool status behave the same on each machine.
I am trying to debug that, hopefully only something on the
configuration
that has changed.
Any ideas?
Anything helpful in the system.log on each of your nodes?
Did you follow all the upgrade notes from your previous release to
2.2.0?
https://github.com/apache/cassandra/blob/cassandra-2.2.0/NEWS.txt
--
Kind regards,
Michael
