Paulo,
I can confirm that the problem is as you stated. Some or all of the other
nodes are keeping a connection in CLOSE_WAIT state. Those nodes are seen as
DN from the point of the node I have restarted the Cassandra service on.
But nodetool disablegossip did not fix the problem.
This sounds like an issue that can potentially affect many users. Is it not
the case?
Do we have a solution for this?
--------
Here is netstat and nodetool status output
1. right after stopping cassandra service on 10.4.68.222:
--------------------------------------
ip-10-4-54-176
tcp 0 0 10.4.54.176:51268 10.4.68.222:7000
TIME_WAIT
tcp 0 0 10.4.54.176:56135 10.4.68.222:7000
TIME_WAIT
tcp 1 0 10.4.54.176:43697 10.4.68.222:7000
CLOSE_WAIT
tcp 0 0 10.4.54.176:52372 10.4.68.222:7000
TIME_WAIT
--------------------------------------
--------------------------------------
ip-10-4-54-177
tcp 0 0 10.4.54.177:56960 10.4.68.222:7000
TIME_WAIT
tcp 0 0 10.4.54.177:54539 10.4.68.222:7000
TIME_WAIT
tcp 0 0 10.4.54.177:32823 10.4.68.222:7000
TIME_WAIT
tcp 1 0 10.4.54.177:48985 10.4.68.222:7000
CLOSE_WAIT
--------------------------------------
--------------------------------------
ip-10-4-68-222
tcp 0 0 10.4.68.222:7000 10.4.54.176:43697
FIN_WAIT2
tcp 0 0 10.4.68.222:7000 10.4.54.177:48985
FIN_WAIT2
tcp 0 0 10.4.68.222:7000 10.4.68.222:54419
TIME_WAIT
tcp 0 0 10.4.68.222:7000 10.4.43.65:43197
FIN_WAIT2
tcp 0 0 10.4.68.222:7000 10.4.68.221:44149
FIN_WAIT2
tcp 0 0 10.4.68.222:7000 10.4.68.222:41302
TIME_WAIT
tcp 0 0 10.4.68.222:7000 10.4.43.66:54321
FIN_WAIT2
--------------------------------------
--------------------------------------
ip-10-4-68-221
tcp 0 0 10.4.68.221:49599 10.4.68.222:7000
TIME_WAIT
tcp 0 0 10.4.68.221:55033 10.4.68.222:7000
TIME_WAIT
tcp 0 0 10.4.68.221:51628 10.4.68.222:7000
TIME_WAIT
tcp 1 0 10.4.68.221:44149 10.4.68.222:7000
CLOSE_WAIT
--------------------------------------
--------------------------------------
ip-10-4-43-66
tcp 0 0 10.4.43.66:55930 10.4.68.222:7000
TIME_WAIT
tcp 1 0 10.4.43.66:54321 10.4.68.222:7000
CLOSE_WAIT
tcp 0 0 10.4.43.66:60968 10.4.68.222:7000
TIME_WAIT
tcp 0 0 10.4.43.66:49087 10.4.68.222:7000
TIME_WAIT
--------------------------------------
--------------------------------------
ip-10-4-43-65
tcp 1 0 10.4.43.65:43197 10.4.68.222:7000
CLOSE_WAIT
tcp 0 0 10.4.43.65:36467 10.4.68.222:7000
TIME_WAIT
tcp 0 0 10.4.43.65:53317 10.4.68.222:7000
TIME_WAIT
tcp 0 0 10.4.43.65:54897 10.4.68.222:7000
TIME_WAIT
--------------------------------------
2. a bit after stopping cassandra service on 10.4.68.222:
--------------------------------------
ip-10-4-54-176
tcp 1 0 10.4.54.176:43697 10.4.68.222:7000
CLOSE_WAIT
--------------------------------------
--------------------------------------
ip-10-4-54-177
--------------------------------------
--------------------------------------
ip-10-4-68-222
--------------------------------------
--------------------------------------
ip-10-4-68-221
tcp 1 0 10.4.68.221:44149 10.4.68.222:7000
CLOSE_WAIT
--------------------------------------
--------------------------------------
ip-10-4-43-66
tcp 1 0 10.4.43.66:54321 10.4.68.222:7000
CLOSE_WAIT
--------------------------------------
--------------------------------------
ip-10-4-43-65
tcp 1 0 10.4.43.65:43197 10.4.68.222:7000
CLOSE_WAIT
--------------------------------------
3. after starting cassandra service on 10.4.68.222:
--------------------------------------
ip-10-4-54-176
tcp 0 0 10.4.54.176:42460 10.4.68.222:7000
ESTABLISHED
tcp 1 303403 10.4.54.176:43697 10.4.68.222:7000
CLOSE_WAIT
tcp 0 0 10.4.54.176:42109 10.4.68.222:7000
ESTABLISHED
--------------------------------------
--------------------------------------
ip-10-4-54-177
tcp 0 0 10.4.54.177:43687 10.4.68.222:7000
ESTABLISHED
tcp 0 0 10.4.54.177:56107 10.4.68.222:7000
ESTABLISHED
tcp 0 0 10.4.54.177:39426 10.4.68.222:7000
ESTABLISHED
--------------------------------------
--------------------------------------
ip-10-4-68-222
tcp 0 0 10.4.68.222:7000 0.0.0.0:*
LISTEN
tcp 0 0 10.4.68.222:7000 10.4.54.176:42109
ESTABLISHED
tcp 0 0 10.4.68.222:7000 10.4.54.177:43687
ESTABLISHED
tcp 0 0 10.4.68.222:7000 10.4.54.176:42460
ESTABLISHED
tcp 0 0 10.4.68.222:7000 10.4.43.66:55168
ESTABLISHED
tcp 0 0 10.4.68.222:7000 10.4.43.65:60239
ESTABLISHED
tcp 0 0 10.4.68.222:7000 10.4.54.177:39426
ESTABLISHED
tcp 0 0 10.4.68.222:7000 10.4.43.65:43480
ESTABLISHED
tcp 0 0 10.4.68.222:7000 10.4.68.221:54490
ESTABLISHED
tcp 0 0 10.4.68.222:7000 10.4.68.221:59771
ESTABLISHED
tcp 0 0 10.4.68.222:7000 10.4.54.177:56107
ESTABLISHED
tcp 0 0 10.4.68.222:7000 10.4.43.66:55581
ESTABLISHED
--------------------------------------
--------------------------------------
ip-10-4-68-221
tcp 0 0 10.4.68.221:54490 10.4.68.222:7000
ESTABLISHED
tcp 0 0 10.4.68.221:59771 10.4.68.222:7000
ESTABLISHED
tcp 1 304316 10.4.68.221:44149 10.4.68.222:7000
CLOSE_WAIT
--------------------------------------
--------------------------------------
ip-10-4-43-66
tcp 1 322344 10.4.43.66:54321 10.4.68.222:7000
CLOSE_WAIT
tcp 0 0 10.4.43.66:55581 10.4.68.222:7000
ESTABLISHED
tcp 0 0 10.4.43.66:55168 10.4.68.222:7000
ESTABLISHED
--------------------------------------
--------------------------------------
ip-10-4-43-65
tcp 1 376331 10.4.43.65:43197 10.4.68.222:7000
CLOSE_WAIT
tcp 0 0 10.4.43.65:43480 10.4.68.222:7000
ESTABLISHED
tcp 0 0 10.4.43.65:60239 10.4.68.222:7000
ESTABLISHED
--------------------------------------
4. nodetool status on all nodes after starting cassandra service on
10.4.68.222:
ip-10-4-54-176
Datacenter: us-east
===================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID
Rack
UN 10.4.54.176 127.67 GB 256 47.5%
7163bf77-2fef-4e33-81c1-0e61038dece1 1b
UN 10.4.43.65 124.19 GB 256 46.2%
80265afb-8beb-4887-a696-fc9b75956894 1a
UN 10.4.54.177 136.06 GB 256 50.7%
b9010e24-4e92-4212-8a17-65892ea9ff66 1b
UN 10.4.43.66 141.94 GB 256 52.3%
b00fdf10-1075-4953-8a96-caf375221684 1a
UN 10.4.68.221 137.12 GB 256 50.7%
37479ec3-7b6d-4537-975c-f9d95e92ee1d 1d
UN 10.4.68.222 141.89 GB 256 52.7%
8df87657-c39b-405a-ba54-d60b577c1429 1d
--------------------------------------
--------------------------------------
ip-10-4-54-177
Datacenter: us-east
===================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID
Rack
UN 10.4.54.176 127.63 GB 256 47.5%
7163bf77-2fef-4e33-81c1-0e61038dece1 1b
UN 10.4.43.65 124.19 GB 256 46.2%
80265afb-8beb-4887-a696-fc9b75956894 1a
UN 10.4.54.177 136.06 GB 256 50.7%
b9010e24-4e92-4212-8a17-65892ea9ff66 1b
UN 10.4.43.66 141.94 GB 256 52.3%
b00fdf10-1075-4953-8a96-caf375221684 1a
UN 10.4.68.221 137.12 GB 256 50.7%
37479ec3-7b6d-4537-975c-f9d95e92ee1d 1d
UN 10.4.68.222 141.89 GB 256 52.7%
8df87657-c39b-405a-ba54-d60b577c1429 1d
------------------------
--------------------------------------
ip-10-4-68-222
Datacenter: us-east
===================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID
Rack
DN 10.4.54.176 127.63 GB 256 47.5%
7163bf77-2fef-4e33-81c1-0e61038dece1 1b
DN 10.4.43.65 124.19 GB 256 46.2%
80265afb-8beb-4887-a696-fc9b75956894 1a
UN 10.4.54.177 136.06 GB 256 50.7%
b9010e24-4e92-4212-8a17-65892ea9ff66 1b
DN 10.4.43.66 141.94 GB 256 52.3%
b00fdf10-1075-4953-8a96-caf375221684 1a
DN 10.4.68.221 137.12 GB 256 50.7%
37479ec3-7b6d-4537-975c-f9d95e92ee1d 1d
UN 10.4.68.222 141.89 GB 256 52.7%
8df87657-c39b-405a-ba54-d60b577c1429 1d
--------------------------------------
--------------------------------------
ip-10-4-68-221
Datacenter: us-east
===================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID
Rack
UN 10.4.54.176 127.63 GB 256 47.5%
7163bf77-2fef-4e33-81c1-0e61038dece1 1b
UN 10.4.43.65 124.19 GB 256 46.2%
80265afb-8beb-4887-a696-fc9b75956894 1a
UN 10.4.54.177 136.06 GB 256 50.7%
b9010e24-4e92-4212-8a17-65892ea9ff66 1b
UN 10.4.43.66 141.94 GB 256 52.3%
b00fdf10-1075-4953-8a96-caf375221684 1a
UN 10.4.68.221 137.12 GB 256 50.7%
37479ec3-7b6d-4537-975c-f9d95e92ee1d 1d
UN 10.4.68.222 141.89 GB 256 52.7%
8df87657-c39b-405a-ba54-d60b577c1429 1d
--------------------------------------
--------------------------------------
ip-10-4-43-66
Datacenter: us-east
===================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID
Rack
UN 10.4.54.176 127.67 GB 256 47.5%
7163bf77-2fef-4e33-81c1-0e61038dece1 1b
UN 10.4.43.65 124.19 GB 256 46.2%
80265afb-8beb-4887-a696-fc9b75956894 1a
UN 10.4.54.177 136.06 GB 256 50.7%
b9010e24-4e92-4212-8a17-65892ea9ff66 1b
UN 10.4.43.66 141.95 GB 256 52.3%
b00fdf10-1075-4953-8a96-caf375221684 1a
UN 10.4.68.221 137.12 GB 256 50.7%
37479ec3-7b6d-4537-975c-f9d95e92ee1d 1d
UN 10.4.68.222 141.89 GB 256 52.7%
8df87657-c39b-405a-ba54-d60b577c1429 1d
--------------------------------------
--------------------------------------
ip-10-4-43-65
Datacenter: us-east
===================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID
Rack
UN 10.4.54.176 127.67 GB 256 47.5%
7163bf77-2fef-4e33-81c1-0e61038dece1 1b
UN 10.4.43.65 124.19 GB 256 46.2%
80265afb-8beb-4887-a696-fc9b75956894 1a
UN 10.4.54.177 136.06 GB 256 50.7%
b9010e24-4e92-4212-8a17-65892ea9ff66 1b
UN 10.4.43.66 141.94 GB 256 52.3%
b00fdf10-1075-4953-8a96-caf375221684 1a
UN 10.4.68.221 137.12 GB 256 50.7%
37479ec3-7b6d-4537-975c-f9d95e92ee1d 1d
UN 10.4.68.222 141.89 GB 256 52.7%
8df87657-c39b-405a-ba54-d60b577c1429 1d
--------------------------------------
On Wed, Jul 27, 2016 at 2:04 PM, Farzad Panahi <[email protected]>
wrote:
> Thanks Paulo for the reply.
>
> Cassandra version is 3.0.8. I will test what you said and share the
> results.
>
> On Wed, Jul 27, 2016 at 2:01 PM, Paulo Motta <[email protected]>
> wrote:
>
>> This looks somewhat related to CASSANDRA-9630. What is the C* version?
>>
>> Can you check with netstats if other nodes keep connections with the
>> stopped node in the CLOSE_WAIT state? And also if the problem disappears if
>> you run nodetool disablegossip before stopping the node?
>>
>> 2016-07-26 16:54 GMT-03:00 Farzad Panahi <[email protected]>:
>>
>>> I am new to Cassandra and trying to figure out how the cluster behaves
>>> when things go south.
>>>
>>> I have a 6-node cluster, RF=3.
>>>
>>> I stop Cassandra service on a node for a while. All nodes see the node
>>> as DN. After a while I start the Cassandra service on DN. Interesting point
>>> is that all other nodes see the node now as UN but the node itself sees 4
>>> nodes as DN and only one node as UN. After about 10 minutes the node sees
>>> other nodes as up as well.
>>>
>>> I am trying to figure out where this delay is coming from.
>>>
>>> I have attached part of system.log that looks interesting. Looks like
>>> after Gossiper logs InetAddress <ip> is now UP the node is actually seeing
>>> that node as up even though the node has already handshaked with that node
>>> before.
>>>
>>> Any ideas?
>>>
>>> Cheers
>>>
>>> Farzad
>>>
>>> --------------------------------------
>>> INFO [main] 2016-07-25 21:58:46,044 StorageService.java:533 - Cassandra
>>> version: 3.0.8
>>> INFO [main] 2016-07-25 21:58:46,098 StorageService.java:534 - Thrift
>>> API version: 20.1.0
>>> INFO [main] 2016-07-25 21:58:46,150 StorageService.java:535 - CQL
>>> supported versions: 3.4.0 (default: 3.4.0)
>>> INFO [main] 2016-07-25 21:58:46,284 IndexSummaryManager.java:85 -
>>> Initializing index summary manager with a memory pool size of 198 MB and a
>>> resize interval of 60 minutes
>>> INFO [main] 2016-07-25 21:58:46,343 StorageService.java:554 - Loading
>>> persisted ring state
>>> INFO [main] 2016-07-25 21:58:46,418 StorageService.java:743 - Starting
>>> up server gossip
>>> INFO [main] 2016-07-25 21:58:46,680 TokenMetadata.java:429 - Updating
>>> topology for ip-10-4-43-66.ec2.internal/10.4.43.66
>>> INFO [main] 2016-07-25 21:58:46,707 TokenMetadata.java:429 - Updating
>>> topology for ip-10-4-43-66.ec2.internal/10.4.43.66
>>> INFO [main] 2016-07-25 21:58:46,792 MessagingService.java:557 -
>>> Starting Messaging Service on ip-10-4-43-66.ec2.internal/10.4.43.66:7000
>>> (eth0)
>>>
>>> INFO [HANDSHAKE-/10.4.68.222] 2016-07-25 21:58:46,920
>>> OutboundTcpConnection.java:515 - Handshaking version with /10.4.68.222
>>> INFO [GossipStage:1] 2016-07-25 21:58:47,011 Gossiper.java:1028 - Node /
>>> 10.4.68.221 has restarted, now UP
>>> INFO [HANDSHAKE-/10.4.68.222] 2016-07-25 21:58:47,007
>>> OutboundTcpConnection.java:515 - Handshaking version with /10.4.68.222
>>> INFO [main] 2016-07-25 21:58:47,030 StorageService.java:1902 - Node
>>> ip-10-4-43-66.ec2.internal/10.4.43.66 state jump to NORMAL
>>> INFO [main] 2016-07-25 21:58:47,096 CassandraDaemon.java:644 - Waiting
>>> for gossip to settle before accepting client requests...
>>> INFO [GossipStage:1] 2016-07-25 21:58:47,134 StorageService.java:1902 -
>>> Node /10.4.68.221 state jump to NORMAL
>>> INFO [HANDSHAKE-/10.4.68.221] 2016-07-25 21:58:47,137
>>> OutboundTcpConnection.java:515 - Handshaking version with /10.4.68.221
>>> INFO [GossipStage:1] 2016-07-25 21:58:47,211 TokenMetadata.java:429 -
>>> Updating topology for /10.4.68.221
>>> INFO [GossipStage:1] 2016-07-25 21:58:47,261 TokenMetadata.java:429 -
>>> Updating topology for /10.4.68.221
>>> INFO [GossipStage:1] 2016-07-25 21:58:47,295 Gossiper.java:1028 - Node /
>>> 10.4.68.222 has restarted, now UP
>>> INFO [GossipStage:1] 2016-07-25 21:58:47,337 StorageService.java:1902 -
>>> Node /10.4.68.222 state jump to NORMAL
>>> INFO [GossipStage:1] 2016-07-25 21:58:47,385 TokenMetadata.java:429 -
>>> Updating topology for /10.4.68.222
>>> INFO [GossipStage:1] 2016-07-25 21:58:47,452 TokenMetadata.java:429 -
>>> Updating topology for /10.4.68.222
>>> INFO [GossipStage:1] 2016-07-25 21:58:47,497 Gossiper.java:1028 - Node /
>>> 10.4.54.176 has restarted, now UP
>>> INFO [GossipStage:1] 2016-07-25 21:58:47,544 StorageService.java:1902 -
>>> Node /10.4.54.176 state jump to NORMAL
>>> INFO [HANDSHAKE-/10.4.54.176] 2016-07-25 21:58:47,548
>>> OutboundTcpConnection.java:515 - Handshaking version with /10.4.54.176
>>> INFO [GossipStage:1] 2016-07-25 21:58:47,594 TokenMetadata.java:429 -
>>> Updating topology for /10.4.54.176
>>> INFO [GossipStage:1] 2016-07-25 21:58:47,639 TokenMetadata.java:429 -
>>> Updating topology for /10.4.54.176
>>> WARN [GossipTasks:1] 2016-07-25 21:58:47,678 FailureDetector.java:287 -
>>> Not marking nodes down due to local pause of 43226235115 > 5000000000
>>> INFO [HANDSHAKE-/10.4.43.65] 2016-07-25 21:58:47,679
>>> OutboundTcpConnection.java:515 - Handshaking version with /10.4.43.65
>>> INFO [GossipStage:1] 2016-07-25 21:58:47,757 Gossiper.java:1028 - Node /
>>> 10.4.54.177 has restarted, now UP
>>> INFO [GossipStage:1] 2016-07-25 21:58:47,788 StorageService.java:1902 -
>>> Node /10.4.54.177 state jump to NORMAL
>>> INFO [HANDSHAKE-/10.4.54.177] 2016-07-25 21:58:47,789
>>> OutboundTcpConnection.java:515 - Handshaking version with /10.4.54.177
>>> INFO [GossipStage:1] 2016-07-25 21:58:47,836 TokenMetadata.java:429 -
>>> Updating topology for /10.4.54.177
>>> INFO [GossipStage:1] 2016-07-25 21:58:47,887 TokenMetadata.java:429 -
>>> Updating topology for /10.4.54.177
>>> INFO [GossipStage:1] 2016-07-25 21:58:47,926 Gossiper.java:1028 - Node /
>>> 10.4.43.65 has restarted, now UP
>>> INFO [GossipStage:1] 2016-07-25 21:58:47,976 StorageService.java:1902 -
>>> Node /10.4.43.65 state jump to NORMAL
>>> INFO [GossipStage:1] 2016-07-25 21:58:48,036 TokenMetadata.java:429 -
>>> Updating topology for /10.4.43.65
>>> INFO [GossipStage:1] 2016-07-25 21:58:48,076 TokenMetadata.java:429 -
>>> Updating topology for /10.4.43.65
>>> INFO [GossipStage:1] 2016-07-25 21:58:48,122 StorageService.java:1902 -
>>> Node /10.4.43.65 state jump to NORMAL
>>> INFO [HANDSHAKE-/10.4.54.176] 2016-07-25 21:58:48,123
>>> OutboundTcpConnection.java:515 - Handshaking version with /10.4.54.176
>>> INFO [HANDSHAKE-/10.4.68.221] 2016-07-25 21:58:48,123
>>> OutboundTcpConnection.java:515 - Handshaking version with /10.4.68.221
>>> INFO [GossipStage:1] 2016-07-25 21:58:48,165 TokenMetadata.java:429 -
>>> Updating topology for /10.4.43.65
>>> INFO [SharedPool-Worker-1] 2016-07-25 21:58:48,170 Gossiper.java:994 -
>>> InetAddress /10.4.54.177 is now UP
>>> INFO [GossipStage:1] 2016-07-25 21:58:48,198 TokenMetadata.java:429 -
>>> Updating topology for /10.4.43.65
>>> INFO [GossipStage:1] 2016-07-25 21:58:48,278 StorageService.java:1902 -
>>> Node /10.4.54.177 state jump to NORMAL
>>> INFO [GossipStage:1] 2016-07-25 21:58:48,345 TokenMetadata.java:429 -
>>> Updating topology for /10.4.54.177
>>> INFO [GossipStage:1] 2016-07-25 21:58:48,388 TokenMetadata.java:429 -
>>> Updating topology for /10.4.54.177
>>> INFO [HANDSHAKE-/10.4.43.65] 2016-07-25 21:58:48,444
>>> OutboundTcpConnection.java:515 - Handshaking version with /10.4.43.65
>>> INFO [HANDSHAKE-/10.4.54.177] 2016-07-25 21:58:48,444
>>> OutboundTcpConnection.java:515 - Handshaking version with /10.4.54.177
>>> INFO [main] 2016-07-25 21:58:55,155 CassandraDaemon.java:675 - No
>>> gossip backlog; proceeding
>>> INFO [main] 2016-07-25 21:58:55,436 NativeTransportService.java:70 -
>>> Netty using native Epoll event loop
>>> INFO [main] 2016-07-25 21:58:55,594 Server.java:159 - Using Netty
>>> Version: [netty-buffer=netty-buffer-4.0.23.Final.208198c,
>>> netty-codec=netty-codec-4.0.23.Final.208198c,
>>> netty-codec-http=netty-codec-http-4.0.23.Final.208198c,
>>> netty-codec-socks=netty-codec-socks-4.0.23.Final.208198c,
>>> netty-common=netty-common-4.0.23.Final.208198c,
>>> netty-handler=netty-handler-4.0.23.Final.208198c,
>>> netty-transport=netty-transport-4.0.23.Final.208198c,
>>> netty-transport-rxtx=netty-transport-rxtx-4.0.23.Final.208198c,
>>> netty-transport-sctp=netty-transport-sctp-4.0.23.Final.208198c,
>>> netty-transport-udt=netty-transport-udt-4.0.23.Final.208198c]
>>> INFO [main] 2016-07-25 21:58:55,691 Server.java:160 - Starting
>>> listening for CQL clients on /0.0.0.0:9042 (unencrypted)...
>>> INFO [main] 2016-07-25 21:58:55,842 ThriftServer.java:119 - Binding
>>> thrift service to /0.0.0.0:9160
>>> INFO [Thread-3] 2016-07-25 21:58:55,896 ThriftServer.java:136 -
>>> Listening for thrift clients...
>>> INFO [HANDSHAKE-/10.4.54.177] 2016-07-25 21:59:24,753
>>> OutboundTcpConnection.java:515 - Handshaking version with /10.4.54.177
>>> INFO [HANDSHAKE-ip-10-4-43-66.ec2.internal/10.4.43.66] 2016-07-25
>>> 21:59:33,375 OutboundTcpConnection.java:515 - Handshaking version with
>>> ip-10-4-43-66.ec2.internal/10.4.43.66
>>> INFO [SharedPool-Worker-4] 2016-07-25 22:14:19,273 Gossiper.java:994 -
>>> InetAddress /10.4.43.65 is now UP
>>> INFO [SharedPool-Worker-2] 2016-07-25 22:14:19,274 Gossiper.java:994 -
>>> InetAddress /10.4.43.65 is now UP
>>> INFO [SharedPool-Worker-7] 2016-07-25 22:14:19,307 Gossiper.java:994 -
>>> InetAddress /10.4.43.65 is now UP
>>> INFO [SharedPool-Worker-36] 2016-07-25 22:14:19,313 Gossiper.java:994 -
>>> InetAddress /10.4.43.65 is now UP
>>> INFO [SharedPool-Worker-4] 2016-07-25 22:14:19,347 Gossiper.java:994 -
>>> InetAddress /10.4.43.65 is now UP
>>> INFO [SharedPool-Worker-2] 2016-07-25 22:14:19,356 Gossiper.java:994 -
>>> InetAddress /10.4.43.65 is now UP
>>> INFO [SharedPool-Worker-7] 2016-07-25 22:14:19,369 Gossiper.java:994 -
>>> InetAddress /10.4.43.65 is now UP
>>> INFO [SharedPool-Worker-36] 2016-07-25 22:14:19,372 Gossiper.java:994 -
>>> InetAddress /10.4.43.65 is now UP
>>> INFO [SharedPool-Worker-2] 2016-07-25 22:14:19,426 Gossiper.java:994 -
>>> InetAddress /10.4.43.65 is now UP
>>> INFO [SharedPool-Worker-36] 2016-07-25 22:14:19,426 Gossiper.java:994 -
>>> InetAddress /10.4.43.65 is now UP
>>> INFO [SharedPool-Worker-4] 2016-07-25 22:14:19,434 Gossiper.java:994 -
>>> InetAddress /10.4.43.65 is now UP
>>> INFO [SharedPool-Worker-7] 2016-07-25 22:14:19,442 Gossiper.java:994 -
>>> InetAddress /10.4.43.65 is now UP
>>> INFO [SharedPool-Worker-4] 2016-07-25 22:14:19,486 Gossiper.java:994 -
>>> InetAddress /10.4.43.65 is now UP
>>> INFO [SharedPool-Worker-2] 2016-07-25 22:14:19,500 Gossiper.java:994 -
>>> InetAddress /10.4.43.65 is now UP
>>> INFO [SharedPool-Worker-36] 2016-07-25 22:14:19,508 Gossiper.java:994 -
>>> InetAddress /10.4.43.65 is now UP
>>> INFO [SharedPool-Worker-7] 2016-07-25 22:14:19,514 Gossiper.java:994 -
>>> InetAddress /10.4.43.65 is now UP
>>> INFO [SharedPool-Worker-1] 2016-07-25 22:14:19,687 Gossiper.java:994 -
>>> InetAddress /10.4.54.176 is now UP
>>> INFO [SharedPool-Worker-5] 2016-07-25 22:14:19,699 Gossiper.java:994 -
>>> InetAddress /10.4.54.176 is now UP
>>> INFO [SharedPool-Worker-33] 2016-07-25 22:14:19,719 Gossiper.java:994 -
>>> InetAddress /10.4.54.176 is now UP
>>> INFO [SharedPool-Worker-7] 2016-07-25 22:14:19,721 Gossiper.java:994 -
>>> InetAddress /10.4.54.176 is now UP
>>> INFO [SharedPool-Worker-5] 2016-07-25 22:14:19,756 Gossiper.java:994 -
>>> InetAddress /10.4.54.176 is now UP
>>> INFO [SharedPool-Worker-1] 2016-07-25 22:14:19,757 Gossiper.java:994 -
>>> InetAddress /10.4.54.176 is now UP
>>> INFO [SharedPool-Worker-33] 2016-07-25 22:14:19,762 Gossiper.java:994 -
>>> InetAddress /10.4.54.176 is now UP
>>> INFO [SharedPool-Worker-7] 2016-07-25 22:14:19,778 Gossiper.java:994 -
>>> InetAddress /10.4.54.176 is now UP
>>> INFO [SharedPool-Worker-1] 2016-07-25 22:14:19,809 Gossiper.java:994 -
>>> InetAddress /10.4.54.176 is now UP
>>> INFO [SharedPool-Worker-5] 2016-07-25 22:14:19,814 Gossiper.java:994 -
>>> InetAddress /10.4.54.176 is now UP
>>> INFO [SharedPool-Worker-7] 2016-07-25 22:14:19,819 Gossiper.java:994 -
>>> InetAddress /10.4.54.176 is now UP
>>> INFO [SharedPool-Worker-33] 2016-07-25 22:14:19,822 Gossiper.java:994 -
>>> InetAddress /10.4.54.176 is now UP
>>> INFO [SharedPool-Worker-1] 2016-07-25 22:14:19,872 Gossiper.java:994 -
>>> InetAddress /10.4.54.176 is now UP
>>> INFO [SharedPool-Worker-5] 2016-07-25 22:14:19,874 Gossiper.java:994 -
>>> InetAddress /10.4.54.176 is now UP
>>> INFO [SharedPool-Worker-7] 2016-07-25 22:14:19,879 Gossiper.java:994 -
>>> InetAddress /10.4.54.176 is now UP
>>> INFO [SharedPool-Worker-33] 2016-07-25 22:14:22,489 Gossiper.java:994 -
>>> InetAddress /10.4.68.222 is now UP
>>> INFO [SharedPool-Worker-36] 2016-07-25 22:14:22,491 Gossiper.java:994 -
>>> InetAddress /10.4.68.222 is now UP
>>> INFO [SharedPool-Worker-34] 2016-07-25 22:14:22,491 Gossiper.java:994 -
>>> InetAddress /10.4.68.222 is now UP
>>> INFO [SharedPool-Worker-35] 2016-07-25 22:14:22,491 Gossiper.java:994 -
>>> InetAddress /10.4.68.222 is now UP
>>> INFO [SharedPool-Worker-36] 2016-07-25 22:14:22,533 Gossiper.java:994 -
>>> InetAddress /10.4.68.222 is now UP
>>> INFO [SharedPool-Worker-34] 2016-07-25 22:14:22,549 Gossiper.java:994 -
>>> InetAddress /10.4.68.222 is now UP
>>> INFO [SharedPool-Worker-33] 2016-07-25 22:14:22,570 Gossiper.java:994 -
>>> InetAddress /10.4.68.222 is now UP
>>> INFO [SharedPool-Worker-35] 2016-07-25 22:14:22,574 Gossiper.java:994 -
>>> InetAddress /10.4.68.222 is now UP
>>> INFO [SharedPool-Worker-36] 2016-07-25 22:14:22,637 Gossiper.java:994 -
>>> InetAddress /10.4.68.222 is now UP
>>> INFO [SharedPool-Worker-34] 2016-07-25 22:14:22,655 Gossiper.java:994 -
>>> InetAddress /10.4.68.222 is now UP
>>> INFO [SharedPool-Worker-33] 2016-07-25 22:14:22,672 Gossiper.java:994 -
>>> InetAddress /10.4.68.222 is now UP
>>> INFO [SharedPool-Worker-35] 2016-07-25 22:14:22,685 Gossiper.java:994 -
>>> InetAddress /10.4.68.222 is now UP
>>> INFO [SharedPool-Worker-36] 2016-07-25 22:14:22,703 Gossiper.java:994 -
>>> InetAddress /10.4.68.222 is now UP
>>> INFO [SharedPool-Worker-33] 2016-07-25 22:14:22,710 Gossiper.java:994 -
>>> InetAddress /10.4.68.222 is now UP
>>> INFO [SharedPool-Worker-34] 2016-07-25 22:14:22,725 Gossiper.java:994 -
>>> InetAddress /10.4.68.222 is now UP
>>> INFO [SharedPool-Worker-35] 2016-07-25 22:14:22,740 Gossiper.java:994 -
>>> InetAddress /10.4.68.221 is now UP
>>> INFO [SharedPool-Worker-36] 2016-07-25 22:14:22,748 Gossiper.java:994 -
>>> InetAddress /10.4.68.221 is now UP
>>> INFO [SharedPool-Worker-33] 2016-07-25 22:14:22,758 Gossiper.java:994 -
>>> InetAddress /10.4.68.221 is now UP
>>> INFO [SharedPool-Worker-34] 2016-07-25 22:14:22,764 Gossiper.java:994 -
>>> InetAddress /10.4.68.221 is now UP
>>> INFO [SharedPool-Worker-36] 2016-07-25 22:14:22,788 Gossiper.java:994 -
>>> InetAddress /10.4.68.221 is now UP
>>> INFO [SharedPool-Worker-35] 2016-07-25 22:14:22,806 Gossiper.java:994 -
>>> InetAddress /10.4.68.221 is now UP
>>> INFO [SharedPool-Worker-33] 2016-07-25 22:14:22,808 Gossiper.java:994 -
>>> InetAddress /10.4.68.221 is now UP
>>> INFO [SharedPool-Worker-34] 2016-07-25 22:14:22,836 Gossiper.java:994 -
>>> InetAddress /10.4.68.221 is now UP
>>> INFO [SharedPool-Worker-36] 2016-07-25 22:14:22,843 Gossiper.java:994 -
>>> InetAddress /10.4.68.221 is now UP
>>> INFO [SharedPool-Worker-33] 2016-07-25 22:14:22,849 Gossiper.java:994 -
>>> InetAddress /10.4.68.221 is now UP
>>> INFO [SharedPool-Worker-35] 2016-07-25 22:14:22,862 Gossiper.java:994 -
>>> InetAddress /10.4.68.221 is now UP
>>> INFO [SharedPool-Worker-33] 2016-07-25 22:14:22,892 Gossiper.java:994 -
>>> InetAddress /10.4.68.221 is now UP
>>> INFO [SharedPool-Worker-34] 2016-07-25 22:14:22,900 Gossiper.java:994 -
>>> InetAddress /10.4.68.221 is now UP
>>> INFO [SharedPool-Worker-36] 2016-07-25 22:14:22,904 Gossiper.java:994 -
>>> InetAddress /10.4.68.221 is now UP
>>> INFO [SharedPool-Worker-35] 2016-07-25 22:14:22,944 Gossiper.java:994 -
>>> InetAddress /10.4.68.221 is now UP
>>> INFO [SharedPool-Worker-33] 2016-07-25 22:14:22,945 Gossiper.java:994 -
>>> InetAddress /10.4.68.221 is now UP
>>>
>>>
>>
>>
>
