OK. I upgraded one node from 1.1.6 to 1.2.2 today. Despite some new
problems that I had and I posted them in a separate email, this issue still
exists but now it is only on 1.2.2 node. This means that the nodes running
1.1.6 see all other nodes including 1.2.2 as Up. Here is the ring and
gossip from nodes with 1.1.6 for example. Bold denotes upgraded node:
Address DC Rack Status State Load
Effective-Ownership Token
141784319550391026443072753098378663700
XX.180.36 us-east 1b Up Normal 49.47 GB 25.00%
1808575600
*XX.231.121 us-east 1c Up Normal 47.08 GB 25.00%
7089215977519551322153637656637080005*
XX.177.177 us-east 1d Up Normal 33.64 GB 25.00%
14178431955039102644307275311465584410
XX.7.148 us-east 1b Up Normal 41.27 GB 25.00%
42535295865117307932921825930779602030
XX.20.9 us-east 1c Up Normal 38.51 GB 25.00%
49624511842636859255075463585608106435
XX.86.255 us-east 1d Up Normal 34.78 GB 25.00%
56713727820156410577229101240436610840
XX.63.230 us-east 1b Up Normal 38.11 GB 25.00%
85070591730234615865843651859750628460
XX.163.36 us-east 1c Up Normal 44.25 GB 25.00%
92159807707754167187997289514579132865
XX.31.234 us-east 1d Up Normal 44.66 GB 25.00%
99249023685273718510150927169407637270
XX.132.169 us-east 1b Up Normal 44.2 GB 25.00%
127605887595351923798765477788721654890
XX.71.63 us-east 1c Up Normal 38.74 GB 25.00%
134695103572871475120919115443550159295
XX.197.209 us-east 1d Up Normal 41.5 GB 25.00%
141784319550391026443072753098378663700
/XX.71.63
RACK:1c
SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
LOAD:4.1598705272E10
DC:us-east
INTERNAL_IP:XX.194.92
STATUS:NORMAL,134695103572871475120919115443550159295
RPC_ADDRESS:XX.194.92
RELEASE_VERSION:1.1.6
/XX.86.255
RACK:1d
SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
LOAD:3.734334162E10
DC:us-east
INTERNAL_IP:XX.6.195
STATUS:NORMAL,56713727820156410577229101240436610840
RPC_ADDRESS:XX.6.195
RELEASE_VERSION:1.1.6
/XX.7.148
RACK:1b
SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
LOAD:4.4316975808E10
DC:us-east
INTERNAL_IP:XX.47.250
STATUS:NORMAL,42535295865117307932921825930779602030
RPC_ADDRESS:XX.47.250
RELEASE_VERSION:1.1.6
/XX.63.230
RACK:1b
SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
LOAD:4.0918593305E10
DC:us-east
INTERNAL_IP:XX.89.127
STATUS:NORMAL,85070591730234615865843651859750628460
RPC_ADDRESS:XX.89.127
RELEASE_VERSION:1.1.6
/XX.132.169
RACK:1b
SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
LOAD:4.745883458E10
DC:us-east
INTERNAL_IP:XX.94.161
STATUS:NORMAL,127605887595351923798765477788721654890
RPC_ADDRESS:XX.94.161
RELEASE_VERSION:1.1.6
/XX.180.36
RACK:1b
SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
LOAD:5.311963027E10
DC:us-east
INTERNAL_IP:XX.123.112
STATUS:NORMAL,1808575600
RPC_ADDRESS:XX.123.112
RELEASE_VERSION:1.1.6
/XX.163.36
RACK:1c
SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
LOAD:4.7516755022E10
DC:us-east
INTERNAL_IP:XX.163.180
STATUS:NORMAL,92159807707754167187997289514579132865
RPC_ADDRESS:XX.163.180
RELEASE_VERSION:1.1.6
/XX.31.234
RACK:1d
SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
LOAD:4.7954372912E10
DC:us-east
INTERNAL_IP:XX.192.159
STATUS:NORMAL,99249023685273718510150927169407637270
RPC_ADDRESS:XX.192.159
RELEASE_VERSION:1.1.6
/XX.197.209
RACK:1d
SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
LOAD:4.4558968005E10
DC:us-east
INTERNAL_IP:XX.66.205
STATUS:NORMAL,141784319550391026443072753098378663700
RPC_ADDRESS:XX.66.205
RELEASE_VERSION:1.1.6
/XX.177.177
RACK:1d
SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
LOAD:3.6115572697E10
DC:us-east
INTERNAL_IP:XX.65.57
STATUS:NORMAL,14178431955039102644307275311465584410
RPC_ADDRESS:XX.65.57
RELEASE_VERSION:1.1.6
/XX.20.9
RACK:1c
SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
LOAD:4.1352503882E10
DC:us-east
INTERNAL_IP:XX.33.229
STATUS:NORMAL,49624511842636859255075463585608106435
RPC_ADDRESS:XX.33.229
RELEASE_VERSION:1.1.6
*/XX.231.121*
* RACK:1c*
* SCHEMA:09487aa5-3380-33ab-b9a5-bcc8476066b0*
* X4:9c765678-d058-4d85-a588-638ce10ff984*
* X3:7*
* DC:us-east*
* INTERNAL_IP:XX.223.241*
* RPC_ADDRESS:XX.223.241*
* RELEASE_VERSION:1.2.2*
Now the nodetool on the 1.2.2 node shows all nodes as Down but itself.
Gossipinfo looks gook though:
Datacenter: us-east
==========
Replicas: 3
Address Rack Status State Load Owns
Token
56713727820156410577229101240436610840
XX.132.169 1b Down Normal 44.2 GB 25.00%
127605887595351923798765477788721654890
XX.7.148 1b Down Normal 41.27 GB 25.00%
42535295865117307932921825930779602030
XX.180.36 1b Down Normal 49.47 GB 25.00%
1808575600
XX.63.230 1b Down Normal 38.11 GB 25.00%
85070591730234615865843651859750628460
*XX.231.121 1c Up Normal 47.25 GB 25.00%
7089215977519551322153637656637080005*
XX.71.63 1c Down Normal 38.74 GB 25.00%
134695103572871475120919115443550159295
XX.177.177 1d Down Normal 33.64 GB 25.00%
14178431955039102644307275311465584410
XX.31.234 1d Down Normal 44.66 GB 25.00%
99249023685273718510150927169407637270
XX.20.9 1c Down Normal 38.51 GB 25.00%
49624511842636859255075463585608106435
XX.163.36 1c Down Normal 44.25 GB 25.00%
92159807707754167187997289514579132865
XX.197.209 1d Down Normal 41.5 GB 25.00%
141784319550391026443072753098378663700
XX.86.255 1d Down Normal 34.78 GB 25.00%
56713727820156410577229101240436610840
/XX.71.63
RACK:1c
RPC_ADDRESS:XX.194.92
RELEASE_VERSION:1.1.6
INTERNAL_IP:XX.194.92
STATUS:NORMAL,134695103572871475120919115443550159295
SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
DC:us-east
LOAD:4.1598705272E10
/XX.86.255
RACK:1d
RPC_ADDRESS:XX.6.195
RELEASE_VERSION:1.1.6
INTERNAL_IP:XX.6.195
STATUS:NORMAL,56713727820156410577229101240436610840
SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
DC:us-east
LOAD:3.7343205002E10
/XX.7.148
RACK:1b
RPC_ADDRESS:XX.47.250
RELEASE_VERSION:1.1.6
INTERNAL_IP:XX.47.250
STATUS:NORMAL,42535295865117307932921825930779602030
SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
DC:us-east
LOAD:4.4316975808E10
/XX.63.230
RACK:1b
RPC_ADDRESS:XX.89.127
RELEASE_VERSION:1.1.6
INTERNAL_IP:XX.89.127
STATUS:NORMAL,85070591730234615865843651859750628460
SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
DC:us-east
LOAD:4.0918456687E10
/XX.132.169
RACK:1b
RPC_ADDRESS:XX.94.161
RELEASE_VERSION:1.1.6
INTERNAL_IP:XX.94.161
STATUS:NORMAL,127605887595351923798765477788721654890
SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
DC:us-east
LOAD:4.745883458E10
/XX.180.36
RACK:1b
RPC_ADDRESS:XX.123.112
RELEASE_VERSION:1.1.6
INTERNAL_IP:XX.123.112
STATUS:NORMAL,1808575600
SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
DC:us-east
LOAD:5.311963027E10
/XX.163.36
RACK:1c
RPC_ADDRESS:XX.163.180
RELEASE_VERSION:1.1.6
INTERNAL_IP:XX.163.180
STATUS:NORMAL,92159807707754167187997289514579132865
SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
DC:us-east
LOAD:4.7516755022E10
/XX.31.234
RACK:1d
RPC_ADDRESS:XX.192.159
RELEASE_VERSION:1.1.6
INTERNAL_IP:XX.192.159
STATUS:NORMAL,99249023685273718510150927169407637270
SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
DC:us-east
LOAD:4.7954372912E10
/XX.197.209
RACK:1d
RPC_ADDRESS:XX.66.205
RELEASE_VERSION:1.1.6
INTERNAL_IP:XX.66.205
STATUS:NORMAL,141784319550391026443072753098378663700
SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
DC:us-east
LOAD:4.4559013211E10
/XX.177.177
RACK:1d
RPC_ADDRESS:XX.65.57
RELEASE_VERSION:1.1.6
INTERNAL_IP:XX.65.57
STATUS:NORMAL,14178431955039102644307275311465584410
SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
DC:us-east
LOAD:3.6115572697E10
/XX.20.9
RACK:1c
RPC_ADDRESS:XX.33.229
RELEASE_VERSION:1.1.6
INTERNAL_IP:XX.33.229
STATUS:NORMAL,49624511842636859255075463585608106435
SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
DC:us-east
LOAD:4.1352367264E10
*/XX.231.121*
* HOST_ID:9c765678-d058-4d85-a588-638ce10ff984*
* RACK:1c*
* RPC_ADDRESS:XX.223.241*
* RELEASE_VERSION:1.2.2*
* INTERNAL_IP:XX.223.241*
* STATUS:NORMAL,7089215977519551322153637656637080005*
* NET_VERSION:7*
* SCHEMA:8b8948f5-d56f-3a96-8005-b9452e42cd67*
* SEVERITY:0.0*
* DC:us-east*
* LOAD:5.0710624207E10*
Is this just a display bug in nodetool or this upgraded node really sees
the other ones as dead?
-Arya
On Mon, Feb 25, 2013 at 8:10 PM, Arya Goudarzi <gouda...@gmail.com> wrote:
> No I did not look at nodetool gossipinfo but from the ring on both
> pre-upgrade and post upgrade nodes to 1.2.1, what I observed was the
> described behavior.
>
>
> On Sat, Feb 23, 2013 at 1:26 AM, Michael Kjellman <mkjell...@barracuda.com
> > wrote:
>
>> This was a bug with 1.2.0 but resolved in 1.2.1. Did you take a capture
>> of nodetool gossipinfo and nodetool ring by chance?
>>
>> On Feb 23, 2013, at 12:26 AM, "Arya Goudarzi" <gouda...@gmail.com> wrote:
>>
>> > Hi C* users,
>> >
>> > I just upgrade a 12 node test cluster from 1.1.6 to 1.2.1. What I
>> noticed from nodetool ring was that the new upgraded nodes only saw each
>> other as Normal and the rest of the cluster which was on 1.1.6 as Down.
>> Vise versa was true for the nodes running 1.1.6. They saw each other as
>> Normal but the 1.2.1 nodes as down. I don't see a note in upgrade docs that
>> this would be an issue. Has anyone else observed this problem?
>> >
>> > In the debug logs I could see messages saying attempting to connect to
>> node IP and then saying it is down.
>> >
>> > Cheers,
>> > -Arya
>>
>> Copy, by Barracuda, helps you store, protect, and share all your amazing
>>
>> things. Start today: www.copy.com.
>>
>
>
