Thanks Jonathan, Utkarsh and Jeff.
We will try to find a way for our Mesos framework to support upgrading the
nodes in-place.
On Mon, Oct 10, 2016 at 5:11 PM, Jonathan Haddad <j...@jonhaddad.com> wrote:
> During the upgrade you'll want to avoid the following operations that
> result in data streaming:
>
> 1. Bootstrapping nodes
> 2. Decomissioning nodes
> 3. Repair
>
>
> On Mon, Oct 10, 2016 at 5:00 PM Jeff Jirsa <jeff.ji...@crowdstrike.com>
> wrote:
>
>>
>>
>> No need to cc dev@, user@ is the right list for this question.
>>
>>
>>
>> As Jon mentioned, you can’t stream (bootstrap/rebuild/repair) across
>> major versions, so don’t try to destroy the cluster – just upgrade in
>> place. It IS a good idea to do one DC at a time, but an in-place upgrade is
>> pretty straightforward – flush, drain, stop Cassandra, replace binaries,
>> start Cassandra, run nodetool upgradesstables -a.
>>
>>
>>
>> Note that you can run nodetool upgradesstables on more than one node at a
>> time if you can tolerate the hit to your read latencies.
>>
>>
>>
>> It IS common, I imagine, for there to be schema mismatches temporarily
>> while you have a mixed version cluster – this isn’t necessarily a huge
>> problem, but do try to get to 3.0.8 as quickly as possible once you start,
>> and if you can avoid administrative tasks (such as those that will change
>> the schema) during the process, that’s generally advisable.
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> *From: *Abhishek Verma <ve...@uber.com>
>> *Reply-To: *"user@cassandra.apache.org" <user@cassandra.apache.org>
>> *Date: *Monday, October 10, 2016 at 4:34 PM
>> *To: *"user@cassandra.apache.org" <user@cassandra.apache.org>, "
>> d...@cassandra.apache.org" <d...@cassandra.apache.org>
>> *Subject: *Bootstrapping data from Cassandra 2.2.5 datacenter to 3.0.8
>> datacenter fails because of streaming errors
>>
>>
>>
>> Hi Cassandra users,
>>
>>
>>
>> We are trying to upgrade our Cassandra version from 2.2.5 to 3.0.8
>> (running on Mesos, but that's besides the point). We have two datacenters,
>> so in order to preserve our data, we are trying to upgrade one datacenter
>> at a time.
>>
>>
>>
>> Initially both DCs (dc1 and dc2) are running 2.2.5. The idea is to tear
>> down dc1 completely (delete all the data in it), bring it up with 3.0.8,
>> let data replicate from dc2 to dc1, and then tear down dc2, bring it up
>> with 3.0.8 and replicate data from dc1.
>>
>>
>>
>> I am able to reproduce the problem on bare metal clusters running on 3
>> nodes. I am using Oracle's server-jre-8u74-linux-x64 JRE.
>>
>>
>>
>> *Node A*: Downloaded 2.2.5-bin.tar.gz, changed the seeds to include its
>> own IP address, changed listen_address and rpc_address to its own IP and
>> changed endpoint_snitch to GossipingPropertyFileSnitch. I
>> changed conf/cassandra-rackdc.properties to
>>
>> dc=dc2
>>
>> rack=rack2
>>
>> This node started up fine and is UN in nodetool status in dc2.
>>
>>
>>
>> I used CQL shell to create a table and insert 3 rows:
>>
>> verma@xxxxx:~/apache-cassandra-2.2.5$ bin/cqlsh $HOSTNAME
>>
>> Connected to Test Cluster at xxxxx:9042.
>>
>> [cqlsh 5.0.1 | Cassandra 2.2.5 | CQL spec 3.3.1 | Native protocol v4]
>>
>> Use HELP for help.
>>
>> cqlsh> desc tmp
>>
>>
>>
>> CREATE KEYSPACE tmp WITH replication = {'class':
>> 'NetworkTopologyStrategy', 'dc1': '1', 'dc2': '1'} AND durable_writes =
>> true;
>>
>>
>>
>> CREATE TABLE tmp.map (
>>
>> key text PRIMARY KEY,
>>
>> value text
>>
>> )...;
>>
>> cqlsh> select * from tmp.map;
>>
>>
>>
>> key | value
>>
>> -----+-------
>>
>> k1 | v1
>>
>> k3 | v3
>>
>> k2 | v2
>>
>>
>>
>>
>>
>> *Node B:* Downloaded 3.0.8-bin.tar.gz, changed the seeds to include
>> itself and node A, changed listen_address and rpc_address to its own IP,
>> changed endpoint_snitch to GossipingPropertyFileSnitch. I did not change
>> conf/cassandra-rackdc.properties and its contents are
>>
>> dc=dc1
>>
>> rack=rack1
>>
>>
>>
>> In the logs, I see:
>>
>> INFO [main] 2016-10-10 22:42:42,850 MessagingService.java:557 - Starting
>> Messaging Service on /10.164.32.29:7000
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.164.32.29-3A7000&d=DQMFaQ&c=08AGY6txKsvMOP6lYkHQpPMRA1U6kqhAwGa8-0QCg3M&r=yfYEBHVkX6l0zImlOIBID0gmhluYPD5Jje-3CtaT3ow&m=aDxgr_M4K2TNU2MqOBLrREEaFoUbWmFpnP4l67CROag&s=t4xNW-nnA0EVZWnt-PSAoq4vx6Lep2ObgkG3ZuUUg7c&e=>
>> (eth0)
>>
>> INFO [main] 2016-10-10 22:42:42,864 StorageService.java:784 - This node
>> will not auto bootstrap because it is configured to be a seed node.
>>
>>
>>
>> So I start a third node:
>>
>> *Node C:* Downloaded 3.0.8-bin.tar.gz, changed the seeds to include node
>> A and node B, changed listen_address and rpc_address to its own IP, changed
>> endpoint_snitch to GossipingPropertyFileSnitch. I did not change
>> conf/cassandra-rackdc.properties.
>>
>> Now, nodetool status shows:
>>
>>
>>
>> verma@xxxxxxx:~/apache-cassandra-3.0.8$ bin/nodetool status
>>
>> Datacenter: dc1
>>
>> ===============
>>
>> Status=Up/Down
>>
>> |/ State=Normal/Leaving/Joining/Moving
>>
>> -- Address Load Tokens Owns (effective) Host ID
>> Rack
>>
>> UJ <Node C IP> 87.81 KB 256 ?
>> 9064832d-ed5c-4c42-ad5a-f754b52b670c rack1
>>
>> UN <Node B IP> 107.72 KB 256 100.0%
>> 28b1043f-115b-46a5-b6b6-8609829cde76 rack1
>>
>> Datacenter: dc2
>>
>> ===============
>>
>> Status=Up/Down
>>
>> |/ State=Normal/Leaving/Joining/Moving
>>
>> -- Address Load Tokens Owns (effective) Host ID
>> Rack
>>
>> UN <Node A IP> 73.2 KB 256 100.0%
>> 09cc542c-2299-45a5-a4d1-159c239ded37 rack2
>>
>>
>>
>> Nodetool describe cluster shows:
>>
>> verma@xxxxxxx:~/apache-cassandra-3.0.8$ bin/nodetool describecluster
>>
>> Cluster Information:
>>
>> Name: Test Cluster
>>
>> Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
>>
>> Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
>>
>> Schema versions:
>>
>> c2a2bb4f-7d31-3fb8-a216-00b41a643650: [<Node B IP>, <Node C IP>]
>>
>>
>>
>> 9770e3c5-3135-32e2-b761-65a0f6d8824e: [<Node A IP>]
>>
>>
>>
>> Note that there are two schema versions and they don't match.
>>
>>
>>
>> I see the following in the system.log:
>>
>>
>>
>> INFO [InternalResponseStage:1] 2016-10-10 22:48:36,055
>> ColumnFamilyStore.java:390 - Initializing system_auth.roles
>>
>> INFO [main] 2016-10-10 22:48:36,316 StorageService.java:1149 - JOINING:
>> waiting for schema information to complete
>>
>> INFO [main] 2016-10-10 22:48:36,316 StorageService.java:1149 - JOINING:
>> schema complete, ready to bootstrap
>>
>> INFO [main] 2016-10-10 22:48:36,316 StorageService.java:1149 - JOINING:
>> waiting for pending range calculation
>>
>> INFO [main] 2016-10-10 22:48:36,317 StorageService.java:1149 - JOINING:
>> calculation complete, ready to bootstrap
>>
>> INFO [main] 2016-10-10 22:48:36,319 StorageService.java:1149 - JOINING:
>> getting bootstrap token
>>
>> INFO [main] 2016-10-10 22:48:36,357 StorageService.java:1149 - JOINING:
>> sleeping 30000 ms for pending range setup
>>
>> INFO [main] 2016-10-10 22:49:06,358 StorageService.java:1149 - JOINING:
>> Starting to bootstrap...
>>
>> INFO [main] 2016-10-10 22:49:06,494 StreamResultFuture.java:87 - [Stream
>> #bfb5e470-8f3b-11e6-b69a-1b451159408e] Executing streaming plan for
>> Bootstrap
>>
>> INFO [StreamConnectionEstablisher:1] 2016-10-10 22:49:06,495
>> StreamSession.java:242 - [Stream #bfb5e470-8f3b-11e6-b69a-1b451159408e]
>> Starting streaming to /<Node A IP>
>>
>> INFO [StreamConnectionEstablisher:2] 2016-10-10 22:49:06,495
>> StreamSession.java:242 - [Stream #bfb5e470-8f3b-11e6-b69a-1b451159408e]
>> Starting streaming to /<Node B IP>
>>
>> INFO [StreamConnectionEstablisher:2] 2016-10-10 22:49:06,500
>> StreamCoordinator.java:213 - [Stream #bfb5e470-8f3b-11e6-b69a-1b451159408e,
>> ID#0] Beginning stream session with /<Node B IP>
>>
>> INFO [STREAM-IN-/<Node B IP>] 2016-10-10 22:49:06,590
>> StreamResultFuture.java:183 - [Stream #bfb5e470-8f3b-11e6-b69a-1b451159408e]
>> Session with /<Node B IP> is complete
>>
>> INFO [StreamConnectionEstablisher:1] 2016-10-10 22:49:06,635
>> StreamCoordinator.java:213 - [Stream #bfb5e470-8f3b-11e6-b69a-1b451159408e,
>> ID#0] Beginning stream session with /<Node A IP>
>>
>> ERROR [STREAM-IN-/<Node A IP>] 2016-10-10 22:49:06,639
>> StreamSession.java:528 - [Stream #bfb5e470-8f3b-11e6-b69a-1b451159408e]
>> Streaming error occurred
>>
>> java.io.IOException: Connection reset by peer
>>
>> at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[na:1.8.0_102]
>>
>> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
>> ~[na:1.8.0_102]
>>
>> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
>> ~[na:1.8.0_102]
>>
>> at sun.nio.ch.IOUtil.read(IOUtil.java:197) ~[na:1.8.0_102]
>>
>> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
>> ~[na:1.8.0_102]
>>
>> at sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:206)
>> ~[na:1.8.0_102]
>>
>> at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
>> ~[na:1.8.0_102]
>>
>> at java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385)
>> ~[na:1.8.0_102]
>>
>> at
>> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:54)
>> ~[apache-cassandra-3.0.8.jar:3.0.8]
>>
>> at org.apache.cassandra.streaming.ConnectionHandler$
>> IncomingMessageHandler.run(ConnectionHandler.java:287)
>> ~[apache-cassandra-3.0.8.jar:3.0.8]
>>
>> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]
>>
>> INFO [STREAM-IN-/<Node A IP>] 2016-10-10 22:49:06,639
>> StreamResultFuture.java:183 - [Stream #bfb5e470-8f3b-11e6-b69a-1b451159408e]
>> Session with /<Node A IP> is complete
>>
>> WARN [STREAM-IN-/<Node A IP>] 2016-10-10 22:49:06,640
>> StreamResultFuture.java:210 - [Stream #bfb5e470-8f3b-11e6-b69a-1b451159408e]
>> Stream failed
>>
>> WARN [STREAM-IN-/<Node A IP>] 2016-10-10 22:49:06,640
>> StorageService.java:1208 - Error during bootstrap.
>>
>> org.apache.cassandra.streaming.StreamException: Stream failed
>>
>> at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.
>> onFailure(StreamEventJMXNotifier.java:85) ~[apache-cassandra-3.0.8.jar:
>> 3.0.8]
>>
>> at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
>> [guava-18.0.jar:na]
>>
>> at com.google.common.util.concurrent.MoreExecutors$
>> DirectExecutor.execute(MoreExecutors.java:457) [guava-18.0.jar:na]
>>
>> at com.google.common.util.concurrent.ExecutionList.
>> executeListener(ExecutionList.java:156) [guava-18.0.jar:na]
>>
>> at com.google.common.util.concurrent.ExecutionList.
>> execute(ExecutionList.java:145) [guava-18.0.jar:na]
>>
>> at com.google.common.util.concurrent.AbstractFuture.
>> setException(AbstractFuture.java:202) [guava-18.0.jar:na]
>>
>> at
>> org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:211)
>> [apache-cassandra-3.0.8.jar:3.0.8]
>>
>> at org.apache.cassandra.streaming.StreamResultFuture.
>> handleSessionComplete(StreamResultFuture.java:187)
>> [apache-cassandra-3.0.8.jar:3.0.8]
>>
>> at org.apache.cassandra.streaming.StreamSession.
>> closeSession(StreamSession.java:429) [apache-cassandra-3.0.8.jar:3.0.8]
>>
>> at org.apache.cassandra.streaming.StreamSession.
>> onError(StreamSession.java:534) [apache-cassandra-3.0.8.jar:3.0.8]
>>
>> at org.apache.cassandra.streaming.ConnectionHandler$
>> IncomingMessageHandler.run(ConnectionHandler.java:305)
>> [apache-cassandra-3.0.8.jar:3.0.8]
>>
>> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]
>>
>> ERROR [main] 2016-10-10 22:49:06,641 StorageService.java:1218 - Error
>> while waiting on bootstrap to complete. Bootstrap will have to be restarted.
>>
>> java.util.concurrent.ExecutionException:
>> org.apache.cassandra.streaming.StreamException:
>> Stream failed
>>
>> at com.google.common.util.concurrent.AbstractFuture$
>> Sync.getValue(AbstractFuture.java:299) ~[guava-18.0.jar:na]
>>
>> at com.google.common.util.concurrent.AbstractFuture$
>> Sync.get(AbstractFuture.java:286) ~[guava-18.0.jar:na]
>>
>> at
>> com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
>> ~[guava-18.0.jar:na]
>>
>> at
>> org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1213)
>> [apache-cassandra-3.0.8.jar:3.0.8]
>>
>> at
>> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:889)
>> [apache-cassandra-3.0.8.jar:3.0.8]
>>
>> at
>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:663)
>> [apache-cassandra-3.0.8.jar:3.0.8]
>>
>> at
>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:528)
>> [apache-cassandra-3.0.8.jar:3.0.8]
>>
>> at
>> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:339)
>> [apache-cassandra-3.0.8.jar:3.0.8]
>>
>> at
>> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:557)
>> [apache-cassandra-3.0.8.jar:3.0.8]
>>
>> at
>> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:685)
>> [apache-cassandra-3.0.8.jar:3.0.8]
>>
>> Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
>>
>> at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.
>> onFailure(StreamEventJMXNotifier.java:85) ~[apache-cassandra-3.0.8.jar:
>> 3.0.8]
>>
>> at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
>> ~[guava-18.0.jar:na]
>>
>> at com.google.common.util.concurrent.MoreExecutors$
>> DirectExecutor.execute(MoreExecutors.java:457) ~[guava-18.0.jar:na]
>>
>> at com.google.common.util.concurrent.ExecutionList.
>> executeListener(ExecutionList.java:156) ~[guava-18.0.jar:na]
>>
>> at com.google.common.util.concurrent.ExecutionList.
>> execute(ExecutionList.java:145) ~[guava-18.0.jar:na]
>>
>> at com.google.common.util.concurrent.AbstractFuture.
>> setException(AbstractFuture.java:202) ~[guava-18.0.jar:na]
>>
>> at
>> org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:211)
>> ~[apache-cassandra-3.0.8.jar:3.0.8]
>>
>> at org.apache.cassandra.streaming.StreamResultFuture.
>> handleSessionComplete(StreamResultFuture.java:187)
>> ~[apache-cassandra-3.0.8.jar:3.0.8]
>>
>> at org.apache.cassandra.streaming.StreamSession.
>> closeSession(StreamSession.java:429) ~[apache-cassandra-3.0.8.jar:3.0.8]
>>
>> at org.apache.cassandra.streaming.StreamSession.
>> onError(StreamSession.java:534) ~[apache-cassandra-3.0.8.jar:3.0.8]
>>
>> at org.apache.cassandra.streaming.ConnectionHandler$
>> IncomingMessageHandler.run(ConnectionHandler.java:305)
>> ~[apache-cassandra-3.0.8.jar:3.0.8]
>>
>> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_102]
>>
>> WARN [main] 2016-10-10 22:49:06,646 StorageService.java:944 - Some data
>> streaming failed. Use nodetool to check bootstrap state and resume. For
>> more, see `nodetool help bootstrap`. IN_PROGRESS
>>
>> INFO [main] 2016-10-10 22:49:06,647 CassandraDaemon.java:644 - Waiting
>> for gossip to settle before accepting client requests...
>>
>> INFO [main] 2016-10-10 22:49:14,648 CassandraDaemon.java:675 - No gossip
>> backlog; proceeding
>>
>> INFO [main] 2016-10-10 22:49:14,694 NativeTransportService.java:70 -
>> Netty using native Epoll event loop
>>
>> INFO [main] 2016-10-10 22:49:14,726 Server.java:159 - Using Netty
>> Version: [netty-buffer=netty-buffer-4.0.23.Final.208198c,
>> netty-codec=netty-codec-4.0.23.Final.208198c,
>> netty-codec-http=netty-codec-http-4.0.23.Final.208198c,
>> netty-codec-socks=netty-codec-socks-4.0.23.Final.208198c,
>> netty-common=netty-common-4.0.23.Final.208198c,
>> netty-handler=netty-handler-4.0.23.Final.208198c, netty-transport=netty-
>> transport-4.0.23.Final.208198c, netty-transport-rxtx=netty-
>> transport-rxtx-4.0.23.Final.208198c, netty-transport-sctp=netty-
>> transport-sctp-4.0.23.Final.208198c, netty-transport-udt=netty-
>> transport-udt-4.0.23.Final.208198c]
>>
>> INFO [main] 2016-10-10 22:49:14,726 Server.java:160 - Starting listening
>> for CQL clients on /<Node C IP>:9042 (unencrypted)...
>>
>> INFO [main] 2016-10-10 22:49:14,748 CassandraDaemon.java:477 - Not
>> starting RPC server as requested. Use JMX (StorageService->startRPCServer())
>> or nodetool (enablethrift) to start it
>>
>>
>>
>> I tried resuming bootstrap but it fails with the same streaming errors:
>>
>>
>>
>> verma@<Node C>:~/apache-cassandra-3.0.8$ bin/nodetool bootstrap resume
>>
>> Resuming bootstrap
>>
>> [2016-10-10 23:15:11,816] session with /<Node B IP> complete (progress:
>> 0%)
>>
>> [2016-10-10 23:15:11,939] session with /<Node A IP> complete (progress:
>> 0%)
>>
>> [2016-10-10 23:15:11,940] Stream failed
>>
>>
>>
>> and I see the same error in the system.log:
>>
>>
>>
>> StreamSession.java:528 - [Stream #64b73a20-8f3f-11e6-b69a-1b451159408e]
>> Streaming error occurred
>>
>> java.io.IOException: Connection reset by peer
>>
>> ...
>>
>>
>>
>> Does Cassandra support upgrading from 2.2.5 to 3.0.8 in this way? Am I
>> missing something?
>>
>>
>>
>> Thanks for your time.
>>
>> -Abhishek.
>> ____________________________________________________________________
>> CONFIDENTIALITY NOTE: This e-mail and any attachments are confidential
>> and may be legally privileged. If you are not the intended recipient, do
>> not disclose, copy, distribute, or use this email or any attachments. If
>> you have received this in error please let the sender know and then delete
>> the email and all attachments.
>>
>
