Hi, Here are some thoughts:
running 1.2.18. I plan to upgrade them to 2.2.latest > Going 1 major release at the time is probably the safest way to go indeed. > 1. Install 2.0.latest on one node at a time, start and wait for it to > join the ring. > 2. Run upgradesstables on this node. > 3. Repeat Step 1,2 on each node installing cassandra2.0 in a rolling > manner and running upgradesstables in parallel. (Please let me know if > running upgrades stables in parallel is not right here. My cluster is not > under much load really) > > I would: - Upgrade one node, check for cluster health (monitoring, logs, nodeool commands), having a special attention to the 2.0 node. - If everything is ok, then go for more nodes, if using distinct racks I would go per rack; sequentially, node per node, all the nodes from DC1-rack1, then DC1-rack2, then DC1-rack3. Then move to the next DC if everything is fine. - Start the 'upgradesstables' when the cluster is completely and successfully running with the new version (2.0.17). It is perfectly fine to run this in parallel as the last part of the upgrade. As you guessed, it is good to keep monitoring the cluster load. 4. Now I will have both my DCs running 2.0.latest. Without really having any strong argument, I would let it run for "some time" like this, hours at least, maybe days. In any case, you will probably have some work to prepare before the next upgrade, so you will have time to check how the cluster is doing. 6. Do I need to run upgradesstables here again after the node has started > and joined? (I think yes, but seek advice. https://docs.datastax. > com/en/latest-upgrade/upgrade/cassandra/upgrdCassandra.html) Yes, every time you run a major upgrade. Anyway, nodetool upgradesstables will skip any sstable that do not need to be upgraded (as long as you don't add the option to force it), so it is probably better to run it when you have a doubt. As additional information, I would prepare, for each upgrade: - The new Cassandra configuration (cassandra.yaml and cassandra-sh.yaml mainly but also other configuration files) To do that, I use to merge the current file in use (your configuration on C* 1.2.18) and the Cassandra version file from github for the new version (i.e. https://github.com/apache/cassandra/tree/cassandra-2.0.17/conf). This allows you to - Acknowledge and consider the new and removed configuration settings - Keep comments and default values in the configuration files up to date - Be fully exhaustive, and learn as you parse the files - Make sure clients will still work with the new version (see the doc, do the tests) - Cassandra metrics changed in the latest versions, you might have to rework your dashboards. Anticipating the dashboard creation for new versions would prevent you from loosing metrics when you need them the most. Finally keep in mind that you should not perform any streaming while running multiple version and as long as 'nodetool upgradesstables' is not completely done. Meaning you should not add, remove, replace, move or repair a node. Also, I would limit schema changes as much as possible while running multiple versions as it caused troubles in the past. During an upgrade, almost nothing else than the normal load due to the service and the upgrade itself should happen. We always try to keep this time window as short as possible. C*heers, ----------------------- Alain Rodriguez - @arodream - al...@thelastpickle.com France The Last Pickle - Apache Cassandra Consulting http://www.thelastpickle.com 2016-12-21 20:36 GMT+01:00 Aiman Parvaiz <ai...@steelhouse.com>: > Hi everyone, > I have 2 C* DCs with 12 nodes in each running 1.2.18. I plan to upgrade > them to 2.2.latest and wanted to run by you experts my plan. > > > 1. Install 2.0.latest on one node at a time, start and wait for it to > join the ring. > 2. Run upgradesstables on this node. > 3. Repeat Step 1,2 on each node installing cassandra2.0 in a rolling > manner and running upgradesstables in parallel. (Please let me know if > running upgrades stables in parallel is not right here. My cluster is not > under much load really) > 4. Now I will have both my DCs running 2.0.latest. > 5. Install cassandra 2.1.latest on one node at a time (same as above) > 6. Do I need to run upgradesstables here again after the node has > started and joined? (I think yes, but seek advice. > https://docs.datastax.com/en/latest-upgrade/upgrade/ > cassandra/upgrdCassandra.html > > <https://docs.datastax.com/en/latest-upgrade/upgrade/cassandra/upgrdCassandra.html> > ) > 7. Following the above pattern, I would install cassandra2.1 in a > rolling manner across 2 DCs (depending on response to 6 I might or might > not run upgradesstables) > 8. At this point both DCs would have 2.1.latest and again in rolling > manner I install 2.2.8. > > > My assumption is that while this upgrade would be happening C* would still > be able to serve reads and writes and running different versions at various > point in the upgrade process will not affect the apps reading/writing from > C*. > > Thanks > >
