Also before you get started. Make sure: 1) no one attempts to change schema during the process 2) no one attempts to run a repair 3) no one attempts to join a node 4) no one attempts to remove/move nodes from the cluster
Each of these things trigger repair sessions and stream data which do not work in a mix cluster On Thu, Dec 22, 2016 at 3:07 PM, Aiman Parvaiz <ai...@steelhouse.com> wrote: > Thanks Alain. This was extremely helpful, really grateful. > > Aiman > > On Dec 22, 2016, at 5:00 AM, Alain RODRIGUEZ <arodr...@gmail.com> wrote: > > Hi, > > Here are some thoughts: > > running 1.2.18. I plan to upgrade them to 2.2.latest >> > > Going 1 major release at the time is probably the safest way to go indeed. > > >> 1. Install 2.0.latest on one node at a time, start and wait for it to >> join the ring. >> 2. Run upgradesstables on this node. >> 3. Repeat Step 1,2 on each node installing cassandra2.0 in a rolling >> manner and running upgradesstables in parallel. (Please let me know if >> running upgrades stables in parallel is not right here. My cluster is not >> under much load really) >> >> > I would: > > - Upgrade one node, check for cluster health (monitoring, logs, nodeool > commands), having a special attention to the 2.0 node. > - If everything is ok, then go for more nodes, if using distinct racks I > would go per rack; sequentially, node per node, all the nodes from > DC1-rack1, then DC1-rack2, then DC1-rack3. Then move to the next DC if > everything is fine. > - Start the 'upgradesstables' when the cluster is completely and > successfully running with the new version (2.0.17). It is perfectly fine to > run this in parallel as the last part of the upgrade. As you guessed, it is > good to keep monitoring the cluster load. > > 4. Now I will have both my DCs running 2.0.latest. > > > Without really having any strong argument, I would let it run for "some > time" like this, hours at least, maybe days. In any case, you will probably > have some work to prepare before the next upgrade, so you will have time to > check how the cluster is doing. > > 6. Do I need to run upgradesstables here again after the node has started >> and joined? (I think yes, but seek advice. https://docs.datastax. >> com/en/latest-upgrade/upgrade/cassandra/upgrdCassandra.html) > > > Yes, every time you run a major upgrade. Anyway, nodetool upgradesstables > will skip any sstable that do not need to be upgraded (as long as you don't > add the option to force it), so it is probably better to run it when you > have a doubt. > > > As additional information, I would prepare, for each upgrade: > > > - The new Cassandra configuration (cassandra.yaml and > cassandra-sh.yaml mainly but also other configuration files) > > To do that, I use to merge the current file in use (your configuration > on C* 1.2.18) and the Cassandra version file from github for the new > version (i.e. https://github.com/apache/cassandra/tree/ > cassandra-2.0.17/conf). > > This allows you to > - Acknowledge and consider the new and removed configuration > settings > - Keep comments and default values in the configuration files up to > date > - Be fully exhaustive, and learn as you parse the files > > - Make sure clients will still work with the new version (see the > doc, do the tests) > - Cassandra metrics changed in the latest versions, you might have to > rework your dashboards. Anticipating the dashboard creation for new > versions would prevent you from loosing metrics when you need them the > most. > > > Finally keep in mind that you should not perform any streaming while > running multiple version and as long as 'nodetool upgradesstables' is not > completely done. Meaning you should not add, remove, replace, move or > repair a node. Also, I would limit schema changes as much as possible while > running multiple versions as it caused troubles in the past. > > During an upgrade, almost nothing else than the normal load due to the > service and the upgrade itself should happen. We always try to keep this > time window as short as possible. > > C*heers, > ----------------------- > Alain Rodriguez - @arodream - al...@thelastpickle.com > France > > The Last Pickle - Apache Cassandra Consulting > http://www.thelastpickle.com > > 2016-12-21 20:36 GMT+01:00 Aiman Parvaiz <ai...@steelhouse.com>: > >> Hi everyone, >> I have 2 C* DCs with 12 nodes in each running 1.2.18. I plan to upgrade >> them to 2.2.latest and wanted to run by you experts my plan. >> >> >> 1. Install 2.0.latest on one node at a time, start and wait for it to >> join the ring. >> 2. Run upgradesstables on this node. >> 3. Repeat Step 1,2 on each node installing cassandra2.0 in a rolling >> manner and running upgradesstables in parallel. (Please let me know if >> running upgrades stables in parallel is not right here. My cluster is not >> under much load really) >> 4. Now I will have both my DCs running 2.0.latest. >> 5. Install cassandra 2.1.latest on one node at a time (same as above) >> 6. Do I need to run upgradesstables here again after the node has >> started and joined? (I think yes, but seek advice. >> https://docs.datastax.com/en/latest-upgrade/upgrade/ >> cassandra/upgrdCassandra.html >> >> <https://docs.datastax.com/en/latest-upgrade/upgrade/cassandra/upgrdCassandra.html> >> ) >> 7. Following the above pattern, I would install cassandra2.1 in a >> rolling manner across 2 DCs (depending on response to 6 I might or might >> not run upgradesstables) >> 8. At this point both DCs would have 2.1.latest and again in rolling >> manner I install 2.2.8. >> >> >> My assumption is that while this upgrade would be happening C* would >> still be able to serve reads and writes and running different versions at >> various point in the upgrade process will not affect the apps >> reading/writing from C*. >> >> Thanks >> >> > >
