Hello Rob and thanks for your reply, At the end we had to wait for the upgradesstables ti finish on every node. Just to eliminate the possibility of this being the reason of any weird behaviour after the upgrade. However, this process might take a long time in a cluster with a large number of nodes which means no new work can be done for that period.
1) TRUNCATE requires all known nodes to be available to succeed, if you are > restarting one, it won't be available. > I suppose all means all, not all replicas here, is that right? Not directly related to the original question, but that might explain why we end up with peculiar behaviour some times when we run TRUNCATE. We've now taken the approach DROP it and do it again when possible (even though this is still problematic when using the same CF name). > 2) in theory, the newly upgraded nodes might not get the DDL schema update > properly due to some incompatible change > > To check for 2, do : > " > nodetool gossipinfo | grep SCHEMA |sort | uniq -c | sort -n > " > > Before and after and make sure the schema propagates correctly. There > should be a new version on all nodes between each DDL change, if there is > you will likely be able to see the new schema on all the new nodes. > > Yeas, this makes perfect sense. We monitor the schema changes every minutes across the cluster with Nagios by checking the JMX console. It is an important thing to monitor in several situations (running migrations for example, or during upgrades like you describe here). Is there a way to find out if the upgradesstables has been run against a particular node or not? Many Thanks, Vasilis
