When you say add 2 nodes, do you mean bootstrap, or upgrade in place? On Wed, Jul 1, 2015 at 11:37 AM David Aronchick <aronch...@gmail.com> wrote:
> This helps - so let me understand: > > Starting point: > - 4 nodes running 2.1.4 > - System is healthy > > Decide to upgrade: > - Add 2 nodes running 2.1.5 > - Run nodetool upgradestable > - Wait until system is healthy > - Stop 2 nodes running 2.1.4 > - Run nodetool upgradestable > - Add 2 nodes running 2.1.5 > - Run nodetool upgradestable > - Stop 2 nodes running 2.1.4 > - Run nodetool upgradestable > > Finished? > > On Wed, Jul 1, 2015 at 1:49 AM, Alain RODRIGUEZ <arodr...@gmail.com> > wrote: > >> Basically, when you add nodes, add them on the correct version to avoid >> schema / network issues in your streams. Also, try to update all the node >> using rolling restarts in a reduced time frame after stopping repairs, with >> all the nodes up, etc. >> >> You must have a healthy cluster before performing an upgrade. >> >> Remember running a nodetool upgradesstable after any upgrade (if not >> needed it will end fast, so I would run it always as a best practice, just >> in case). >> >> C*heers, >> >> Alain >> >> 2015-07-01 2:16 GMT+02:00 David Aronchick <aronch...@gmail.com>: >> >>> That is a GREAT lead! So it looks like I can't add a few nodes to the >>> cluster of the new version, have it settle down, and then upgrade the rest? >>> >>> On Tue, Jun 30, 2015 at 11:58 AM, Alain RODRIGUEZ <arodr...@gmail.com> >>> wrote: >>> >>>> "Would it matter that I'm mixing cassandra versions? " >>>> >>>> From: >>>> http://docs.datastax.com/en/upgrade/doc/upgrade/datastax_enterprise/upgrdLim.html >>>> >>>> "General upgrade limitations¶ >>>> >>>> Do not run nodetool repair. >>>> Do not enable new features. >>>> Do not issue these types of queries during a rolling restart: DDL, >>>> TRUNCATE >>>> *During upgrades, the nodes on different versions show a schema >>>> disagreement*." >>>> >>>> I think this is a good lead. >>>> >>>> C*heers, >>>> >>>> Alain >>>> >>>> 2015-06-30 20:22 GMT+02:00 David Aronchick <aronch...@gmail.com>: >>>> >>>>> I appreciate the thoughts! My issue is that it seems to work >>>>> perfectly, until the node goes away. Would it matter that I'm mixing >>>>> cassandra versions? (2.1.4 and 2.1.5)? >>>>> >>>>> On Tue, Jun 30, 2015 at 5:23 AM, Alain RODRIGUEZ <arodr...@gmail.com> >>>>> wrote: >>>>> >>>>>> Hi David ? >>>>>> >>>>>> What does a "nodetool describecluster" output look like ? >>>>>> >>>>>> My guess is you might be having a schema version desynchronisation. >>>>>> If you see a node with different schema version you might want to try a >>>>>> "nodetool resetlocal*schema* - Reset node's local *schema* and >>>>>> resync" >>>>>> >>>>>> You asked for any thoughts, this is a thought, not sure if it will >>>>>> help, I hope so. >>>>>> >>>>>> C*heers, >>>>>> >>>>>> Alain >>>>>> >>>>>> 2015-06-30 1:44 GMT+02:00 Robert Coli <rc...@eventbrite.com>: >>>>>> >>>>>>> On Mon, Jun 29, 2015 at 2:43 PM, David Aronchick < >>>>>>> aronch...@gmail.com> wrote: >>>>>>> >>>>>>>> Ping--- any thoughts here? >>>>>>>> >>>>>>> >>>>>>> I don't have any thoughts on your specific issue at this time, but >>>>>>> FWIW #cassandra on freenode is sometimes a better forum for interactive >>>>>>> debugging of operational edge cases. >>>>>>> >>>>>>> =Rob >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >> >
