Thanks for the detailed reply Ken, this really helps. I also realized that I wasn't doing a 'nodetool rebuild' after reading your email. I was following the steps mentioned here http://www.datastax.com/documentation/cassandra/1.2/cassandra/operations/ops_decomission_dc_t.html
I do a test with nodetool rebuild and see what happens. On Thu, Aug 7, 2014 at 1:27 PM, Ken Hancock <ken.hanc...@schange.com> wrote: > My reading is it didn't forget the schema. It lost the data. > > My reading is decomissioning worked fine. Possibly when you changed the > replication on a keyspace to include a second data center, the data didn't > get replicated. > Probably not because I could see the sstables for the keyspace from the other datacenter created. My understanding could be wrong though. > > When you ADD a datacenter, you need to do a nodetool rebuild to get the > data streamed to the new data center. When you alter a keyspace to include > another datacenter in its replication schema, a nodetool repair is required > -- was this done? > http://www.datastax.com/documentation/cql/3.0/cql/cql_using/update_ks_rf_t.html > I missed the 'nodetool rebuild' step that could be my issue, yes I did run repair. > > When you use nodetool decomission, you're effectively deleting the > parititioning token from the cluster. The node being decommissioned will > stream its data to the new owners of its original token range. This > streaming in no way should affect any other datacenter because you have not > changed the tokens or data ownership for any datacenter but the one in > which you are decomissioning a node. > That is what my understanding was, but when I decommission it does clear out (removes) all the keyspaces. > > When you eventually decomission the last node in the datacenter, all data > is gone as there are no tokens in that datacenter to own any data. > > If you had a keyspace that was only replicated within that datacenter, > that data is gone (though you could probably add nodes back in and > ressurect it). > The (now outdated) documentation [1] says that data remains on the node even after decommissioning. So I do not understand why the data would go away. > > If you had a keyspace where you changed the replication to include another > datacenter, if that datacenter had never received the data, then it may > have the schema but would have none of the data (other than new data that > was written AFTER you change the replication). > I would expect the repair to fix this, i.e. to stream the old data to the newly added datacenter. So, does nodetool rebuild help here ? [1] https://wiki.apache.org/cassandra/Operations#Removing_nodes_entirely > > > > > On Thu, Aug 7, 2014 at 2:11 PM, srmore <comom...@gmail.com> wrote: > >> >> >> >> On Thu, Aug 7, 2014 at 12:27 PM, Robert Coli <rc...@eventbrite.com> >> wrote: >> >>> On Thu, Aug 7, 2014 at 10:04 AM, srmore <comom...@gmail.com> wrote: >>> >>>> Sorry for being ambiguous. By "deletes" I mean that running >>>> decommission I can no longer see any keyspaces owned by this node or >>>> replicated by other nodes using the cfstats command. I am also seeing the >>>> same behavior when I remove a single node from a cluster (without >>>> datacenters). >>>> >>> >>> I'm still not fully parsing you, but clusters should never "forget" >>> schema as a result of decommission. >>> >>> Is that what you are saying is happening? >>> >> >> Yes, this is what is happening. >> >> >>> >>> (In fact, even the decommissioned node itself does not forget its >>> schema, which I personally consider a bug.) >>> >> >> >> Ok, so I am assuming this is not a normal behavior and possibly a bug - >> is this correct ? >> >> >>> >>> =Rob >>> >>> >> > > > -- > *Ken Hancock *| System Architect, Advanced Advertising > SeaChange International > 50 Nagog Park > Acton, Massachusetts 01720 > ken.hanc...@schange.com | www.schange.com | NASDAQ:SEAC > <http://www.schange.com/en-US/Company/InvestorRelations.aspx> > Office: +1 (978) 889-3329 | [image: Google Talk:] ken.hanc...@schange.com > | [image: Skype:]hancockks | [image: Yahoo IM:]hancockks [image: > LinkedIn] <http://www.linkedin.com/in/kenhancock> > > [image: SeaChange International] > <http://www.schange.com/>This e-mail and any attachments may contain > information which is SeaChange International confidential. The information > enclosed is intended only for the addressees herein and may not be copied > or forwarded without permission from SeaChange International. >
