Nope, there was definitely no intersection of the seed nodes between the two clusters so I'm fairly certain that the second cluster found out about the first through what was in the LocationInfo* system tables. Also, I don't think that procedure will really help because I don't actually want the schema on cass-analysis-1 to be consistent with the schema in the original cluster -- I just want to totally remove it.
On Thu, Oct 13, 2011 at 8:01 AM, Mohit Anchlia <mohitanch...@gmail.com>wrote: > Do you have same seed node specified in cass-analysis-1 as cass-1,2,3? > I am thinking that changing the seed node in cass-analysis-2 and > following the directions in > http://wiki.apache.org/cassandra/FAQ#schema_disagreement might solve > the problem. Somone please correct me. > > On Thu, Oct 13, 2011 at 12:05 AM, Eric Czech <e...@nextbigsound.com> > wrote: > > I don't think that's what I'm after here since the unwanted nodes were > > originally assimilated into the cluster with the same initial_token > values > > as other nodes that were already in the cluster (that have, and still do > > have, useful data). I know this is an awkward situation so I'll try to > > depict it in a simpler way: > > Let's say I have a simplified version of our production cluster that > looks > > like this - > > cass-1 token = A > > cass-2 token = B > > cass-3 token = C > > Then I tried to create a second cluster that looks like this - > > cass-analysis-1 token = A (and contains same data as cass-1) > > cass-analysis-2 token = B (and contains same data as cass-2) > > cass-analysis-3 token = C (and contains same data as cass-3) > > But after starting the second cluster, things got crossed up between the > > clusters and here's what the original cluster now looks like - > > cass-1 token = A (has data and schema) > > cass-2 token = B (has data and schema) > > cass-3 token = C (had data and schema) > > cass-analysis-1 token = A (has *no* data and is not part of the ring, > but > > is trying to be included in cluster schema) > > A simplified version of "describe cluster" for the original cluster now > > shows: > > Cluster Information: > > Schema versions: > > SCHEMA-UUID-1: [cass-1, cass-2, cass-3] > > SCHEMA-UUID-2: [cass-analysis-1] > > But the simplified ring looks like this (has only 3 nodes instead of 4): > > Host Owns Token > > cass-1 33% A > > cass-2 33% B > > cass-3 33% C > > The original cluster is still working correctly but all live schema > updates > > are failing because of the inconsistent schema versions introduced by the > > unwanted node. > > From my perspective, a simple fix seems to be for cassandra to exclude > nodes > > that aren't part of the ring from the schema consistency requirements. > Any > > reason that wouldn't work? > > And aside from a possible code patch, any recommendations as to how I can > > best fix this given the current 8.4 release? > > > > On Thu, Oct 13, 2011 at 12:14 AM, Jonathan Ellis <jbel...@gmail.com> > wrote: > >> > >> Does nodetool removetoken not work? > >> > >> On Thu, Oct 13, 2011 at 12:59 AM, Eric Czech <e...@nextbigsound.com> > >> wrote: > >> > Not sure if anyone has seen this before but it's really killing me > right > >> > now. Perhaps that was too long of a description of the issue so > here's > >> > a > >> > more succinct question -- How do I remove nodes associated with a > >> > cluster > >> > that contain no data and have no reason to be associated with the > >> > cluster > >> > whatsoever? > >> > My last resort here is to stop cassandra (after recording all tokens > for > >> > each node), set the initial token for each node in the cluster in > >> > cassandra.yaml, manually delete the LocationInfo* sstables in the > system > >> > keyspace, and then restart. I'm hoping there's a simpler, less > >> > seemingly > >> > risky way to do this so please, please let me know if that's true! > >> > Thanks again. > >> > - Eric > >> > On Tue, Oct 11, 2011 at 11:55 AM, Eric Czech <e...@nextbigsound.com> > >> > wrote: > >> >> > >> >> Hi, I'm having what I think is a fairly uncommon schema issue -- > >> >> My situation is that I had a cluster with 10 nodes and a consistent > >> >> schema. Then, in an experiment to setup a second cluster with the > same > >> >> information (by copying the raw sstables), I left the LocationInfo* > >> >> sstables > >> >> in the system keyspace in the new cluster and after starting the > second > >> >> cluster, I realized that the two clusters were discovering each other > >> >> when > >> >> they shouldn't have been. Since then, I changed the cluster name for > >> >> the > >> >> second cluster and made sure to delete the LocationInfo* sstables > >> >> before > >> >> starting it and the two clusters are now operating independent of one > >> >> another for the most part. The only remaining connection between the > >> >> two > >> >> seems to be that the first cluster is still maintaining references to > >> >> nodes > >> >> in the second cluster in the schema versions despite those nodes not > >> >> actually being part of the ring. > >> >> Here's what my "describe cluster" looks like on the original cluster: > >> >> Cluster Information: > >> >> Snitch: org.apache.cassandra.locator.SimpleSnitch > >> >> Partitioner: org.apache.cassandra.dht.RandomPartitioner > >> >> Schema versions: > >> >> 48971cb0-e9ff-11e0-0000-eb9eab7d90bf: [<INTENTIONAL_IP1>, > >> >> <INTENTIONAL_IP2>, ..., <INTENTIONAL_IP10>] > >> >> 848bcfc0-eddf-11e0-0000-8a3bb58f08ff: [<NOT_INTENTIONAL_IP1>, > >> >> <NOT_INTENTIONAL_IP2>] > >> >> The second cluster, however, contains no schema versions involving > >> >> nodes > >> >> from the first cluster. > >> >> My question then is, how can I remove those schema versions from the > >> >> original cluster that are associated with the unwanted nodes from the > >> >> second > >> >> cluster? Is there any way to remove or evict an IP from a cluster > >> >> instead > >> >> of just a token? > >> >> Thanks in advance! > >> >> - Eric > >> > > >> > >> > >> > >> -- > >> Jonathan Ellis > >> Project Chair, Apache Cassandra > >> co-founder of DataStax, the source for professional Cassandra support > >> http://www.datastax.com > > > > >
