Do you have same seed node specified in cass-analysis-1 as cass-1,2,3? I am thinking that changing the seed node in cass-analysis-2 and following the directions in http://wiki.apache.org/cassandra/FAQ#schema_disagreement might solve the problem. Somone please correct me.
On Thu, Oct 13, 2011 at 12:05 AM, Eric Czech <e...@nextbigsound.com> wrote: > I don't think that's what I'm after here since the unwanted nodes were > originally assimilated into the cluster with the same initial_token values > as other nodes that were already in the cluster (that have, and still do > have, useful data). I know this is an awkward situation so I'll try to > depict it in a simpler way: > Let's say I have a simplified version of our production cluster that looks > like this - > cass-1 token = A > cass-2 token = B > cass-3 token = C > Then I tried to create a second cluster that looks like this - > cass-analysis-1 token = A (and contains same data as cass-1) > cass-analysis-2 token = B (and contains same data as cass-2) > cass-analysis-3 token = C (and contains same data as cass-3) > But after starting the second cluster, things got crossed up between the > clusters and here's what the original cluster now looks like - > cass-1 token = A (has data and schema) > cass-2 token = B (has data and schema) > cass-3 token = C (had data and schema) > cass-analysis-1 token = A (has *no* data and is not part of the ring, but > is trying to be included in cluster schema) > A simplified version of "describe cluster" for the original cluster now > shows: > Cluster Information: > Schema versions: > SCHEMA-UUID-1: [cass-1, cass-2, cass-3] > SCHEMA-UUID-2: [cass-analysis-1] > But the simplified ring looks like this (has only 3 nodes instead of 4): > Host Owns Token > cass-1 33% A > cass-2 33% B > cass-3 33% C > The original cluster is still working correctly but all live schema updates > are failing because of the inconsistent schema versions introduced by the > unwanted node. > From my perspective, a simple fix seems to be for cassandra to exclude nodes > that aren't part of the ring from the schema consistency requirements. Any > reason that wouldn't work? > And aside from a possible code patch, any recommendations as to how I can > best fix this given the current 8.4 release? > > On Thu, Oct 13, 2011 at 12:14 AM, Jonathan Ellis <jbel...@gmail.com> wrote: >> >> Does nodetool removetoken not work? >> >> On Thu, Oct 13, 2011 at 12:59 AM, Eric Czech <e...@nextbigsound.com> >> wrote: >> > Not sure if anyone has seen this before but it's really killing me right >> > now. Perhaps that was too long of a description of the issue so here's >> > a >> > more succinct question -- How do I remove nodes associated with a >> > cluster >> > that contain no data and have no reason to be associated with the >> > cluster >> > whatsoever? >> > My last resort here is to stop cassandra (after recording all tokens for >> > each node), set the initial token for each node in the cluster in >> > cassandra.yaml, manually delete the LocationInfo* sstables in the system >> > keyspace, and then restart. I'm hoping there's a simpler, less >> > seemingly >> > risky way to do this so please, please let me know if that's true! >> > Thanks again. >> > - Eric >> > On Tue, Oct 11, 2011 at 11:55 AM, Eric Czech <e...@nextbigsound.com> >> > wrote: >> >> >> >> Hi, I'm having what I think is a fairly uncommon schema issue -- >> >> My situation is that I had a cluster with 10 nodes and a consistent >> >> schema. Then, in an experiment to setup a second cluster with the same >> >> information (by copying the raw sstables), I left the LocationInfo* >> >> sstables >> >> in the system keyspace in the new cluster and after starting the second >> >> cluster, I realized that the two clusters were discovering each other >> >> when >> >> they shouldn't have been. Since then, I changed the cluster name for >> >> the >> >> second cluster and made sure to delete the LocationInfo* sstables >> >> before >> >> starting it and the two clusters are now operating independent of one >> >> another for the most part. The only remaining connection between the >> >> two >> >> seems to be that the first cluster is still maintaining references to >> >> nodes >> >> in the second cluster in the schema versions despite those nodes not >> >> actually being part of the ring. >> >> Here's what my "describe cluster" looks like on the original cluster: >> >> Cluster Information: >> >> Snitch: org.apache.cassandra.locator.SimpleSnitch >> >> Partitioner: org.apache.cassandra.dht.RandomPartitioner >> >> Schema versions: >> >> 48971cb0-e9ff-11e0-0000-eb9eab7d90bf: [<INTENTIONAL_IP1>, >> >> <INTENTIONAL_IP2>, ..., <INTENTIONAL_IP10>] >> >> 848bcfc0-eddf-11e0-0000-8a3bb58f08ff: [<NOT_INTENTIONAL_IP1>, >> >> <NOT_INTENTIONAL_IP2>] >> >> The second cluster, however, contains no schema versions involving >> >> nodes >> >> from the first cluster. >> >> My question then is, how can I remove those schema versions from the >> >> original cluster that are associated with the unwanted nodes from the >> >> second >> >> cluster? Is there any way to remove or evict an IP from a cluster >> >> instead >> >> of just a token? >> >> Thanks in advance! >> >> - Eric >> > >> >> >> >> -- >> Jonathan Ellis >> Project Chair, Apache Cassandra >> co-founder of DataStax, the source for professional Cassandra support >> http://www.datastax.com > >
