Out of interest, how out of sync where they ? Cheers
----------------- Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 14/09/2012, at 6:53 AM, Ben Frank <b...@airlust.com> wrote: > Hi Sergey, > That was exactly it, thank you! > > -Ben > > On Thu, Sep 13, 2012 at 12:07 AM, Sergey Tryuber <stryu...@gmail.com> wrote: > Hi Ben, > > We've seen similar behavior once a time. The cause was unsynchronized time on > the nodes. So I can recommend to check that system time is equal over the > nodes (especially on that one "bad" server). > > > On 13 September 2012 05:15, Ben Frank <b...@airlust.com> wrote: > Hey all, > I'm setting up a new ring on 11 machines with cassandra 1.1.5. All seems > to install fine and startup ok (from tarball) but I'm having an issue when > updating the schema. There's one node that just doesn't want to receive the > schema change. I've tried blowing away my /var/lib/cassandra/* directories > and trying it a few times with the same results. I also tried the > instructions here: http://wiki.apache.org/cassandra/FAQ#schema_disagreement > > If I update the schema with cassandra-cli I get the following error: > > Waiting for schema agreement... > The schema has not settled in 10 seconds; further migrations are ill-advised > until it does. > Versions are 3b172ce8-d8e8-362f-b955-79fe6b8a35e4:[12.19.103.103, > 12.19.110.103, 12.19.102.137, 12.19.103.111, 12.19.102.131, 12.19.102.129, > 12.19.110.105, 172.19.102.135, 12.19.103.105, 12.19.102.133], > b83fe28a-2851-34cb-bd8d-7d2621c1d872:[12.19.110.109] > > then if I do a 'describe cluster' I get: > > Cluster Information: > Snitch: org.apache.cassandra.locator.SimpleSnitch > Partitioner: org.apache.cassandra.dht.RandomPartitioner > Schema versions: > 3b172ce8-d8e8-362f-b955-79fe6b8a35e4: [12.19.103.103, 12.19.110.103, > 12.19.102.137, 12.19.103.111, 12.19.102.131, 12.19.102.129, 12.19.110.105, > 12.19.102.135, 12.19.103.105, 12.19.102.133] > > a0f80d21-ad98-31df-b6dc-871061b1bcf9: [12.19.110.109] > > if I keep running describe cluster, both the schema UUID's(?) above keep > changing every few seconds. Looking at the logs shows the nodes are > continually doing a flush and compact. > > if I start over by blowing away /var/lib/cassandra, and then try the schema > update on the machine which is having trouble receiving it everything works > ok and all nodes are in sync. > > Anyone know what's going on here, or what I should look at to troubleshoot > this? > > -Ben > >
