Hi Sergey, That was exactly it, thank you! -Ben
On Thu, Sep 13, 2012 at 12:07 AM, Sergey Tryuber <stryu...@gmail.com> wrote: > Hi Ben, > > We've seen similar behavior once a time. The cause was unsynchronized time > on the nodes. So I can recommend to check that system time is equal over > the nodes (especially on that one "bad" server). > > > On 13 September 2012 05:15, Ben Frank <b...@airlust.com> wrote: > >> Hey all, >> I'm setting up a new ring on 11 machines with cassandra 1.1.5. All >> seems to install fine and startup ok (from tarball) but I'm having an issue >> when updating the schema. There's one node that just doesn't want to >> receive the schema change. I've tried blowing away my /var/lib/cassandra/* >> directories and trying it a few times with the same results. I also tried >> the instructions here: >> http://wiki.apache.org/cassandra/FAQ#schema_disagreement >> >> If I update the schema with cassandra-cli I get the following error: >> >> Waiting for schema agreement... >> The schema has not settled in 10 seconds; further migrations are >> ill-advised until it does. >> Versions are 3b172ce8-d8e8-362f-b955-79fe6b8a35e4:[12.19.103.103, >> 12.19.110.103, 12.19.102.137, 12.19.103.111, 12.19.102.131, 12.19.102.129, >> 12.19.110.105, 172.19.102.135, 12.19.103.105, 12.19.102.133], >> b83fe28a-2851-34cb-bd8d-7d2621c1d872:[12.19.110.109] >> >> then if I do a 'describe cluster' I get: >> >> Cluster Information: >> Snitch: org.apache.cassandra.locator.SimpleSnitch >> Partitioner: org.apache.cassandra.dht.RandomPartitioner >> Schema versions: >> 3b172ce8-d8e8-362f-b955-79fe6b8a35e4: [12.19.103.103, 12.19.110.103, >> 12.19.102.137, 12.19.103.111, 12.19.102.131, 12.19.102.129, 12.19.110.105, >> 12.19.102.135, 12.19.103.105, 12.19.102.133] >> >> a0f80d21-ad98-31df-b6dc-871061b1bcf9: [12.19.110.109] >> >> if I keep running describe cluster, both the schema UUID's(?) above keep >> changing every few seconds. Looking at the logs shows the nodes >> are continually doing a flush and compact. >> >> if I start over by blowing away /var/lib/cassandra, and then try the >> schema update on the machine which is having trouble receiving >> it everything works ok and all nodes are in sync. >> >> Anyone know what's going on here, or what I should look at to >> troubleshoot this? >> >> -Ben >> > >
