Interesting that you are finding excessive drift from public time servers. I only once saw that problem with AWS' time servers. To be conservative I sometimes recommend that clients spool up their own time server, but realize IT will also drift if the public time servers do! Somewhat different if in your own DC, but same time server drift issues.
Google has resorted to putting tier one time server(s) (cesium clock or whatever) in every data center due to the public drift issues. Does anyone know if AWS' time server is now stratum 1 backed? However, it is better to have two (at least) in AWS, make sure their private IP's are not in the same 24 CIDR subnet! Of course this can get troublesome if load sharing between e.g. AWS East and West. *.......* *“Life should not be a journey to the grave with the intention of arriving safely in apretty and well preserved body, but rather to skid in broadside in a cloud of smoke,thoroughly used up, totally worn out, and loudly proclaiming “Wow! What a Ride!” - Hunter ThompsonDaemeon C.M. ReiydelleUSA (+1) 415.501.0198London (+44) (0) 20 8144 9872* On Tue, Mar 31, 2015 at 10:49 PM, Saurabh Sethi <saurabh_se...@symantec.com> wrote: > Thanks Mark. A great post indeed and saved me a lot of trouble. > > - Saurabh > From: Mark Greene <green...@gmail.com> > Reply-To: "user@cassandra.apache.org" <user@cassandra.apache.org> > Date: Tuesday, March 31, 2015 at 10:15 PM > To: "user@cassandra.apache.org" <user@cassandra.apache.org> > > Subject: Re: Column value not getting updated > > Hey Saurabh, > > We're actually preparing for this ourselves and spinning up our own NTP > server pool. The public NTP pools have a lot of drift and should not be > relied upon for cluster technology that is sensitive to time skew like C*. > > The folks at Logentries did a great write up about this which we used as a > guide. > > > > - > > https://blog.logentries.com/2014/03/synchronizing-clocks-in-a-cassandra-cluster-pt-1-the-problem/ > - > > https://blog.logentries.com/2014/03/synchronizing-clocks-in-a-cassandra-cluster-pt-2-solutions/ > > > -Mark > > On Tue, Mar 31, 2015 at 5:59 PM, Saurabh Sethi <saurabh_se...@symantec.com > > wrote: > >> That’s what I found out that the clocks were not in sync. >> >> But I have setup NTP on all 3 nodes and would expect the clocks to be in >> sync. >> >> From: Nate McCall <n...@thelastpickle.com> >> Reply-To: "user@cassandra.apache.org" <user@cassandra.apache.org> >> Date: Tuesday, March 31, 2015 at 2:50 PM >> To: Cassandra Users <user@cassandra.apache.org> >> Subject: Re: Column value not getting updated >> >> You would see that if the servers' clocks were out of sync. >> >> Make sure the time on the servers is in sync or set the client timestamps >> explicitly. >> >> On Tue, Mar 31, 2015 at 3:23 PM, Saurabh Sethi < >> saurabh_se...@symantec.com> wrote: >> >>> I have written a unit test that creates a column family, inserts a row >>> in that column family and then updates the value of one of the columns. >>> >>> After updating, unit test immediately tries to read the updated value >>> for that column, but Cassandra returns the old value. >>> >>> - I am using QueryBuilder API and not CQL directly. >>> - I am using the consistency level of QUORUM for everything – >>> insert, update and read. >>> - Cassandra is running as a 3 node cluster with replication factor >>> of 3. >>> >>> >>> Anyone has any idea what is going on here? >>> >>> Thanks, >>> Saurabh >>> >> >> >> >> -- >> ----------------- >> Nate McCall >> Austin, TX >> @zznate >> >> Co-Founder & Sr. Technical Consultant >> Apache Cassandra Consulting >> http://www.thelastpickle.com >> > >
