Bug: https://lkml.org/lkml/2012/6/30/122
Simple fix to reset the leap second flag: date; date `date +"%m%d%H%M%C%y.%S"`; date; /Henrik On Mon, Jul 2, 2012 at 1:56 PM, Jean Paul Adant <jean.paul.ad...@gmail.com>wrote: > Hi, > > I did have the same problem with cassandra 1.1.1 on Ubuntu 11.10 > I had to reboot all nodes > I'm interested in any information about this. > > Thanks > > Jean Paul > > 2012/7/2 Filippo Diotalevi <fili...@ntoklo.com> > >> Hi, >> we had some really weird issues during the weekend, with our cassandra >> nodes starting marking as dead other (working) nodes in the cluster. That >> happened all Sunday, and it's still happening. Node are marked dead and up >> all the time…. >> >> Some example logs: >> >> INFO [GossipTasks:1] 2012-07-02 06:55:01,804 Gossiper.java (line 818) >> InetAddress /xx.xx.xx.233 is now dead. >> INFO [GossipTasks:1] 2012-07-02 06:55:01,805 Gossiper.java (line 818) >> InetAddress /xx.xx.xx.235 is now dead. >> INFO [GossipStage:1] 2012-07-02 06:55:21,748 Gossiper.java (line 804) >> InetAddress /xx.xx.xx.233 is now UP >> INFO [GossipStage:1] 2012-07-02 06:55:21,893 Gossiper.java (line 804) >> InetAddress /xx.xx.xx.235 is now UP >> INFO [GossipTasks:1] 2012-07-02 06:56:03,877 Gossiper.java (line 818) >> InetAddress /xx.xx.xx.235 is now dead. >> INFO [GossipTasks:1] 2012-07-02 06:57:58,537 Gossiper.java (line 818) >> InetAddress /xx.xx.xx.233 is now dead. >> INFO [GossipStage:1] 2012-07-02 06:59:06,444 Gossiper.java (line 804) >> InetAddress /xx.xx.xx.233 is now UP >> >> >> I couldn't find any real exception in the logs, but I noticed that the >> first error occurred at >> INFO [GossipTasks:1] 2012-07-01 02:00:31,169 Gossiper.java (line 818) >> InetAddress /xx.xx.xx.234 is now dead. >> >> 2012-07-01 02:00:31,169, in the German timezone were the machine is >> hosted, is June 30th 23:59:60 UTC, the leap second that caused quite a few >> issues this weekend. >> >> Can it be the cause of the cluster failure? Has anybody noticed similar >> issues? ( also see >> https://twitter.com/redditstatus/status/219244389044731904 ) >> >> I'm running Ubuntu 10.04.3 LTS. >> >> Many thanks, >> -- >> Filippo Diotalevi >> >> > > > -- > ----------------------------------------------------- > Jean Paul Adant - Créative-Ingénierie > jean.paul.ad...@gmail.com > > > >
