Isn't there some other way we can sync the time as a triage that would avoid a large test on the system vm template? Like adding a time flag to one of the commonly-run system vm scripts (or even just the ones that trigger tasks that require accurate time), which would run a 'date' command in the system vm, and modifying the applicable CitrixResourceBase methods to pass epoch timestamp along when these scripts are called?
On Mon, May 20, 2013 at 2:05 PM, Chip Childers <chip.child...@sungard.com> wrote: > We appear to be at an impasse here as well. I'm going to start a VOTE > on this issue, given that we have no easy answer. > > > On Wed, May 15, 2013 at 02:49:41PM -0700, Chiradeep Vittal wrote: >> Well, I disagree, from the perspective of hundreds of production clouds. >> No harm has been perceived in those clouds due to this defect. If they can >> live with it for several years, then they can live with it for a few more >> months. >> >> On 5/15/13 2:35 PM, "Wido den Hollander" <w...@widodh.nl> wrote: >> >> > >> > >> >On 05/15/2013 11:33 PM, John Burwell wrote: >> >> Chiradeep, >> >> >> >> I disagree regarding the impact of this issue. Anyone running an SSVM >> >>will be affected by this issue because clocks will eventually drift >> >>(sooner rather than later) and when they do, any timestamps rendered by >> >>a system VM will unreliable (e.g. files creation and modified times, log >> >>entries, etc). When I put my sys admin hat on, that type behavior would >> >>4.1 an unacceptable release to me. >> >> >> > >> >Ack. Clocks should always be on time. No matter what a machine is doing. >> >Have the correct time is crucial. >> > >> >Think about log lines, debugging, etc, etc. >> > >> >Nothing wrong with running NTP in KVM System VM btw. >> > >> >Wido >> > >> >> Thanks, >> >> -John >> >> >> >> On May 15, 2013, at 5:24 PM, Chiradeep Vittal >> >><chiradeep.vit...@citrix.com> wrote: >> >> >> >>> Sure, I agree. But I'd also point out that for the vast majority of >> >>> CloudStack users (4.1 at least), this is not going to be an issue. I >> >>> suggest deferring this to 4.1.1 >> >>> A new template (easy or not) does require a full regression QA round. >> >>> >> >>> On 5/15/13 2:07 PM, "John Burwell" <jburw...@basho.com> wrote: >> >>> >> >>>> Chiradeep, >> >>>> >> >>>> As I think thought it, I don't think a documentation note will >> >>>>sufficient >> >>>> because the SSVM can be destroyed and respawned by CloudStack without >> >>>> intervention by a human. Therefore, we can get into a situation >> >>>>where an >> >>>> operator installs and configures NTP, and then at some point in the >> >>>> future, the SSVM gets respawned and clocks drift. The worst part >> >>>>about >> >>>> this scenario is that the failures for S3 sync become silent since we >> >>>> have no mechanism to surface the failure to a dashboard or monitoring >> >>>> system. >> >>>> >> >>>> How much effort (i.e. hours/days) would it be to build a new image? >> >>>> >> >>>> Thanks, >> >>>> -John >> >>>> >> >>>> On May 15, 2013, at 4:59 PM, Chiradeep Vittal >> >>>> <chiradeep.vit...@citrix.com> wrote: >> >>>> >> >>>>> Did some further digging around as to why >> >>>>> /proc/sys/xen/independent_wallclock is not there on the Debian system >> >>>>> vm. >> >>>>> >> >>>>> TLDR; the kernel is PvOps (I.e., just a regular kernel that works >> >>>>>like a >> >>>>> PV kernel, not a specialized paravirt kernel). To eliminate >> >>>>> special-casing, the independent_wallclock feature was dropped. >> >>>>>However, >> >>>>> this means that the domU clock is NO LONGER synch'ed to dom0 and NTP >> >>>>>is >> >>>>> required on ANY domU. So the clock on the domU is only sync'ed at >> >>>>>domU >> >>>>> creation time. I suspect Citrix XenServer might have some workaround >> >>>>> >> >>>>> >> >>>>> http://www.gossamer-threads.com/lists/xen/users/234750 >> >>>>> >> >>>>> >> >>>>> What this means is that we *may* need to run ntp on system vms on >> >>>>>Xen. >> >>>>> This is of course a non-trivial change involving updates to >> >>>>>systemvms. >> >>>>> >> >>>>> I suspect that it does not matter much for virtual routers or console >> >>>>> proxy vms. >> >>>>> >> >>>>> We could have an advisory that for those users that care (e.g., those >> >>>>> using S3 sync) that they need to run ntp after the SSVM has been >> >>>>> created. >> >>>>> I.e, login to the SSVM and run >> >>>>> apt-get update >> >>>>> apt-get install ntp >> >>>>> >> >>>>> -- >> >>>>> Chiradeep >> >>>>> >> >>>>> >> >>>>> >> >>>>> On 5/15/13 1:11 PM, "Chiradeep Vittal" <chiradeep.vit...@citrix.com> >> >>>>> wrote: >> >>>>> >> >>>>>> For VMWare, the command >> >>>>>> vmware-toolbox-cmd timesync status returns 'Disabled'. I can submit >> >>>>>>a >> >>>>>> patch for /etc/init.d/cloud-early-config to enable it >> >>>>>> >> >>>>>> >> >>>>>> On 5/15/13 12:23 PM, "Chiradeep Vittal" >> >>>>>><chiradeep.vit...@citrix.com> >> >>>>>> wrote: >> >>>>>> >> >>>>>>> I am not sure why it is missing, but I will refer to >> >>>>>>> Citrix XenServer 6.0 Virtual Machine Installation Guide >> >>>>>>> >> >>>>>>> http://s.apache.org/YGn >> >>>>>>> >> >>>>>>> And quote >> >>>>>>> "Time Handling in Linux VMs >> >>>>>>> By default, the clocks in a Linux VM are synchronized to the clock >> >>>>>>> running >> >>>>>>> on the control domain, and cannot be >> >>>>>>> independently changed. This mode is a convenient default, since >> >>>>>>>only >> >>>>>>> the >> >>>>>>> control domain needs to be running the >> >>>>>>> NTP service to keep accurate time across all VMs. Upon installation >> >>>>>>> of a >> >>>>>>> new Linux VM, make sure you change the >> >>>>>>> time-zone from the default UTC to your local value (see the section >> >>>>>>> called >> >>>>>>> ³Release Notes² for specific distribution >> >>>>>>> instructions). >> >>>>>>> To set individual Linux VMs to maintain independent times >> >>>>>>> 1. From a root prompt on the VM, run the command: echo 1 > >> >>>>>>> /proc/sys/xen/independent_wallclock >> >>>>>>> 2. This can be persisted across reboots by changing the >> >>>>>>> /etc/sysctl.conf >> >>>>>>> configuration file and adding: >> >>>>>>> # Set independent wall clock time >> >>>>>>> xen.independent_wallclock=1 >> >>>>>>> 3. As a third alternative, independent_wallclock=1 may also be >> >>>>>>>passed >> >>>>>>> as >> >>>>>>> a >> >>>>>>> boot parameter to the VM. >> >>>>>>> " >> >>>>>>> >> >>>>>>> >> >>>>>>> On 5/15/13 12:09 PM, "Chip Childers" <chip.child...@sungard.com> >> >>>>>>> wrote: >> >>>>>>> >> >>>>>>>> On Wed, May 15, 2013 at 12:02:41PM -0700, Chiradeep Vittal wrote: >> >>>>>>>>> /proc/sys is not a regular filesystem and cannot be added to from >> >>>>>>>>> the >> >>>>>>>>> shell. >> >>>>>>>>> Drivers need to add nodes into this filesystem. >> >>>>>>>> >> >>>>>>>> Backing up a bit... >> >>>>>>>> >> >>>>>>>> This is the current system VM image that the 4.1 software should >> >>>>>>>>be >> >>>>>>>> using on Xen: >> >>>>>>>> >> >>>>>>>> http://download.cloud.com/templates/acton/acton- >> >>>>>>>> systemvm-02062012.vhd.bz2 >> >>>>>>>> >> >>>>>>>> Does this system VM have the correct drives installed to set >> >>>>>>>> /proc/sys/xen to use the Dom0 time for sync? >> >>>>>>>> >> >>>>>>>> If so, then is this a devcloud only issue for Xen? If that's the >> >>>>>>>> case, >> >>>>>>>> then we shouldn't block a release to simply improve things. >> >>>>>>>> >> >>>>>>>> We do know that a patch for kvm was needed. >> >>>>>>>> >> >>>>>>>> We still don't have any >> >>>>>>>> answer or comments about the VMware system VM (specifically this >> >>>>>>>> template: http://download.cloud.com/templates/burbank/burbank- >> >>>>>>>> systemvm-08012012.ova >> >>>>>>>> >> >>>>>>> >> >>>>>> >> >>>>> >> >>>> >> >>> >> >> >>
