Hi, I just posted in our operations blog how CERN is dealing with quotas synchronization problem.
http://openstack-in-production.blogspot.fr/2015/03/nova-quota-usage-synchronization.html Hope it helps, cheers, Belmiro On Sat, Mar 21, 2015 at 12:55 AM, Sam Morrison <sorri...@gmail.com> wrote: > I’d need to go through the logs to see exactly the occurrence maybe 10 > minutes was a bit much. We run the cron job every 10 mins. > > The cron job stopped a few weeks back and we had at least half a dozen > projects with out of sync quotas within a few hours. > > Cheers, > Sam > > > > On 21 Mar 2015, at 4:01 am, Mike Dorman <mdor...@godaddy.com> wrote: > > > > Hey Sam, > > > > When you say it occurs every 10 minutes, what exactly do you mean? The > > quota refresh? Or a quota getting out of sync? > > > > I am surprised you have max_age set so low. I would think that would > > basically trigger a quota refresh on every single reservation for most > > users, right? > > > > Mike > > > > > > > > > > > > On 3/20/15, 4:18 AM, "Sam Morrison" <sorri...@gmail.com> wrote: > > > >> We’ve had the following for a year or so but doesn’t help much, we still > >> see it occurring every 10 mins or so. > >> > >> max_age = 10 > >> until_refresh = 5 > >> reservation_expire=600 > >> > >> We have a cron job that runs every 10 mins that figures out what > projects > >> are out of sync and corrects them. > >> We’ve always been scared of setting these to zero but we probably > should. > >> > >> Sam > >> > >> > >>> On 15 Mar 2015, at 2:53 pm, Mike Dorman <mdor...@godaddy.com> wrote: > >>> > >>> Yeah the default is just ‘0’ for both, which disables the refresh. > >>> > >>> > >>> > >>> The one downside is that it may not be 100% transparent to the user. > >>> If > >>> the quota is already (incorrectly) too high, and exceeding the quota > >>> limit, the reservation that triggers the refresh will still fail. I.e. > >>> the reservation is attempted based on the quota usage values _before_ > >>> the > >>> refresh. But then after that the quota should be fixed and it will > >>> work > >>> again on the next reservation. > >>> > >>> But my thinking is that most quota issues happen slowly over time. If > >>> we > >>> are correcting them often and automatically, they hopefully never get > >>> to > >>> the point where they’re bad enough to manifest reservation errors to > >>> the > >>> user. > >>> > >>> I don’t have any information re: db load. I assume it regenerates > >>> based > >>> on what’s in the instances or reservations table. I imagine the load > >>> for > >>> doing a single refresh is probably comparable to doing a ‘nova list’. > >>> > >>> Mike > >>> > >>> > >>> > >>> On 3/14/15, 2:27 PM, "Tim Bell" <tim.b...@cern.ch> wrote: > >>> > >>>> Interesting... what are the defaults ? > >>>> > >>>> Assuming no massive DB load, getting synced within a day would seem > >>>> reasonable. Is the default no max age ? > >>>> > >>>> Tim > >>>> > >>>>> -----Original Message----- > >>>>> From: Jesse Keating [mailto:j...@bluebox.net] > >>>>> Sent: 14 March 2015 16:59 > >>>>> To: openstack-operators@lists.openstack.org > >>>>> Subject: Re: [Openstack-operators] max_age and until_refresh for > >>>>> fixing > >>>>> Nova > >>>>> quotas > >>>>> > >>>>> On 3/14/15 8:11 AM, Mike Dorman wrote: > >>>>>> I did short write-up here http://t.co/Q5X1hTgJG1 if you are > >>>>>> interested > >>>>>> in the details. > >>>>>> > >>>>> > >>>>> Thanks for sharing Matt! That's an excellent write up. > >>>>> > >>>>> -- > >>>>> -jlk > >>>>> > >>>>> _______________________________________________ > >>>>> OpenStack-operators mailing list > >>>>> OpenStack-operators@lists.openstack.org > >>>>> > >>>>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators > >>>> > >>>> _______________________________________________ > >>>> OpenStack-operators mailing list > >>>> OpenStack-operators@lists.openstack.org > >>>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators > >>> _______________________________________________ > >>> OpenStack-operators mailing list > >>> OpenStack-operators@lists.openstack.org > >>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators > >> > > _______________________________________________ > > OpenStack-operators mailing list > > OpenStack-operators@lists.openstack.org > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators > > > _______________________________________________ > OpenStack-operators mailing list > OpenStack-operators@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators >
_______________________________________________ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
