> -----Original Message----- > From: Prasanna Santhanam [mailto:prasanna.santha...@citrix.com] > Sent: Thursday, July 12, 2012 4:34 PM > To: cloudstack-dev@incubator.apache.org > Subject: Re: Limits set for a resource can be exceeded > > On Wed, Jul 11, 2012 at 05:50:47AM -0400, Devdeep Singh wrote: > > Hi, > > > > I observed this issue while looking into how resource limiting works. > > When a request is placed to create a resource (instance, volume etc.) > > the check for resource limits and incrementing the resource count is > > done in the following order > > > > 1. Check the limits will not be exceeded by the addition of another > > resource. Raise an exception if so. 2. Create a db entry for the > > resource. 3. Increment the resource count. > > > > The check and increment of a resource are not synchronized to be > > atomic. This can lead to scenarios where two different requests can > > successfully check the limits for the same resource and increment the > > count beyond the limits on boundary conditions. > > > > For example: Suppose domain limits for a vm instance is set to 5 and > > there are already 4 instances created for the domain. Two requests are > > placed by two different users of the domain to create an instance. 1. > > User 1 request checks the limit. Verifies that domain limit of 5 will > > not be exceeded. 2. Similarly User 2 request checks the limit and > > verifies the domain limit of 5 will not be exceeded. > > This is because the first request still hasn't incremented the > > resource count. 3. Both the requests go ahead and increment the count > > (to 6) and exceed the limit. > > > > I have observed this issue with instance creation. It may be present > > at other places too (volumes, public ips etc.). Before I go ahead and > > file a bug, I wanted to check if it is a valid concern or am I missing > > something. > > > > FWIW, this counter's atomicity used to be a real problem back in the last few > 2.2.y series of releases and partly the reason why the updateResourceCount > API came into being. But I remember this was fixed for accounts limits. > Whether domain-limits and other resource limits follow the same > increment/decrement code paths I'm not sure. If they don't we most likely > have the same bug. > > Can we reproduce this trivially or will we have to hold the threads in the > debugger to cause this situation? > > -- > Prasanna.,
I reproduced the bug for limits on instances in a debugger by suspending one thread after it had checked for limits and letting the second request go through (and then resuming the first thread). The problem with limits on snapshots is easier to reproduce because snapshot upload to secondary storage takes time and the increment is done after upload is successful. http://bugs.cloudstack.org/browse/CS-15430 was filed for the same. A fix for this has already been proposed. Regards, Devdeep
