On 05/12/2016 08:26 PM, Ilya Chukhnakov wrote: > Hi everyone. > > I’ve recently found that straightforward use of NeutronDbObject is prone to > concurrency-related problems. > > I’ve submitted a patch set [3] with some tests to show that without special > treatment using NeutronDbObject could lead to unexpected results. > > Further patch sets will provide acquire_object/acquire_objects contextmanager > methods to the NeutronDbObject class. These methods are to be used in place of > get_object/get_objects whenever the user intends to make changes to the > object. > These methods would start an autonested_transaction. > > There are (at least) two potential options for the implementation: > > 1. Based on the DB locks (e.g. SELECT FOR UPDATE/SqlAlchemy with_for_update). > > pros: > - the object is guaranteed to not be changed while within the context > > cons: > - prone to deadlocks ([1] and potentially when locking multiple objects) > > 2. Lock-free CAS based on object version counter. Can use SqlAlchemy version > counter [2] or add our own. If conflicting changes are detected upon > exiting > the context (i.e. version counter held differs from the one in the DB), > will > raise OSLO RetryRequest exception. > > pros: > - does not require locking > > cons: > - require an additional field in the models > > While opt.2 only prevents the conflicting changes, but does not guarantee that > the object does not change while within the context, opt.1 may seem > preferential. But even with opt.1 the user should not expect that the changes > made to the object while within the context will get to the database as the > autonested_transaction could fail on flush/commit. > > So I’d like to hear others’ opinion on the problem and which of the two > implementation options would be preferred? Or maybe someone has a better idea. > > [1] > https://wiki.openstack.org/wiki/OpenStack_and_SQLAlchemy#MySQLdb_.2B_eventlet_.3D_sad > [2] http://docs.sqlalchemy.org/en/rel_0_9/orm/versioning.html > > [3] https://review.openstack.org/#/c/315705/
In Cinder we're handling similar problems related to race conditions between status check and status change with conditional updates. Basically we're doing "UPDATE table SET status='abc' WHERE id=1 AND status='status_allowing_transition_to_foo';". You can check out o.vo layer of that stuff at [1]. You could provide fields you care not to be modified in the expected_values and retry DB operations on failure. In general the problem you're mentioning is related more to the concurrent DB updates and o.vo never aimed for magically solving that problem. I believe you've had same problem with raw SQLA objects. [1] https://github.com/openstack/cinder/blob/master/cinder/objects/base.py#L173-L289 __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
