Reviewed: https://review.openstack.org/489295
Committed:
https://git.openstack.org/cgit/openstack/neutron/commit/?id=ec709767e6ca7fa7d8ceb81b6ef76fdde1580289
Submitter: Jenkins
Branch: master
commit ec709767e6ca7fa7d8ceb81b6ef76fdde1580289
Author: Kevin Benton <ke...@benton.pub>
Date: Mon Jul 31 11:28:02 2017 -0700
Check if record is stale after bulk pull
We need to check if the results from a bulk_pull are stale because
the resource might have been updated and concurrently pushed from
the server while the bulk_pull query was being fulfilled.
Change-Id: I755a1cb2e0037ec2316161a09ad462bc4b09f397
Closes-Bug: #1707699
** Changed in: neutron
Status: In Progress => Fix Released
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1707699
Title:
resource cache can push in stale data on server query
Status in neutron:
Fix Released
Bug description:
There is a race condition in the push notifications resource cache
where a query to the server can wipe out newer data received for a
given resource.
Order of operations:
* resource_cache issues bulk_pull request to server
* server starts to build response in one thread
* server updates one of the resources in another thread and pushes out the
resource onto the queue
* client receives the updated resource and updates the cache
* bulk_pull finally returns and wipes out the newer version of the object
with the one in the response.
Snip from
http://logs.openstack.org/22/410422/27/gate/gate-tempest-dsvm-py35-ubuntu-xenial/436680c/logs/screen-q-agt.txt.gz
on relevant messages showing stale data:
Jul 31 15:09:43.914823 ubuntu-xenial-internap-mtl01-10175685
neutron-openvswitch-agent[19186]: DEBUG neutron.agent.resource_cache [None
req-d91a8fc6-2545-49e7-a7aa-90f764483e5e service neutron] Resource Port
680f28df-4950-4af0-adf6-67f03d096a46 updated (revision_number 4->5). Old
fields: {'binding_levels': [], 'binding':
PortBinding(host='ubuntu-xenial-internap-mtl01-10175685',port_id=680f28df-4950-4af0-adf6-67f03d096a46,profile={},status='ACTIVE',vif_details=None,vif_type='unbound',vnic_type='normal')}
New fields: {'binding_levels':
[PortBindingLevel(driver='openvswitch',host='ubuntu-xenial-internap-mtl01-10175685',level=0,port_id=680f28df-4950-4af0-adf6-67f03d096a46,segment=NetworkSegment(7fff9c50-c9c8-47f5-87ad-3a1aed38f02c))],
'binding':
PortBinding(host='ubuntu-xenial-internap-mtl01-10175685',port_id=680f28df-4950-4af0-adf6-67f03d096a46,profile={},status='ACTIVE',vif_details={"ovs_hybrid_plug":
true, "datapath_type": "system", "port_filter":
true},vif_type='ovs',vnic_type='nor
mal')} {{(pid=19186) record_resource_update
/opt/stack/new/neutron/neutron/agent/resource_cache.py:180}}
Jul 31 15:09:44.180728 ubuntu-xenial-internap-mtl01-10175685
neutron-openvswitch-agent[19186]: DEBUG neutron.agent.resource_cache [None
req-b9f3b846-1d32-43dd-acd7-c68fc5da9a87 None None] 19 resources returned for
queries {('Port', ('security_group_ids',
('76bbd669-2595-4e44-bd7d-cb1f5096cb28',)))} {{(pid=19186)
_flood_cache_for_query
/opt/stack/new/neutron/neutron/agent/resource_cache.py:80}}
Jul 31 15:09:45.308162 ubuntu-xenial-internap-mtl01-10175685
neutron-openvswitch-agent[19186]: WARNING neutron.agent.rpc [None
req-b9f3b846-1d32-43dd-acd7-c68fc5da9a87 None None] Device
Port(admin_state_up=True,allowed_address_pairs=[],binding=PortBinding,binding_levels=[],created_at=2017-07-31T15:09:39Z,data_plane_status=<?>,description='',device_id='c139f8f1-0be7-4a2e-bcbe-50df320effcb',device_owner='compute:nova',dhcp_options=[],distributed_binding=None,dns=None,fixed_ips=[IPAllocation],id=680f28df-4950-4af0-adf6-67f03d096a46,mac_address=fa:16:3e:12:5d:f7,name='',network_id=96ad3ed5-7530-4e26-a0a9-99abe3157e0c,project_id='330410e9ec0044f494be2def53f589b1',qos_policy_id=None,revision_number=4,security=PortSecurity(680f28df-4950-4af0-adf6-67f03d096a46),security_group_ids=set([76bbd669-2595-4e44-bd7d-cb1f5096cb28]),status='DOWN',updated_at=2017-07-31T15:09:43Z)
is not bound.
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1707699/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
