Reviewed: https://review.openstack.org/301685 Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=1cb43734808eded87210d2957d56b70c514d55c3 Submitter: Jenkins Branch: master
commit 1cb43734808eded87210d2957d56b70c514d55c3 Author: Oleg Bondarev <[email protected]> Date: Tue Apr 5 16:18:03 2016 +0300 ADDRESS_SCOPE_MARK_IDS should not be global for L3 agent Otherwise agent becomes unable to handle more than 1024 routers (including deleted routers) and starts failing. It should be enough to distinguish address scopes inside router namespace, so this patch moves ADDRESS_SCOPE_MARK_IDS set to the RouterInfo class. Closes-Bug: #1566291 Change-Id: I1e43bb3e68db4db93cc1dfc1383af0311bfb0f2d ** Changed in: neutron Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1566291 Title: L3 agent: at some point an agent becomes unable to handle new routers Status in neutron: Fix Released Bug description: Following seen in l3 agent logs: 2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent [-] Failed to process compatible router 'e341e0e2-5089-46e9-91f9-2099a156b27f' 2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent Traceback (most recent call last): 2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 497, in _process_router_update 2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent self._process_router_if_compatible(router) 2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 434, in _process_router_if_compatible 2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent self._process_added_router(router) 2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 439, in _process_added_router 2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent self._router_added(router['id'], router) 2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 340, in _router_added 2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent ri = self._create_router(router_id, router) 2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 337, in _create_router 2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent return legacy_router.LegacyRouter(*args, **kwargs) 2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 61, in __init__ 2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent DEFAULT_ADDRESS_SCOPE: ADDRESS_SCOPE_MARK_IDS.pop()} 2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent KeyError: 'pop from an empty set' 2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent So agent is constantly resyncing (causing load on neutron server) and unable to handle new routers. I believe that set "ADDRESS_SCOPE_MARK_IDS = set(range(1024, 2048))" from router_info.py should not be agent global but it should be ADDRESS_SCOPE_MARK_IDS per router. Or at least need to return values back to the set when router is deleted. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1566291/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
