Public bug reported: I used cProfile to profile neutron-ovs-agent (from neutron kilo 2015.1.0) as VMs are provisioned (see code sample below to reproduce).
I find a couple of functions in the IptablesManager scaling poorly with # of VMs (_modify_rules, and its callee find_last_entry). As the # of current VMs doubles, the time spent in these functions to provision 10 new VMs also roughly doubles, While we wait for the new IptablesTables firewall driver: https://blueprints.launchpad.net/neutron/+spec/new-iptables-driver Can we improve the performance of the current iptables firewall on those 2 functions which do a lot of string processing ? Current: #VMs: 20, # SG rules: 657, provision 10 new VMs 60 0.143 0.002 3.979 0.066 iptables_manager.py:511(_modify_rules) 25989 2.752 0.000 3.332 0.000 iptables_manager.py:504(_find_last_entry) Current #VMs: 40, # SG rules: 1277 , provision 10 new VMs 65 0.220 0.003 7.974 0.123 iptables_manager.py:511(_modify_rules) 38891 5.782 0.000 6.986 0.000 iptables_manager.py:504(_find_last_entry) Current #VMs: 80, # SG rules: 2517 , provision 10 new VMs 30 0.274 0.009 20.496 0.683 iptables_manager.py:511(_modify_rules) 43862 15.920 0.000 19.292 0.000 iptables_manager.py:504(_find_last_entry) current #VMs: 160, #SG rules: 4997, provision 10 new VMs 20 0.375 0.019 49.255 2.463 iptables_manager.py:511(_modify_rules) 56478 39.275 0.001 47.629 0.001 iptables_manager.py:504(_find_last_entry) To Reproduce: ------------------- Make following change to neutron_ovs_agent.py to enable/disable cProfile for a given scenario. < import cProfile < import os.path < pr_enabled = False < pr = None In OVSNeutronAgent add method: < def toggle_cprofile(self): < global pr, pr_enabled < start = False < data = "" < fname = "vm.profile" < try: < if os.path.isfile("/tmp/cprof"): < start = True < except IOError as e: < LOG.warn("Error %s", e.strerror) < < if start and not pr_enabled: < pr = cProfile.Profile() < pr.enable() < pr_enabled = True < LOG.warn("enabled cprofile") < if not start and pr_enabled: < pr.disable() < pr.create_stats() < pr.dump_stats("/tmp/%s"%fname) < pr_enabled = False < LOG.warn("disabled cprofile") In polling loop: < self.toggle_cprofile() ** Affects: neutron Importance: Undecided Status: New ** Description changed: + I used cProfile to profile neutron-ovs-agent (from neutron 2015.1.0) as + VMs are provisioned (see code sample below to reproduce). - I used cProfile to profile neutron-ovs-agent (from neutron 2015.1.0) as VMs are provisioned (see code sample below to reproduce). - - I find a couple of functions in the IptablesManager scaling poorly with # of VMs (_modify_rules, and its callee find_last_entry). + I find a couple of functions in the IptablesManager scaling poorly with # of VMs (_modify_rules, and its callee find_last_entry). As the # of current VMs doubles, the time spent in these functions to provision 10 new VMs also roughly doubles, While we wait for the new IptablesTables firewall driver: https://blueprints.launchpad.net/neutron/+spec/new-iptables-driver - Can we improve the performance of the current iptables firewall driver to perform better ? + Can we improve the performance of the current iptables firewall on those 2 functions which do a lot of string processing ? - - Current: #VMs: 20, # SG rules: 657, + Current: #VMs: 20, # SG rules: 657, provision 10 new VMs - 60 0.143 0.002 3.979 0.066 iptables_manager.py:511(_modify_rules) - 25989 2.752 0.000 3.332 0.000 iptables_manager.py:504(_find_last_entry) + 60 0.143 0.002 3.979 0.066 iptables_manager.py:511(_modify_rules) + 25989 2.752 0.000 3.332 0.000 iptables_manager.py:504(_find_last_entry) - Current #VMs: 40, # SG rules: 1277 , + Current #VMs: 40, # SG rules: 1277 , provision 10 new VMs - 65 0.220 0.003 7.974 0.123 iptables_manager.py:511(_modify_rules) - 38891 5.782 0.000 6.986 0.000 iptables_manager.py:504(_find_last_entry) + 65 0.220 0.003 7.974 0.123 iptables_manager.py:511(_modify_rules) + 38891 5.782 0.000 6.986 0.000 iptables_manager.py:504(_find_last_entry) - Current #VMs: 80, # SG rules: 2517 , + Current #VMs: 80, # SG rules: 2517 , provision 10 new VMs - 30 0.274 0.009 20.496 0.683 iptables_manager.py:511(_modify_rules) - 43862 15.920 0.000 19.292 0.000 iptables_manager.py:504(_find_last_entry) + 30 0.274 0.009 20.496 0.683 iptables_manager.py:511(_modify_rules) + 43862 15.920 0.000 19.292 0.000 iptables_manager.py:504(_find_last_entry) - - current #VMs: 160, #SG rules: 4997, + current #VMs: 160, #SG rules: 4997, provision 10 new VMs - 20 0.375 0.019 49.255 2.463 iptables_manager.py:511(_modify_rules) - 56478 39.275 0.001 47.629 0.001 iptables_manager.py:504(_find_last_entry) - + 20 0.375 0.019 49.255 2.463 iptables_manager.py:511(_modify_rules) + 56478 39.275 0.001 47.629 0.001 iptables_manager.py:504(_find_last_entry) To Reproduce: ------------------- Make following change to neutron_ovs_agent.py to enable/disable cProfile for a given scenario. < import cProfile < import os.path < pr_enabled = False < pr = None In OVSNeutronAgent add method: < def toggle_cprofile(self): < global pr, pr_enabled < start = False < data = "" < fname = "vm.profile" < try: < if os.path.isfile("/tmp/cprof"): < start = True < except IOError as e: < LOG.warn("Error %s", e.strerror) - < + < < if start and not pr_enabled: < pr = cProfile.Profile() < pr.enable() < pr_enabled = True < LOG.warn("enabled cprofile") < if not start and pr_enabled: < pr.disable() < pr.create_stats() < pr.dump_stats("/tmp/%s"%fname) < pr_enabled = False - < LOG.warn("disabled cprofile") + < LOG.warn("disabled cprofile") In polling loop: < self.toggle_cprofile() -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1492456 Title: cProfile - fix Security Groups hotfunctions Status in neutron: New Bug description: I used cProfile to profile neutron-ovs-agent (from neutron kilo 2015.1.0) as VMs are provisioned (see code sample below to reproduce). I find a couple of functions in the IptablesManager scaling poorly with # of VMs (_modify_rules, and its callee find_last_entry). As the # of current VMs doubles, the time spent in these functions to provision 10 new VMs also roughly doubles, While we wait for the new IptablesTables firewall driver: https://blueprints.launchpad.net/neutron/+spec/new-iptables-driver Can we improve the performance of the current iptables firewall on those 2 functions which do a lot of string processing ? Current: #VMs: 20, # SG rules: 657, provision 10 new VMs 60 0.143 0.002 3.979 0.066 iptables_manager.py:511(_modify_rules) 25989 2.752 0.000 3.332 0.000 iptables_manager.py:504(_find_last_entry) Current #VMs: 40, # SG rules: 1277 , provision 10 new VMs 65 0.220 0.003 7.974 0.123 iptables_manager.py:511(_modify_rules) 38891 5.782 0.000 6.986 0.000 iptables_manager.py:504(_find_last_entry) Current #VMs: 80, # SG rules: 2517 , provision 10 new VMs 30 0.274 0.009 20.496 0.683 iptables_manager.py:511(_modify_rules) 43862 15.920 0.000 19.292 0.000 iptables_manager.py:504(_find_last_entry) current #VMs: 160, #SG rules: 4997, provision 10 new VMs 20 0.375 0.019 49.255 2.463 iptables_manager.py:511(_modify_rules) 56478 39.275 0.001 47.629 0.001 iptables_manager.py:504(_find_last_entry) To Reproduce: ------------------- Make following change to neutron_ovs_agent.py to enable/disable cProfile for a given scenario. < import cProfile < import os.path < pr_enabled = False < pr = None In OVSNeutronAgent add method: < def toggle_cprofile(self): < global pr, pr_enabled < start = False < data = "" < fname = "vm.profile" < try: < if os.path.isfile("/tmp/cprof"): < start = True < except IOError as e: < LOG.warn("Error %s", e.strerror) < < if start and not pr_enabled: < pr = cProfile.Profile() < pr.enable() < pr_enabled = True < LOG.warn("enabled cprofile") < if not start and pr_enabled: < pr.disable() < pr.create_stats() < pr.dump_stats("/tmp/%s"%fname) < pr_enabled = False < LOG.warn("disabled cprofile") In polling loop: < self.toggle_cprofile() To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1492456/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
