Public bug reported: There is a huge performance issue with the security groups when using the iptables implementation: If you have a security group with say 500 rules it will take minutes for the RPC server to create the the port configuration. You will see this when you restart the neutron-linuxbridge-agent on a compute node with an instance with the security group with a lot of rules. In the agent log you will see "Preparing filters for devices" and this will take minutes for a single port when having a significate amount of rules in the security group.
After some investigation this seems to be cause: In the commit below stateful functionality was added for iptables implementation: https://opendev.org/openstack/neutron/commit/cbc473e066d#diff-7d7a372d8ed39ad8489a39ff7c3f3d783235218c However there is a huge performance impact, in the following function in neutron/db/securitygroups_rpc_base.py def security_group_info_for_ports For EACH rule in the security group rule in a a group it will do a database lookup to check what the setting is on the group: stateful = self._is_security_group_stateful(context, security_group_id) Which will call: def _is_security_group_stateful(self, context, sg_id): return sg_obj.SecurityGroup.get_sg_by_id(context, sg_id).stateful So if you have say 500 rules it will go 500 times(!) to the database to check the exact same property on the group object which absolutely tanks performance. I played around with caching the stateful property for the group (since it is not even changeable on a security group if there are rules present) and the function went from taking multiple minutes to about a second. ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2045950 Title: Security group performance issue for iptables driver due to "stateless feature" Status in neutron: New Bug description: There is a huge performance issue with the security groups when using the iptables implementation: If you have a security group with say 500 rules it will take minutes for the RPC server to create the the port configuration. You will see this when you restart the neutron-linuxbridge-agent on a compute node with an instance with the security group with a lot of rules. In the agent log you will see "Preparing filters for devices" and this will take minutes for a single port when having a significate amount of rules in the security group. After some investigation this seems to be cause: In the commit below stateful functionality was added for iptables implementation: https://opendev.org/openstack/neutron/commit/cbc473e066d#diff-7d7a372d8ed39ad8489a39ff7c3f3d783235218c However there is a huge performance impact, in the following function in neutron/db/securitygroups_rpc_base.py def security_group_info_for_ports For EACH rule in the security group rule in a a group it will do a database lookup to check what the setting is on the group: stateful = self._is_security_group_stateful(context, security_group_id) Which will call: def _is_security_group_stateful(self, context, sg_id): return sg_obj.SecurityGroup.get_sg_by_id(context, sg_id).stateful So if you have say 500 rules it will go 500 times(!) to the database to check the exact same property on the group object which absolutely tanks performance. I played around with caching the stateful property for the group (since it is not even changeable on a security group if there are rules present) and the function went from taking multiple minutes to about a second. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/2045950/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
