Public bug reported: Upgrading neutron from rocky -> stein and get a considerable slow down when listing all security groups for a project. Goes from ~2 seconds to almost 2 minutes. Looking into the code it looks like it is very inefficient because it gets all rules from the DB and then filters after the fact. We have around 7000 rules in our QA env.
Very keen to get this sorted but don't know the neutron code base that well so can offer testing of patches if there are any out there already. It looks like this happened with listing ports too for stein and found this https://bugzilla.redhat.com/show_bug.cgi?id=1737012 so wonder if this is related? With Rocky: time openstack security group rule list +--------------------------------------+-------------+-----------+--------------------+------------+--------------------------------------+--------------------------------------+ | ID | IP Protocol | Ethertype | IP Range | Port Range | Remote Security Group | Security Group | +--------------------------------------+-------------+-----------+--------------------+------------+--------------------------------------+--------------------------------------+ | 01b877cc-1621-44cd-8e69-1345ab01a1ef | None | IPv4 | 0.0.0.0/0 | | None | 3dcbd4fa-d017-4361-b0b0-b7508e923087 | | 0c744788-6319-42e5-931a-5e7b0df166c4 | None | IPv6 | ::/0 | | None | 3dcbd4fa-d017-4361-b0b0-b7508e923087 | | 0fc6b79d-d211-4201-ac76-60fb8ea40c9c | None | IPv4 | 0.0.0.0/0 | | None | 8f55c18b-cd8c-4d84-afef-f8b83d5eb128 | | 17d6c8a3-7894-42a6-92f2-1bd56a30ef1d | tcp | IPv4 | 0.0.0.0/0 | 80:80 | None | ed257fd7-d825-4014-96a8-c16adfea70f0 | | 19d3ba79-65f1-4c89-a1c2-b32049ceb25a | None | IPv6 | ::/0 | | None | 008510a7-d176-4ee5-87e2-e74da06c55ba | | 21f1d173-b99f-47a7-9983-6926f7bc58f3 | icmp | IPv4 | 0.0.0.0/0 | | None | 008510a7-d176-4ee5-87e2-e74da06c55ba | | 3321d5ff-11c3-4104-be13-107c789e4bf8 | None | IPv6 | ::/0 | | None | 57cb14de-dd5f-4f0c-b0cf-a7effc36fca5 | | 381c6816-9b5c-42b7-9dd3-dae12a49c08b | None | IPv4 | 0.0.0.0/0 | | None | 3f63cfbb-87ee-4aa2-8193-7e86cb542881 | | 3886ad10-99ea-4f60-a36c-ffbe80d92907 | None | IPv6 | ::/0 | | None | ed257fd7-d825-4014-96a8-c16adfea70f0 | | 5be4853a-75d1-435c-87ca-56c54a243f70 | None | IPv4 | 0.0.0.0/0 | | None | 57cb14de-dd5f-4f0c-b0cf-a7effc36fca5 | | 71656249-4454-410e-8e7d-24910df127ba | None | IPv6 | ::/0 | | None | 8f55c18b-cd8c-4d84-afef-f8b83d5eb128 | | 783324ac-6844-4d4d-985c-936015bcb66e | icmp | IPv4 | 0.0.0.0/0 | | None | 3f63cfbb-87ee-4aa2-8193-7e86cb542881 | | 7ca7f0cc-b4df-401f-aaa4-662f17afcfb0 | None | IPv4 | 0.0.0.0/0 | | None | 008510a7-d176-4ee5-87e2-e74da06c55ba | | 825a33ff-b693-456d-811e-a0b494e8e308 | None | IPv6 | ::/0 | | 008510a7-d176-4ee5-87e2-e74da06c55ba | 008510a7-d176-4ee5-87e2-e74da06c55ba | | 89fd2d18-45d3-4a86-a020-09d240912e5c | tcp | IPv4 | 128.250.116.173/32 | 22:22 | None | 008510a7-d176-4ee5-87e2-e74da06c55ba | | 8a1f45f1-e4c8-41e4-b6f3-80ab48b7e38d | None | IPv6 | ::/0 | | None | bf7abb53-e5ca-428d-9fce-6a2e37a25ee0 | | 9ebc6d15-e3eb-4d20-88d4-6737367ffc08 | None | IPv4 | 0.0.0.0/0 | | None | ed257fd7-d825-4014-96a8-c16adfea70f0 | | 9f29f539-a80a-4a8d-89cc-f714224b5f8c | icmp | IPv4 | 0.0.0.0/0 | | None | 8f55c18b-cd8c-4d84-afef-f8b83d5eb128 | | a1bc8f05-3a20-48c2-bae5-a60f4ffed514 | None | IPv4 | 0.0.0.0/0 | | 008510a7-d176-4ee5-87e2-e74da06c55ba | 008510a7-d176-4ee5-87e2-e74da06c55ba | | bef999d6-669a-47f6-988c-e69bab6df87a | tcp | IPv4 | 0.0.0.0/0 | 22:22 | 57cb14de-dd5f-4f0c-b0cf-a7effc36fca5 | bf7abb53-e5ca-428d-9fce-6a2e37a25ee0 | | c5ce339b-cd92-492c-9af4-6eab875027ce | tcp | IPv4 | 0.0.0.0/0 | 80:80 | 008510a7-d176-4ee5-87e2-e74da06c55ba | 008510a7-d176-4ee5-87e2-e74da06c55ba | | d9ec0eba-d80d-4331-a588-e4f8c1c75533 | None | IPv6 | ::/0 | | None | 3f63cfbb-87ee-4aa2-8193-7e86cb542881 | | de760e03-92a9-4183-8acc-1d82addc3604 | None | IPv4 | 0.0.0.0/0 | | None | bf7abb53-e5ca-428d-9fce-6a2e37a25ee0 | | f4bc4616-1d18-4488-84bb-546516c053bc | tcp | IPv4 | 0.0.0.0/0 | 443:443 | None | ed257fd7-d825-4014-96a8-c16adfea70f0 | +--------------------------------------+-------------+-----------+--------------------+------------+--------------------------------------+--------------------------------------+ real 0m2.499s user 0m0.642s sys 0m0.053s With Stein: time openstack security group rule list +--------------------------------------+-------------+-----------+--------------------+------------+--------------------------------------+--------------------------------------+ | ID | IP Protocol | Ethertype | IP Range | Port Range | Remote Security Group | Security Group | +--------------------------------------+-------------+-----------+--------------------+------------+--------------------------------------+--------------------------------------+ | 01b877cc-1621-44cd-8e69-1345ab01a1ef | None | IPv4 | 0.0.0.0/0 | | None | 3dcbd4fa-d017-4361-b0b0-b7508e923087 | | 0c744788-6319-42e5-931a-5e7b0df166c4 | None | IPv6 | ::/0 | | None | 3dcbd4fa-d017-4361-b0b0-b7508e923087 | | 0fc6b79d-d211-4201-ac76-60fb8ea40c9c | None | IPv4 | 0.0.0.0/0 | | None | 8f55c18b-cd8c-4d84-afef-f8b83d5eb128 | | 17d6c8a3-7894-42a6-92f2-1bd56a30ef1d | tcp | IPv4 | 0.0.0.0/0 | 80:80 | None | ed257fd7-d825-4014-96a8-c16adfea70f0 | | 19d3ba79-65f1-4c89-a1c2-b32049ceb25a | None | IPv6 | ::/0 | | None | 008510a7-d176-4ee5-87e2-e74da06c55ba | | 21f1d173-b99f-47a7-9983-6926f7bc58f3 | icmp | IPv4 | 0.0.0.0/0 | | None | 008510a7-d176-4ee5-87e2-e74da06c55ba | | 3321d5ff-11c3-4104-be13-107c789e4bf8 | None | IPv6 | ::/0 | | None | 57cb14de-dd5f-4f0c-b0cf-a7effc36fca5 | | 381c6816-9b5c-42b7-9dd3-dae12a49c08b | None | IPv4 | 0.0.0.0/0 | | None | 3f63cfbb-87ee-4aa2-8193-7e86cb542881 | | 3886ad10-99ea-4f60-a36c-ffbe80d92907 | None | IPv6 | ::/0 | | None | ed257fd7-d825-4014-96a8-c16adfea70f0 | | 5be4853a-75d1-435c-87ca-56c54a243f70 | None | IPv4 | 0.0.0.0/0 | | None | 57cb14de-dd5f-4f0c-b0cf-a7effc36fca5 | | 71656249-4454-410e-8e7d-24910df127ba | None | IPv6 | ::/0 | | None | 8f55c18b-cd8c-4d84-afef-f8b83d5eb128 | | 783324ac-6844-4d4d-985c-936015bcb66e | icmp | IPv4 | 0.0.0.0/0 | | None | 3f63cfbb-87ee-4aa2-8193-7e86cb542881 | | 7ca7f0cc-b4df-401f-aaa4-662f17afcfb0 | None | IPv4 | 0.0.0.0/0 | | None | 008510a7-d176-4ee5-87e2-e74da06c55ba | | 825a33ff-b693-456d-811e-a0b494e8e308 | None | IPv6 | ::/0 | | 008510a7-d176-4ee5-87e2-e74da06c55ba | 008510a7-d176-4ee5-87e2-e74da06c55ba | | 89fd2d18-45d3-4a86-a020-09d240912e5c | tcp | IPv4 | 128.250.116.173/32 | 22:22 | None | 008510a7-d176-4ee5-87e2-e74da06c55ba | | 8a1f45f1-e4c8-41e4-b6f3-80ab48b7e38d | None | IPv6 | ::/0 | | None | bf7abb53-e5ca-428d-9fce-6a2e37a25ee0 | | 9ebc6d15-e3eb-4d20-88d4-6737367ffc08 | None | IPv4 | 0.0.0.0/0 | | None | ed257fd7-d825-4014-96a8-c16adfea70f0 | | 9f29f539-a80a-4a8d-89cc-f714224b5f8c | icmp | IPv4 | 0.0.0.0/0 | | None | 8f55c18b-cd8c-4d84-afef-f8b83d5eb128 | | a1bc8f05-3a20-48c2-bae5-a60f4ffed514 | None | IPv4 | 0.0.0.0/0 | | 008510a7-d176-4ee5-87e2-e74da06c55ba | 008510a7-d176-4ee5-87e2-e74da06c55ba | | bef999d6-669a-47f6-988c-e69bab6df87a | tcp | IPv4 | 0.0.0.0/0 | 22:22 | 57cb14de-dd5f-4f0c-b0cf-a7effc36fca5 | bf7abb53-e5ca-428d-9fce-6a2e37a25ee0 | | c5ce339b-cd92-492c-9af4-6eab875027ce | tcp | IPv4 | 0.0.0.0/0 | 80:80 | 008510a7-d176-4ee5-87e2-e74da06c55ba | 008510a7-d176-4ee5-87e2-e74da06c55ba | | d9ec0eba-d80d-4331-a588-e4f8c1c75533 | None | IPv6 | ::/0 | | None | 3f63cfbb-87ee-4aa2-8193-7e86cb542881 | | de760e03-92a9-4183-8acc-1d82addc3604 | None | IPv4 | 0.0.0.0/0 | | None | bf7abb53-e5ca-428d-9fce-6a2e37a25ee0 | | f4bc4616-1d18-4488-84bb-546516c053bc | tcp | IPv4 | 0.0.0.0/0 | 443:443 | None | ed257fd7-d825-4014-96a8-c16adfea70f0 | +--------------------------------------+-------------+-----------+--------------------+------------+--------------------------------------+--------------------------------------+ real 1m51.921s user 0m0.624s sys 0m0.077s ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1863201 Title: stein regression listing security group rules Status in neutron: New Bug description: Upgrading neutron from rocky -> stein and get a considerable slow down when listing all security groups for a project. Goes from ~2 seconds to almost 2 minutes. Looking into the code it looks like it is very inefficient because it gets all rules from the DB and then filters after the fact. We have around 7000 rules in our QA env. Very keen to get this sorted but don't know the neutron code base that well so can offer testing of patches if there are any out there already. It looks like this happened with listing ports too for stein and found this https://bugzilla.redhat.com/show_bug.cgi?id=1737012 so wonder if this is related? With Rocky: time openstack security group rule list +--------------------------------------+-------------+-----------+--------------------+------------+--------------------------------------+--------------------------------------+ | ID | IP Protocol | Ethertype | IP Range | Port Range | Remote Security Group | Security Group | +--------------------------------------+-------------+-----------+--------------------+------------+--------------------------------------+--------------------------------------+ | 01b877cc-1621-44cd-8e69-1345ab01a1ef | None | IPv4 | 0.0.0.0/0 | | None | 3dcbd4fa-d017-4361-b0b0-b7508e923087 | | 0c744788-6319-42e5-931a-5e7b0df166c4 | None | IPv6 | ::/0 | | None | 3dcbd4fa-d017-4361-b0b0-b7508e923087 | | 0fc6b79d-d211-4201-ac76-60fb8ea40c9c | None | IPv4 | 0.0.0.0/0 | | None | 8f55c18b-cd8c-4d84-afef-f8b83d5eb128 | | 17d6c8a3-7894-42a6-92f2-1bd56a30ef1d | tcp | IPv4 | 0.0.0.0/0 | 80:80 | None | ed257fd7-d825-4014-96a8-c16adfea70f0 | | 19d3ba79-65f1-4c89-a1c2-b32049ceb25a | None | IPv6 | ::/0 | | None | 008510a7-d176-4ee5-87e2-e74da06c55ba | | 21f1d173-b99f-47a7-9983-6926f7bc58f3 | icmp | IPv4 | 0.0.0.0/0 | | None | 008510a7-d176-4ee5-87e2-e74da06c55ba | | 3321d5ff-11c3-4104-be13-107c789e4bf8 | None | IPv6 | ::/0 | | None | 57cb14de-dd5f-4f0c-b0cf-a7effc36fca5 | | 381c6816-9b5c-42b7-9dd3-dae12a49c08b | None | IPv4 | 0.0.0.0/0 | | None | 3f63cfbb-87ee-4aa2-8193-7e86cb542881 | | 3886ad10-99ea-4f60-a36c-ffbe80d92907 | None | IPv6 | ::/0 | | None | ed257fd7-d825-4014-96a8-c16adfea70f0 | | 5be4853a-75d1-435c-87ca-56c54a243f70 | None | IPv4 | 0.0.0.0/0 | | None | 57cb14de-dd5f-4f0c-b0cf-a7effc36fca5 | | 71656249-4454-410e-8e7d-24910df127ba | None | IPv6 | ::/0 | | None | 8f55c18b-cd8c-4d84-afef-f8b83d5eb128 | | 783324ac-6844-4d4d-985c-936015bcb66e | icmp | IPv4 | 0.0.0.0/0 | | None | 3f63cfbb-87ee-4aa2-8193-7e86cb542881 | | 7ca7f0cc-b4df-401f-aaa4-662f17afcfb0 | None | IPv4 | 0.0.0.0/0 | | None | 008510a7-d176-4ee5-87e2-e74da06c55ba | | 825a33ff-b693-456d-811e-a0b494e8e308 | None | IPv6 | ::/0 | | 008510a7-d176-4ee5-87e2-e74da06c55ba | 008510a7-d176-4ee5-87e2-e74da06c55ba | | 89fd2d18-45d3-4a86-a020-09d240912e5c | tcp | IPv4 | 128.250.116.173/32 | 22:22 | None | 008510a7-d176-4ee5-87e2-e74da06c55ba | | 8a1f45f1-e4c8-41e4-b6f3-80ab48b7e38d | None | IPv6 | ::/0 | | None | bf7abb53-e5ca-428d-9fce-6a2e37a25ee0 | | 9ebc6d15-e3eb-4d20-88d4-6737367ffc08 | None | IPv4 | 0.0.0.0/0 | | None | ed257fd7-d825-4014-96a8-c16adfea70f0 | | 9f29f539-a80a-4a8d-89cc-f714224b5f8c | icmp | IPv4 | 0.0.0.0/0 | | None | 8f55c18b-cd8c-4d84-afef-f8b83d5eb128 | | a1bc8f05-3a20-48c2-bae5-a60f4ffed514 | None | IPv4 | 0.0.0.0/0 | | 008510a7-d176-4ee5-87e2-e74da06c55ba | 008510a7-d176-4ee5-87e2-e74da06c55ba | | bef999d6-669a-47f6-988c-e69bab6df87a | tcp | IPv4 | 0.0.0.0/0 | 22:22 | 57cb14de-dd5f-4f0c-b0cf-a7effc36fca5 | bf7abb53-e5ca-428d-9fce-6a2e37a25ee0 | | c5ce339b-cd92-492c-9af4-6eab875027ce | tcp | IPv4 | 0.0.0.0/0 | 80:80 | 008510a7-d176-4ee5-87e2-e74da06c55ba | 008510a7-d176-4ee5-87e2-e74da06c55ba | | d9ec0eba-d80d-4331-a588-e4f8c1c75533 | None | IPv6 | ::/0 | | None | 3f63cfbb-87ee-4aa2-8193-7e86cb542881 | | de760e03-92a9-4183-8acc-1d82addc3604 | None | IPv4 | 0.0.0.0/0 | | None | bf7abb53-e5ca-428d-9fce-6a2e37a25ee0 | | f4bc4616-1d18-4488-84bb-546516c053bc | tcp | IPv4 | 0.0.0.0/0 | 443:443 | None | ed257fd7-d825-4014-96a8-c16adfea70f0 | +--------------------------------------+-------------+-----------+--------------------+------------+--------------------------------------+--------------------------------------+ real 0m2.499s user 0m0.642s sys 0m0.053s With Stein: time openstack security group rule list +--------------------------------------+-------------+-----------+--------------------+------------+--------------------------------------+--------------------------------------+ | ID | IP Protocol | Ethertype | IP Range | Port Range | Remote Security Group | Security Group | +--------------------------------------+-------------+-----------+--------------------+------------+--------------------------------------+--------------------------------------+ | 01b877cc-1621-44cd-8e69-1345ab01a1ef | None | IPv4 | 0.0.0.0/0 | | None | 3dcbd4fa-d017-4361-b0b0-b7508e923087 | | 0c744788-6319-42e5-931a-5e7b0df166c4 | None | IPv6 | ::/0 | | None | 3dcbd4fa-d017-4361-b0b0-b7508e923087 | | 0fc6b79d-d211-4201-ac76-60fb8ea40c9c | None | IPv4 | 0.0.0.0/0 | | None | 8f55c18b-cd8c-4d84-afef-f8b83d5eb128 | | 17d6c8a3-7894-42a6-92f2-1bd56a30ef1d | tcp | IPv4 | 0.0.0.0/0 | 80:80 | None | ed257fd7-d825-4014-96a8-c16adfea70f0 | | 19d3ba79-65f1-4c89-a1c2-b32049ceb25a | None | IPv6 | ::/0 | | None | 008510a7-d176-4ee5-87e2-e74da06c55ba | | 21f1d173-b99f-47a7-9983-6926f7bc58f3 | icmp | IPv4 | 0.0.0.0/0 | | None | 008510a7-d176-4ee5-87e2-e74da06c55ba | | 3321d5ff-11c3-4104-be13-107c789e4bf8 | None | IPv6 | ::/0 | | None | 57cb14de-dd5f-4f0c-b0cf-a7effc36fca5 | | 381c6816-9b5c-42b7-9dd3-dae12a49c08b | None | IPv4 | 0.0.0.0/0 | | None | 3f63cfbb-87ee-4aa2-8193-7e86cb542881 | | 3886ad10-99ea-4f60-a36c-ffbe80d92907 | None | IPv6 | ::/0 | | None | ed257fd7-d825-4014-96a8-c16adfea70f0 | | 5be4853a-75d1-435c-87ca-56c54a243f70 | None | IPv4 | 0.0.0.0/0 | | None | 57cb14de-dd5f-4f0c-b0cf-a7effc36fca5 | | 71656249-4454-410e-8e7d-24910df127ba | None | IPv6 | ::/0 | | None | 8f55c18b-cd8c-4d84-afef-f8b83d5eb128 | | 783324ac-6844-4d4d-985c-936015bcb66e | icmp | IPv4 | 0.0.0.0/0 | | None | 3f63cfbb-87ee-4aa2-8193-7e86cb542881 | | 7ca7f0cc-b4df-401f-aaa4-662f17afcfb0 | None | IPv4 | 0.0.0.0/0 | | None | 008510a7-d176-4ee5-87e2-e74da06c55ba | | 825a33ff-b693-456d-811e-a0b494e8e308 | None | IPv6 | ::/0 | | 008510a7-d176-4ee5-87e2-e74da06c55ba | 008510a7-d176-4ee5-87e2-e74da06c55ba | | 89fd2d18-45d3-4a86-a020-09d240912e5c | tcp | IPv4 | 128.250.116.173/32 | 22:22 | None | 008510a7-d176-4ee5-87e2-e74da06c55ba | | 8a1f45f1-e4c8-41e4-b6f3-80ab48b7e38d | None | IPv6 | ::/0 | | None | bf7abb53-e5ca-428d-9fce-6a2e37a25ee0 | | 9ebc6d15-e3eb-4d20-88d4-6737367ffc08 | None | IPv4 | 0.0.0.0/0 | | None | ed257fd7-d825-4014-96a8-c16adfea70f0 | | 9f29f539-a80a-4a8d-89cc-f714224b5f8c | icmp | IPv4 | 0.0.0.0/0 | | None | 8f55c18b-cd8c-4d84-afef-f8b83d5eb128 | | a1bc8f05-3a20-48c2-bae5-a60f4ffed514 | None | IPv4 | 0.0.0.0/0 | | 008510a7-d176-4ee5-87e2-e74da06c55ba | 008510a7-d176-4ee5-87e2-e74da06c55ba | | bef999d6-669a-47f6-988c-e69bab6df87a | tcp | IPv4 | 0.0.0.0/0 | 22:22 | 57cb14de-dd5f-4f0c-b0cf-a7effc36fca5 | bf7abb53-e5ca-428d-9fce-6a2e37a25ee0 | | c5ce339b-cd92-492c-9af4-6eab875027ce | tcp | IPv4 | 0.0.0.0/0 | 80:80 | 008510a7-d176-4ee5-87e2-e74da06c55ba | 008510a7-d176-4ee5-87e2-e74da06c55ba | | d9ec0eba-d80d-4331-a588-e4f8c1c75533 | None | IPv6 | ::/0 | | None | 3f63cfbb-87ee-4aa2-8193-7e86cb542881 | | de760e03-92a9-4183-8acc-1d82addc3604 | None | IPv4 | 0.0.0.0/0 | | None | bf7abb53-e5ca-428d-9fce-6a2e37a25ee0 | | f4bc4616-1d18-4488-84bb-546516c053bc | tcp | IPv4 | 0.0.0.0/0 | 443:443 | None | ed257fd7-d825-4014-96a8-c16adfea70f0 | +--------------------------------------+-------------+-----------+--------------------+------------+--------------------------------------+--------------------------------------+ real 1m51.921s user 0m0.624s sys 0m0.077s To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1863201/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
