Public bug reported: With upgrade from 2023.1 to 2024.1 with driver ML2/OVS we've spotted a significant (10 times) performance regression on some operations.
As best example - we can take security groups operations. Neutron is running in eventlet, since uWSGI is not yet fully functional for 2024.1 (see https://review.opendev.org/c/openstack/neutron/+/926922). So neutron-server is just being launched with exactly same database and config, just from different venvs. ``` # cat /etc/systemd/system/neutron-server.service [Unit] Description = neutron-server service After = network-online.target After = syslog.target [Service] Type = simple User = neutron Group = neutron ExecStart = /openstack/venvs/neutron-29.0.2/bin/neutron-server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini ExecReload = /bin/kill -HUP $MAINPID # Give a reasonable amount of time for the server to start up/shut down TimeoutSec = 120 Restart = on-failure RestartSec = 2 # This creates a specific slice which all services will operate from # The accounting options give us the ability to see resource usage through # the `systemd-cgtop` command. Slice = neutron.slice # Set Accounting CPUAccounting = True BlockIOAccounting = True MemoryAccounting = True TasksAccounting = True # Set Sandboxing PrivateTmp = False PrivateDevices = False PrivateNetwork = False PrivateUsers = False [Install] WantedBy = multi-user.target # time curl -X GET http://127.0.0.1:9696/v2.0/security-groups?project_id=${OS_PROJECT_ID} -H "X-Auth-Token: ${TOKEN}" ... real 0m24.450s user 0m0.008s sys 0m0.010s # time curl -X GET http://127.0.0.1:9696/v2.0/security-groups/${security_group_uuid} -H "X-Auth-Token: ${TOKEN}" ... real 0m54.841s user 0m0.010s sys 0m0.012s # sed -i 's/29.0.2/27.4.0/g' /etc/systemd/system/neutron-server.service # systemctl daemon-reload # systemctl restart neutron-server # time curl -X GET http://127.0.0.1:9696/v2.0/security-groups?project_id=${OS_PROJECT_ID} -H "X-Auth-Token: ${TOKEN}" ... real 0m1.040s user 0m0.011s sys 0m0.007s # time curl -X GET http://127.0.0.1:9696/v2.0/security-groups/${security_group_uuid} -H "X-Auth-Token: ${TOKEN}" ... real 0m0.589s user 0m0.012s sys 0m0.007s ``` So as you might see, difference in response time is very significant, while the only change I've made is to use previous codebase for the service. I am also providing pip freeze for both venvs for comparison, though both of them were using upper-constraints: # /openstack/venvs/neutron-27.4.0/bin/pip freeze alembic==1.8.1 amqp==5.1.1 appdirs==1.4.4 attrs==22.1.0 autopage==0.5.1 bcrypt==4.0.0 cachetools==5.2.0 certifi==2023.11.17 cffi==1.15.1 charset-normalizer==2.1.1 cliff==4.2.0 cmd2==2.4.2 cryptography==38.0.2 debtcollector==2.5.0 decorator==5.1.1 dnspython==2.2.1 dogpile.cache==1.1.8 eventlet==0.33.1 extras==1.0.0 fasteners==0.18 fixtures==4.0.1 futurist==2.4.1 greenlet==1.1.3 httplib2==0.20.4 idna==3.4 importlib-metadata==5.0.0 iso8601==1.1.0 Jinja2==3.1.2 jmespath==1.0.1 jsonpatch==1.32 jsonpointer==2.3 jsonschema==4.16.0 keystoneauth1==5.1.2 keystonemiddleware==10.2.0 kombu==5.2.4 logutils==0.3.5 lxml==4.9.1 Mako==1.2.3 MarkupSafe==2.1.1 msgpack==1.0.4 ncclient==0.6.13 netaddr==0.8.0 netifaces==0.11.0 neutron==22.1.1.dev52 neutron-dynamic-routing==22.0.0 neutron-lib==3.4.1 neutron-vpnaas==22.0.1.dev2 openstacksdk==1.0.1 os-client-config==2.1.0 os-ken==2.6.0 os-resource-classes==1.1.0 os-service-types==1.7.0 os-traits==2.10.0 os-vif==3.1.1 osc-lib==2.7.0 oslo.cache==3.3.1 oslo.concurrency==5.1.1 oslo.config==9.1.1 oslo.context==5.6.0 oslo.db==12.3.1 oslo.i18n==6.0.0 oslo.log==5.2.0 oslo.messaging==14.7.1 oslo.metrics==0.6.0 oslo.middleware==5.1.1 oslo.policy==4.1.1 oslo.privsep==3.1.0 oslo.reports==3.0.0 oslo.rootwrap==7.0.1 oslo.serialization==5.1.1 oslo.service==3.1.1 oslo.upgradecheck==2.1.1 oslo.utils==6.1.0 oslo.versionedobjects==3.1.0 osprofiler==3.4.3 ovs==2.17.1.post1 ovsdbapp==2.2.1 packaging==21.3 paramiko==2.11.0 Paste==3.5.2 PasteDeploy==2.1.1 pbr==5.11.1 pecan==1.4.2 prettytable==3.4.1 prometheus-client==0.14.1 psutil==5.9.2 pycadf==3.1.1 pycparser==2.21 pyinotify==0.9.6 pymemcache==3.5.2 PyMySQL==1.0.2 PyNaCl==1.5.0 pyOpenSSL==22.1.0 pyparsing==3.0.9 pyperclip==1.8.2 pyroute2==0.7.3 pyrsistent==0.18.1 python-binary-memcached==0.31.1 python-dateutil==2.8.2 python-designateclient==5.2.0 python-keystoneclient==5.1.0 python-memcached==1.59 python-neutronclient==9.0.0 python-novaclient==18.3.0 pytz==2022.4 PyYAML==6.0 repoze.lru==0.7 requests==2.28.1 requestsexceptions==1.4.0 rfc3986==1.5.0 Routes==2.5.1 setproctitle==1.3.2 simplejson==3.17.6 six==1.16.0 sortedcontainers==2.4.0 SQLAlchemy==1.4.41 sqlalchemy-migrate==0.13.0 sqlparse==0.4.3 statsd==3.3.0 stevedore==5.0.0 systemd-python==234 Tempita==0.5.2 tenacity==6.3.1 testresources==2.0.1 testscenarios==0.5.0 testtools==2.5.0 tooz==3.2.0 uhashring==2.1 urllib3==1.26.12 vine==5.0.0 voluptuous==0.13.1 wcwidth==0.2.5 WebOb==1.8.7 wrapt==1.14.1 yappi==1.4.0 zipp==3.8.1 # /openstack/venvs/neutron-29.0.2/bin/pip freeze alembic==1.9.4 amqp==5.2.0 async-timeout==4.0.3 attrs==23.2.0 autopage==0.5.2 bcrypt==4.1.2 cachetools==5.3.2 certifi==2024.8.30 cffi==1.16.0 charset-normalizer==3.3.2 cliff==4.6.0 cmd2==2.4.3 cryptography==42.0.4 debtcollector==3.0.0 decorator==5.1.1 dnspython==2.6.1 dogpile.cache==1.3.1 eventlet==0.36.1 fasteners==0.19 fixtures==4.1.0 futurist==3.0.0 greenlet==3.0.3 httplib2==0.22.0 idna==3.6 iso8601==2.1.0 Jinja2==3.1.3 jmespath==1.0.1 jsonpatch==1.33 jsonpointer==2.4 jsonschema==4.19.2 jsonschema-specifications==2023.12.1 keystoneauth1==5.6.0 keystonemiddleware==10.6.0 kombu==5.3.5 logutils==0.3.5 lxml==5.1.0 Mako==1.3.2 MarkupSafe==2.1.5 msgpack==1.0.7 ncclient==0.6.15 netaddr==0.10.1 netifaces==0.11.0 neutron==24.0.2.dev15 neutron-dynamic-routing==24.0.1.dev2 neutron-lib==3.11.0 neutron-vpnaas==24.0.2.dev1 openstacksdk==3.0.0 os-client-config==2.1.0 os-ken==2.8.1 os-resource-classes==1.1.0 os-service-types==1.7.0 os-traits==3.0.0 os-vif==3.5.0 osc-lib==3.0.1 oslo.cache==3.7.0 oslo.concurrency==6.0.0 oslo.config==9.4.0 oslo.context==5.5.0 oslo.db==15.0.0 oslo.i18n==6.3.0 oslo.log==5.5.1 oslo.messaging==14.7.1 oslo.metrics==0.8.0 oslo.middleware==6.1.0 oslo.policy==4.3.0 oslo.privsep==3.3.0 oslo.reports==3.3.0 oslo.rootwrap==7.2.0 oslo.serialization==5.4.0 oslo.service==3.4.1 oslo.upgradecheck==2.3.0 oslo.utils==7.1.0 oslo.versionedobjects==3.3.0 osprofiler==4.1.0 ovs==3.1.2 ovsdbapp==2.6.1 packaging==23.2 paramiko==3.4.0 Paste==3.7.1 PasteDeploy==3.1.0 pbr==6.0.0 pecan==1.5.1 platformdirs==4.2.0 prettytable==3.9.0 prometheus_client==0.20.0 psutil==5.9.8 pycadf==3.1.1 pycparser==2.21 pyinotify==0.9.6 PyJWT==2.8.0 pymemcache==4.0.0 PyMySQL==1.1.0 PyNaCl==1.5.0 pyOpenSSL==24.0.0 pyparsing==3.1.1 pyperclip==1.8.2 pyroute2==0.7.12 python-binary-memcached==0.31.2 python-dateutil==2.8.2 python-designateclient==6.0.1 python-keystoneclient==5.4.0 python-memcached==1.62 python-neutronclient==11.2.0 python-novaclient==18.6.0 pytz==2024.1 PyYAML==6.0.1 redis==5.0.1 referencing==0.33.0 repoze.lru==0.7 requests==2.31.0 requestsexceptions==1.4.0 rfc3986==2.0.0 Routes==2.5.1 rpds-py==0.18.0 setproctitle==1.3.3 simplejson==3.19.2 six==1.16.0 sortedcontainers==2.4.0 SQLAlchemy==1.4.51 statsd==4.0.1 stevedore==5.2.0 systemd-python==235 tenacity==8.2.3 testresources==2.0.1 testscenarios==0.5.0 testtools==2.7.1 tooz==6.2.0 typing_extensions==4.9.0 tzdata==2024.1 uhashring==2.3 urllib3==1.26.18 vine==5.1.0 voluptuous==0.14.2 wcwidth==0.2.13 WebOb==1.8.7 wrapt==1.16.0 yappi==1.6.0 # ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2081087 Title: Performance regression in neutron-server from 2023.1 to 2024.1 Status in neutron: New Bug description: With upgrade from 2023.1 to 2024.1 with driver ML2/OVS we've spotted a significant (10 times) performance regression on some operations. As best example - we can take security groups operations. Neutron is running in eventlet, since uWSGI is not yet fully functional for 2024.1 (see https://review.opendev.org/c/openstack/neutron/+/926922). So neutron-server is just being launched with exactly same database and config, just from different venvs. ``` # cat /etc/systemd/system/neutron-server.service [Unit] Description = neutron-server service After = network-online.target After = syslog.target [Service] Type = simple User = neutron Group = neutron ExecStart = /openstack/venvs/neutron-29.0.2/bin/neutron-server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini ExecReload = /bin/kill -HUP $MAINPID # Give a reasonable amount of time for the server to start up/shut down TimeoutSec = 120 Restart = on-failure RestartSec = 2 # This creates a specific slice which all services will operate from # The accounting options give us the ability to see resource usage through # the `systemd-cgtop` command. Slice = neutron.slice # Set Accounting CPUAccounting = True BlockIOAccounting = True MemoryAccounting = True TasksAccounting = True # Set Sandboxing PrivateTmp = False PrivateDevices = False PrivateNetwork = False PrivateUsers = False [Install] WantedBy = multi-user.target # time curl -X GET http://127.0.0.1:9696/v2.0/security-groups?project_id=${OS_PROJECT_ID} -H "X-Auth-Token: ${TOKEN}" ... real 0m24.450s user 0m0.008s sys 0m0.010s # time curl -X GET http://127.0.0.1:9696/v2.0/security-groups/${security_group_uuid} -H "X-Auth-Token: ${TOKEN}" ... real 0m54.841s user 0m0.010s sys 0m0.012s # sed -i 's/29.0.2/27.4.0/g' /etc/systemd/system/neutron-server.service # systemctl daemon-reload # systemctl restart neutron-server # time curl -X GET http://127.0.0.1:9696/v2.0/security-groups?project_id=${OS_PROJECT_ID} -H "X-Auth-Token: ${TOKEN}" ... real 0m1.040s user 0m0.011s sys 0m0.007s # time curl -X GET http://127.0.0.1:9696/v2.0/security-groups/${security_group_uuid} -H "X-Auth-Token: ${TOKEN}" ... real 0m0.589s user 0m0.012s sys 0m0.007s ``` So as you might see, difference in response time is very significant, while the only change I've made is to use previous codebase for the service. I am also providing pip freeze for both venvs for comparison, though both of them were using upper-constraints: # /openstack/venvs/neutron-27.4.0/bin/pip freeze alembic==1.8.1 amqp==5.1.1 appdirs==1.4.4 attrs==22.1.0 autopage==0.5.1 bcrypt==4.0.0 cachetools==5.2.0 certifi==2023.11.17 cffi==1.15.1 charset-normalizer==2.1.1 cliff==4.2.0 cmd2==2.4.2 cryptography==38.0.2 debtcollector==2.5.0 decorator==5.1.1 dnspython==2.2.1 dogpile.cache==1.1.8 eventlet==0.33.1 extras==1.0.0 fasteners==0.18 fixtures==4.0.1 futurist==2.4.1 greenlet==1.1.3 httplib2==0.20.4 idna==3.4 importlib-metadata==5.0.0 iso8601==1.1.0 Jinja2==3.1.2 jmespath==1.0.1 jsonpatch==1.32 jsonpointer==2.3 jsonschema==4.16.0 keystoneauth1==5.1.2 keystonemiddleware==10.2.0 kombu==5.2.4 logutils==0.3.5 lxml==4.9.1 Mako==1.2.3 MarkupSafe==2.1.1 msgpack==1.0.4 ncclient==0.6.13 netaddr==0.8.0 netifaces==0.11.0 neutron==22.1.1.dev52 neutron-dynamic-routing==22.0.0 neutron-lib==3.4.1 neutron-vpnaas==22.0.1.dev2 openstacksdk==1.0.1 os-client-config==2.1.0 os-ken==2.6.0 os-resource-classes==1.1.0 os-service-types==1.7.0 os-traits==2.10.0 os-vif==3.1.1 osc-lib==2.7.0 oslo.cache==3.3.1 oslo.concurrency==5.1.1 oslo.config==9.1.1 oslo.context==5.6.0 oslo.db==12.3.1 oslo.i18n==6.0.0 oslo.log==5.2.0 oslo.messaging==14.7.1 oslo.metrics==0.6.0 oslo.middleware==5.1.1 oslo.policy==4.1.1 oslo.privsep==3.1.0 oslo.reports==3.0.0 oslo.rootwrap==7.0.1 oslo.serialization==5.1.1 oslo.service==3.1.1 oslo.upgradecheck==2.1.1 oslo.utils==6.1.0 oslo.versionedobjects==3.1.0 osprofiler==3.4.3 ovs==2.17.1.post1 ovsdbapp==2.2.1 packaging==21.3 paramiko==2.11.0 Paste==3.5.2 PasteDeploy==2.1.1 pbr==5.11.1 pecan==1.4.2 prettytable==3.4.1 prometheus-client==0.14.1 psutil==5.9.2 pycadf==3.1.1 pycparser==2.21 pyinotify==0.9.6 pymemcache==3.5.2 PyMySQL==1.0.2 PyNaCl==1.5.0 pyOpenSSL==22.1.0 pyparsing==3.0.9 pyperclip==1.8.2 pyroute2==0.7.3 pyrsistent==0.18.1 python-binary-memcached==0.31.1 python-dateutil==2.8.2 python-designateclient==5.2.0 python-keystoneclient==5.1.0 python-memcached==1.59 python-neutronclient==9.0.0 python-novaclient==18.3.0 pytz==2022.4 PyYAML==6.0 repoze.lru==0.7 requests==2.28.1 requestsexceptions==1.4.0 rfc3986==1.5.0 Routes==2.5.1 setproctitle==1.3.2 simplejson==3.17.6 six==1.16.0 sortedcontainers==2.4.0 SQLAlchemy==1.4.41 sqlalchemy-migrate==0.13.0 sqlparse==0.4.3 statsd==3.3.0 stevedore==5.0.0 systemd-python==234 Tempita==0.5.2 tenacity==6.3.1 testresources==2.0.1 testscenarios==0.5.0 testtools==2.5.0 tooz==3.2.0 uhashring==2.1 urllib3==1.26.12 vine==5.0.0 voluptuous==0.13.1 wcwidth==0.2.5 WebOb==1.8.7 wrapt==1.14.1 yappi==1.4.0 zipp==3.8.1 # /openstack/venvs/neutron-29.0.2/bin/pip freeze alembic==1.9.4 amqp==5.2.0 async-timeout==4.0.3 attrs==23.2.0 autopage==0.5.2 bcrypt==4.1.2 cachetools==5.3.2 certifi==2024.8.30 cffi==1.16.0 charset-normalizer==3.3.2 cliff==4.6.0 cmd2==2.4.3 cryptography==42.0.4 debtcollector==3.0.0 decorator==5.1.1 dnspython==2.6.1 dogpile.cache==1.3.1 eventlet==0.36.1 fasteners==0.19 fixtures==4.1.0 futurist==3.0.0 greenlet==3.0.3 httplib2==0.22.0 idna==3.6 iso8601==2.1.0 Jinja2==3.1.3 jmespath==1.0.1 jsonpatch==1.33 jsonpointer==2.4 jsonschema==4.19.2 jsonschema-specifications==2023.12.1 keystoneauth1==5.6.0 keystonemiddleware==10.6.0 kombu==5.3.5 logutils==0.3.5 lxml==5.1.0 Mako==1.3.2 MarkupSafe==2.1.5 msgpack==1.0.7 ncclient==0.6.15 netaddr==0.10.1 netifaces==0.11.0 neutron==24.0.2.dev15 neutron-dynamic-routing==24.0.1.dev2 neutron-lib==3.11.0 neutron-vpnaas==24.0.2.dev1 openstacksdk==3.0.0 os-client-config==2.1.0 os-ken==2.8.1 os-resource-classes==1.1.0 os-service-types==1.7.0 os-traits==3.0.0 os-vif==3.5.0 osc-lib==3.0.1 oslo.cache==3.7.0 oslo.concurrency==6.0.0 oslo.config==9.4.0 oslo.context==5.5.0 oslo.db==15.0.0 oslo.i18n==6.3.0 oslo.log==5.5.1 oslo.messaging==14.7.1 oslo.metrics==0.8.0 oslo.middleware==6.1.0 oslo.policy==4.3.0 oslo.privsep==3.3.0 oslo.reports==3.3.0 oslo.rootwrap==7.2.0 oslo.serialization==5.4.0 oslo.service==3.4.1 oslo.upgradecheck==2.3.0 oslo.utils==7.1.0 oslo.versionedobjects==3.3.0 osprofiler==4.1.0 ovs==3.1.2 ovsdbapp==2.6.1 packaging==23.2 paramiko==3.4.0 Paste==3.7.1 PasteDeploy==3.1.0 pbr==6.0.0 pecan==1.5.1 platformdirs==4.2.0 prettytable==3.9.0 prometheus_client==0.20.0 psutil==5.9.8 pycadf==3.1.1 pycparser==2.21 pyinotify==0.9.6 PyJWT==2.8.0 pymemcache==4.0.0 PyMySQL==1.1.0 PyNaCl==1.5.0 pyOpenSSL==24.0.0 pyparsing==3.1.1 pyperclip==1.8.2 pyroute2==0.7.12 python-binary-memcached==0.31.2 python-dateutil==2.8.2 python-designateclient==6.0.1 python-keystoneclient==5.4.0 python-memcached==1.62 python-neutronclient==11.2.0 python-novaclient==18.6.0 pytz==2024.1 PyYAML==6.0.1 redis==5.0.1 referencing==0.33.0 repoze.lru==0.7 requests==2.31.0 requestsexceptions==1.4.0 rfc3986==2.0.0 Routes==2.5.1 rpds-py==0.18.0 setproctitle==1.3.3 simplejson==3.19.2 six==1.16.0 sortedcontainers==2.4.0 SQLAlchemy==1.4.51 statsd==4.0.1 stevedore==5.2.0 systemd-python==235 tenacity==8.2.3 testresources==2.0.1 testscenarios==0.5.0 testtools==2.7.1 tooz==6.2.0 typing_extensions==4.9.0 tzdata==2024.1 uhashring==2.3 urllib3==1.26.18 vine==5.1.0 voluptuous==0.14.2 wcwidth==0.2.13 WebOb==1.8.7 wrapt==1.16.0 yappi==1.6.0 # To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/2081087/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
