On 9/18/2014 12:35 AM, Morgan Fainberg wrote:
-----Original Message-----
From: Dean Troyer <[email protected]>
Reply: OpenStack Development Mailing List (not for usage questions)
<[email protected]>>
Date: September 17, 2014 at 21:21:47
To: OpenStack Development Mailing List (not for usage questions)
<[email protected]>>
Subject: Re: [openstack-dev] PostgreSQL jobs slow in the gate
Clark Boylan Wrotr :
It appears that keystone eventlet is the source of the slowness in this
job. With keystone eventlet all of the devstack jobs are slower and with
keystone wsgi all of the jobs are quicker. Probably need to collect a
bit more data but this doesn't look good for keystone eventlet.
On Wed, Sep 17, 2014 at 11:02 PM, Morgan Fainberg <
[email protected]> wrote:
I've kicked off a test[1] as well to check into some tunable options
(eventlet workers) for improving keystone eventlet performance. I'll circle
back with the infra team once we have a little more data on both fronts.
The Keystone team will use this data to figure out the best way to approach
this issue.
Brant submitted https://review.openstack.org/#/c/121384/ to up the Keystone
workers when API_WORKERS is set.
I submitted https://review.openstack.org/#/c/122013/ to set a scaling
default for API_WORKERS based on the available CPUs ((nproc+1)/2). There
is a summary in that commit message of the current reviews addressing the
workers in various projects.
I think it has become clear that DevStack needs to set a default for most
services that are currently either too big (nproc) or too small (Keystone
at 1). Of course, moving things to mod_wsgi moots all of that, but it'll
be a while before everything moves.
dt
--
Dean Troyer
[email protected]
Dean,
We should probably look at increasing the default number of workers as well in
the Keystone configuration (rather than just devstack). It looks like, with
limited datasets, we are seeing a real improvement with Keystone and 4 workers
(from my previously mentioned quick test). Thanks for the extra data points.
This helps to confirm the issue is Keystone under eventlet.
Cheers,
Morgan
_______________________________________________
OpenStack-dev mailing list
[email protected]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
I pointed this out in Brant's devstack patch but we had a product team
internally bring up this same point in Icehouse, they were really
limited due to the eventlet workers issue in Keystone and once we
provided the option (backported) it increased their throughput by 20%.
We've been running with that in our internal Tempest runs (setting
workers equal to number of CPUs / 2) and so far so good.
--
Thanks,
Matt Riedemann
_______________________________________________
OpenStack-dev mailing list
[email protected]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
