If for whatever reason you don't want to use the memcache token driver, I've seen an order-of-magnitude increase in performance simply from adding an index to the "expires" column in the token table. I opened a blueprint ( https://blueprints.launchpad.net/keystone/+spec/index-token-expiry) to get this in mainline, but until that lands in the version you're using, you may benefit from adding the index manually.
On Fri, Jul 26, 2013 at 12:03 PM, Jay Pipes <jaypi...@gmail.com> wrote: > On 07/25/2013 11:23 PM, Chu Duc Minh wrote: > >> On Thu, Jul 25, 2013 at 7:30 PM, Jay Pipes <jaypi...@gmail.com >> <mailto:jaypi...@gmail.com>> wrote: >> >> You will need to provide more details than "old" vs. "new" OpenStack. >> >> For example... >> >> 1) What is your network model in the old vs. new >> >> Old: nova-network + FlatDHCP >> New: Quantum + OpenvSwitch + network-namespace + metadata-ns-proxy >> > > OK. If you do a time ping -c1 <IP> from a VM to the metadata IP, what kind > of times do you see in each setup? > > > 2) What version of OpenStack is the old >> >> Old: Essex >> New: Grizzly >> > > OK. > > > 3) Is Keystone used in old and new? If so, what drivers are used in >> Keystone? >> >> Yes, both use Keystone with SQL backend >> > > I recommend switching to the memcache Token driver in Keystone. > > > 4) Do you have errors in any of your log files (usually an >> indication that something like a timeout or failure on >> >> RPC which may affect performance) >> >> No, i didn't found it. >> > > OK. > > > 5) Are you using nova-conductor in the new? >> >> I have nova-conductor process run on Controller node, but seem >> nova-compute don't use this. >> > > OK, good to know, but unlikely to be the cause of a metadata API query > slowdown. > > > 6) What database backend are you using? >> >> MySQL. >> > > OK. > > 7) Do a time keystone user-list on both old and new >> >> # keystone user-list >> Old: >> Run 1st time: >> real *0m2.494s* >> >> user 0m0.112s >> sys 0m0.040s >> 2nd time: >> real *0m0.279s* >> >> user 0m0.112s >> sys 0m0.020s >> >> >> New: >> Run 1st time: >> real *0m1.687s* >> >> user 0m0.176s >> sys 0m0.012s >> 2nd time: >> real *0m0.213s* >> user 0m0.160s >> sys 0m0.040s >> > > Interesting, so Keystone is actually faster in Grizzly than in Essex... or > at least it seems to be given the above rudimentary timing. > > > 8) Pastebin your conf files, with passwords removed >> >> My config files are quite large. >> I can sum up that all services & API are HA-ed using HAproxy, RabbitMQ >> Cluster, Database replication, v.v... >> > > OK. > > > This morning, i just do some tuning memcached for Dashboard service >> (HA-ed using HAProxy), then my Dashboard become faster significantly. >> >> The questions remain are: >> 1. Why metadata query so slow? Possible problems? ( "curl >> http://169.254.169.254/**openstack <http://169.254.169.254/openstack>" >> took 2.5-5 seconds to complete, >> "ec2metadata" took 14-17 seconds to complete - i tested many times.) >> > > Could be a number of things. I thought Vish had redesigned the metadata > API endpoint in Folsom and removed the problematic slow queries that used > to be in there. If you check your MySQL slow log (hopefully you have it > enabled), look to see if any of the queries in there reference the > instance_metadata table. > > > 2. Why API calls using *-client reduce time from 2nd time on my old >> Essex deployment, but not on my new Grizzly deployment? Maybe I need >> some "cache" settings? >> > > Looks to me that both 2nd time calls were reduced...in Essex as well as > Grizzly zones. > > Best, > -jay > > Thank you very much! >> >> >> The more information you give, the better folks can help you. >> >> Best, >> -jay >> >> >> On 07/25/2013 07:14 AM, Chu Duc Minh wrote: >> >> Check some more API (I run these command below from Controller >> node): >> # time quantum subnet-list >> (...have 4 subnet) >> real 0m0.676s >> user 0m0.196s >> sys 0m0.020s >> >> # time quantum router-list >> (...have 1 router) >> real 0m0.496s >> user 0m0.164s >> sys 0m0.052s >> >> # time nova list --all_tenants=1 >> (...have 5 instances) >> real 0m1.290s >> user 0m0.308s >> sys 0m0.040s >> >> Compare with my old OpenStack deployment on weaker servers, it >> took 1/3 >> in times. >> >> >> >> On Thu, Jul 25, 2013 at 5:43 PM, Peter Cheung >> <mcheun...@hotmail.com <mailto:mcheun...@hotmail.com> >> <mailto:mcheun...@hotmail.com <mailto:mcheun...@hotmail.com>** >> >__> >> >> wrote: >> >> I am having a problem about calling API speed is up and down, >> something need 0.1s, something it needs 3s >> >> Thanks >> from Peter >> >> >> >> ------------------------------**__----------------------------** >> --__------------ >> >> >> Date: Thu, 25 Jul 2013 17:41:11 +0700 >> From: chu.ducm...@gmail.com <mailto:chu.ducm...@gmail.com> >> <mailto:chu.ducm...@gmail.com <mailto:chu.ducm...@gmail.com>**> >> To: openst...@lists.launchpad.net >> >> <mailto:openstack@lists.**launchpad.net<openst...@lists.launchpad.net> >> > >> <mailto:openstack@lists.__laun**chpad.net<http://launchpad.net> >> >> >> <mailto:openstack@lists.**launchpad.net<openst...@lists.launchpad.net> >> >>; >> openstack@lists.openstack.org >> >> <mailto:openstack@lists.**openstack.org<openstack@lists.openstack.org> >> > >> <mailto:openstack@lists.__open**stack.org <http://openstack.org> >> >> >> <mailto:openstack@lists.**openstack.org<openstack@lists.openstack.org> >> >> >> >> Subject: [Openstack] Call to API very slow [Grizzly] >> >> >> All operations in my Openstack dashboard very slow (compare >> to my >> old Openstack deployment) >> Then i do some check on an instance: >> >> $ time curl >> http://169.254.169.254/__**openstack<http://169.254.169.254/__openstack> >> >> <http://169.254.169.254/**openstack<http://169.254.169.254/openstack> >> > >> 2012-08-10 >> 2013-04-04 >> latest >> real 0m*5.605s* >> >> user 0m0.004s >> sys 0m0.004s >> >> 5 seconds for a simple API query !?? >> >> >> in quantum-ns-metadata-proxyxxxx.**__log, i saw: >> >> 2013-07-25 *17:17:09 * DEBUG >> >> [quantum.agent.metadata.__**namespace_proxy] Request: GET >> >> /openstack >> HTTP/1.0 >> Accept: */* >> Content-Type: text/plain >> Host: 169.254.169.254 >> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 >> OpenSSL/1.0.1 zlib/1.2.3.4 <http://1.2.3.4> >> <http://1.2.3.4> libidn/1.23 librtmp/2.3 >> 2013-07-25 *17:17:14* DEBUG >> >> [quantum.agent.metadata.__**namespace_proxy] {'date': 'Thu, >> >> 25 Jul 2013 >> 10:17:14 GMT', 'status': '200', 'content-length': '28', >> 'content-type': 'text/html; charset=UTF-8', >> 'content-location': >> >> u'http://169.254.169.254/__**openstack<http://169.254.169.254/__openstack> >> >> <http://169.254.169.254/**openstack<http://169.254.169.254/openstack> >> >'} >> 2013-07-25 17:17:14 DEBUG >> [quantum.agent.metadata.__**namespace_proxy] 2012-08-10 >> >> 2013-04-04 >> latest >> >> I take a look at metadata-agent.log, and saw almost >> request/response >> finished @*17:17:09 >> * >> But the last finished *@**17:17:14 >> *2013-07-25 *17:17:14* DEBUG >> [quantum.agent.metadata.agent] >> >> {'date': 'Thu, 25 Jul 2013 10:17:14 GMT', 'status': '200', >> 'content-length': '28', 'content-type': 'text/html; >> charset=UTF-8', >> 'content-location': >> u'http://172.30.1.14:8775/__**openstack<http://172.30.1.14:8775/__openstack> >> >> >> <http://172.30.1.14:8775/**openstack<http://172.30.1.14:8775/openstack> >> >'} >> * >> * >> >> I enabled slow query log on MySql, but can't find any slow >> query. >> >> Do you know possible problems in this situation? >> Thank you very much! >> >> >> ______________________________**___________________ Mailing >> list: >> >> https://launchpad.net/~__**openstack<https://launchpad.net/~__openstack> >> >> >> <https://launchpad.net/~**openstack<https://launchpad.net/~openstack>> >> Post to : >> openst...@lists.launchpad.net >> >> <mailto:openstack@lists.**launchpad.net<openst...@lists.launchpad.net> >> > >> <mailto:openstack@lists.__laun**chpad.net <http://launchpad.net> >> >> <mailto:openstack@lists.**launchpad.net<openst...@lists.launchpad.net> >> >> >> >> Unsubscribe : >> https://launchpad.net/~__**openstack<https://launchpad.net/~__openstack> >> >> <https://launchpad.net/~**openstack<https://launchpad.net/~openstack>> >> More help : >> >> https://help.launchpad.net/__**ListHelp<https://help.launchpad.net/__ListHelp> >> >> <https://help.launchpad.net/**ListHelp<https://help.launchpad.net/ListHelp> >> > >> >> >> >> >> ______________________________**___________________ >> Mailing list: >> https://launchpad.net/~__**openstack<https://launchpad.net/~__openstack> >> >> <https://launchpad.net/~**openstack<https://launchpad.net/~openstack> >> > >> Post to : openst...@lists.launchpad.net >> >> <mailto:openstack@lists.**launchpad.net<openst...@lists.launchpad.net> >> > >> Unsubscribe : >> https://launchpad.net/~__**openstack<https://launchpad.net/~__openstack> >> <https://launchpad.net/~**openstack<https://launchpad.net/~openstack> >> > >> More help : >> https://help.launchpad.net/__**ListHelp<https://help.launchpad.net/__ListHelp> >> >> <https://help.launchpad.net/**ListHelp<https://help.launchpad.net/ListHelp> >> > >> >> >> >> ______________________________**___________________ >> Mailing list: >> https://launchpad.net/~__**openstack<https://launchpad.net/~__openstack> >> >> <https://launchpad.net/~**openstack<https://launchpad.net/~openstack> >> > >> Post to : openst...@lists.launchpad.net >> <mailto:openstack@lists.**launchpad.net<openst...@lists.launchpad.net> >> > >> Unsubscribe : >> https://launchpad.net/~__**openstack<https://launchpad.net/~__openstack> >> <https://launchpad.net/~**openstack<https://launchpad.net/~openstack> >> > >> More help : >> https://help.launchpad.net/__**ListHelp<https://help.launchpad.net/__ListHelp> >> >> <https://help.launchpad.net/**ListHelp<https://help.launchpad.net/ListHelp> >> > >> >> >> >> >> >> ______________________________**_________________ >> Mailing list: %(list_owner)s >> Post to : %(list_post)s >> Unsubscribe : %(list_unsubscribe)s >> More help : %(list_help)s >> >> > > ______________________________**_________________ > Mailing list: %(list_owner)s > Post to : %(list_post)s > Unsubscribe : %(list_unsubscribe)s > More help : %(list_help)s > >
_______________________________________________ Mailing list: %(list_owner)s Post to : %(list_post)s Unsubscribe : %(list_unsubscribe)s More help : %(list_help)s
