On 07/25/2013 11:23 PM, Chu Duc Minh wrote:
On Thu, Jul 25, 2013 at 7:30 PM, Jay Pipes <jaypi...@gmail.com
<mailto:jaypi...@gmail.com>> wrote:
You will need to provide more details than "old" vs. "new" OpenStack.
For example...
1) What is your network model in the old vs. new
Old: nova-network + FlatDHCP
New: Quantum + OpenvSwitch + network-namespace + metadata-ns-proxy
OK. If you do a time ping -c1 <IP> from a VM to the metadata IP, what
kind of times do you see in each setup?
2) What version of OpenStack is the old
Old: Essex
New: Grizzly
OK.
3) Is Keystone used in old and new? If so, what drivers are used in
Keystone?
Yes, both use Keystone with SQL backend
I recommend switching to the memcache Token driver in Keystone.
4) Do you have errors in any of your log files (usually an
indication that something like a timeout or failure on
RPC which may affect performance)
No, i didn't found it.
OK.
5) Are you using nova-conductor in the new?
I have nova-conductor process run on Controller node, but seem
nova-compute don't use this.
OK, good to know, but unlikely to be the cause of a metadata API query
slowdown.
6) What database backend are you using?
MySQL.
OK.
7) Do a time keystone user-list on both old and new
# keystone user-list
Old:
Run 1st time:
real *0m2.494s*
user 0m0.112s
sys 0m0.040s
2nd time:
real *0m0.279s*
user 0m0.112s
sys 0m0.020s
New:
Run 1st time:
real *0m1.687s*
user 0m0.176s
sys 0m0.012s
2nd time:
real *0m0.213s*
user 0m0.160s
sys 0m0.040s
Interesting, so Keystone is actually faster in Grizzly than in Essex...
or at least it seems to be given the above rudimentary timing.
8) Pastebin your conf files, with passwords removed
My config files are quite large.
I can sum up that all services & API are HA-ed using HAproxy, RabbitMQ
Cluster, Database replication, v.v...
OK.
This morning, i just do some tuning memcached for Dashboard service
(HA-ed using HAProxy), then my Dashboard become faster significantly.
The questions remain are:
1. Why metadata query so slow? Possible problems? ( "curl
http://169.254.169.254/openstack"; took 2.5-5 seconds to complete,
"ec2metadata" took 14-17 seconds to complete - i tested many times.)
Could be a number of things. I thought Vish had redesigned the metadata
API endpoint in Folsom and removed the problematic slow queries that
used to be in there. If you check your MySQL slow log (hopefully you
have it enabled), look to see if any of the queries in there reference
the instance_metadata table.
2. Why API calls using *-client reduce time from 2nd time on my old
Essex deployment, but not on my new Grizzly deployment? Maybe I need
some "cache" settings?
Looks to me that both 2nd time calls were reduced...in Essex as well as
Grizzly zones.
Best,
-jay
Thank you very much!
The more information you give, the better folks can help you.
Best,
-jay
On 07/25/2013 07:14 AM, Chu Duc Minh wrote:
Check some more API (I run these command below from Controller
node):
# time quantum subnet-list
(...have 4 subnet)
real 0m0.676s
user 0m0.196s
sys 0m0.020s
# time quantum router-list
(...have 1 router)
real 0m0.496s
user 0m0.164s
sys 0m0.052s
# time nova list --all_tenants=1
(...have 5 instances)
real 0m1.290s
user 0m0.308s
sys 0m0.040s
Compare with my old OpenStack deployment on weaker servers, it
took 1/3
in times.
On Thu, Jul 25, 2013 at 5:43 PM, Peter Cheung
<mcheun...@hotmail.com <mailto:mcheun...@hotmail.com>
<mailto:mcheun...@hotmail.com <mailto:mcheun...@hotmail.com>>__>
wrote:
I am having a problem about calling API speed is up and down,
something need 0.1s, something it needs 3s
Thanks
from Peter
------------------------------__------------------------------__------------
Date: Thu, 25 Jul 2013 17:41:11 +0700
From: chu.ducm...@gmail.com <mailto:chu.ducm...@gmail.com>
<mailto:chu.ducm...@gmail.com <mailto:chu.ducm...@gmail.com>>
To: openst...@lists.launchpad.net
<mailto:openst...@lists.launchpad.net>
<mailto:openstack@lists.__launchpad.net
<mailto:openst...@lists.launchpad.net>>;
openstack@lists.openstack.org
<mailto:openstack@lists.openstack.org>
<mailto:openstack@lists.__openstack.org
<mailto:openstack@lists.openstack.org>>
Subject: [Openstack] Call to API very slow [Grizzly]
All operations in my Openstack dashboard very slow (compare
to my
old Openstack deployment)
Then i do some check on an instance:
$ time curl http://169.254.169.254/__openstack
<http://169.254.169.254/openstack>
2012-08-10
2013-04-04
latest
real 0m*5.605s*
user 0m0.004s
sys 0m0.004s
5 seconds for a simple API query !??
in quantum-ns-metadata-proxyxxxx.__log, i saw:
2013-07-25 *17:17:09 * DEBUG
[quantum.agent.metadata.__namespace_proxy] Request: GET
/openstack
HTTP/1.0
Accept: */*
Content-Type: text/plain
Host: 169.254.169.254
User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0
OpenSSL/1.0.1 zlib/1.2.3.4 <http://1.2.3.4>
<http://1.2.3.4> libidn/1.23 librtmp/2.3
2013-07-25 *17:17:14* DEBUG
[quantum.agent.metadata.__namespace_proxy] {'date': 'Thu,
25 Jul 2013
10:17:14 GMT', 'status': '200', 'content-length': '28',
'content-type': 'text/html; charset=UTF-8', 'content-location':
u'http://169.254.169.254/__openstack
<http://169.254.169.254/openstack>'}
2013-07-25 17:17:14 DEBUG
[quantum.agent.metadata.__namespace_proxy] 2012-08-10
2013-04-04
latest
I take a look at metadata-agent.log, and saw almost
request/response
finished @*17:17:09
*
But the last finished *@**17:17:14
*2013-07-25 *17:17:14* DEBUG [quantum.agent.metadata.agent]
{'date': 'Thu, 25 Jul 2013 10:17:14 GMT', 'status': '200',
'content-length': '28', 'content-type': 'text/html;
charset=UTF-8',
'content-location': u'http://172.30.1.14:8775/__openstack
<http://172.30.1.14:8775/openstack>'}
*
*
I enabled slow query log on MySql, but can't find any slow
query.
Do you know possible problems in this situation?
Thank you very much!
_________________________________________________ Mailing list:
https://launchpad.net/~__openstack
<https://launchpad.net/~openstack> Post to :
openst...@lists.launchpad.net
<mailto:openst...@lists.launchpad.net>
<mailto:openstack@lists.__launchpad.net
<mailto:openst...@lists.launchpad.net>>
Unsubscribe : https://launchpad.net/~__openstack
<https://launchpad.net/~openstack> More help :
https://help.launchpad.net/__ListHelp
<https://help.launchpad.net/ListHelp>
_________________________________________________
Mailing list: https://launchpad.net/~__openstack
<https://launchpad.net/~openstack>
Post to : openst...@lists.launchpad.net
<mailto:openst...@lists.launchpad.net>
Unsubscribe : https://launchpad.net/~__openstack
<https://launchpad.net/~openstack>
More help : https://help.launchpad.net/__ListHelp
<https://help.launchpad.net/ListHelp>
_________________________________________________
Mailing list: https://launchpad.net/~__openstack
<https://launchpad.net/~openstack>
Post to : openst...@lists.launchpad.net
<mailto:openst...@lists.launchpad.net>
Unsubscribe : https://launchpad.net/~__openstack
<https://launchpad.net/~openstack>
More help : https://help.launchpad.net/__ListHelp
<https://help.launchpad.net/ListHelp>
_______________________________________________
Mailing list: %(list_owner)s
Post to : %(list_post)s
Unsubscribe : %(list_unsubscribe)s
More help : %(list_help)s
_______________________________________________
Mailing list: %(list_owner)s
Post to : %(list_post)s
Unsubscribe : %(list_unsubscribe)s
More help : %(list_help)s
