On 01/03/2014 02:08 PM, Joshua Harlow wrote:
Thinking more on reproducible environment, I thought to experiment with /docker/ to configure Keystone and Rally. This would allow us to quickly compare different configuration of Keystone, though I not sure about the overhead of /docker/. I have put down the details at:-Awesome!!I am looking forward to any results u get. For said results when they arrive make sure they are reproducible and accurate, and then people can start solving any performance problems in a way that can be compared to the initial baseline...
https://github.com/nkhare/keystone-performance Any comments and suggestions are welcome. Regards, Neependra
2014 the year of rally :P Sent from my really tiny device...On Jan 2, 2014, at 11:32 PM, "Neependra Khare" <nkh...@redhat.com <mailto:nkh...@redhat.com>> wrote:Hi All, I wanted to share the current status of this work.Thanks to Boris and the team for implementing keystone benchmark with Rally.It has been merged with upstream Rally code base couple of days back. I have tested above with devstack on Fedora 19 and Ubuntu 12.04.With Rally Deploy engine we can deploy OpenStack distribution like DevStackbefore running the benchmark. https://wiki.openstack.org/wiki/Rally/DeployEngines#DevstackEngineI am planning to prepare the /localrc/ file for /devstack/ by which we can configure keystone with different configuration like with PKI/UUID tokens, with or withoutmemcached etc. This would help us is automating the process. I would be documenting everything and would share it with the community. Regards, Neependra On 12/20/2013 02:09 PM, Neependra Khare wrote:Hi Joshua, On 12/20/2013 01:34 PM, Joshua Harlow wrote:Awesome i will see if I can make it, maybe u guys can send out a summary afterwards? Glad to see performance work (and the associated tooling) get the attention it deserves!We had a meeting yesterday. Etherpad has most of things we discussed:- https://etherpad.openstack.org/p/Rally_Keystone_BenchmarksWe decided to have specific pattern in workload (like user name) for keystonebenchmarks as that would make clean-up easier.INIT methods like creating N users, would setup environment before running the actual setup. Jay suggested set that may be we can use some form of snapshot for images, which have per-populated dataset. So that, we do not have to spendtime in INIT methods every time we run the tests. Next steps- Boris would have the clean-up and basic INIT functionality written down for Keystone, which would then help us to write our first Keystone benchmark.- Meanwhile we'll update the benchmark we want to run with Rally in /"//List of benchmarksthat we would like to have//"/ section of the etherpad. Boris, Jay, Hugh, Tristan and others, If I missed any important point then please feel free to add here. Regards, NeependraSent from my really tiny device...On Dec 18, 2013, at 5:32 AM, "Neependra Khare" <nkh...@redhat.com <mailto:nkh...@redhat.com>> wrote:On 12/17/2013 05:19 PM, Neependra Khare wrote:On 12/17/2013 12:48 AM, Joshua Harlow wrote:Another thing that u might consider. The rally project[1] has been using a tool called tomograph[2] and making tomograph better and it has been collecting some similar use-cases and test-cases around various openstack performance related work (also it has been working on defining said methodology, how to setup for performance tests, and more...). Some examples of various keystone/nova/glance related calls (see the low level trace information gathered there):Thanks Josh. We have started a blueprint to have keystone related benchmarks with Rally. https://blueprints.launchpad.net/rally/+spec/keystone-benchmarkWe are planning to have g+ hangout tomorrow (Dec 19, 2013) to discuss Keystone related benchmarks with Rally at 11:00 EST https://etherpad.openstack.org/p/Rally_Keystone_BenchmarksFeel free to join it. Just update your name on above mentioned etherpadunder Interested participants section. Regards, NeependraRegards, Neependra- https://raw.github.com/stackforge/tomograph/master/doc/screenshots/zipkin-g lance-image-list.png -http://www.slideshare.net/mirantis/rally-benchmarkingatscale/24 It would seem like some of our scripts/test-cases would actually fit quite nicely into rally instead of being one-offs. I know that boris-42 would appreciate a single solution instead of one-offs. It seems like rally is becoming that. [1]https://wiki.openstack.org/wiki/Rally [2]https://github.com/stackforge/tomograph Jump in IRC and the rally team I'm sure would love to chat (#openstack-rally, freenode). -Josh On 12/16/13, 10:00 AM, "Jay Pipes"<jaypi...@gmail.com> wrote:On 12/16/2013 02:25 AM, Neependra Khare wrote:Hi Jay, Thanks for your comments. Please find my reply in-line. On 12/14/2013 12:58 AM, Jay Pipes wrote:I have listed down the methodology I'll be following for this test:-https://wiki.openstack.org/wiki/KeystonePerformance#Identify_CPU.2C_Dis k.2C_Memory.2C_Database_bottlenecksMy first suggestion would be to rework the performance benchmarking work items to have clearer indications regarding *what are the metrics being tested* in each work item.Performance characterization is an iterative process. I am open to rework on the work-items as we go along.Right, but the smaller the work item, the easier the iterations are :)For example, the first work item is "Identify CPU, Disk, Memory, and Database Bottlenecks". The first test case listed is: "Test #1, Create users in parallel and look for CPU, disk or memory bottleneck." I think that is a bit too big of an initial bite ;) Instead, it may be more effective to instead break down the performance analysis based on the metrics you wish to test and the relative conclusions you wish your work to generate. For example, consider this possible work item: "Determine the maximum number of token authentication calls that can be performed"Tests like these would be very subjective to the hardware and software resources we have like no. of CPUs, Memcahced etc. Its is very important to see if we can find any obvious bottlenecks.No, that's not my point. When you have a specific metric like "number of token authentication calls that can be performed in X minutes", you can iterate based on singular changes -- not to the hardware, but to the configuration of the software. If you are trying to solve the problem of "where are my bottlenecks", without first identifying what metrics will describe how a piece of software scales, then you are putting the cart before the horse.Within that work item, you can then further expand a testing matrix, like so: * Measure the total number of token authentication calls performed by a single client against a single-process, Python-only Keystone server * Measure the total number of token authentication calls performed by a single client against a multi-process Keystone server running inside an nginx or Apache container server -- with 2, 4, 8, 16, and 32 pre-forked processesAny pointers on configuring multi-process Keystone would be helpful. I see a method mentioned in "Run N keystone Processes" section of following:- http://blog.gridcentric.com/bid/318277/Boosting-OpenStack-s-Parallel-Perf ormance"Absolutely. You can spawn Keystone server in multiple pre-forked Apache processes by configuring Keystone in an Apache vhost. Some general docs: http://docs.openstack.org/developer/keystone/apache-httpd.html Take a look at provision.sh script in eNovance's keystone-wsgi-apache repo: https://github.com/enovance/keystone-wsgi-apache/blob/master/provision.sh# L152* Measure the above using increasing numbers of concurrent clients -- 10, 50, 100, 500, 1000. There's, of course, nothing wrong with measuring things like CPU, disk and I/O performance during tests, however there should be a clear metric that is being measured for each test.Agreed. Let me start collecting results from the tests you suggested above and I mentioned on the wiki. Once we have those, we can rework on the work-items. Does that sound OK ?Sure, absolutely. I'm just big on first defining the metrics by which scale can be described, and THEN describing the test variations and iterations...My second suggestion would be to drop the requirement of using RDO -- or any version of OpenStack for that matter.My end goal would be to have scripts that one can run on any of the OpenStack distribution. RDO is mentioned here an example here.Probably worth just removing the RDO reference entirely from the wiki, since, as you agree below, benchmarking Keystone actually does not require installing OpenStack as a whole at all... Best, -jayIn these kinds of tests, where you are not measuring the integrated performance of multiple endpoints, but are instead measuring the performance of a single endpoint (Keystone), there's no reason, IMHO, to install all of OpenStack. Installing and serving the Keystone server (and it's various drivers) is all that is needed. The fewer "balls up in the air" during a benchmarking session, the fewer side-effects are around to effect the outcome of the benchmark...Agreed. As mentioned in following I suggested to install just Keystone on the instances, where the tests would be performed :- https://wiki.openstack.org/wiki/KeystonePerformance#Test_.231.2C_Create_u sers_in_parallel_and_look_for_CPU.2C_disk_or_memory_bottleneck. Thanks, NeependraBest, -jay _______________________________________________ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to :openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack_______________________________________________ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to :openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack_______________________________________________ Mailing list:http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to :openstack@lists.openstack.org Unsubscribe :http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack_______________________________________________Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org <mailto:openstack@lists.openstack.org> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
_______________________________________________ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
