Another thing that u might consider. The rally project[1] has been using a tool called tomograph[2] and making tomograph better and it has been collecting some similar use-cases and test-cases around various openstack performance related work (also it has been working on defining said methodology, how to setup for performance tests, and more...).
Some examples of various keystone/nova/glance related calls (see the low level trace information gathered there): - https://raw.github.com/stackforge/tomograph/master/doc/screenshots/zipkin-g lance-image-list.png - http://www.slideshare.net/mirantis/rally-benchmarkingatscale/24 It would seem like some of our scripts/test-cases would actually fit quite nicely into rally instead of being one-offs. I know that boris-42 would appreciate a single solution instead of one-offs. It seems like rally is becoming that. [1] https://wiki.openstack.org/wiki/Rally [2] https://github.com/stackforge/tomograph Jump in IRC and the rally team I'm sure would love to chat (#openstack-rally, freenode). -Josh On 12/16/13, 10:00 AM, "Jay Pipes" <jaypi...@gmail.com> wrote: >On 12/16/2013 02:25 AM, Neependra Khare wrote: >> Hi Jay, >> >> Thanks for your comments. Please find my reply in-line. >> >> On 12/14/2013 12:58 AM, Jay Pipes wrote: >>> I have listed down the methodology I'll be following for this test:- >>>> >>>>https://wiki.openstack.org/wiki/KeystonePerformance#Identify_CPU.2C_Dis >>>>k.2C_Memory.2C_Database_bottlenecks >>>> >>> >>> My first suggestion would be to rework the performance benchmarking >>> work items to have clearer indications regarding *what are the metrics >>> being tested* in each work item. >> Performance characterization is an iterative process. I am open to >> rework on the work-items as we >> go along. > >Right, but the smaller the work item, the easier the iterations are :) > >>> For example, the first work item is "Identify CPU, Disk, Memory, and >>> Database Bottlenecks". >>> >>> The first test case listed is: >>> >>> "Test #1, Create users in parallel and look for CPU, disk or memory >>> bottleneck." >>> >>> I think that is a bit too big of an initial bite ;) >>> >>> Instead, it may be more effective to instead break down the >>> performance analysis based on the metrics you wish to test and the >>> relative conclusions you wish your work to generate. >>> >>> For example, consider this possible work item: >>> >>> "Determine the maximum number of token authentication calls that can >>> be performed" >> Tests like these would be very subjective to the hardware and software >> resources we have like >> no. of CPUs, Memcahced etc. Its is very important to see if we can find >> any obvious bottlenecks. > >No, that's not my point. When you have a specific metric like "number of >token authentication calls that can be performed in X minutes", you can >iterate based on singular changes -- not to the hardware, but to the >configuration of the software. If you are trying to solve the problem of >"where are my bottlenecks", without first identifying what metrics will >describe how a piece of software scales, then you are putting the cart >before the horse. > >>> Within that work item, you can then further expand a testing matrix, >>> like so: >>> >>> * Measure the total number of token authentication calls performed by >>> a single client against a single-process, Python-only Keystone server >>> * Measure the total number of token authentication calls performed by >>> a single client against a multi-process Keystone server running inside >>> an nginx or Apache container server -- with 2, 4, 8, 16, and 32 >>> pre-forked processes >> Any pointers on configuring multi-process Keystone would be helpful. I >> see a method >> mentioned in "Run N keystone Processes" section of following:- >> >>http://blog.gridcentric.com/bid/318277/Boosting-OpenStack-s-Parallel-Perf >>ormance" > >Absolutely. You can spawn Keystone server in multiple pre-forked Apache >processes by configuring Keystone in an Apache vhost. Some general docs: > >http://docs.openstack.org/developer/keystone/apache-httpd.html > >Take a look at provision.sh script in eNovance's keystone-wsgi-apache >repo: > >https://github.com/enovance/keystone-wsgi-apache/blob/master/provision.sh# >L152 > >>> * Measure the above using increasing numbers of concurrent clients -- >>> 10, 50, 100, 500, 1000. >>> >>> There's, of course, nothing wrong with measuring things like CPU, disk >>> and I/O performance during tests, however there should be a clear >>> metric that is being measured for each test. >> Agreed. Let me start collecting results from the tests you suggested >> above and I mentioned >> on the wiki. Once we have those, we can rework on the work-items. Does >> that sound OK ? > >Sure, absolutely. I'm just big on first defining the metrics by which >scale can be described, and THEN describing the test variations and >iterations... > >>> My second suggestion would be to drop the requirement of using RDO -- >>> or any version of OpenStack for that matter. >> My end goal would be to have scripts that one can run on any of the >> OpenStack distribution. >> RDO is mentioned here an example here. > >Probably worth just removing the RDO reference entirely from the wiki, >since, as you agree below, benchmarking Keystone actually does not >require installing OpenStack as a whole at all... > >Best, >-jay > >>> In these kinds of tests, where you are not measuring the integrated >>> performance of multiple endpoints, but are instead measuring the >>> performance of a single endpoint (Keystone), there's no reason, IMHO, >>> to install all of OpenStack. Installing and serving the Keystone >>> server (and it's various drivers) is all that is needed. The fewer >>> "balls up in the air" during a benchmarking session, the fewer >>> side-effects are around to effect the outcome of the benchmark... >> Agreed. As mentioned in following I suggested to install just Keystone >> on the instances, where the tests would be performed :- >> >>https://wiki.openstack.org/wiki/KeystonePerformance#Test_.231.2C_Create_u >>sers_in_parallel_and_look_for_CPU.2C_disk_or_memory_bottleneck. >> >> >> Thanks, >> Neependra >> >> >>> >>> Best, >>> -jay >>> >>> >>> >>> _______________________________________________ >>> Mailing list: >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >>> Post to : openstack@lists.openstack.org >>> Unsubscribe : >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >> > > >_______________________________________________ >Mailing list: >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >Post to : openstack@lists.openstack.org >Unsubscribe : >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack _______________________________________________ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
