David,
Yes, that is the goal. We should be able to get information about "how" it works, not only that it works. And get such interesting results: https://wiki.openstack.org/wiki/Rally#How_influence_amqp_rpc_single_reply_queue_on_performance Best regards, Boris Pavlovic --- Mirantis Inc. On Fri, Oct 18, 2013 at 8:57 PM, David Kranz <dkr...@redhat.com> wrote: > On 10/18/2013 12:17 PM, Boris Pavlovic wrote: > > John, > > Actually seems like a pretty good suggestion IMO, at least something > worth some investigation and consideration before quickly discounting it. > Rather than "that's not what tempest is", maybe it's something tempest > "could do". Don't know, not saying one way or the other, just wondering if > it's worth some investigation or thought. > > > These investigations I made before start working around Rally. It was > about 3 months ago. > It is not "quickly discounting" I didn't have yesterday time to make long > response, so I will write it today: > > I really don't like to make a copy of another projects, so I tried to > reuse all projects & libs that we already have. > > To explain why we shouldn't merge Rally & Tempest in one project (and > should keep both) we should analyze their use cases. > > > 1. DevStack - one "click" and get your OpenStack cloud from sources > > 2. Tempest - one "click" and get your OpenStack Cloud verified > > Both of these projects are great, because they are very useful and solve > complicated tasks without "pain" for end user. (and I like them) > > 3. Rally is also one "click" system that solve OpenStack benchmarking. > > To clarify situation. We should analyze what I mean by one "click" > benchmarking and what are the use cases. > > Use Cases: > 1. Investigate how deployments influence on OS performance (find the set > of good OpenStack deployment architectures) > 2. Automatically get numbers & profiling info about how your changes > influence on OS performance > 3. Using Rally profiler detect scale & performance issues. > Like here when we are trying to delete 3 VMs by one request they are > deleted one by one because of DB lock on quotas table > http://37.58.79.43:8080/traces/0011f252c9d98e31 > 4. Determine maximal load that could handle production cloud > > To cover these cases we should actually test OpenStack deployments > making simultaneously OpenStack API calls. > > So to get results we have to: > 1. Deploy OpenStack cloud somewhere. (Or get existing cloud) > 2. Verify It > 3. Run Benchmarks > 4. Collect all results & present it in human readable form. > > > The goal of Rally was designed to automate these steps: > 1.a Use existing cloud. > 1.b.1 Automatically get (virtual) Servers from (soft layer, Amazon, > RackSpace or you private public cloud, or OpenStack cloud) > 1.b.2 DeployOpenStack on these servers from source (using Devstack, Anvli, > Fuel or TrippleO...). > 1.b.3 Patch this OpenStack with tomograph to get profiling information (I > hope we will merge these patches into upstream). > 2. Using tempest verify this cloud (we are going to switch from > fuel-ostf-tests) > 3. Run specified parametrized (to be able to make different load) > benchmark scenarios > 4. Collect all information about execution & present it in human readable > form. (Tomograph, Zipking, matplotlib...) > > > So I am not sure that we should put inside Tempest Rally, because Rally > use tempest. It is something like putting Nova into Cinder =) > Also putting Tempest into Rally is not a good idea. (same as putting > Cinder back to Nova). > > > Best regards, > Boris Pavlovic > --- > Mirantis Inc. > > > On Thu, Oct 17, 2013 at 11:56 PM, John Griffith < > john.griff...@solidfire.com> wrote: > >> >> >> >> On Thu, Oct 17, 2013 at 1:44 PM, Jay Pipes <jaypi...@gmail.com> wrote: >> >>> On 10/17/2013 03:32 PM, Boris Pavlovic wrote: >>> >>>> Jay, >>>> >>>> >>>> Or, alternately, just have Rally as part of Tempest. >>>> >>>> >>>> Actually, tempest is used only to verify that cloud works properly. >>>> And verification is only small part of the Rally. >>>> >>>> At this moment we are using fuel-ostf-tests, but we are going to use >>>> tempest to verify cloud. >>>> >>> >>> OK, cool... was just a suggestion :) Tempest has a set of stress tests >>> [1] which are kind of related, which is the only reason I brought it up. >>> >>> Best, >>> -jay >>> >>> [1] https://github.com/openstack/tempest/tree/master/tempest/stress >>> >>> >>> _______________________________________________ >>> OpenStack-dev mailing list >>> OpenStack-dev@lists.openstack.org >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >> >> Actually seems like a pretty good suggestion IMO, at least something >> worth some investigation and consideration before quickly discounting it. >> Rather than "that's not what tempest is", maybe it's something tempest >> "could do". Don't know, not saying one way or the other, just wondering if >> it's worth some investigation or thought. >> >> By the way, VERY COOL!! >> >> >> _______________________________________________ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> > > > _______________________________________________ > OpenStack-dev mailing > listOpenStack-dev@lists.openstack.orghttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > Thanks, Boris. This is really great. I took a look and there is a lot of > similarity between the tempest stress framework and the rally benchmark > driver and some code could be shared. But that code is only a very small > part of each of these projects and IMO there is no reason to try and > integrate the two more deeply. Rally will be beneficial to the OpenStack QA > community which needs to grow beyond "OpenStackQA == tempest". > > -David > > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >
_______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
