On Wed, Jan 15, 2014 at 11:20:22AM -0500, Yair Fried wrote: > Hi Guys > As Maru pointed out - NetworkBasicOps scenario has grown out of proportion > and is no longer "basic" ops.
Is your issue here that it's just called basic ops and you don't think that's reflective of what is being tested in that file anymore. If that's the case then just change the name. > > So, I started breaking it down to smaller test cases that can fail > independently. I'm not convinced this is needed. Some scenarios are going to be very involved and complex. Each scenario tests is designed to simulate real use cases in the cloud, so obviously some of them will be fairly large. The solution for making debugging easier in these cases is to make sure that any failures have clear messages. Also make sure there is plenty of signposting debug log messages so when something goes wrong you know what state the test was in. If you split things up into smaller individual tests you'll most likely end up making tests that are really aren't scenario tests. They'd be closer to API tests, just using the official clients, which really shouldn't be in the scenario tests. > > Those test cases share the same setup and tear-down code: > 1. create network resources (and verify them) > 2. create VM with floating IP. > > I see 2 options to manage these resources: > 1. Completely isolated - resources are created and cleaned using setUp() and > tearDown() methods [1]. Moving cleanup to tearDown revealed this bug [2]. > Apparently the previous way (with tearDownClass) wasn't as fast). This has > the side effect of having expensive resources (ie VMs and floating IPs) > created and discarded multiple times though they are unchanged. > > 2. Shared Resources - Using the idea of (or actually using) Fixtures - use > the same resources unless a test case fails, in which case resources are > deleted and created by the next test case [3]. If you're doing this and splitting things into smaller tests then it has to be option 1. Scenarios have to be isolated if there are resources shared between scenario tests that really is only one scenario and shouldn't be split. In fact I've been working on a change that fixes the scenario test tearDowns that has the side effect of enforcing this policy. Also just for the record we've tried doing option 2 in the past, for example there used to be a tenant-reuse config option. The problem with doing that was actually tends to cause more non-deterministic failures or adding a not insignificant wait time to ensure the state is clean when you start the next test. Which is why we ended up pulling this out of tree. What ends up happening is that you get leftover state from previous tests and the second test ends up failing because things aren't in the clean state that the test case assumes. If you look at some of the oneserver files in the API that is the only place we still do this in the tempest, and we've had many issues on making that work reliably. Those tests are in a relatively good place now but those are much simpler tests. Also between each test setUp has to check and ensure that the shared server is in the proper state. If it's not then the shared server has to be rebuilt. This methodology would become far more involved for the scenario tests where you have to manage more than one shared resource. > > I would like to hear your opinions, and know if anyone has any thoughts or > ideas on which direction is best, and why. > > Once this is completed, we can move on to other scenarios as well > > Regards > Yair > > [1] fully isolated - https://review.openstack.org/#/c/66879/ > [2] https://bugs.launchpad.net/nova/+bug/1269407/+choose-affected-product > [3] shared resources - https://review.openstack.org/#/c/64622/ -Matt Treinish _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
