Thanks Gaurav for putting up this guidelines page. Good work! ~Talluri
On 19/03/15 10:59 am, "Gaurav Aradhye" <gaurav.arad...@clogeny.com> wrote: >I have added Wiki page for this in QA section. > >https://cwiki.apache.org/confluence/display/CLOUDSTACK/Guidelines+for+test >+developers > >Not a very polished one, but I will be adding and polishing it soon. > >Regards, >Gaurav > >On Thu, Mar 19, 2015 at 9:52 AM, Gaurav Aradhye ><gaurav.arad...@clogeny.com> >wrote: > >> Paul, >> >> Some pages on wiki talk about general guidelines to Dev, but there is no >> page stating detailed guidelines for test developers. >> I would like to add a new page in QA section for this. Will drop >>separate >> mail for write access to wiki. >> >> Daan, >> >> I will have to check on this. I will find out how we can include static >> analysis for tests similar to that it is in place for dev code >>(findbugs). >> >> Regards, >> Gaurav >> >> On Wed, Mar 18, 2015 at 8:22 PM, Daan Hoogland <daan.hoogl...@gmail.com> >> wrote: >> >>> good write-up Gaurav, I hope that most of these can be >>> validated/verified checkstyle-style. pep8 can of course. Some others >>> must remain judged by humanoids, like the one where functions are >>> pulled up to higher abstraction layers. Maybe you could shine your >>> light on what we can automate? i.e. can pyflakes be added to a target? >>> >>> > -----Original Message----- >>> > From: Gaurav Aradhye [mailto:gaurav.arad...@clogeny.com] >>> > Sent: 18 March 2015 07:16 >>> > To: dev@cloudstack.apache.org >>> > Subject: Guidelines for test developers >>> > >>> > Hello all, >>> > >>> > Last year after improving Marvin framework, we have been continuously >>> spending time in improving old test cases which are written in old >>>style or >>> they don't abide to certain guidelines, also which don't use new >>>functions >>> available in marvin. Many times a test developer who is adding test >>>case >>> for the first time or feature developers adding Basic Validation Tests >>> > (BVTs) tend to copy paste the code available in certain test case and >>> try to modify it according to feature and commit it. This adds to >>> inconsistencies. >>> > >>> > As and when I touch a file for fixing an issue or adding/editing it, >>>I >>> try to incorporate below guidelines and improve the existing code. But >>> still few test files are not up to the mark. Our final goal is to have >>>good >>> code in every file.So writing this mail to consolidate few rules that >>> should be known/considered by everyone adding tests to Marvin. Also, >>>if you >>> touch a file, feel free to remove any inconsistencies that are already >>> present in the file. >>> > >>> > *1. Import * should always be avoided*. When I started two years >>>back, >>> and tried to understand the code flow/ test cases, I could not easily >>> understand from where the particular module is imported. The imports >>>must >>> be specific. >>> > >>> > When the imports are specific, it eliminates the possibility of test >>> case failure due to invalid import when specific import is removed from >>> dependent module. >>> > >>> > E.g. If your test case has following import. >>> > >>> > from A import * >>> > >>> > And it uses time module which is not imported explicitly in test case >>> and comes from module A. Then the test case will start failing when >>>"import >>> time"is removed from module A. You certainly don't want this to happen. >>> > >>> > *2. Maintaining Pep8 standards for python code.* >>> > >>> > The code is read more often that it is written. Pep8 standards >>>improve >>> the readability of the code making it consistent in style. There is a >>>tool >>> named "*autopep8*" which you can install with pip install and then you >>>can >>> run following command on your test file. >>> > >>> > autopep8 -i -a -a testFile.py >>> > >>> > This will make the file pep8 consistent and will also remove the >>>white >>> spaces. But some issues need human intervention and can't be fixed with >>> tool. For fixing those, check the pep8 issues with "pep8 testFile.py" >>>and >>> fix manually. >>> > >>> > *3. Keep only imports which are used* in the test suite and remove >>> unwanted imports. >>> > >>> > *4. Keep all the configuration parameters* (such as data which is >>> passed to API while creating Network offering, Service offering, >>>account >>> etc...) *in tools/marvin/marvin/config/test_data.py file*. Don't >>>include >>> them in test suite itself. >>> > >>> > Many of the dictionaries are reusable and if you are adding a new >>>test, >>> there are only a few dictionaries you will have to add in the file. >>> > >>> > If any of the data contains URLs or any data which should be changed >>> according to setup/env, then include the dict in "*configurableData*" >>> > section in test_data.py file. This makes it easier to identify which >>> data needs to be dynamic according to the setup and which data doesn't >>>need >>> to be touched when env is changed. >>> > >>> > *5. Before committing a test case, run it* with the latest branch >>> against which you are adding the test case and attach the results in >>>Pull >>> Request. >>> > If in case change is very small, and doesn't need to be run, then at >>> least check syntactical errors with python command and also with the >>>help >>> of tools such as pyflakes. >>> > >>> > 6. If you add a new function in your test case and you think it can >>>be >>> used in future by other test cases, then please *add that function to >>> common or utils file* in Marvin. Don't keep it local to test case. This >>> will prevent multiple contributors adding same functions in their test >>>case >>> to achieve a particular goal. >>> > >>> > *7. Please make sure all the resources created through the test cases >>> are deleted *when test case execution completes, or even when the test >>>case >>> fails. >>> > >>> > 8. If same test case is to be run with different configuration or >>> setting, you can *make use of ddt library*. For example, if you have >>>added >>> test case for isolated networks, and you need to run the same code for >>> shared and VPC networks, then you don't need to add 3 test cases. Just >>>add >>> relevant tags to the test case and you are good to go. Although you >>>will >>> need to write code for handling those tags. It is already used in few >>>test >>> cases. A simple grep over component folder and you can see how it is >>>used. >>> > >>> > This blog explains how it works. >>> > >>> >>>https://technomilk.wordpress.com/2012/02/12/multiplying-python-unit-test >>>-cases-with-different-sets-of-data/ >>> > >>> > I will check if this is in any wiki page currently, and edit it. Or >>> will add a new page. >>> > I hope everyone adding test cases follows above guidelines. Feel free >>> to add more. >>> > >>> > Regards, >>> > Gaurav >>> >>> -- >>> Daan >>> >> >>
