On 8/14/12 12 :01PM, "Jeff Conrad" <jeff...@gmail.com> wrote:
>Hi, > >I'd like to help the project get to a point to where we can run the entire >test suite for the sdk in 10 minutes or less. I think that's a worthy >goal, and I'm willing to help make that a reality. > >If we get the testing time down to being that fast, that will mean we can >review a lot more contributions, and it could also mean that potential >contributors can submit patches that they know don't fail any tests before >submitting the patch. It also means that if one of those tests fail, the >person writing the code still has the context of what they did fresh in >their mind and can fix it quickly. > >For reference, I ran the entire Mustella suite last night, and it took 4 >hours, 3 minutes, and 50 seconds to run ./mini_run.sh -createImages -all >last night on a quad-core machine running Windows 7 using the Git Bash >(yay! no cygwin!). To make it work with the Git Bash, I changed the shell >variable in mustella/build.xml to just sh.exe. As a plus, shellrunner.sh >somewhat intelligently parallel-ized the compilation of all the test swfs >so it was compiling 4 swfs at a time, and then the ant script ran all the >test swfs one at a time. I know mustella is more a functional / >integration test suite, so by definition it's going to run slower than a >suite of unit tests. > >I also know that at least a few people on this list want to refactor the >sdk so it's unit-testable. I definitely support this and would like to >help in any way I can on that front. > >I want the entire suite: unit, integration, and functional to be able to >run in 10 minutes or less. I have two ideas as to how we can make this >happen, and I'm open to more. > >One idea would be to intelligently look at the files that a given patch / >changeset affects and only build and run the tests that test that >functionality. That's what we used to do at Adobe. We had what we called a cyclone server that we submitted a patch to and it ran the subset of tests based on the changed files. It used to take more than 10 minutes to get the results though. The problem with our implementation is the set of tests that were run was based on a database that was manually put together, not based on code inspection. Over time the db became less and less accurate because it wasn't kept up to date (and it might not ever have been totally accurate because it was manually created). As to your other question about "hurricane". I don't know what that is. Maybe Alex remembers. I was working on mustella/build.xml and mini_run.sh last Friday. I suspect there is a lot of dead code in there but we need to get everything working first before we "clean". The mustella directory was previously maintained by the QE group for their setups. I think there are code paths in there that no longer apply. BTW I'm not sure you can get the run down to 10 minutes but I think there are an awful lot of duplicate tests and if we could figure which were duplicates we could probably just throw away many tests. Carol
