The problem with this approach is that we shouldn't rely on just mobile-spec for our testing. In this example, I wasn't able to fully reproduce what he was doing because the person who filed it refused to provide me with a project. That being said, I was able to find that I was getting weird TIMEOUT errors with the code due to it going through the wrong method. While on my devices, it was a really minor error in logcat that would be easily ignored, it apparently causes meltdowns on the emulator.
The fact is that on Android, if you're doing changes to the core, you should be running the JUnit tests incldued with the project. These tests are completely independent of mobile-spec and catch a majority of the native use cases that we currently have running. mobile-spec is the cross-platform test suite, and every platform should have its own native test suite to test its internals to make sure we're not totally screwing things up. If there's an error from Cordova that appears after running it, chances are that something is wrong, even if the tests pass. Remember that for certain bugs, like bugs where you exit the app, which this one was, it's really hard to test those even with JUnit. As far as this being a recurring problem, I do think that it is starting recurring problem, since our users are saying that they haven't seen a quality release of Cordova-Android since 2.4. That means that we broke something in the past four months that meant that they couldn't upgrade. Some of it was stuff we couldn't avoid, like the WebSQL vs WebStorage choice, or deprecations of Plugins.java, but most of it was stuff like this and the Splashscreen error, where we'd accept a patch without testing. So, please write more JUnit tests as well as mobile-spec. mobile-spec is not a silver bullet for testing on Cordova, since it really only tests Cordova JS and the API. On Wed, Jun 12, 2013 at 2:07 PM, Marcel Kinard <cmarc...@gmail.com> wrote: > +1 to Ian's comments. > > How about if: > - the developer (contributor or committer) is responsible to test their own > code and correct any problems before a pull request is submitted (contributor > authored) or it lands in the stream (committer authored). The testing > includes both verifying the function they added/touched, plus running > mobile-spec to verify there are no regressions. > - if a pull request, the committer should sanity check the contributor's code. > - if a pull request, the committer should see that the contributor tested it. > Perhaps that could be a simple comment in Jira, similar to how Mike and Lisa > worked as contributors here: https://issues.apache.org/jira/browse/CB-3521 . > > I find it odd that "Processing Pull Requests" in > http://wiki.apache.org/cordova/CommitterWorkflow doesn't mention anything > about testing or sanity checking / reviewing what comes from the contributor. > If there is consensus, I can add the verbage above to "Processing Pull > Requests". (Hmm, shouldn't there also be something there about doc for > external behaviors / interfaces / requirements?) > > Based on other commercial projects I've seen, and how aggressive you want to > be, I'd also be tempted to suggest: > - every new function and defect fix should include an automated test (where > reasonably feasible) that lands in mobile-spec at the same time as the new > function / fix. What this does over time is to build up coverage in the > automated test suite that gets a lot better at catching breakages / > regressions. Yes, this takes time to do, but when mobile-spec is run, you've > got confidence. > > -- Marcel Kinard
