As Alex Sorokin kindly pointed out to me on Slack, we have Appium UI automation already existing for the contacts [1] and camera [2] plugins. Our CI system also executes on this to automate the UI end-to-end and guard against further errors [3]. The CI tosses this job over to Sauce Labs to do the heavy lifting: Sauce runs the test [4] from the example from [3], and, based on the logs, looks to execute all that fun web + native context UI automation - unfortunately Sauce only stores assets around for a month, so we can't see the video or screenshots (it's been over a month since the last contacts PR).
But all that's to say that, with that in place, I think it's mostly a matter of applying this approach to other plugins. I think the value of the UI automation-based tests is wayyyy higher than the autotests; asserting that a JavaScript function exists inside the Cordova webview is a good sanity check, but asserting that you can pick a contact via a native UI and have the relevant contact details show up back in the Cordova webview, in my opinion, is a much more thorough test of the functions of the plugin APIs. Lastly, I think documenting how to set up UI automation tests such as the ones in the camera and contacts plugins in the cordova-plugin-test-framework readme makes sense. Let me know what y'all think. [1] https://github.com/apache/cordova-plugin-contacts/blob/master/appium-tests [2] https://github.com/apache/cordova-plugin-camera/blob/master/appium-tests [3] http://cordova-ci.cloudapp.net:8080/view/Pull%20requests/job/cordova-plugin-contacts-pr/lastSuccessfulBuild/PLATFORM=ios/consoleText [4] https://saucelabs.com/beta/tests/b014505bd8894ff58b383471bdaf4759 On Wed, Dec 7, 2016 at 6:14 AM, Filip Maj <maj....@gmail.com> wrote: > I am planning on putting a proof of concept together for a few plugins, and > having something like mobile spec be able to string multiple plugins' UI > automation tests together and assert pass/failure (via its defineAutoTests / > defineManualTests-like functions). > > I'll post more when I have something to show :) > > The idea would then be to help guide how to do this kind of testing via the > framework Cordova has built up for plugin testing via the > cordova-plugin-test-framework plugin. > > On Dec 7, 2016 05:56, "julio cesar sanchez" <jcesarmob...@gmail.com> wrote: >> >> I was the other way around, I got Appium working easily, but I was never >> able to make calabash work in iOS because it always failed to instrument >> my >> app. >> Both worked fine for Android though. >> >> >> 2016-12-07 12:48 GMT+01:00 Trevor Brindle <tabrin...@gmail.com>: >> >> > As little market share as windows has, it's hard to say how much effort >> > should be dedicated to additional testing for it. That said, if windows >> > testing is deemed necessary, I don't think we have much choice. >> > >> > Just keep in mind the new windows support and new iOS driver are all >> > appium >> > 1.6, only released less than 60 days ago. >> > >> > Perhaps a PoC project should be done for picking the appropriate >> > technology? >> > >> > We are midway through a similar PoC at my place of work, I could >> > volunteer >> > some time. >> > >> > On Tue, Dec 6, 2016 at 10:02 PM Filip Maj <maj....@gmail.com> wrote: >> > >> > > Full disclosure: I contribute(d) to appium and worked for Sauce Labs, >> > > >> > > so I am pretty biased ;) >> > > >> > > >> > > >> > > Calabash, unfortunately, is iOS and Android only and does not support >> > > >> > > Windows app, whereas Appium does via WinAppDriver [1]. >> > > >> > > >> > > >> > > I agree that appium, at least earlier on, was difficult to set up. >> > > >> > > These days it's a simple `npm install`. I think appium has a much more >> > > >> > > prolific committership to boot (MSFT contributes to it, for example). >> > > >> > > The github network stats for each project back that up as well. >> > > >> > > >> > > >> > > [1] https://github.com/microsoft/winappdriver >> > > >> > > >> > > >> > > On Tue, Dec 6, 2016 at 4:02 PM, Trevor Brindle <tabrin...@gmail.com> >> > > wrote: >> > > >> > > > In my experience with appium, it is troublesome to set up and >> > > > maintain. >> > > We >> > > >> > > > may investigate other frameworks that have similar functionality. I >> > have >> > > >> > > > had great luck with calabash. >> > > >> > > > >> > > >> > > > >> > > >> > > > On Tue, Dec 6, 2016 at 5:30 PM julio cesar sanchez < >> > > jcesarmob...@gmail.com> >> > > >> > > > wrote: >> > > >> > > > >> > > >> > > >> +1 to appium >> > > >> > > >> >> > > >> > > >> >> > > >> > > >> >> > > >> > > >> 2016-12-05 23:33 GMT+01:00 Filip Maj <maj....@gmail.com>: >> > > >> > > >> >> > > >> > > >> >> > > >> > > >> >> > > >> > > >> > Hi, it's me again! >> > > >> > > >> >> > > >> > > >> > >> > > >> > > >> >> > > >> > > >> > How I'd like to contribute to Cordova is to help automate the >> > > >> > stuff >> > > >> > > >> >> > > >> > > >> > that saves committers having to take the manual time to do >> > themselves. >> > > >> > > >> >> > > >> > > >> > I think a good first goal would be to help automate as much of >> > > >> > > >> >> > > >> > > >> > platform release testing as possible [1]. The autotests seem to >> > > >> > be >> > > >> > > >> >> > > >> > > >> > handled relatively well by the CI system so far (if anyone feels >> > this >> > > >> > > >> >> > > >> > > >> > is not true, please speak up! I want to hear about what works and >> > what >> > > >> > > >> >> > > >> > > >> > does not). The other part of mobile-spec-based platform release >> > > >> > > >> >> > > >> > > >> > testing is to "run the manual tests". After reading the >> > > >> > > >> >> > > >> > > >> > cordova-plugin-test-framework README [2], I thought this would be >> > > >> > a >> > > >> > > >> >> > > >> > > >> > good place to give plugin developers some better tools to deal >> > > >> > with >> > > >> > > >> >> > > >> > > >> > testing complex UI interactions that the autotests can't handle >> > > >> > on >> > > >> > > >> >> > > >> > > >> > their own. I was thinking appium [3] would be a good tool to >> > > >> > > >> >> > > >> > > >> > complement that in this case. It gives us UI hooks into both web >> > > >> > and >> > > >> > > >> >> > > >> > > >> > natives contexts within hybrid applications, plus it also allows >> > > >> > us >> > to >> > > >> > > >> >> > > >> > > >> > inject JavaScript into the web context. Wondering what others >> > > >> > think? >> > > >> > > >> >> > > >> > > >> > >> > > >> > > >> >> > > >> > > >> > I could then foresee, with a little bit of scaffolding, a way to >> > > >> > > >> >> > > >> > > >> > string plugins' appium tests together to fully automate the >> > > >> > 'manual' >> > > >> > > >> >> > > >> > > >> > testing of plugin tests during platform release testing. >> > > >> > > >> >> > > >> > > >> > >> > > >> > > >> >> > > >> > > >> > Let me know what y'all think! >> > > >> > > >> >> > > >> > > >> > >> > > >> > > >> >> > > >> > > >> > [1] https://github.com/apache/cordova-coho/blob/master/docs/ >> > > >> > > >> >> > > >> > > >> > platforms-release-process.md#what-to-test >> > > >> > > >> >> > > >> > > >> > [2] https://github.com/apache/cordova-plugin-test-framework# >> > > >> > > >> >> > > >> > > >> > defining-manual-tests >> > > >> > > >> >> > > >> > > >> > [3] appium.io >> > > >> > > >> >> > > >> > > >> > >> > > >> > > >> >> > > >> > > >> > ------------------------------------------------------------ >> > --------- >> > > >> > > >> >> > > >> > > >> > To unsubscribe, e-mail: dev-unsubscr...@cordova.apache.org >> > > >> > > >> >> > > >> > > >> > For additional commands, e-mail: dev-h...@cordova.apache.org >> > > >> > > >> >> > > >> > > >> > >> > > >> > > >> >> > > >> > > >> > >> > > >> > > >> >> > > >> > > >> >> > > >> > > >> > > >> > > --------------------------------------------------------------------- >> > > >> > > To unsubscribe, e-mail: dev-unsubscr...@cordova.apache.org >> > > >> > > For additional commands, e-mail: dev-h...@cordova.apache.org >> > > >> > > >> > > >> > > >> > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@cordova.apache.org For additional commands, e-mail: dev-h...@cordova.apache.org
