On Wed, Dec 11, 2013 at 6:29 PM, Paul Larson <paul.lar...@canonical.com>wrote:
> On Wed, Dec 11, 2013 at 10:28 AM, Olivier Tilloy > <olivier.til...@canonical.com> wrote: > > Is it common for the webbrowser-app tests to fail, be re-run, and then > pass? > > Does that happen weekly, daily, more often? Has it always been the case, > or > > did it start happening recently? > > When it does happen, does the second run always pass, or how many re-runs > > are necessary to reach the green state? > Not *super* common, but it does happen from time to time. Last week, > we re-ran a couple of webbrowser tests. All tests have become > considerably more stable since the mtp fixes went in. Here are a > couple with failures that we reran and got better results the second > time last week: > > http://q-jenkins.ubuntu-ci:8080/job/trusty-touch-maguro-smoke-webbrowser-app-autopilot/47/ > > http://q-jenkins.ubuntu-ci:8080/job/trusty-touch-mako-smoke-webbrowser-app-autopilot/57/ > > > >> For instance, here is on image 57 webbrowser with a lot of failing > tests: > >> > http://ci.ubuntu.com/smokeng/trusty/touch/mako/57:20131210.1:20131203/5375/webbrowser-app-autopilot/ > . > >> Would be interesting to know the root case, it seems something is > screwed > >> at some point, and then all tests are failing, can you have a look at > that > >> case and try to be more robust? (or pass to the CI team if it's an infra > >> issue). > > > > > > This specific run looks like an infrastructure issue to me, after a given > > test all the subsequent tests fail when starting the app (it might be > that > > the app actually starts, but autopilot fails to find it). There are no > crash > > files in the artifacts, so it doesn’t look like an issue in the app > itself. > +Thomi - is there any way to tell the difference between the app not > being started and autopilot not finding it? > Unfortunately I don't have much to go on here either, but the maguro > test of the same image passed fine and several builds around it > passed, so it seems likely that mako would have passed if we reran it. > Do the webbrowser tests depend on external websites or does it just > connect locally on the device? > The tests spawn a local test server, no connection to the outside world is made.
-- Mailing list: https://launchpad.net/~ubuntu-phone Post to : ubuntu-phone@lists.launchpad.net Unsubscribe : https://launchpad.net/~ubuntu-phone More help : https://help.launchpad.net/ListHelp
