On Fri, Jul 7, 2017 at 12:15 PM, Ehsan Akhgari <ehsan.akhg...@gmail.com> wrote:
> FWIW years ago I decided to act on trying to detect some of the patterns > indicated on that page automatically, see https://bugzilla.mozilla.org/s > how_bug.cgi?id=649012. It didn't use linting (because we didn't really > have any linting support back then) but the idea is the same, just with a > different syntax. That project never really received much (well, any!) > traction since I always was hoping that people would triage the initial set > of "opt-out" annotations that it landed with, and do something about those > tests, but that never happened. But hopefully this helped a bit by > plugging one of the holes in terms of the unintended sources of > intermittent oranges. > > In this case the way to opt out was adding a single line to the test. > This wasn't hard per se, but if my memory serves it used to trip people for > quite a while, even though the failure message you would get if you forgot > to do it would indicate how to opt out of the extra check. In hindsight, I > think some people may have found the check more annoying than helpful, > because in the cases where it helps, you'd get green tests, so nobody would > consciously know that this is helping and in the cases where it would get > in your way, people would be rightfully unhappy since their workflow got > disturbed. > I forgot mochitest already checks for flaky timeouts! Maybe people would be less opposed to a lint rule as many teams (especially frontend) have already ingrained eslint into their workflows and know how to deal with enabling/disabling rules. I guess in the setTimeout case we'd need to choose one or the other, either eslint or requestFlakyTimeout. Doing both at once would be a pain for developers. Fwiw, the eslint rule for no-arbitrary-setTimeout turned out to be pretty easy to implement. > >> 1. Do any of the other patterns on that page look feasible to >> implement as linters? >> >> I always struggled with this one. Not many of them are, in my opinion. > The example of using Date()s is, for example, but in my experience that is > a super rare cause of intermittent failures... A lot of these require some > kind of dynamic check at runtime, and aren't easily detectible by just > looking at the code of the test without running it. > >> >> 1. Are there additional things not listed on that page that we could >> lint for? >> >> Almost certainly. Ideally when people fix an intermittent orange, they > would think about whether this was a pattern that could reappear in other > places, and whether it could be detected using a linter. It is unclear to > me how many of these new unique patterns we encounter though. (For > example, *many* intermittent oranges boil down to code expecting the wrong > ordering of events, and that's not easily detectible using a linter.... > But there may be a long tail of super rare error cases that are detectible > using linters.) > > Hope this helps, > Ehsan > Thanks for the insight. If we can block maybe 1 or 2 major causes of intermittents with this, I'd be pretty happy. Maybe the bigger benefit of this would be laying a foundation for adding new such linters in the future, even if they only avoid minor causes of intermittents. -Andrew _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
