On Fri, Aug 07, 2020 at 10:35:58PM +0200, Michael Niedermayer wrote: > On Thu, Aug 06, 2020 at 10:08:33PM +0300, Martin Storsjö wrote: > > On Mon, 3 Aug 2020, Martin Storsjö wrote: > > > > > On Mon, 3 Aug 2020, Michael Niedermayer wrote: > > > > > > > is this removing the only test which tests the "-re" flag ? > > > > if so that would reduce test coverage > > > > > > Well, tehcnically yes, but this test doesn't actually test whether the > > > "-re" flag does what it's supposed to or anything like that - it only > > > tests that adding the flag doesn't error out and doesn't alter the > > > output. > > yes > > > > > And it looks very much unintentional here. > > maybe > > > > > > Do you want to follow up the discussion here? I'd like at least some sort of > > acknowledgement back on the discussion before going forward with it in any > > direction. > > sorry for the lack of reply, theres always more to reply to and do then i seem > to have time. > > About it not fully testing things, that is of course not great but testing > things halfway is still better than not testing at all. > > Also it may make sense given the large number of features we have to test > multiple per test. > For example with 200 features, it would take 200 individual tests to test each > in its own test, but only 10 tests if 4 of these would always test the same > feature. this of course assumes all kinds of things, its more meant to show > that a bit of "chaos" in what each test tests could actually improve the > amount of information we could obtain from the pattern of failing tests
To elaborate here a bit in relation to the context of "-re" If you have 1 test that tests "-re" in addition to other things like is the case in master. Its failure will not tell you if its "-re" or something else. But if 2 tests which are otherwise unlrelated test "-re" in addition then both failing will give a strong hint its "-re" without further debuging. So if a test like the -re test is added on top of other tests it should be to more than 1 if it doesnt have its own specialized test. That way its easier to identify the cause. Also obviously for the specific case of "-re" it should be added in such a way that its not slowing any test down and doesnt cause any time dependant test failures So in the end for -re it probably makes sense to have a specialized test instead as its a bit special and different from other features. But for other features adding them to some existing tests seems a intriguing idea IMO thx [...] -- Michael GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB If a bugfix only changes things apparently unrelated to the bug with no further explanation, that is a good sign that the bugfix is wrong.
signature.asc
Description: PGP signature
_______________________________________________ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
