Thanks for yours and Chesnay's efforts in this topic. I also think it would be a good idea to just transcribe your email to some documentation/wiki page, so that it can be easily found in the future.
Piotrek czw., 11 lut 2021 o 10:36 Till Rohrmann <trohrm...@apache.org> napisał(a): > Thanks for letting the community know Arvid. Can we document this specific > behaviour somewhere easy to find for others? Maybe we could add it to the > Flink wiki or the how to contribute guide. > > Cheers, > Till > > On Wed, Feb 10, 2021 at 2:44 PM Arvid Heise <ar...@apache.org> wrote: > > > Dear devs, > > > > I just merged FLINK-19520, which adds randomization to all tests that use > > the MiniCluster(WithResource). The overall goal is to provide a faster > and > > better coverage for future changes in the network stack in particular in > > conjunction with checkpointing. > > > > We implemented it in a way to fulfill these requirements: > > > > - Randomization should be seeded in a way that both builds on Azure > > pipelines and local builds will result in the same settings to ease > > debugging and ensure reproducibility. > > - Randomized options should be shown in the test log. > > - Execution order of test cases will not influence the randomization. > > - Randomization is hidden, no change on any test is needed. > > - Randomization only happens during local/remote test execution. User > > deployments are not affected. > > - Test developers are able to avoid randomization by explicitly > > providing values for the randomized configs. > > > > To achieve these goals, we use the git commit id of HEAD + test name to > > calculate the seed of the randomization. So if you experience a test > > failure on AZP and checkout the same branch/commit, you will get the > exact > > same configuration. > > > > If for some reason, git commit id changed (e.g., you added some bugfix > > commit on top and want to test it) or git commit id cannot be extracted, > > you can overwrite the seed with the maven property > > <test.randomization.seed> > > (test name will still be used additionally to that value). > > > > Lastly, you can also add your own randomizations. [1] But please use it > > conservatively until we have gained enough experience with the setup. > > > > Best, > > > > Arvid > > > > [1] > > > > > https://github.com/AHeise/flink/blob/3d69b21f05abca0de64a2e778c292a7d0694b263/flink-test-utils-parent/flink-test-utils/src/main/java/org/apache/flink/streaming/util/TestStreamEnvironment.java#L97 > > >
