Hi I am a new flink developer and I also feel we need to improve the testing side of flink.
>From Tony list: 1-stream logic control: we may consider reactiveX as a reference for stream testing: http://reactivex.io/. They have a test scheduler and use marble diagram to write unit test: for example '--a--b--|' represent event a arriving at slot time 3 and event b arriving at slot time 6 and an end of stream at slot time 9. The same diagram can be use for the expected events at the output of our stream. It is very efficient. 3-state testing: I also feel there is a lack of checkpoint testing. On my side, checkpoint is mostly observe at runtime. I would also suggest on a design and testing point of view to use less reflection and have more interfaces. It is easier to test, to mock... Quick description of the use case I had: We run flink to stream web event, more that 1 billion a day. Those events are transformed by flink and push to kafka. We are using FlinkKafkaProducer010 to push in kafka. >From time to time, the kafka leader node goes down and so it generate an exception in flink which stop the stream. We don't want that behavior so I implemented a decorator to catch the exception and recreate the producer. On a design and testing point of view, addSink needs a “SinkFunction[T]”. It means I should decorate the SinkFunction interface. I wrote the code and I wrote the test. Code and unit tests run ok until I went to runtime and discovered nothing was push to kafka. The interface does not describe the behavior and so the test mocking a SinkFunction generating an exception. The new code for the decorator and the test are much more complicated as it is not using interface but several class extension. Regards Matthieu Bonneviot Le lun. 8 oct. 2018 à 03:47, Tony Wei <tony19920...@gmail.com> a écrit : > > Hi all, > > Does anyone want to share something about how to provide more better > testing tools or good documentations? > > @Till, @Aljoscha > Do you have further suggestions about the improvements of testing tools and > what is the next step we can do? > > Best, > Tony Wei > > > Tony Wei <tony19920...@gmail.com> 於 2018年9月29日 週六 上午12:20寫道： > > > Hi Till, > > > > Thanks for your feedback. I didn't think about this before. It will be > > better if we can provide such tools instead of > > let user deal with testing in operator level directly. Because test > > harness was introduced for contributors, who > > may have more knowledge about internal design, to test their patches > > easily, it sometimes may not be intuitive > > to users to use test harness. And that's why I want to provide some > > examples in docs to ease the pain from such > > problem in the beginning. > > > > If Aljoscha has already made some efforts on it and let such tool be > > available in the near future, I will look forward > > to seeing this happen. And if there is anything I can help, I'm glad to > > give a hand. > > > > To follow up the discussion about users' requirements, I can provide some > > from my experience. What I want to > > achieve is to do unit testing on my operators with Flink features, such as > > states, event time and checkpoint. > > > > From the testing doc [1] I provided, I concluded some scenarios I would > > test in my flink application: > > > > 1. Easily control stream records to see the changes about the states > > and outputs, no matter it is one input > > stream operator or two. > > 2. Manually control watermark progress to test "out of order" > > problem, "late data" problem or the behavior of > > "event time service". > > 3. Preserve states from every version for testing states evolution and > > compatibility or just to test the change > > of Flink version. > > 4. Test customized stateful source function. In current version, > > source function still need to be implemented > > by a long running method. How to easily block it and verify the states > > is helpful. > > 5. A simple way to expose states to verify the exactly value stored in > > states, instead of testing it indirectly by > > outputs. I archived this by getting state backend from test harness > > and refer to some examples from Flink > > project to access the keyed states I needed, but it is too deeper into > > the internal implementations to make > > it as an example in my testing doc [1]. > > > > Best, > > Tony Wei > > > > [1] > > https://github.com/apache/flink/compare/master...tony810430:flink-testing-doc > > > > 2018-09-28 21:48 GMT+08:00 Till Rohrmann <trohrm...@apache.org>: > > > >> Hi Tony, > >> > >> I think this is a long sought-after feature to provide better testing > >> tools > >> for our users. Thus, I'm strongly in favour of adding something like this. > >> If I remember correctly Aljoscha already spend some brain cycles on this > >> and he also gave a training about the current state at FF SF 2018. I've > >> pulled him in to give more details. > >> > >> The first two things we can do is to collect requirements from our users > >> and to see what the current state is. Based on that we could plan which > >> things to add in which order. > >> > >> Cheers, > >> Till > >> > >> On Fri, Sep 28, 2018 at 3:13 PM Tony Wei <tony19920...@gmail.com> wrote: > >> > >> > Hi all, > >> > > >> > @Ken, Thanks for your positive feedback. I have a similar experience > >> with > >> > test harness and that's why > >> > I want to provide more contents on testing doc to prevent from this > >> kind of > >> > problems. > >> > > >> > Does anyone have any feedback and advice? I would like to collect more > >> > opinions from every developers > >> > and users. Please let me know what you think about this topic. All > >> > suggestions are welcome. Thank you. > >> > > >> > Best, > >> > Tony Wei > >> > > >> > 2018-09-26 2:20 GMT+08:00 Ken Krugler <kkrugler_li...@transpac.com>: > >> > > >> > > Hi Tony, > >> > > > >> > > I think this would be great - we’ve been building out tests using > >> > > AbstractStreamOperator, and the lack of documentation has made it > >> > > challenging. > >> > > > >> > > For example, there was this exchange I had with Piotr about a month > >> ago: > >> > > > >> > > > You made a small mistake when restoring from state using test > >> harness, > >> > > that I myself have also done in the past. Problem is with an ordering > >> of > >> > > those calls: > >> > > > > >> > > > result.open(); > >> > > > if (savedState != null) { > >> > > > result.initializeState(savedState); > >> > > > } > >> > > > > >> > > > Open is supposed to be called after initializeState, and if you look > >> > > into the code of AbstractStreamOperatorTestHarness#open, if it is > >> called > >> > > before initialize, it will initialize harness without any state. > >> > > > > >> > > > Unfortunate is that this is implicit behaviour that doesn’t throw > >> any > >> > > error (test harness is not part of a Flink’s public api). I will try > >> to > >> > fix > >> > > this: https://issues.apache.org/jira/browse/FLINK-10159 < > >> > > https://issues.apache.org/jira/browse/FLINK-10159> > >> > > — Ken > >> > > > >> > > > On Sep 25, 2018, at 3:30 AM, Tony Wei <tony19920...@gmail.com> > >> wrote: > >> > > > > >> > > > Hi all, > >> > > > > >> > > > It seems that there are more and more users from user mailing list > >> ask > >> > > how > >> > > > to do unit test with Flink > >> > > > features like states or timer. And the community usually tends to > >> > suggest > >> > > > them using > >> > > > `AbstractStreamOperator` and provide an example from Flink github > >> repo. > >> > > > Here I sort out some > >> > > > examples and write them down in the testing documentation [1]. And I > >> > > would > >> > > > link to contribute back > >> > > > to the Flink. > >> > > > > >> > > > The reason why I ask it first in dev mailing list is that > >> > > > `AbstractStreamOperator` is an internal API and > >> > > > could be changed at any time. I'm not sure if it is worth to provide > >> > > these > >> > > > examples on testing > >> > > > document, so I want to collect some feedbacks before I go to open a > >> > JIRA > >> > > > ticket. > >> > > > > >> > > > If this is feasible and valuable, then I will open the corresponding > >> > JIRA > >> > > > ticket and we can discuss > >> > > > more details of what examples are good to have in the document or > >> how > >> > to > >> > > > structure the content. > >> > > > > >> > > > I would really appreciate any feedback from you. Thanks in advance. > >> > > > > >> > > > Best Regards, > >> > > > Tony Wei > >> > > > > >> > > > [1] > >> > > > https://github.com/apache/flink/compare/master... > >> > > tony810430:flink-testing-doc > >> > > > >> > > -------------------------- > >> > > Ken Krugler > >> > > +1 530-210-6378 > >> > > http://www.scaleunlimited.com > >> > > Custom big data solutions & training > >> > > Flink, Solr, Hadoop, Cascading & Cassandra > >> > > > >> > > > >> > > >> > > > > -- Matthieu Bonneviot Senior Engineer, DataDome M +33 7 68 29 79 34 E matthieu.bonnev...@datadome.co W www.datadome.co DataDome ranked 'Strong Performer' in latest Forrester Bot management report
