Re: Testing a streaming data flow

2016-10-17 Thread Stefan Richter
Hi, I think there are some things that could be helpful for testing your algorithm. From the top of my head, first thing is that you could try to test in a more „unit-testing“ style, i.e. just write small drivers that inject records to your UDFs and check if the output is as expected. Other t

Testing a streaming data flow

2016-10-17 Thread bart.kastermans
What are the standard approaches for testing a streaming algorithm? I have been able to come up with the below where I 1) create a data source that emits events in bunches with set times so that I know the events will be in the same window, 2) end the stream with a mapWithState where the state ch