Re: Published test artifacts for flink streaming

2015-12-15 Thread lofifnc
scalatest support. As soon as I've finished the first two points and made the core compatible with scala I will release maven artifacts. Best Alex! -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Published-test-artifacts-for-flink-stream

Re: Published test artifacts for flink streaming

2015-12-14 Thread Nick Dimiduk
; > > > > > -- > View this message in context: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Published-test-artifacts-for-flink-streaming-tp3379p3638.html > Sent from the Apache Flink User Mailing List archive. mailing list archive > at Nabble.com. >

Re: Published test artifacts for flink streaming

2015-11-23 Thread lofifnc
production. Best, Alex. -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Published-test-artifacts-for-flink-streaming-tp3379p3638.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Re: Published test artifacts for flink streaming

2015-11-20 Thread Nick Dimiduk
nning to have a > first release candidate at the end of the month. > > Best, Alex > > > > > -- > View this message in context: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Published-test-artifacts-for-flink-streaming-tp3379p3584.html > Sent from the Apache Flink User Mailing List archive. mailing list archive > at Nabble.com. >

Re: Published test artifacts for flink streaming

2015-11-19 Thread lofifnc
d28eb Unfortunately It's still a work in progress. But I'm planning to have a first release candidate at the end of the month. Best, Alex -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Published-test-artifacts-for-flink-streaming-tp3379p3584

Re: Published test artifacts for flink streaming

2015-11-18 Thread Nick Dimiduk
Please see the above gist: my test makes no assertions until after the env.execute() call. Adding setParallelism(1) to my sink appears to stabilize my test. Indeed, very helpful. Thanks a lot! -n On Wed, Nov 18, 2015 at 9:15 AM, Stephan Ewen wrote: > Okay, I think I misunderstood your problem.

Re: Published test artifacts for flink streaming

2015-11-18 Thread Stephan Ewen
Okay, I think I misunderstood your problem. Usually you can simply execute tests one after another by waiting until "env.execute()" returns. The streaming jobs terminate by themselves once the sources reach end of stream (finite streams are supported that way) but make sure all records flow throug

Re: Published test artifacts for flink streaming

2015-11-18 Thread Nick Dimiduk
Sorry Stephan but I don't follow how global order applies in my case. I'm merely checking the size of the sink results. I assume all tuples from a given test invitation have sunk before the next test begins, which is clearly not the case. Is there a way I can place a barrier in my tests to ensure o

Re: Published test artifacts for flink streaming

2015-11-18 Thread Stephan Ewen
There is no global order in parallel streams, it is something that applications need to work with. We are thinking about adding operations to introduce event-time order (at the cost of some delay), but that is only plans at this point. What I do in my tests is run the test streams in parallel, bu

Re: Published test artifacts for flink streaming

2015-11-12 Thread Nick Dimiduk
On Thu, Nov 5, 2015 at 6:58 PM, Nick Dimiduk wrote: > Thanks Stephan, I'll check that out in the morning. Generally speaking, it > would be great to have some single-jvm example tests for those of us > getting started. Following the example of WindowingIntegrationTest is > mostly working, though

Re: Published test artifacts for flink streaming

2015-11-06 Thread Till Rohrmann
No that is not possible since you cannot access DataSets from inside UDFs. And select and where operations are translated into a filter operation on a DataSet. ​ On Fri, Nov 6, 2015 at 6:03 PM, Nick Dimiduk wrote: > Promising observation, Till. Is it possible to access Table API's > select and w

Re: Published test artifacts for flink streaming

2015-11-06 Thread Nick Dimiduk
Promising observation, Till. Is it possible to access Table API's select and where operators from within such a flatMap? -n On Fri, Nov 6, 2015 at 6:19 AM, Till Rohrmann wrote: > Hi Nick, > > I think a flatMap operation which is instantiated with your list of > predicates should do the job. Thus

Re: Published test artifacts for flink streaming

2015-11-06 Thread Till Rohrmann
Hi Nick, I think a flatMap operation which is instantiated with your list of predicates should do the job. Thus, there shouldn’t be a need to dig deeper than the DataStream for the first version. Cheers, Till ​ On Fri, Nov 6, 2015 at 3:58 AM, Nick Dimiduk wrote: > Thanks Stephan, I'll check th

Re: Published test artifacts for flink streaming

2015-11-05 Thread Nick Dimiduk
Thanks Stephan, I'll check that out in the morning. Generally speaking, it would be great to have some single-jvm example tests for those of us getting started. Following the example of WindowingIntegrationTest is mostly working, though reusing my single sink instance with its static collection res

Re: Published test artifacts for flink streaming

2015-11-05 Thread Stephan Ewen
Hey! There is also a collect() sink in the "flink-streaming-contrib" project, see here: https://github.com/apache/flink/blob/master/flink-contrib/flink-streaming-contrib/src/main/java/org/apache/flink/contrib/streaming/DataStreamUtils.java It should work well locally for testing. In that case you

Re: Published test artifacts for flink streaming

2015-11-05 Thread Nick Dimiduk
Hi Robert, It seems "type" was what I needed. This it also looks like the test jar has an undeclared dependency. In the end, the following allowed me to use TestStreamEnvironment for my integration test. Thanks a lot! -n org.apache.flink flink-streaming-core ${flink.versio

Re: Published test artifacts for flink streaming

2015-11-05 Thread Robert Metzger
Hi Nick, we are usually publishing the test artifacts. Can you try and replace the tag by test-jar: org.apache.flink flink-streaming-core ${flink.version} test-jar test On Thu, Nov 5, 2015 at 8:18 PM, Nick Dimiduk wrote: > Hello, > > I'm attempting integration tests for my

Published test artifacts for flink streaming

2015-11-05 Thread Nick Dimiduk
Hello, I'm attempting integration tests for my streaming flows. I'd like to produce an input stream of java objects and sink the results into a collection for verification via JUnit asserts. StreamExecutionEnvironment provides methods for the former, however, how to achieve the latter is not evide