Re: Strange behaviour of the flatMap Collector

2016-08-11 Thread Yassin Marzouki
e correct. > > Stephan > > > On Thu, Aug 11, 2016 at 2:29 PM, Yassin Marzouki > wrote: > >> Hi all, >> >> When I use out.collect() twice inside a faltMap, the output is sometimes >> and randomly skewed. Take this example: >> >

Strange behaviour of the flatMap Collector

2016-08-11 Thread Yassin Marzouki
Hi all, When I use out.collect() twice inside a faltMap, the output is sometimes and randomly skewed. Take this example: final StreamExecutionEnvironment env = StreamExecutionEnvironment.createLocalEnvironment(); env.generateSequence(1, 10) .flatMap((Long t, Collector out) -> {

Re: No output when using event time with multiple Kafka partitions

2016-07-27 Thread Yassin Marzouki
ave a 4 CPU cores, but what is the explanation of that? So I suppose that if I don't specify the source parallelism, it is set automatically to 4. Isn't it supposed to be set to the number of topic patitions (= 2) by default? On Wed, Jul 27, 2016 at 2:33 PM, Yassin Marzouki wrot

Re: No output when using event time with multiple Kafka partitions

2016-07-27 Thread Yassin Marzouki
ndingTimestampExtractor() { > @Override > public long extractAscendingTimestamp(Request req) { > return req.ts; > } > }) > > > This at least will tell us if reading from Kafka works as expected. > > Kostas > > On Jul 25, 2016, at 3:39 PM, Yassin M

Re: If I chain two windows, what event-time would the second window have?

2016-07-27 Thread Yassin Marzouki
uggest checking out the tests in the > WindowOperatorTest class. > > There you can do experiments and figure out how Flinkā€™s windowOperator > works internally and what is the > interplay between windowAssingers, triggers, and the windowOperator. > > Hope this helps, > Kostas > &g

If I chain two windows, what event-time would the second window have?

2016-07-26 Thread Yassin Marzouki
Hi all, Say I assign timestamps to a stream and then apply a transformation like this: stream.keyBy(0).timeWindow(Time.hours(5)).reduce(count).timeWindowAll(Time.days(1)).apply(transformation) Now, when the first window is applied, events are aggregated based on their timestamps, but I don't und

No output when using event time with multiple Kafka partitions

2016-07-25 Thread Yassin Marzouki
Hi everyone, I am reading messages from a Kafka topic with 2 partitions and using event time. This is my code: .assignTimestampsAndWatermarks(new AscendingTimestampExtractor() { @Override public long extractAscendingTimestamp(Request req) { return req.ts; } }) .windowAll(Tumbl

Re: Variable not initialized in the open() method of RichMapFunction

2016-07-22 Thread Yassin Marzouki
d with the wrong signature. The right >>> signature would be "open(Configuration cfg) {...}". You probably overlooked >>> this because you missed the "@Override" annotation. >>> >>> On Fri, Jul 22, 2016 at 4:49 PM, Yassin Marzouki >>>

Variable not initialized in the open() method of RichMapFunction

2016-07-22 Thread Yassin Marzouki
Hi everyone, I want to convert a stream of json strings to POJOs using Jackson, so I did the following: .map(new RichMapFunction() { private ObjectMapper objectMapper; public void open() { objectMapper = new ObjectMapper(); } @Override publi