Re: emit partial state in window (streaming)

2016-11-03 Thread Luis Mariano Guerra
; > Cheers, > Kostas > > > > On Nov 3, 2016, at 3:29 PM, Luis Mariano Guerra < > mari...@event-fabric.com> wrote: > > > > On Thu, Nov 3, 2016 at 2:06 PM, Kostas Kloudas < > k.klou...@data-artisans.com> wrote: > > Hi Luis, > > > > C

Re: emit partial state in window (streaming)

2016-11-03 Thread Luis Mariano Guerra
always have to wait for an element > outside the partial window interval to arrive. > then I think I understood the purpose of registering the event time timer wrong, isn't "ctx.registerEventTimeTimer(window.getEnd())" called to register a timer to call onEventTime?

Re: emit partial state in window (streaming)

2016-11-03 Thread Luis Mariano Guerra
/Introducing-windows.html > [2] https://www.mapr.com/blog/essential-guide-streaming- > first-processing-apache-flink > > 2016-10-27 16:31 GMT+02:00 Luis Mariano Guerra : > >> hi, >> >> I need to calculate some counts for the day but also emit the partial >> co

emit partial state in window (streaming)

2016-10-27 Thread Luis Mariano Guerra
hi, I need to calculate some counts for the day but also emit the partial counts periodically, I think triggers may help me, I'm searching around but there's not much content about it, any tip? for example I'm counting access by location to different services, I want to accumulate access during

Re: Side effects or multiple sinks on streaming jobs?

2016-10-26 Thread Luis Mariano Guerra
ote: > Hi Luis, > > You can define as many data sinks as you want in a Flink job topology. > So its not a problem for your use case to define two Kafka sinks, sending > data to different topics. > > Regards, > Robert > > > On Tue, Oct 25, 2016 at 3:30 PM, Luis Mariano

Side effects or multiple sinks on streaming jobs?

2016-10-25 Thread Luis Mariano Guerra
hi, I'm migrating some samza jobs to flink streaming, and on samza we sent the errors to a kafka topic to make it easier to display on dashboards, I would like to do the same on flink, what do you recommend?

Re: solution for this kryo serialization problem? (also: update to kryo 4.0.0?)

2016-09-27 Thread Luis Mariano Guerra
ssues.apache.org/jira/browse/FLINK-3154). You could open a PR >> if you like. >> >> Timo >> >> Am 27/09/16 um 13:10 schrieb Luis Mariano Guerra: >> >> On Tue, Sep 27, 2016 at 11:22 AM, Luis Mariano Guerra < >> mari...@event-fabric.com> wrote: &

Re: solution for this kryo serialization problem? (also: update to kryo 4.0.0?)

2016-09-27 Thread Luis Mariano Guerra
On Tue, Sep 27, 2016 at 11:22 AM, Luis Mariano Guerra < mari...@event-fabric.com> wrote: > hi, I created an issue at kryo with a simple project that reproduces the > problem, but still asking here in case anyone knows a solution: > > https://github.com/EsotericSoftware/kryo/iss

solution for this kryo serialization problem? (also: update to kryo 4.0.0?)

2016-09-27 Thread Luis Mariano Guerra
hi, I created an issue at kryo with a simple project that reproduces the problem, but still asking here in case anyone knows a solution: https://github.com/EsotericSoftware/kryo/issues/462 basically, I get an error trying to serialize a simple tree like structure, the error happens with a 2 node,

Re: flink run throws NPE, JobSubmissionResult is null when interactive and not isDetached()

2016-09-26 Thread Luis Mariano Guerra
; [1] https://issues.apache.org/jira/browse/FLINK-4677 > > On Tue, Sep 20, 2016 at 2:00 PM, Luis Mariano Guerra > wrote: > > On Tue, Sep 20, 2016 at 12:49 PM, Maximilian Michels > wrote: > >> > >> Hi Luis, > >> > >> That looks like a

Re: flink run throws NPE, JobSubmissionResult is null when interactive and not isDetached()

2016-09-20 Thread Luis Mariano Guerra
hing an exception during setup and logging >> the error, but for some reason logging doesn't log at that point, is there >> a way to avoid the "log and print" problem during setup? or should I just >> print? >> >> >>> >>> Thanks,

Re: flink run throws NPE, JobSubmissionResult is null when interactive and not isDetached()

2016-09-20 Thread Luis Mariano Guerra
> > Thanks, Fabian > > 2016-09-19 17:30 GMT+02:00 Luis Mariano Guerra : > >> context: I have two other similar jobs in the same project that run >> without problem. >> >> On Mon, Sep 19, 2016 at 4:28 PM, Luis Mariano Guerra < >> mari...@event-fabric.

Re: flink run throws NPE, JobSubmissionResult is null when interactive and not isDetached()

2016-09-19 Thread Luis Mariano Guerra
context: I have two other similar jobs in the same project that run without problem. On Mon, Sep 19, 2016 at 4:28 PM, Luis Mariano Guerra < mari...@event-fabric.com> wrote: > hi > > submitting a job I get a NPE here: > https://github.com/apache/flink/blob/master/flink- >

flink run throws NPE, JobSubmissionResult is null when interactive and not isDetached()

2016-09-19 Thread Luis Mariano Guerra
hi submitting a job I get a NPE here: https://github.com/apache/flink/blob/master/flink-clients/src/main/java/org/apache/flink/client/CliFrontend.java#L781 building from source and adding some prints I got that this.lastJobExecutionResult here seems to be null: https://github.com/apache/flink/blo

Re: RichMapFunction in DataStream, how do I set the parameters received in open?

2016-09-12 Thread Luis Mariano Guerra
tDeclaredField(fieldName); } ... } > > Regards, > Chesnay > > > On 12.09.2016 18:27, Luis Mariano Guerra wrote: > > hi! I'm trying to pass Configuration parameters to a RichMapFunction in > flink streaming and I can't find the way to do it > >

how to unit test streaming window jobs?

2016-09-12 Thread Luis Mariano Guerra
hi, is there a way to test window jobs? I would like to build the job, give some inputs, "fast forward" to the next window, collect the results and assert them.

RichMapFunction in DataStream, how do I set the parameters received in open?

2016-09-12 Thread Luis Mariano Guerra
hi! I'm trying to pass Configuration parameters to a RichMapFunction in flink streaming and I can't find the way to do it I need to pass two strings to the MapFunction and I was getting a serialization error, so I tried RichMapFunction and open() but I can't find a way to set the the parameters I

Re: emit a single Map per window

2016-09-05 Thread Luis Mariano Guerra
I have to add a line to specify partition?) > On Fri, 2 Sep 2016 at 18:26 Luis Mariano Guerra > wrote: > >> On Fri, Sep 2, 2016 at 5:24 PM, Aljoscha Krettek >> wrote: >> >>> Hi, >>> from this I would expect to get as many HashMaps as you have keys. Th

Re: emit a single Map per window

2016-09-02 Thread Luis Mariano Guerra
eeing? > yes, it's the behaviour I'm seeing, I'm looking for a way to merge those HashMaps from the same window into a single one, I can't find how. > > Cheers, > Aljoscha > > On Fri, 2 Sep 2016 at 17:37 Luis Mariano Guerra > wrote: > >> hi! &

emit a single Map per window

2016-09-02 Thread Luis Mariano Guerra
hi! I'm trying to collect some metrics by key per window and emiting the full result at the end of the window to kafka, I started with a simple count by key to test it but my requirements are a little more complex than that. what I want to do is to fold the stream events as they come and then at