Re: Flushing the result of a groupReduce to a Sink before all reduces complete

2016-10-28 Thread Paul Wilson
e a group > until all data is consumed. > > Please let me know if you have further questions. > > Best, Fabian > > > 2016-10-26 19:07 GMT+02:00 Paul Wilson : > >> Hi, >> >> DataSet API >> Flink 1.1.3 >> >> I have an application where I'd

Flushing the result of a groupReduce to a Sink before all reduces complete

2016-10-26 Thread Paul Wilson
Hi, DataSet API Flink 1.1.3 I have an application where I'd like to perform some mapping before batching the results and passing them to the sink. I'm performing a 'composite' key selection to group the items by their natural key as well as a batch (itemCount / batchSize). When I reduce the batch

Re: AW: REST Interface to JobManager

2016-09-23 Thread Paul Wilson
Hi, We went a different way and used the same underlying ClusterClient classes from the command line tool, to connect to the JobManager and submit JARs that way. >From the YARN RM address, we search for the Flink application and use the YarnClusterClient to get the connection details from the disc

Remote upload and execute

2016-09-02 Thread Paul Wilson
Hi, I'd like to write a client that can execute an already 'uploaded' JAR (i.e. the JAR is deployed and available by some other external process). This is similar to what the web console allows which consists of 2 steps: upload the JAR followed by a submit with parameters. I'm looking at the Flin

Re: Custom Barrier?

2016-06-14 Thread Paul Wilson
... and those events are in order On 14 Jun 2016 14:04, "Paul Wilson" wrote: > Hi, > > No these super-structure events only serve the purpose of defining the > boundaries of a join, and do not relate to the keys of the sub-events. > > Thanks, > Paul > > On 14

Re: Custom Barrier?

2016-06-14 Thread Paul Wilson
after A-End then I > don't see an easy way to do it. > > Let me know if you need to know more about assigners/triggers. > > Cheers, > Aljoscha > > On Mon, 13 Jun 2016 at 16:29 Paul Wilson wrote: > >> Hi, >> >> I've been evaluating Flin

Custom Barrier?

2016-06-13 Thread Paul Wilson
Hi, I've been evaluating Flink and wondering if it was possible to define a window that is based on characteristics of the data (data driven) but not contained in the data stream directly. Consider 'nested events' where lower level events belong to a wider event where the wider event serves only