Re: Making batches of small messages

2017-01-12 Thread Kostas Kloudas
18:32 > To: user@flink.apache.org > Subject: Re: Making batches of small messages > > Hi, > > I think this is a case for the ProcessFunction that was recently added and > will be included in Flink 1.2. > ProcessFunction allows to register timers (so the 5 secs timeout c

RE: Making batches of small messages

2017-01-12 Thread Gwenhael Pasquiers
Thanks, We are waiting for the 1.2 release eagerly ☺ From: Fabian Hueske [mailto:fhue...@gmail.com] Sent: mercredi 11 janvier 2017 18:32 To: user@flink.apache.org Subject: Re: Making batches of small messages Hi, I think this is a case for the ProcessFunction that was recently added and will

Re: Making batches of small messages

2017-01-11 Thread Fabian Hueske
Hi, I think this is a case for the ProcessFunction that was recently added and will be included in Flink 1.2. ProcessFunction allows to register timers (so the 5 secs timeout can be addressed). You can maintain the fault tolerance guarantees if you collect the records in managed state. That way th

Making batches of small messages

2017-01-11 Thread Gwenhael Pasquiers
Hi, Sorry if this was already asked. For performances reasons (streaming as well as batch) I'd like to "group" messages (let's say by batches of 1000) before sending them to my sink (kafka, but mainly ES) so that I have a smaller overhead. I've seen the "countWindow" operation but if I'm not w