Re: how does flink assign windows to task

2016-07-30 Thread Sameer Wadkar
Vishnu, I would imagine based on Max's explanation and how other systems like MapReduce and Spark partition keys, if you have 100 keys and 50 slots, 2 keys would be assigned to each slot. Each slot would maintain one or more windows (more for time based windows) and each window would have upto

Re: Flink - Once and once processing

2016-07-30 Thread milind parikh
Flink operates in conjunction with sources and sinks. So ,yes, there are things that an underlying sink (or a source) must support in conjunction with Flink to enable a particular semantic. On Jul 30, 2016 11:46 AM, "M Singh" wrote: > Thanks Konstantin. > > Just to clarify - unless the target

Re: Flink - Once and once processing

2016-07-30 Thread M Singh
Thanks Konstantin. Just to clarify - unless the target database is resilient to duplicates, Flink's once-only configuration will not avoid duplicate updates. Mans On Saturday, July 30, 2016 7:40 AM, Konstantin Knauf wrote: Hi Mans, depending on the number of operations and the particu

AW: Serialization of "not a valid POJO type"

2016-07-30 Thread Paschek, Robert
Hi again, I implemented the scenario with the combiner and answered my 3rd question by myself: The combiners start after the mapper finished, so the reducer will not start processing partial results until the mappers are completely done. Regards Robert Von: Paschek, Robert [mailto:robert.pasc

Re: Flink - Once and once processing

2016-07-30 Thread Konstantin Knauf
Hi Mans, depending on the number of operations and the particular database, you might be able to use transactions. Maybe you can also find a data model, which is more resilient to these kind of failures. Cheers, Konstantin On 29.07.2016 19:26, M Singh wrote: > Hi: > > I have a use case where

Serialization of "not a valid POJO type"

2016-07-30 Thread Paschek, Robert
Hi Mailing List, according to my questions (and your answers!) at this topic http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Performance-issues-with-GroupBy-td8130.html I have eliminated my ArrayList in my collect methods. Additional I want to emit partial results. My mapper