Re: Message guarantees with S3 Sink

Thanks Gary! Sure, there are issues with updates in S3. You may want to look over EMRFS guarantees of the consistent view [1]. I'm not sure, is it possible in non-EMR AWS system or not. I'm creating a JIRA issue regarding data loss possibility in S3. IMHO, Flink docs should mention about possible

RE: How to Flink can solve this example

Hi I have tried this "homework" about 2 months now.. I have tried for example CEP (with Scala) and iterative condition, but I don't really understand how it would work. Or does it work in my case ? It seems difficult to store values of previous state to use next state(s). And even between diff

Re: sharebuffer prune code

Hi guys, Were you able to RCA this? I think I'm hitting the same issue on 1.4.0, but not really able to reproduce it through a test case. In an IterativeCondition (using AfterMatchSkipStrategy.skipPastLastEvent), while looking up previously matched events, it is hitting a java.util.NoSuchElementEx

Multiple hdfs

Hi, I want to store my data in one hdfs and the flink checkpoints in another hdfs. I didn't find a way to do it, anyone can point me a direction? Thanks in advance, Raul

Strange Behaviour with task manager oom ?

Hi, I am running flink batch job . My job is running fine if i use 4 task manger and 8 slots = 32 parallelism with 6GB memory per task manager. As soon I increase task mangers to 5 with 6 task per task manager = 30 parallelism (6GB memory per task manager) I am getting oom error . I am not able

Checkpointing when reading from files?

I want to add checkpointing to my program that reads from a set of files in a directory. Without checkpointing I use readFile(): DataStream text = env.readFile( new TextInputFormat(new Path(inputPath)), inputPath,

Re: Checkpointing when reading from files?

Hi Alex, StreamingExecutionEnvironment#readFile is a helper function to create file reader data streaming source. It uses ContinuousFileReaderOperator and ContinuousFileMonitoringFunction internally. As both file reader operator and monitoring function uses checkpointing so is readFile [1], you c

Re: Strange Behaviour with task manager oom ?

Hi, Could you share log of job and impacted task manager? How much memory you have allocated to the Job Manager? -- Thanks, Amit On Mon, May 21, 2018 at 8:46 PM, sohimankotia wrote: > Hi, > > I am running flink batch job . > > My job is running fine if i use 4 task manger and 8 slots = 32 parall

Re: Strange Behaviour with task manager oom ?

Hi Amit , Thanks for response . Meanwhile I figured out the issue . I had /Class X extending RichMapFunction/ and this class was preparing some heavy data required for map function . I just moved that code to *open()* function and it worked fine . So I have one doubt , was it because flink was