Flink serialization errors at a batch job

Hi, We're running a large Flink batch job and sometimes it throws serialization errors in the middle of the job. It is always the same operator but the error can be different. Then the following attempts work. Or sometimes attempts get exhausted, then retrying the job. The job is basically readin

Flink recovers even cancelled jobs after zookeeper failure

Hi, We are using Flink 1.4.0 at zookeeper high availability mode and with externalized checkpoints. Today after we have restarted a zookeeper node, several Flink clusters have lost connection to the zookeeper. This triggered a leader election at effected clusters. After the leader election, the new

Kafka Producer writeToKafkaWithTimestamps; name, uid, parallelism fails

Hi, I am using Flink 1.3.2. When I try to use KafkaProducer with timestamps it fails to set name, uid or parallelism. It uses default values. ——— FlinkKafkaProducer010.FlinkKafkaProducer010Configuration producer = FlinkKafkaProducer010 .writeToKafkaWithTimestamps(stream, topicName, schema

Re: CustomPartitioner that simulates ForwardPartitioner and watermarks

throughput in taskA >> and sending it to a side output with a taksID, then broadcasting the side >> output to a downstream operator >> which is sth like a coprocess function (taskB) and receives the original >> stream and the side output, and >> this is the one that

Re: CustomPartitioner that simulates ForwardPartitioner and watermarks

> This way we may figure out another approach to achieve your goal. > In fact, I am not sure if you earn anything by broadcasting the watermark, > other than > re-implementing (to some extent) Flink’s windowing mechanism. > > Thanks, > Kostas > >> On Sep 27, 2017,

CustomPartitioner that simulates ForwardPartitioner and watermarks

Hi, I have a simple streaming job such as: source.process(taskA) .process(taskB) I want taskB to access minimum watermark of all parallel taskA instances, but the data is ordered and should not be shuffled. ForwardPartitioner uses watermark of only one predecessor. So, I have used a

CoGroupedStreams.WithWindow sideOutputLateData and allowedLateness

Hi, WindowedStream has sideOutputLateData and allowedLateness methods to handle late data. A similar functionality at CoGroupedStreams would have been nice. As it is, it silently ignores late data and it is error-prone. - Is there a reason it does not exist? - Any suggested workaround?

Synchronized Kafka sources

Hi, Is it possible to synchronize two kafka sources? So they can consume from different Kafka topics in close enough event times. My use case is, I have two Kafka topics: A(very large) and B(large). There is a mapping of one to one or zero between A and B. Topology is simply join A and B in a tum