Flink stateful functions and Event Driven microservices

2020-09-23 Thread Mazen Ezzeddine
Hello, What are the differences between Flink stateful functions and Event driven microservices are they almost the same concept? Indeed I am aware that flink stateful functions provide out of the box functionalities like Exaclty once processing gurantees on Failure and recovery, stateful middle t

BufferTimeout throughput vs latency.

2020-09-16 Thread Mazen Ezzeddine
Hi all, I have read the below in the documentation : "To maximize throughput, set setBufferTimeout(-1) which will remove the timeout and buffers will only be flushed when they are full. To minimize latency, set the timeout to a value close to 0 (for example 5 or 10 ms). A buffer timeout of 0 sh

Connecting two streams and order of their processing

2020-09-16 Thread Mazen Ezzeddine
Hello all, If an event is available now to Flink keyedcoprocess operator, and if another event will be available 1 minute later to that operator (same key), as a result of connecting the two streams, Flink does not provide any guarantee that the event available now will be processed (processElemen

KeyedCoProcessFunction, processElement1, processElement2, onTimer timeout

2020-09-15 Thread Mazen Ezzeddine
Hey all, I am using the KeyedCoProcessFunction class in Flink DataStream APIs to implement a timeout like use case. The scenario is as follows: I have an input kafka topic and an output Kafka topic, a service reads from the input topic processes it (for variable amount of time) and then publishes