I posted my related observation here in a separated thread.
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Does-making-synchronize-call-might-choke-the-whole-pipeline-tc28383.html
--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
private static void doWork(long tid) throws InterruptedException
{
if (!sortedTid.contains(tid)) {
sortedTid.add(tid);
}
// simulate a straggler, make the thread with the lowest tid a
slow
processor
if (
Hi,
What kind of function do you use to implement the operator that has the
blocking call?
Did you have a look at the AsyncIO operator? It was designed for exactly
such use cases.
It issues multiple asynchronous requests to an external service and waits
for the response.
Best, Fabian
Am Mo., 24.
Fabian,
Does the above stack trace looks like a deadlock?
at
org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:539)
- locked <0x0007baf84040> (a java.util.ArrayDeque)
at
org.apache.flink.runtime.io.netwo
Fabian,
Thank you for replying.
If I understand your previous comment correctly, I setup up a consumer with
parallelism 1 and connect a worker task with parallelism 2.
If worker thread one is making a block call and stuck for 60s, the consumer
thread should continue fetching from the partition
Hi Ben,
Flink's Kafka consumers track their progress independent of any worker.
They keep track of the reading offset for themselves (committing progress
to Kafka is optional and only necessary to have progress monitoring in
Kafka's metrics).
As soon as a consumer reads and forwards an event, it i
Dear Flink experts,
I am experimenting Flink for a use case where there is a tight latency
requirements.
A stackoverflow article suggests that I can use setParallism(n) to process
a Kafka partition in a multi-threaded way. My understanding is there is
still one kafka consumer per partition, but b
