Re: Backpressure rationale

2017-01-12 Thread Dean Wampler
You can find longer explanations for backpressure on teh Interwebs, but briefly, 1. Buffers are never infinite. Even if you're willing to write and maintain a nontrivial scheme for flushing to disk, you might still have a major failure scenario where the consumer is gone forever. Depending on volu

Backpressure rationale

2017-01-12 Thread Gábor Gévay
Hello, I would like to ask about the rationale behind the backpressure mechanism in Flink. As I understand it, backpressure is for handling the problem of one operator (or source) producing records faster then the next operator can consume them. However, an alternative solution would be to have a