This goes back to the idea that streaming applications should never go
down. I'd much rather consume at max capacity and knowingly drop some
portion of the incoming pipe than have the streaming job crash. Of course,
once the job itself is robust, I still need the runtime to be robust --
YARN vs (po
Fantastic. Sounds like things are moving in the right direction. I'm hoping
this will be tiered storage.
Thanks!
On Fri, 15 Jan 2016, 17:04 Aljoscha Krettek wrote:
> Hi,
> don’t worry, it’s good to get questions about this stuff. :D
>
> You are right, if Flink is not clever about the state your
@Robert: Is it possible to add a "fallback" strategy to the consumer?
Something like "if offsets cannot be found, use latest"?
I would make this an optional feature to activate. I would think it is
quite surprising to users if records start being skipped in certain
situations. But I can see that t
Hi Nick,
I'm sorry you ran into the issue. Is it possible that Flink's Kafka
consumer falls back in the topic so far that the offsets it's requesting
are invalid?
For that, the retention time of Kafka has to be pretty short.
Skipping records under load is something currently not supported by Fli
