Re: How does Flink handle backpressure in EMR

[Nguyen, Michael]

Thank you for the response Roman and Piotrek!

@Roman - can you clarify on what you mean when you mentioned Flink propagating 
it back to the sources? 

Also, if one of my Flink operators is processing records too slowly and is 
getting further away from the latest record of my source data stream, is there 
a way to detect this slow processing in Flink? Would this be detected by 
Flink's backpressure mechanism?

Thanks,
Michael

On 12/5/19, 7:57 AM, "Piotr Nowojski" <pi...@data-artisans.com on behalf of 
pi...@ververica.com> wrote:

    [External]
    
    
    Hi Michael,
    
    As Roman pointed out Flink currently doesn’t support the auto-scaling. It’s 
on our roadmap but it requires quite a bit of preliminary work to happen before.
    
    Piotrek
    
    > On 5 Dec 2019, at 15:32, r_khachatryan <khachatryan.ro...@gmail.com> 
wrote:
    >
    > Hi Michael
    >
    > Flink *does* detect backpressure but currently, it only propagates it back
    > to sources.
    > And so it doesn't support auto-scaling.
    >
    > Regards,
    > Roman
    >
    >
    > Nguyen, Michael wrote
    >> How does Flink handle backpressure (caused by an increase in traffic) in 
a
    >> Flink job when it’s being hosted in an EMR cluster? Does Flink detect the
    >> backpressure and auto-scales the EMR cluster to handle the workload to
    >> relieve the backpressure? Once the backpressure is gone, then the EMR
    >> cluster would scale back down?
    >
    >
    >
    >
    >
    > --
    > Sent from: 
https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fapache-flink-user-mailing-list-archive.2336050.n4.nabble.com%2F&amp;data=02%7C01%7CMichael.Nguyen79%40t-mobile.com%7Cfda964dacbf04cc2d2cb08d7799bc347%7Cbe0f980bdd994b19bd7bbc71a09b026c%7C0%7C0%7C637111582216065152&amp;sdata=FIHbAWbkH7Tq8AjG%2BVSZoxNrOl0Bn6SukN%2BY1PyhSHg%3D&amp;reserved=0