Hi, Unfortunately, I don't have anything to add. Yes, back pressure doesn't work correctly for functions that do work outside the main thread and iterations currently don't work well and can lead to deadlocks.
Did you already open issues for those by now? Best, Aljoscha > On 10. Nov 2017, at 22:46, Ufuk Celebi <u...@apache.org> wrote: > > Hey Ken, > > thanks for your message. Both your comments are correct (see inline). > > On Fri, Nov 10, 2017 at 10:31 PM, Ken Krugler > <kkrugler_li...@transpac.com> wrote: >> 1. A downstream function in the iteration was (significantly) increasing the >> number of tuples - it would get one in, and sometimes emit 100+. >> >> The output would loop back as input via the iteration. >> >> This eventually caused the network buffers to fill up, and that’s why the >> job got stuck. >> >> I had to add my own tracking/throttling in one of my custom function, to >> avoid having too many “active” tuples. >> >> So maybe something to note in documentation on iterations, if it’s not there >> already. > > Yes, iterations are prone to deadlock due to the way that data is > exchanged between the sink and head nodes. There have been multiple > attempts to fix these shortcomings, but I don't know what the latest > state is. Maybe Aljoscha (CC'd) has some input... > >> 2. The back pressure calculation doesn’t take into account AsyncIO > > Correct, the back pressure monitoring only takes the main task thread > into account. Every operator that uses a separate thread to emit > records (like Async I/O oder Kafka source) is therefore not covered by > the back pressure monitoring. > > – Ufuk
