@Shannon: One could have a "static" broker to share the same netty across slots in the same JVM. Implicitly, Flink does the same with broadcast variables.
On Wed, Sep 21, 2016 at 5:04 PM, Shannon Carey <sca...@expedia.com> wrote: > David, > > I just wanted to say "thanks" for making this proposal! I'm also > interested in performing nonblocking I/O (multiplexing threads/reactive > programming) within Flink operators so that we can, for example, > communicate with external web services with Netty/RxNetty without blocking > an entire Flink slot (aka a thread) while we wait for the operation to > complete. It looks like your FLIP will enable that use case. > > I'm not sure whether it will be possible to share one Netty EventLoopGroup > (or the equivalent for any other non-blocking framework, connection pool, > etc.) among multiple slots in a single JVM though. Flink supports > open/close operation on a RichFunction, but that's on a per-slot basis. I > don't know of a way to open/close objects on a per-job-JVM basis. But I > suppose that's an issue that should be discussed and resolved separately. > > -Shannon > >
