Not to derail this thread onto another topic but the problem with using a static instance is that there's no way to shut it down when the job stops. So if, for example, it starts threads, I don't think those threads will stop when the job stops. I'm not very well versed in how various Java 8 implementations perform unloading of classloaders & class definitions/statics therein, but it seems problematic unless the job provides a shutdown hook to which user code can subscribe.
On 9/21/16, 8:05 PM, "David Wang" <dwan...@gmail.com> wrote: >Hi Shannon, > >That's right. This FLIP aims to boost TPS of the task workers with async >i/o operation. > >As what Stephan has mentioned, by placing static attribute to shared >resources(like event pool, connection), it is possible to share those >resources among different slots in the same JVM. > >I will make a note in the FLIP about how to share resources ;D > >Thanks, >David > >2016-09-22 1:46 GMT+08:00 Stephan Ewen <se...@apache.org>: > >> @Shannon: One could have a "static" broker to share the same netty across >> slots in the same JVM. Implicitly, Flink does the same with broadcast >> variables. >> >> On Wed, Sep 21, 2016 at 5:04 PM, Shannon Carey <sca...@expedia.com> wrote: >> >> > David, >> > >> > I just wanted to say "thanks" for making this proposal! I'm also >> > interested in performing nonblocking I/O (multiplexing threads/reactive >> > programming) within Flink operators so that we can, for example, >> > communicate with external web services with Netty/RxNetty without >> blocking >> > an entire Flink slot (aka a thread) while we wait for the operation to >> > complete. It looks like your FLIP will enable that use case. >> > >> > I'm not sure whether it will be possible to share one Netty >> EventLoopGroup >> > (or the equivalent for any other non-blocking framework, connection pool, >> > etc.) among multiple slots in a single JVM though. Flink supports >> > open/close operation on a RichFunction, but that's on a per-slot basis. I >> > don't know of a way to open/close objects on a per-job-JVM basis. But I >> > suppose that's an issue that should be discussed and resolved separately. >> > >> > -Shannon >> > >> > >>
