Re: [PROPOSAL] Custom JVM initialization for Beam workers

2019-04-15 Thread Ahmet Altay
On Mon, Apr 15, 2019 at 9:35 AM Udi Meiri wrote: > Is this like the way Python SDK allows for a custom setup.py? > example: > https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/complete/juliaset/setup.py > custom setup.py is slightly different. It will execute a custom p

Re: [PROPOSAL] Custom JVM initialization for Beam workers

2019-04-15 Thread Udi Meiri
Is this like the way Python SDK allows for a custom setup.py? example: https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/complete/juliaset/setup.py On Fri, Apr 12, 2019 at 10:51 AM Lukasz Cwik wrote: > +1 on the use cases that Ahmet pointed out and the solution that Bri

Re: [PROPOSAL] Custom JVM initialization for Beam workers

2019-04-12 Thread Lukasz Cwik
+1 on the use cases that Ahmet pointed out and the solution that Brian put forth. I like how the change is being applied to the Beam Java SDK harness and not just Dataflow so all portable runner users get this as well. On Wed, Apr 10, 2019 at 9:03 PM Kenneth Knowles wrote: > > > On Wed, Apr 10,

Re: [PROPOSAL] Custom JVM initialization for Beam workers

2019-04-10 Thread Kenneth Knowles
On Wed, Apr 10, 2019 at 8:18 PM Ahmet Altay wrote: > > > On Wed, Apr 10, 2019 at 7:59 PM Kenneth Knowles wrote: > >> TL;DR I like the simple approach better than the ServiceLoader solution >> when a particular DoFn depends on the result. The ServiceLoader solution >> fits when it is somewhat ind

Re: [PROPOSAL] Custom JVM initialization for Beam workers

2019-04-10 Thread Ahmet Altay
On Wed, Apr 10, 2019 at 7:59 PM Kenneth Knowles wrote: > TL;DR I like the simple approach better than the ServiceLoader solution > when a particular DoFn depends on the result. The ServiceLoader solution > fits when it is somewhat independent of a particular DoFn (I'm not sure the > use case(s)).

Re: [PROPOSAL] Custom JVM initialization for Beam workers

2019-04-10 Thread Kenneth Knowles
TL;DR I like the simple approach better than the ServiceLoader solution when a particular DoFn depends on the result. The ServiceLoader solution fits when it is somewhat independent of a particular DoFn (I'm not sure the use case(s)). On Wed, Apr 10, 2019 at 4:10 PM Brian Hulette wrote: > - Each

[PROPOSAL] Custom JVM initialization for Beam workers

2019-04-10 Thread Brian Hulette
Hi everyone, In Dataflow we've had some Java SDK user issues that could be easily solved if users were able to define some one-time initialization code (e.g. set a system property) that their workers execute before they begin to process data. One solution we've found that works with currently relea