Re: How does SimpleDoFnRunner supports stateful ParDo?

Hi Kenn, Thanks a lot! Shen On Fri, Feb 24, 2017 at 6:37 PM, Kenneth Knowles wrote: > On Thu, Feb 23, 2017 at 9:02 PM, Shen Li wrote: > > > Thanks a lot for explaining. As the SimpleDoFnRunner only takes one > > StepContext object in its arguments, do you mean the runner should > create a > >

Re: How does SimpleDoFnRunner supports stateful ParDo?

On Thu, Feb 23, 2017 at 9:02 PM, Shen Li wrote: > Thanks a lot for explaining. As the SimpleDoFnRunner only takes one > StepContext object in its arguments, do you mean the runner should create a > new SimpleDoFnRunner for each key? > Yes, that is right. You have some flexibility how to manage t

Re: tf.Transform library for using TensorFlow with Beam

That's great! many people have asked me about that and I'm glad to see this happening. Anyone know if there's something at work for the Java SDK (assuming I don't want to wait for Fn API support) ? On Fri, Feb 24, 2017 at 8:44 AM Jean-Baptiste Onofré wrote: > Fantastic ! > > That's a great addit

RE: IT issues faced for HadoopInputFormat IO on Spark and Dataflow runners

Hi JB, We got past the issue finally! The issue was due to InputSplit Serialization/Deserialization. During Deserialization, the ESInputSplit/CqlInputSplit object could not be constructed back, as the classes ESInputSplit and CqlInputSPlit are protected classes, and hence they are not accessib