Re: [Apache Beam] Custom DataSourceV2 instanciation: parameters passing and Encoders

2018-12-18 Thread Etienne Chauchot
Hi everyone, Does anyone have comments on this question? CCing user ML ThanksEtienne Le mardi 11 décembre 2018 à 19:02 +0100, Etienne Chauchot a écrit : > Hi Spark guys, > I'm Etienne Chauchot and I'm a committer on the Apache Beam project. > We have what we call runners. They are pieces of soft

[Apache Beam] Custom DataSourceV2 instanciation: parameters passing and Encoders

2018-12-11 Thread Etienne Chauchot
Hi Spark guys, I'm Etienne Chauchot and I'm a committer on the Apache Beam project. We have what we call runners. They are pieces of software that translate pipelines written using Beam API into pipelines that use native execution engine API. Currently, the Spark runner uses old RDD / DStream