RE: IT issues faced for HadoopInputFormat IO on Spark and Dataflow runners

2017-02-24 Thread Dipti Kulkarni
, February 20, 2017 1:54 AM To: dev@beam.apache.org Subject: Re: IT issues faced for HadoopInputFormat IO on Spark and Dataflow runners OK, I just checked and it's part of the PR. I'm checking out locally to check. I will let you know. Regards JB On 02/19/2017 08:33 PM, Jean-Baptist

Re: IT issues faced for HadoopInputFormat IO on Spark and Dataflow runners

2017-02-19 Thread Jean-Baptiste Onofré
OK, I just checked and it's part of the PR. I'm checking out locally to check. I will let you know. Regards JB On 02/19/2017 08:33 PM, Jean-Baptiste Onofré wrote: Hi Dipti, regarding Elasticsearch, it seems that the classloader doesn't contain the EsInputSplit. Using spark-submit, you have to

Re: IT issues faced for HadoopInputFormat IO on Spark and Dataflow runners

2017-02-19 Thread Jean-Baptiste Onofré
Hi Dipti, regarding Elasticsearch, it seems that the classloader doesn't contain the EsInputSplit. Using spark-submit, you have to provide the corresponding jar as package. For Cassandra, it seems more something about coder/serialization. Are the tests part of the PR (in order for me to take

IT issues faced for HadoopInputFormat IO on Spark and Dataflow runners

2017-02-19 Thread Dipti Kulkarni
Hi folks, I am currently working on running HadoopInputFormatIO's CassandraIT and Elasticsearch IT on Spark and Dataflow runners. However I am facing issues in both ITs for specific Classes not being found at run time only when I use the Dataflow runner profile or Spark runner profile. I see i