Re: spark-submit and the portable runner

2021-06-24 Thread Trevor Kramer
Thanks. That was the issue. The latest release of Beam indicates it supports Spark 3 and I find references in the code to Hadoop 3.2.1. Is it possible to configure beam to run on Hadoop 3.2.1? Trevor On Mon, Jun 21, 2021 at 6:19 PM Kyle Weaver wrote: > Looks like a version mismatch between Hado

spark-submit and the portable runner

2021-06-21 Thread Trevor Kramer
Does anyone have an example of using spark-submit to run a beam job using the portable runner? The documentation indicates this is possible but doesn't give an example of how to do it. I am using the following pipeline options to generate the jar options = PipelineOptions(['--runner=SparkRunner',