Thanks. That was the issue. The latest release of Beam indicates it
supports Spark 3 and I find references in the code to Hadoop 3.2.1. Is it
possible to configure beam to run on Hadoop 3.2.1?
Trevor
On Mon, Jun 21, 2021 at 6:19 PM Kyle Weaver wrote:
> Looks like a version mismatch between Hado
Does anyone have an example of using spark-submit to run a beam job using
the portable runner? The documentation indicates this is possible but
doesn't give an example of how to do it.
I am using the following pipeline options to generate the jar
options = PipelineOptions(['--runner=SparkRunner',
