Missing required configuration "partition.assignment.strategy" [ Kafka + Spark Structured Streaming ]

[Amit Joshi]

Hi All,

I am running the Spark Structured Streaming along with Kafka.
Below is the pom.xml

<properties>
    <maven.compiler.source>1.8</maven.compiler.source>
    <maven.compiler.target>1.8</maven.compiler.target>
    <encoding>UTF-8</encoding>
    <!-- Put the Scala version of the cluster -->
    <scalaVersion>2.12.10</scalaVersion>
    <sparkVersion>3.0.1</sparkVersion>
</properties>

<dependency>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka-clients</artifactId>
    <version>2.1.0</version>
</dependency>

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-core_2.12</artifactId>
    <version>${sparkVersion}</version>
    <scope>provided</scope>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-sql -->
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-sql_2.12</artifactId>
    <version>${sparkVersion}</version>
    <scope>provided</scope>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-sql-kafka-0-10
-->
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-sql-kafka-0-10_2.12</artifactId>
    <version>${sparkVersion}</version>
</dependency>

Building the fat jar with shade plugin. The jar is running as expected
in my local setup with the command

*spark-submit --master local[*] --class com.stream.Main
--num-executors 3 --driver-memory 2g --executor-cores 2
--executor-memory 3g prism-event-synch-rta.jar*

But when I am trying to run same jar in spark cluster using yarn with command:

*spark-submit --master yarn --deploy-mode cluster --class
com.stream.Main --num-executors 4 --driver-memory 2g --executor-cores
1 --executor-memory 4g  gs://jars/prism-event-synch-rta.jar*

Getting the this exception:

        


*at org.apache.spark.sql.execution.streaming.StreamExecution.org
<http://org.apache.spark.sql.execution.streaming.StreamExecution.org>$apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.scala:355)
        at
org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:245)Caused
by: org.apache.kafka.common.config.ConfigException: Missing required
configuration "partition.assignment.strategy" which has no default
value.  at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:124)*

I have tried setting up the "partition.assignment.strategy", then also
its not working.

Please help.


Regards

Amit Joshi