Hi All,
Thnks for the reply.
I did tried removing the client version.
But got the same exception.
Though one point there is some dependent artifacts which I am using, which
contains refrence to the Kafka client saw version.
I am trying to make uber jar, which will choose the closest version.
Thnks
On Monday, December 7, 2020, Gabor Somogyi <gabor.g.somo...@gmail.com>
wrote:
> +1 on the mentioned change, Spark uses the following kafka-clients library:
>
> <kafka.version>2.4.1</kafka.version>
>
> G
>
>
> On Mon, Dec 7, 2020 at 9:30 AM German Schiavon <gschiavonsp...@gmail.com>
> wrote:
>
>> Hi,
>>
>> I think the issue is that you are overriding the kafka-clients that comes
>> with <artifactId>spark-sql-kafka-0-10_2.12</artifactId>
>>
>>
>> I'd try removing the kafka-clients and see if it works
>>
>>
>> On Sun, 6 Dec 2020 at 08:01, Amit Joshi <mailtojoshia...@gmail.com>
>> wrote:
>>
>>> Hi All,
>>>
>>> I am running the Spark Structured Streaming along with Kafka.
>>> Below is the pom.xml
>>>
>>> <properties>
>>> <maven.compiler.source>1.8</maven.compiler.source>
>>> <maven.compiler.target>1.8</maven.compiler.target>
>>> <encoding>UTF-8</encoding>
>>> <!-- Put the Scala version of the cluster -->
>>> <scalaVersion>2.12.10</scalaVersion>
>>> <sparkVersion>3.0.1</sparkVersion>
>>> </properties>
>>>
>>> <dependency>
>>> <groupId>org.apache.kafka</groupId>
>>> <artifactId>kafka-clients</artifactId>
>>> <version>2.1.0</version>
>>> </dependency>
>>>
>>> <dependency>
>>> <groupId>org.apache.spark</groupId>
>>> <artifactId>spark-core_2.12</artifactId>
>>> <version>${sparkVersion}</version>
>>> <scope>provided</scope>
>>> </dependency>
>>> <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-sql -->
>>> <dependency>
>>> <groupId>org.apache.spark</groupId>
>>> <artifactId>spark-sql_2.12</artifactId>
>>> <version>${sparkVersion}</version>
>>> <scope>provided</scope>
>>> </dependency>
>>> <!--
>>> https://mvnrepository.com/artifact/org.apache.spark/spark-sql-kafka-0-10 -->
>>> <dependency>
>>> <groupId>org.apache.spark</groupId>
>>> <artifactId>spark-sql-kafka-0-10_2.12</artifactId>
>>> <version>${sparkVersion}</version>
>>> </dependency>
>>>
>>> Building the fat jar with shade plugin. The jar is running as expected in
>>> my local setup with the command
>>>
>>> *spark-submit --master local[*] --class com.stream.Main --num-executors 3
>>> --driver-memory 2g --executor-cores 2 --executor-memory 3g
>>> prism-event-synch-rta.jar*
>>>
>>> But when I am trying to run same jar in spark cluster using yarn with
>>> command:
>>>
>>> *spark-submit --master yarn --deploy-mode cluster --class com.stream.Main
>>> --num-executors 4 --driver-memory 2g --executor-cores 1 --executor-memory
>>> 4g gs://jars/prism-event-synch-rta.jar*
>>>
>>> Getting the this exception:
>>>
>>>
>>>
>>>
>>> *at org.apache.spark.sql.execution.streaming.StreamExecution.org
>>> <http://org.apache.spark.sql.execution.streaming.StreamExecution.org>$apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.scala:355)
>>> at
>>> org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:245)Caused
>>> by: org.apache.kafka.common.config.ConfigException: Missing required
>>> configuration "partition.assignment.strategy" which has no default value.
>>> at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:124)*
>>>
>>> I have tried setting up the "partition.assignment.strategy", then also its
>>> not working.
>>>
>>> Please help.
>>>
>>>
>>> Regards
>>>
>>> Amit Joshi
>>>
>>>
