Hi Shiva,
*This works on 3.0.1 on prem* but not on Google dataproc with spark
3.1.1-RC2
These are the jar files used for structured streaming
All added under $SPARK_HOME/jars on all nodes
spark-sql-kafka-0-10_2.12-3.0.1.jar
kafka-clients-2.7.0.jar
spark-token-provider-kafka-0-10_2.12-3.0.1.jar
commons-pool2-2.9.0.jar
Also add these under $SPARK_HOME/conf in file spark-defaults.conf all nodes
spark.driver.extraClassPath $SPARK_HOME/jars/*.jar
spark.executor.extraClassPath $SPARK_HOME/jars/*.jar
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 3.0.1
/_/
Using Scala version 2.12.10 (Java HotSpot(TM) 64-Bit Server VM, Java
1.8.0_201)
HTH
LinkedIn *
https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On Tue, 2 Mar 2021 at 11:47, shiva <[email protected]> wrote:
> Hi Mich Talebzadeh,
> Could you please share the spark configuration used to run the job? you
> mentioned it works on 3.0.1 I will check if I am also using the same
> configuration or not.
>
> Regards,
> Shiva
>
>
>
> --
> Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: [email protected]
>
>
