Hi Chetan
I'm having the exact same issue with spark structured streaming and kafka
trying to write to HDFS.
Can you please tell me how did you fixed it?
I'm ussing spark 3.0.1 and hadoop 3.3.0
Thanks!
--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
-
Hello Deng, Thank you for your email.
Issue was with Spark - Hadoop / HDFS configuration settings.
Thanks
On Mon, Jun 10, 2019 at 5:28 AM Deng Ching-Mallete
wrote:
> Hi Chetan,
>
> Best to check if the user account that you're using to run the job has
> permission to write to the path in HDFS.
Hi Chetan,
Best to check if the user account that you're using to run the job has
permission to write to the path in HDFS. I would suggest to write the
parquet files to a different path, perhaps to a project space or user home,
rather than at the root directory.
HTH,
Deng
On Sat, Jun 8, 2019 at
Also anyone has any idea to resolve this issue -
https://stackoverflow.com/questions/56390492/spark-metadata-0-doesnt-exist-while-compacting-batch-9-structured-streaming-er
On Fri, Jun 7, 2019 at 5:59 PM Chetan Khatri
wrote:
> Hello Dear Spark Users,
>
> I am trying to write data from Kafka Topi
