Read hdfs files in spark streaming
I am using spark streaming application to read from kafka. The value coming from kafka message is path to hdfs file. I am using spark 2.x , spark.read.stream. What is the best way to read this path in spark streaming and then read the json stored at the hdfs path , may be using spark.read.json , into a df inside the spark streaming app. Thanks a lot in advance -- Thanks Deepak
Adding Custom finalize method to RDDs.
Hi All, Is there a way to add custom finalize method to RDD objects to add custom logic when RDDs are destructed by JVM ? Thanks, Nasrulla
