RE: Spark structural streaming sinks output late

Hello, Any observations on what am I doing wrong? Thanks, -Kamal From: Kamalanathan Venkatesan Sent: Tuesday, July 09, 2019 7:25 PM To: 'user@spark.apache.org' Subject: Spark structural streaming sinks output late Hello, I have below spark structural streaming code and I was exp

Spark structural streaming sinks output late

Hello, I have below spark structural streaming code and I was expecting the results to be printed on the console every 10 seconds. But, I notice the sink to console happening every ~2 mins and above. What could be the issue def streaming(): Unit = { System.setProperty("hadoop.home.dir", "/D

Need config params while doing rdd.foreach or map

Hello All, I have custom parameter say for example file name added to the conf of spark context example SparkConf.set(INPUT_FILE_NAME, fileName). I need this value inside foreach performed on an RDD, but the when access spark context inside foreach, I receive spark context is null exception!