Thank you for the reply. I implemented my InputDStream to return None when there's no data. After changing it to return empty RDD, the exception is gone.
I am curious as to why all other processings worked correctly with my old incorrect implementation, with or without data? My actual codes, without the count() part, did glom then foreachRDD. I have StatsReportListener registered and when there was no data, there was no StatsReportListener output. And I thought this was Spark's smart logic to avoid launching workers when there was no data. Wouldn't have thought it was actually an indication that I had my InputDStream implementation wrong. On the other hand, why use return type "Option" if None should not be used at all? Thanks for help solving my problem. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/NullPointerException-from-count-foreachRDD-tp2066p12476.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
