Hi, I am developing program using Spark where I am using filter such as: cleanedData = distData.map(json_extractor.extract_json).filter(lambda x: x != None and x != '') cleanedData.saveAsTextFile(sys.argv[3]) It happens to me that there is saved lot of empty files (probably from those partitions that should have been filtered out). Is there some way, how to prevent Spark from saving these empty files? Thank you in advance for any help.
--------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
