Hi, Has any one encountered an issue of missing output partition file in S3 ? My spark job writes output to a S3 location. Occasionally, I noticed one partition file is missing. As a result, one chunk of data was lost. If I rerun the same job, the problem usually goes away. This has been happening pretty random. I observed once or twice a week on a daily run job. I am using Spark 1.2.1.
Very much appreciated on any input, suggestion of fix/workaround.
