Hi, I am getting the following error while trying save a large dataset to s3 using the saveAsHadoopFile command with apache spark-1.0. org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException: S3 PUT failed for '/spark_test%2Fsmaato_one_day_phase_2%2Fsmaato_2014_05_17%2F_temporary%2F_attempt_201407170658_0000_m_000036_276%2Fpart-00036' XML Error Message: <?xml version="1.0" encoding="UTF-8"?><Error><Code>BadDigest</Code><Message>The Content-MD5 you specified did not match what we received.</Message><ExpectedDigest>N808DtNfYiTFzI+i2HxLEw==</ExpectedDigest><CalculatedDigest>66nS+2C1QqQmmcTeFpXOjw==</CalculatedDigest><RequestId>4FB3A3D60B187CE7</RequestId><HostId>H2NznP+RvwspekVHBMvgWGYAupKuO5YceSgmiLym6rOajOh5v5GnyM0VkO+dadyG</HostId></Error>
I have used the same command to write similar content with lesser data to s3 without any problem. When I googled this error message, they say it might be due to md5 checksum mismatch. But will this happen due to load? Regards, lmk -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Bad-Digest-error-while-doing-aws-s3-put-tp10036.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
