I will work on the first two possible causes. For the third one, which I guess is the real problem, Spark treats the testfile.csv object with the url s3a://test-bucket/testfile.csv as a bucket to access _spark_metadata with url s3a://test-bucket/testfile.csv/_spark_metadata testfile.csv is an object and should not be treated as a bucket. But I am not sure how to prevent Spark from doing that.
- [s3a] Spark is not reading s3 object content Amin Mosayyebzadeh
- Re: [s3a] Spark is not reading s3 object content Mich Talebzadeh
- Re: [s3a] Spark is not reading s3 object conte... Amin Mosayyebzadeh
- Re: [s3a] Spark is not reading s3 object c... Mich Talebzadeh
- Re: [s3a] Spark is not reading s3 obje... Amin Mosayyebzadeh
- Re: [s3a] Spark is not reading s3... Mich Talebzadeh
- Re: [s3a] Spark is not readin... Amin Mosayyebzadeh
- Re: [s3a] Spark is not re... Mich Talebzadeh
- Re: [s3a] Spark is not re... Amin Mosayyebzadeh