is caused by SPARK-8458
> <https://issues.apache.org/jira/browse/SPARK-8458>
>
> Thanks.
>
> Zhan Zhang
>
> On Aug 23, 2015, at 12:49 PM, lostrain A
> wrote:
>
> Ted,
> Thanks for the suggestions. Actually I tried both s3n and s3 and the
> result remain
> On Sun, Aug 23, 2015 at 11:03 AM, lostrain A <
> donotlikeworkingh...@gmail.com> wrote:
>
>> Hi Ted,
>> Thanks for the reply. I tried setting both of the keyid and accesskey
>> via
>>
>> sc.hadoopConfiguration.set("fs.s3n.aw
ORC format.
If I change the format to JSON, although the error does not go, the JSON
files can be saved successfully.
On Sun, Aug 23, 2015 at 5:51 AM, Ted Yu wrote:
> You may have seen this:
> http://search-hadoop.com/m/q3RTtdSyM52urAyI
>
>
>
> On Aug 23, 2015, at 1:01 AM, lostrain
Hi,
I'm trying to save a simple dataframe to S3 in ORC format. The code is as
follows:
val sqlContext = new org.apache.spark.sql.hive.HiveContext(sc)
> import sqlContext.implicits._
> val df=sc.parallelize(1 to 1000).toDF()
> df.write.format("orc").save("s3://logs/dummy)
