More info on why toDF is required:
http://spark.apache.org/docs/latest/sql-programming-guide.html#upgrading-from-spark-sql-10-12-to-13
On Tue, Apr 14, 2015 at 6:55 AM, pishen tsai <pishe...@gmail.com> wrote:
> I've changed it to
>
> import sqlContext.implicits._
>
> but it still doesn't work. (I've updated the gist)
>
> BTW, using ".toDF()" do work, thanks for this information.
>
> Regards,
> pishen
>
> 2015-04-14 20:35 GMT+08:00 Todd Nist <tsind...@gmail.com>:
>
>> I think docs are correct. If you follow the example from the docs and
>> add this import shown below, I believe you will get what your looking for:
>>
>> // This is used to implicitly convert an RDD to a DataFrame.import
>> sqlContext.implicits._
>>
>> You could also simply take your rdd and do the following:
>>
>> logs.toDF.saveAsParquetFile("s3n://xxx/xxx")
>>
>>
>> -Todd
>>
>> On Tue, Apr 14, 2015 at 3:50 AM, pishen tsai <pishe...@gmail.com> wrote:
>>
>>> OK, it do work.
>>> Maybe it will be better to update this usage in the official Spark SQL
>>> tutorial:
>>> http://spark.apache.org/docs/latest/sql-programming-guide.html
>>>
>>> Thanks,
>>> pishen
>>>
>>>
>>> 2015-04-14 15:30 GMT+08:00 fightf...@163.com <fightf...@163.com>:
>>>
>>>> Hiļ¼there
>>>>
>>>> If you want to use the saveAsParquetFile, you may want to use
>>>> val log_df = sqlContext.createDataFrame(logs)
>>>>
>>>> And then you can issue log_df.saveAsParquetFile (path)
>>>>
>>>> Best,
>>>> Sun.
>>>>
>>>> ------------------------------
>>>> fightf...@163.com
>>>>
>>>>
>>>> *From:* pishen <pishe...@gmail.com>
>>>> *Date:* 2015-04-14 15:18
>>>> *To:* user <user@spark.apache.org>
>>>> *Subject:* Cannot saveAsParquetFile from a RDD of case class
>>>> Hello,
>>>>
>>>> I tried to follow the tutorial of Spark SQL, but is not able to
>>>> saveAsParquetFile from a RDD of case class.
>>>> Here is my Main.scala and build.sbt
>>>> https://gist.github.com/pishen/939cad3da612ec03249f
>>>>
>>>> At line 34, compiler said that "value saveAsParquetFile is not a member
>>>> of org.apache.spark.rdd.RDD[core.Log]"
>>>>
>>>> Any suggestion on how to solve this?
>>>>
>>>> Thanks,
>>>> pishen
>>>>
>>>> ------------------------------
>>>> View this message in context: Cannot saveAsParquetFile from a RDD of
>>>> case class
>>>> <http://apache-spark-user-list.1001560.n3.nabble.com/Cannot-saveAsParquetFile-from-a-RDD-of-case-class-tp22488.html>
>>>> Sent from the Apache Spark User List mailing list archive
>>>> <http://apache-spark-user-list.1001560.n3.nabble.com/> at Nabble.com.
>>>>
>>>>
>>>
>>
>
