After:
val sqlContext = new org.apache.spark.sql.SQLContext(sc)
Please add:
import sqlContext.implicits._
On Fri, Mar 4, 2016 at 3:03 PM, Mich Talebzadeh <[email protected]>
wrote:
> Hi Ted,
>
> This is my code
>
> import org.apache.spark.SparkConf
> import org.apache.spark.sql.Row
> import org.apache.spark.sql.hive.HiveContext
> import org.apache.spark.sql.types._
> import org.apache.spark.sql.SQLContext
> //
> object Sequence {
> def main(args: Array[String]) {
> val conf = new
> SparkConf().setAppName("Sequence").setMaster("local[*]").set("spark.driver.allowMultipleContexts",
> "true")
> val sc = new SparkContext(conf)
> val sqlContext = new org.apache.spark.sql.SQLContext(sc)
> val HiveContext = new org.apache.spark.sql.hive.HiveContext(sc)
> val a = Seq(("Mich",20), ("Christian", 18), ("James",13), ("Richard",16))
> // Sort option 1 using tempTable
> val b = a.toDF("Name","score").registerTempTable("tmp")
> sql("select Name,score from tmp order by score desc").show
> // Sort option 2 with FP
> a.toDF("Name","score").sort(desc("score")).show
> }
> }
>
> And the error I am getting now is
>
> [info] downloading
> https://repo1.maven.org/maven2/org/scala-lang/jline/2.10.5/jline-2.10.5.jar
> ...
> [info] [SUCCESSFUL ] org.scala-lang#jline;2.10.5!jline.jar (103ms)
> [info] Done updating.
> [info] Compiling 1 Scala source to
> /home/hduser/dba/bin/scala/Sequence/target/scala-2.10/classes...
> [info] 'compiler-interface' not yet compiled for Scala 2.10.5. Compiling...
> [info] Compilation completed in 12.462 s
> [error]
> /home/hduser/dba/bin/scala/Sequence/src/main/scala/Sequence.scala:16: value
> toDF is not a member of Seq[(String, Int)]
> [error] val b = a.toDF("Name","score").registerTempTable("tmp")
> [error] ^
> [error]
> /home/hduser/dba/bin/scala/Sequence/src/main/scala/Sequence.scala:17: not
> found: value sql
> [error] sql("select Name,score from tmp order by score desc").show
> [error] ^
> [error]
> /home/hduser/dba/bin/scala/Sequence/src/main/scala/Sequence.scala:19: value
> toDF is not a member of Seq[(String, Int)]
> [error] a.toDF("Name","score").sort(desc("score")).show
> [error] ^
> [error] three errors found
> [error] (compile:compileIncremental) Compilation failed
> [error] Total time: 88 s, completed Mar 4, 2016 11:12:46 PM
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn *
> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> On 4 March 2016 at 22:52, Ted Yu <[email protected]> wrote:
>
>> Can you show your code snippet ?
>> Here is an example:
>>
>> val sqlContext = new SQLContext(sc)
>> import sqlContext.implicits._
>>
>> On Fri, Mar 4, 2016 at 1:55 PM, Mich Talebzadeh <
>> [email protected]> wrote:
>>
>>> Hi Ted,
>>>
>>> I am getting the following error after adding that import
>>>
>>> [error]
>>> /home/hduser/dba/bin/scala/Sequence/src/main/scala/Sequence.scala:5: not
>>> found: object sqlContext
>>> [error] import sqlContext.implicits._
>>> [error] ^
>>> [error]
>>> /home/hduser/dba/bin/scala/Sequence/src/main/scala/Sequence.scala:15: value
>>> toDF is not a member of Seq[(String, Int)]
>>>
>>>
>>> Dr Mich Talebzadeh
>>>
>>>
>>>
>>> LinkedIn *
>>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>
>>>
>>>
>>> http://talebzadehmich.wordpress.com
>>>
>>>
>>>
>>> On 4 March 2016 at 21:39, Ted Yu <[email protected]> wrote:
>>>
>>>> Can you add the following into your code ?
>>>> import sqlContext.implicits._
>>>>
>>>> On Fri, Mar 4, 2016 at 1:14 PM, Mich Talebzadeh <
>>>> [email protected]> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I have a simple Scala program as below
>>>>>
>>>>> import org.apache.spark.SparkContext
>>>>> import org.apache.spark.SparkContext._
>>>>> import org.apache.spark.SparkConf
>>>>> import org.apache.spark.sql.SQLContext
>>>>> object Sequence {
>>>>> def main(args: Array[String]) {
>>>>> val conf = new SparkConf().setAppName("Sequence")
>>>>> val sc = new SparkContext(conf)
>>>>> val sqlContext = new org.apache.spark.sql.SQLContext(sc)
>>>>> val HiveContext = new org.apache.spark.sql.hive.HiveContext(sc)
>>>>> val a = Seq(("Mich",20), ("Christian", 18), ("James",13),
>>>>> ("Richard",16))
>>>>> // Sort option 1 using tempTable
>>>>> val b = a.toDF("Name","score").registerTempTable("tmp")
>>>>> sql("select Name,score from tmp order by score desc").show
>>>>> // Sort option 2 with FP
>>>>> a.toDF("Name","score").sort(desc("score")).show
>>>>> }
>>>>> }
>>>>>
>>>>> I build this using sbt tool as below
>>>>>
>>>>> cat sequence.sbt
>>>>> name := "Sequence"
>>>>> version := "1.0"
>>>>> scalaVersion := "2.10.5"
>>>>> libraryDependencies += "org.apache.spark" %% "spark-core" % "1.5.0"
>>>>> libraryDependencies += "org.apache.spark" %% "spark-sql" % "1.0.0"
>>>>> libraryDependencies += "org.apache.spark" %% "spark-hive" % "1.5.0"
>>>>>
>>>>>
>>>>> But it fails compilation as below
>>>>>
>>>>> [info] Compilation completed in 12.366 s
>>>>> [error]
>>>>> /home/hduser/dba/bin/scala/Sequence/src/main/scala/Sequence.scala:15:
>>>>> value
>>>>> toDF is not a member of Seq[(String, Int)]
>>>>> [error] val b = a.toDF("Name","score").registerTempTable("tmp")
>>>>> [error] ^
>>>>> [error]
>>>>> /home/hduser/dba/bin/scala/Sequence/src/main/scala/Sequence.scala:16: not
>>>>> found: value sql
>>>>> [error] sql("select Name,score from tmp order by score desc").show
>>>>> [error] ^
>>>>> [error]
>>>>> /home/hduser/dba/bin/scala/Sequence/src/main/scala/Sequence.scala:18:
>>>>> value
>>>>> toDF is not a member of Seq[(String, Int)]
>>>>> [error] a.toDF("Name","score").sort(desc("score")).show
>>>>> [error] ^
>>>>> [error] three errors found
>>>>> [error] (compile:compileIncremental) Compilation failed
>>>>> [error] Total time: 95 s, completed Mar 4, 2016 9:06:40 PM
>>>>>
>>>>> I think I am missing some dependencies here
>>>>>
>>>>>
>>>>> I have a simple
>>>>>
>>>>> Dr Mich Talebzadeh
>>>>>
>>>>>
>>>>>
>>>>> LinkedIn *
>>>>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>>>
>>>>>
>>>>>
>>>>> http://talebzadehmich.wordpress.com
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>
