Re: value toDF is not a member of RDD object

On Wed, May 13, 2015 at 3:00 AM, SLiZn Liu wrote: > Additionally, after I successfully packaged the code, and submitted via > spark-submit > webcat_2.11-1.0.jar, the following error was thrown at the line where > toDF() been called: > > Exception in thread "main" java.lang.NoSuchMethodError: >

Re: value toDF is not a member of RDD object

I believe what Dean Wampler was suggesting is to use the sqlContext not the sparkContext (sc), which is where the createDataFrame function resides: https://spark.apache.org/docs/1.3.1/api/scala/index.html#org.apache.spark.sql.SQLContext HTH. -Todd On Wed, May 13, 2015 at 6:00 AM, SLiZn Liu wro

Fwd: value toDF is not a member of RDD object

class using spark-submit --class WebcatApp --verbose webcat_2.11-1.0.jar to make spark recognize the main class. The error and stack trace remains the same. ​ -- Forwarded message - From: Animesh Baranawal Date: Wed, May 13, 2015 at 6:49 PM Subject: Re: value toDF is not a member of

Re: value toDF is not a member of RDD object

No, creating DF using createDataFrame won’t work: val peopleDF = sqlContext.createDataFrame(people) the code can be compiled but raised the same error as toDF at the line above. On Wed, May 13, 2015 at 6:22 PM Sebastian Alfers [sebastian.alf...@googlemail.com](mailto:sebastian.alf...@googlemail.

Re: value toDF is not a member of RDD object

Additionally, after I successfully packaged the code, and submitted via spark-submit webcat_2.11-1.0.jar, the following error was thrown at the line where toDF() been called: Exception in thread "main" java.lang.NoSuchMethodError: scala.reflect.api.JavaUniverse.runtimeMirror(Ljava/lang/ClassLoader

Re: value toDF is not a member of RDD object

Thanks folks, really appreciate all your replies! I tried each of your suggestions and in particular, *Animesh*‘s second suggestion of *making case class definition global* helped me getting off the trap. Plus, I should have paste my entire code with this mail to help the diagnose. REGARDS, Todd

Re: value toDF is not a member of RDD object

It's the import statement Olivier showed that makes the method available. Note that you can also use `sc.createDataFrame(myRDD)`, without the need for the import statement. I personally prefer this approach. Dean Wampler, Ph.D. Author: Programming Scala, 2nd Edition

Re: value toDF is not a member of RDD object

you need to instantiate a SQLContext : val sc : SparkContext = ... val sqlContext = new SQLContext(sc) import sqlContext.implicits._ Le mar. 12 mai 2015 à 12:29, SLiZn Liu a écrit : > I added `libraryDependencies += "org.apache.spark" % "spark-sql_2.11" % > "1.3.1"` to `build.sbt` but the error

Re: value toDF is not a member of RDD object

I added `libraryDependencies += "org.apache.spark" % "spark-sql_2.11" % "1.3.1"` to `build.sbt` but the error remains. Do I need to import modules other than `import org.apache.spark.sql.{ Row, SQLContext }`? On Tue, May 12, 2015 at 5:56 PM Olivier Girardot wrote: > toDF is part of spark SQL so

Re: value toDF is not a member of RDD object

toDF is part of spark SQL so you need Spark SQL dependency + import sqlContext.implicits._ to get the toDF method. Regards, Olivier. Le mar. 12 mai 2015 à 11:36, SLiZn Liu a écrit : > Hi User Group, > > I’m trying to reproduce the example on Spark SQL Programming Guide >

value toDF is not a member of RDD object

Hi User Group, I’m trying to reproduce the example on Spark SQL Programming Guide , and got a compile error when packaging with sbt: [error] myfile.scala:30: value toDF is not a member of org.ap