Re: Unit test with sqlContext

2016-03-19 Thread Vikas Kawadia
rk-testing-base >>>> >>>> DataFrame examples are here: >>>> https://github.com/holdenk/spark-testing-base/blob/master/src/test/1.3/scala/com/holdenkarau/spark/testing/SampleDataFrameTest.scala >>>> >>>> Thanks, >>>> Silvi

Re: Unit test with sqlContext

2016-02-05 Thread Steve Annessa
m/holdenkarau/spark/testing/SampleDataFrameTest.scala >>> >>> Thanks, >>> Silvio >>> >>> From: Steve Annessa >>> Date: Thursday, February 4, 2016 at 8:36 PM >>> To: "user@spark.apache.org" >>> Subject: Unit test

Re: Unit test with sqlContext

2016-02-04 Thread Rishi Mishra
/1.3/scala/com/holdenkarau/spark/testing/SampleDataFrameTest.scala >> >> Thanks, >> Silvio >> >> From: Steve Annessa >> Date: Thursday, February 4, 2016 at 8:36 PM >> To: "user@spark.apache.org" >> Subject: Unit test with sqlContext >

Re: Unit test with sqlContext

2016-02-04 Thread Holden Karau
e/blob/master/src/test/1.3/scala/com/holdenkarau/spark/testing/SampleDataFrameTest.scala > > Thanks, > Silvio > > From: Steve Annessa > Date: Thursday, February 4, 2016 at 8:36 PM > To: "user@spark.apache.org" > Subject: Unit test with sqlContext > > I'

Re: Unit test with sqlContext

2016-02-04 Thread Silvio Fiorito
/master/src/test/1.3/scala/com/holdenkarau/spark/testing/SampleDataFrameTest.scala Thanks, Silvio From: Steve Annessa mailto:steve.anne...@gmail.com>> Date: Thursday, February 4, 2016 at 8:36 PM To: "user@spark.apache.org<mailto:user@spark.apache.org>" mailto:user@spark.apache.or

Unit test with sqlContext

2016-02-04 Thread Steve Annessa
I'm trying to unit test a function that reads in a JSON file, manipulates the DF and then returns a Scala Map. The function has signature: def ingest(dataLocation: String, sc: SparkContext, sqlContext: SQLContext) I've created a bootstrap spec for spark jobs that instantiates the Spark Context an