scala test is unable to initialize spark context.

is not getting initialized Code:- import org.apache.log4j.LogManager import org.apache.spark.SharedSparkContext import org.scalatest.FunSuite import org.apache.spark.{SparkContext, SparkConf} /** * Created by PSwain on 4/5/2017. */ class Test extends FunSuite with SharedSparkContext

RE: Huge partitioning job takes longer to close after all tasks finished

Hi Swapnil, We are facing same issue , could you please let me know how did you find that partitions are getting merged ? Thanks in advance !! From: Swapnil Shinde [mailto:swapnilushi...@gmail.com] Sent: Thursday, March 09, 2017 1:31 AM To: cht liu Cc: user@spark.apache.org Subject: Re: Huge

Not able to remove header from a text file while creating a data frame .

Hi All, I am reading a text file to create a dataframe . While I am trying to exclude header form the text file I am not able r to do it . Now my concern is how to know what all options are there that I can use while reading from a source , I checked the API , there the Arguments in option metho