Re: Error while reading the CSV

Try the following spark-shell --master yarn-client --name nayan /opt/packages/-data- prepration/target/scala-2.10/-data-prepration-assembly-1.0.jar On Thu, Apr 6, 2017 at 6:36 PM, nayan sharma wrote: > Hi All, > I am getting error while loading CSV file. > > val > datacsv=sqlContext

Re: Error while reading the CSV

Sorry buddy, didn't get your question quite right. Just to test, I created a scala class with spark csv and it seemed to work. Donno if that would help much, but here are the env details: EMR 2.7.3 scalaVersion := "2.11.8" Spark version 2.0.2 On Fri, 7 Apr 2017 at 17:51 nayan sharma wrote: >

Re: Error while reading the CSV

Hi Yash, I know this will work perfect but here I wanted to read the csv using the assembly jar file. Thanks, Nayan > On 07-Apr-2017, at 10:02 AM, Yash Sharma wrote: > > Hi Nayan, > I use the --packages with the spark shell and the spark submit. Could you > please try that and let us know: >

Re: Error while reading the CSV

Hi Nayan, I use the --packages with the spark shell and the spark submit. Could you please try that and let us know: Command: spark-submit --packages com.databricks:spark-csv_2.11:1.4.0 On Fri, 7 Apr 2017 at 00:39 nayan sharma wrote: > spark version 1.6.2 > scala version 2.10.5 > > On 06-Apr-2

Re: Error while reading the CSV

spark version 1.6.2 scala version 2.10.5 > On 06-Apr-2017, at 8:05 PM, Jörn Franke wrote: > > And which version does your Spark cluster use? > > On 6. Apr 2017, at 16:11, nayan sharma > wrote: > >> scalaVersion := “2.10.5" >> >> >> >> >>> On 06-Apr-2017, at

Re: Error while reading the CSV

And which version does your Spark cluster use? > On 6. Apr 2017, at 16:11, nayan sharma wrote: > > scalaVersion := “2.10.5" > > > > >> On 06-Apr-2017, at 7:35 PM, Jörn Franke wrote: >> >> Maybe your Spark is based on scala 2.11, but you compile it for 2.10 or the >> other way around? >>

Re: Error while reading the CSV

scalaVersion := “2.10.5" > On 06-Apr-2017, at 7:35 PM, Jörn Franke wrote: > > Maybe your Spark is based on scala 2.11, but you compile it for 2.10 or the > other way around? > > On 6. Apr 2017, at 15:54, nayan sharma > wrote: > >> In addition I am using spa

Re: Error while reading the CSV

Maybe your Spark is based on scala 2.11, but you compile it for 2.10 or the other way around? > On 6. Apr 2017, at 15:54, nayan sharma wrote: > > In addition I am using spark version 1.6.2 > Is there any chance of error coming because of Scala version or dependencies > are not matching.?I just

Re: Error while reading the CSV

In addition I am using spark version 1.6.2 Is there any chance of error coming because of Scala version or dependencies are not matching.?I just guessed. Thanks, Nayan > On 06-Apr-2017, at 7:16 PM, nayan sharma wrote: > > Hi Jorn, > Thanks for replying. > > jar -tf catalyst-data-prepration-

Re: Error while reading the CSV

Hi Jorn, Thanks for replying. jar -tf catalyst-data-prepration-assembly-1.0.jar | grep csv after doing this I have found a lot of classes under com/databricks/spark/csv/ do I need to check for any specific class ?? Regards, Nayan > On 06-Apr-2017, at 6:42 PM, Jörn Franke wrote: > > Is the li

Re: Error while reading the CSV

Is the library in your assembly jar? > On 6. Apr 2017, at 15:06, nayan sharma wrote: > > Hi All, > I am getting error while loading CSV file. > > val > datacsv=sqlContext.read.format("com.databricks.spark.csv").option("header", > "true").load("timeline.csv") > java.lang.NoSuchMethodError: >

Error while reading the CSV

Hi All, I am getting error while loading CSV file. val datacsv=sqlContext.read.format("com.databricks.spark.csv").option("header", "true").load("timeline.csv") java.lang.NoSuchMethodError: org.apache.commons.csv.CSVFormat.withQuote(Ljava/lang/Character;)Lorg/apache/commons/csv/CSVFormat; I hav