On 31 Dec 2016, at 16:09, Raymond Xie
mailto:xie3208...@gmail.com>> wrote:
Hello Felix,
I followed the instruction and ran the command:
> $SPARK_HOME/bin/spark-shell --packages com.databricks:spark-csv_2.11:1.5.0
and I received the following error message:
java.lang.RuntimeException: java.net
___
From: Raymond Xie mailto:xie3208...@gmail.com>>
Sent: Friday, December 30, 2016 6:46:11 PM
To: user@spark.apache.org<mailto:user@spark.apache.org>
Subject: How to load a big csv to dataframe in Spark 1.6
Hello,
I see there is usually this way to load a c
ote:
> Have you tried the spark-csv package?
>
> https://spark-packages.org/package/databricks/spark-csv
>
>
> --
> *From:* Raymond Xie
> *Sent:* Friday, December 30, 2016 6:46:11 PM
> *To:* user@spark.apache.org
> *Subject:* How to load a
--
> *From:* Raymond Xie
> *Sent:* Friday, December 30, 2016 6:46:11 PM
> *To:* user@spark.apache.org
> *Subject:* How to load a big csv to dataframe in Spark 1.6
>
> Hello,
>
> I see there is usually this way to load a csv to dataframe:
>
> sqlContext = SQLCo
- Original message
From: Raymond Xie
Date: 31/12/2016 10:46 (GMT+08:00)
To: user@spark.apache.org
Subject: How to load a big csv to dataframe in Spark 1.6
Hello,
I see there is usually this way to load a csv to dataframe:
sqlContext = SQLContext(sc)
Employee_rdd = sc.textFile("\
Re: How to load a big csv to dataframe in Spark 1.6
Hi Raymond,
Your problem is to pass those 100 fields to .toDF() method??
Sent from my Samsung device
Original message
From: Raymond Xie
Date: 31/12/2016 10:46 (GMT+08:00)
To: user@spark.apache.org
Subject: How to
Have you tried the spark-csv package?
https://spark-packages.org/package/databricks/spark-csv
From: Raymond Xie
Sent: Friday, December 30, 2016 6:46:11 PM
To: user@spark.apache.org
Subject: How to load a big csv to dataframe in Spark 1.6
Hello,
I see there is
Hi Raymond,
Your problem is to pass those 100 fields to .toDF() method??
Sent from my Samsung device
Original message
From: Raymond Xie
Date: 31/12/2016 10:46 (GMT+08:00)
To: user@spark.apache.org
Subject: How to load a big csv to dataframe in Spark 1.6
Hello,
I
Hello,
I see there is usually this way to load a csv to dataframe:
sqlContext = SQLContext(sc)
Employee_rdd = sc.textFile("\..\Employee.csv")
.map(lambda line: line.split(","))
Employee_df = Employee_rdd.toDF(['Employee_ID','Employee_name'])
Employee_df.show()
However in my cas
