Re: SparkR DataFrame , Out of memory exception for very small file.

2015-11-23 Thread Jeff Zhang
t 4:44 PM, Vipul Rai >>>> wrote: >>>> >>>>> Hello Rui, >>>>> >>>>> Sorry , What I meant was the resultant of the original dataframe to >>>>> which a new column was added gives a new DataFrame. >>>>> >>>

Re: SparkR DataFrame , Out of memory exception for very small file.

2015-11-23 Thread Vipul Rai
> >>>> Check for >>>> WithColumn >>>> >>>> >>>> Thanks, >>>> Vipul >>>> >>>> >>>> On 23 November 2015 at 12:42, Sun, Rui wrote: >>>> >>>>> Vipul, >>>>&

Re: SparkR DataFrame , Out of memory exception for very small file.

2015-11-23 Thread Jeff Zhang
hanks, >>> Vipul >>> >>> >>> On 23 November 2015 at 12:42, Sun, Rui wrote: >>> >>>> Vipul, >>>> >>>> Not sure if I understand your question. DataFrame is immutable. You >>>> can't update a DataFrame. >>>> >&g

Re: SparkR DataFrame , Out of memory exception for very small file.

2015-11-23 Thread Vipul Rai
November 2015 at 12:42, Sun, Rui wrote: >> >>> Vipul, >>> >>> Not sure if I understand your question. DataFrame is immutable. You >>> can't update a DataFrame. >>> >>> Could you paste some log info for the OOM error? >>> >>

Re: SparkR DataFrame , Out of memory exception for very small file.

2015-11-23 Thread Jeff Zhang
M error? >> >> -Original Message- >> From: vipulrai [mailto:vipulrai8...@gmail.com] >> Sent: Friday, November 20, 2015 12:11 PM >> To: user@spark.apache.org >> Subject: SparkR DataFrame , Out of memory exception for very small file. >>

Re: SparkR DataFrame , Out of memory exception for very small file.

2015-11-23 Thread Vipul Rai
vember 20, 2015 12:11 PM > To: user@spark.apache.org > Subject: SparkR DataFrame , Out of memory exception for very small file. > > Hi Users, > > I have a general doubt regarding DataFrames in SparkR. > > I am trying to read a file from Hive and it gets created as DataF

RE: SparkR DataFrame , Out of memory exception for very small file.

2015-11-22 Thread Sun, Rui
e.org Subject: SparkR DataFrame , Out of memory exception for very small file. Hi Users, I have a general doubt regarding DataFrames in SparkR. I am trying to read a file from Hive and it gets created as DataFrame. sqlContext <- sparkRHive.init(sc) #DF sales <- read.df(sqlContext, "h

SparkR DataFrame , Out of memory exception for very small file.

2015-11-19 Thread vipulrai
ery update?? Thanks, Vipul -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/SparkR-DataFrame-Out-of-memory-exception-for-very-small-file-tp25435.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --