Re: Spark Dataframe: Save to hdfs is taking long time

2016-12-28 Thread Raju Bairishetti
2016 7:54:27 PM >> *To:* user @spark >> *Subject:* Spark Dataframe: Save to hdfs is taking long time >> >> Hi, >> >> I am using issue while saving the dataframe back to HDFS. It's taking >> long time to run. >> >> val results_dataframe = sqlCont

Re: Spark Dataframe: Save to hdfs is taking long time

2016-12-15 Thread KhajaAsmath Mohammed
Spark Dataframe: Save to hdfs is taking long time > > Hi, > > I am using issue while saving the dataframe back to HDFS. It's taking long > time to run. > > val results_dataframe = sqlContext.sql("select gt.*,ct.* from > PredictTempTable pt,ClusterTempTable ct,Gami

Re: Spark Dataframe: Save to hdfs is taking long time

2016-12-15 Thread Felix Cheung
What is the format? From: KhajaAsmath Mohammed Sent: Thursday, December 15, 2016 7:54:27 PM To: user @spark Subject: Spark Dataframe: Save to hdfs is taking long time Hi, I am using issue while saving the dataframe back to HDFS. It's taking long time t

Spark Dataframe: Save to hdfs is taking long time

2016-12-15 Thread KhajaAsmath Mohammed
Hi, I am using issue while saving the dataframe back to HDFS. It's taking long time to run. val results_dataframe = sqlContext.sql("select gt.*,ct.* from PredictTempTable pt,ClusterTempTable ct,GamificationTempTable gt where gt.vin=pt.vin and pt.cluster=ct.cluster") results_dataframe.coalesce(num