Re: Ways to partition the RDD

You can try something like this, val kvRdd = sc.textFile("rawdata/").map( m => { val pfUser = m.split("t",2) (pfUser(0) -> pfUser(1))})

Re: SQLContext and HiveContext Query Performance

Any inputs to reduce the time duration for mapPartitions at Exchange.scala:44 from 13 s? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/SQLContext-and-HiveContext-Query-Performance-tp6948p7075.html Sent from the Apache Spark User List mailing list archive a

Re: SQLContext and HiveContext Query Performance

I timed the third line and here are stage timings, collect at SparkPlan.scala:52- 0.5 s mapPartitions at Exchange.scala:58 - 0.7 s RangePartitioner at Exchange.Scala:62 - 0.7 s RangePartitioner at Exchange.Scala:62 - 0.5 s m

Re: SQLContext and HiveContext Query Performance

-- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/SQLContext-and-HiveContext-Query-Performance-tp6948p6976.html Sent from the Apache Spark User List mailing list archive at Nabble.com.