Re: Spark SQL - Long running job

2015-02-23 Thread Cheng Lian
et more partitions when reading instead of original schemaRdd. -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Spark-SQL-Long-running-job-tp10717p10727.html Sent from the Apache Spark Developers List mailing lis

Re: Spark SQL - Long running job

2015-02-22 Thread nitin
(hdfs file system) but I fear that in case my "HDFS block size" > "partition file size", I will get more partitions when reading instead of original schemaRdd. -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Spark-SQL-Long-running

Re: Spark SQL - Long running job

2015-02-22 Thread Cheng Lian
spark job. (running spark in yarn-client mode btw). Thanks -Nitin -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Spark-SQL-Long-running-job-tp10717.html Sent from the Apache Spark Developers List mailing list archive at Nabbl

Spark SQL - Long running job

2015-02-21 Thread nitin
ickly and going out of space as its a long running spark job. (running spark in yarn-client mode btw). Thanks -Nitin -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Spark-SQL-Long-running-job-tp10717.html Sent from the Apache Spark Developers List ma