subject:"\[SPARK\-SQL\] Spark Persist slower than non\-persist calls"

[SPARK-SQL] Spark Persist slower than non-persist calls

2017-09-01 Thread sfbayeng

My settings are: Running Spark 2.1 on 3 node YARN cluster with 160 GB. Dynamic allocation turned on. spark.executor.memory=6G, spark.executor.cores=6 First, I am reading hive tables: orders(329MB) and lineitems(1.43GB) and doing left outer join. Next, I apply 7 different filter conditions based on

[SPARK-SQL] Spark Persist slower than non-persist calls

2017-08-31 Thread saurabh raval

Spark 2.1 My settings are: Running Spark 2.1 on 3 node YARN cluster with 160 GB. Dynamic allocation turned on. spark.executor.memory=6G, spark.executor.cores=6 First, I am reading hive tables: orders(329MB) and lineitems(1.43GB) and doing left outer join.Next, I apply 7 different filter condition