I don't think Spark optimizer supports something like statement cache where
plan is cached and bind variables (like RDBMS) are used for different
values, thus saving the parsing.
What you re stating is that the source and tempTable change but the plan
itself remains the same. I have not seen this
I am using 1.5.2.
I have a data-frame with 10 column and then I pivot 1 column and generate
the 700 columns.
it is like
val df1 = sqlContext.read.parquet("file1")
df1.registerTempTable("df1")
val df2= sqlContext.sql("select col1, col2, sum(case when col3 = 1 then
col4 else 0.0 end ) as col4_1,..
A logical plan should not change assuming the same DAG diagram is used
throughout
Have you tried Spark GUI Page under stages? This is Spark 2
example:
[image: Inline images 1]
HTH
Dr Mich Talebzadeh
LinkedIn *
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8P
Which version are you using here? If the underlying files change,
technically we should go through optimization again.
Perhaps the real "fix" is to figure out why is logical plan creation so
slow for 700 columns.
On Thu, Jun 30, 2016 at 1:58 PM, Darshan Singh
wrote:
> Is there a way I can use
