Jacek,
Thanks for your help. I didn’t want to write a bug/enhancement unless
warranted.
~ Shawn
From: Jacek Laskowski [mailto:ja...@japila.pl]
Sent: Thursday, April 27, 2017 8:39 AM
To: Lavelle, Shawn
Cc: user
Subject: Re: Spark-SQL Query Optimization: overlapping ranges
Hi Shawn,
If
sort of thing. We’re probably going to write our own
> org.apache.spark.sql.catalyst.rules.Rule to handle it.
>
> ~ Shawn
>
>
>
> *From:* Jacek Laskowski [mailto:ja...@japila.pl]
> *Sent:* Wednesday, April 26, 2017 2:55 AM
> *To:* Lavelle, Shawn
> *Cc:* user
> *Subject:* Re: Spark-SQL Q
thing. We’re probably going to write our own
org.apache.spark.sql.catalyst.rules.Rule to handle it.
~ Shawn
From: Jacek Laskowski [mailto:ja...@japila.pl]
Sent: Wednesday, April 26, 2017 2:55 AM
To: Lavelle, Shawn
Cc: user
Subject: Re: Spark-SQL Query Optimization: overlapping ranges
explain it
explain it and you'll know what happens under the covers.
i.e. Use explain on the Dataset.
Jacek
On 25 Apr 2017 12:46 a.m., "Lavelle, Shawn" wrote:
> Hello Spark Users!
>
>Does the Spark Optimization engine reduce overlapping column ranges?
> If so, should it push this down to a Data Sourc
Hello Spark Users!
Does the Spark Optimization engine reduce overlapping column ranges? If so,
should it push this down to a Data Source?
Example,
This: Select * from table where col between 3 and 7 OR col between 5 and 9
Reduces to: Select * from table where col between 3 and 9
