I filed SPARK-10703. Thanks!
Mingyu
From: Reynold Xin
Date: Thursday, September 17, 2015 at 11:22 PM
To: Mingyu Kim
Cc: Zack Sampson, "dev@spark.apache.org", Peter Faiman, Matt Cheah, Michael
Armbrust
Subject: Re: And.eval short circuiting
Please file a ticket and cc me. Thanks.
> Mingyu
>
> From: Reynold Xin
> Date: Wednesday, September 16, 2015 at 1:17 PM
> To: Zack Sampson
> Cc: "dev@spark.apache.org", Mingyu Kim, Peter Faiman, Matt Cheah, Michael
> Armbrust
>
> Subject: Re: And.eval short circuiting
>
> This is "expected&
Kim, Peter Faiman, Matt Cheah, Michael
Armbrust
Subject: Re: And.eval short circuiting
This is "expected" in the sense that DataFrame operations can get re-ordered
under the hood by the optimizer. For example, if the optimizer deems it is
cheaper to apply the 2nd filter first, it
o the second filter. Even weirder is that if you call collect() after the
> first filter you won't see nulls, and if you write the data to disk and
> reread it, the NPE won't happen.
>
> It's bewildering! Is this the intended behavior?
> --------
__
From: Reynold Xin [r...@databricks.com]
Sent: Monday, September 14, 2015 10:14 PM
To: Zack Sampson
Cc: dev@spark.apache.org
Subject: Re: And.eval short circuiting
rxin=# select null and true;
?column?
--
(1 row)
rxin=# select null and false;
?column?
--
rxin=# select null and true;
?column?
--
(1 row)
rxin=# select null and false;
?column?
--
f
(1 row)
null and false should return false.
On Mon, Sep 14, 2015 at 9:12 PM, Zack Sampson wrote:
> It seems like And.eval can avoid calculating right.eval if left.eval
> returns n
It seems like And.eval can avoid calculating right.eval if left.eval returns
null. Is there a reason it's written like it is?
override def eval(input: Row): Any = {
val l = left.eval(input)
if (l == false) {
false
} else {
val r = right.eval(input)
if (r == false) {
false
