Inconsistent results from seqscan and gist indexscan

[Richard Guo]

Here is how it can be reproduced.

create table point_tbl (f1 point);

insert into point_tbl(f1) values ('(5.1, 34.5)');
insert into point_tbl(f1) values (' ( Nan , NaN ) ');
analyze;

create index gpointind on point_tbl using gist (f1);

set enable_seqscan to on;
set enable_indexscan to off;
# select * from point_tbl where f1 <@ polygon
'(0,0),(0,100),(100,100),(100,0),(0,0)';
     f1
------------
 (5.1,34.5)
 (NaN,NaN)
(2 rows)


set enable_seqscan to off;
set enable_indexscan to on;
# select * from point_tbl where f1 <@ polygon
'(0,0),(0,100),(100,100),(100,0),(0,0)';
     f1
------------
 (5.1,34.5)
(1 row)

Seems point_inside() does not handle NaN properly.

BTW, I'm using 15devel. But this issue can be seen in at least 12
version also.

Thanks
Richard