Hi,
Timestamp range filter queries in SQL are not getting pushed down to the
PrunedFilteredScan instances. The filtering is happening at the Spark layer.
The physical plan for timestamp range queries is not showing the pushed
filters where as range queries on other types is working fine as the
physical plan is showing the pushed filters.
Please see below for code and examples.
*Example:*
*1.* Range filter queries on Timestamp types
*code: *
> sqlContext.sql("SELECT * from events WHERE `registration` >= '2015-05-28'
> AND `registration` <= '2015-05-29' ")
*Full example*:
https://github.com/lucidworks/spark-solr/blob/master/src/test/scala/com/lucidworks/spark/EventsimTestSuite.scala#L151
* plan*:
https://gist.github.com/kiranchitturi/4a52688c9f0abe3d4b2bd8b938044421#file-time-range-sql
*2. * Range filter queries on Long types
*code*:
> sqlContext.sql("SELECT * from events WHERE `length` >= '700' and `length`
> <= '1000'")
*Full example*:
https://github.com/lucidworks/spark-solr/blob/master/src/test/scala/com/lucidworks/spark/EventsimTestSuite.scala#L151
*plan*:
https://gist.github.com/kiranchitturi/4a52688c9f0abe3d4b2bd8b938044421#file-length-range-sql
The SolrRelation class we use extends
<https://github.com/lucidworks/spark-solr/blob/master/src/main/scala/com/lucidworks/spark/SolrRelation.scala#L37>
the PrunedFilteredScan.
Since Solr supports date ranges, I would like for the timestamp filters to
be pushed down to the Solr query.
Are there limitations on the type of filters that are passed down with
Timestamp types ?
Is there something that I should do in my code to fix this ?
Thanks,
--
Kiran Chitturi
