Re: Using Spark SQL for temporal data

> > I haven't been paying close attention to the JIRA tickets for > PrunedFilteredScan but I noticed some weird behavior around the filters > being applied when OR expressions were used in the WHERE clause. From what > I was seeing, it looks like it could be possible that the "start" and "end" > ra

Re: Using Spark SQL for temporal data

Ok. I just verified that this is the case with a little test: WHERE (a = 'v1' and b = 'v2')PrunedFilteredScan passes down 2 filters WHERE(a = 'v1' and b = 'v2') or (a = 'v3') PrunedFilteredScan passes down 0 filters On Fri, Feb 13, 2015 at 12:28 AM, Corey Nolet wrote: > Michael, > > I haven

Re: Using Spark SQL for temporal data

Michael, I haven't been paying close attention to the JIRA tickets for PrunedFilteredScan but I noticed some weird behavior around the filters being applied when OR expressions were used in the WHERE clause. From what I was seeing, it looks like it could be possible that the "start" and "end" rang

Re: Using Spark SQL for temporal data

Hi Corey, I would not recommend using the CatalystScan for this. Its lower level, and not stable across releases. You should be able to do what you want with PrunedFilteredScan

Using Spark SQL for temporal data

I have a temporal data set in which I'd like to be able to query using Spark SQL. The dataset is actually in Accumulo and I've already written a CatalystScan implementation and RelationProvider[1] to register with the SQLContext so that I can apply my SQL statements. With my current implementation