Re: Question on Spark 1.3 SQL External Datasource

2015-03-17 Thread Yang Lei
Thanks Cheng for the clarification. Looking forward to this new API mentioned below. Yang Sent from my iPad > On Mar 17, 2015, at 8:05 PM, Cheng Lian wrote: > > Hey Yang, > > My comments are in-lined below. > > Cheng > >> On 3/18/15 6:53 AM, Yang Lei wrote: >> Hello, >> >> I am migrat

Re: Question on Spark 1.3 SQL External Datasource

2015-03-17 Thread Cheng Lian
Hey Yang, My comments are in-lined below. Cheng On 3/18/15 6:53 AM, Yang Lei wrote: Hello, I am migrating my Spark SQL external datasource integration from Spark 1.2.x to Spark 1.3. I noticed, there are a couple of new filters now, e.g. org.apache.spark.sql.sources.And. However, for a sq

Question on Spark 1.3 SQL External Datasource

2015-03-17 Thread Yang Lei
Hello, I am migrating my Spark SQL external datasource integration from Spark 1.2.x to Spark 1.3. I noticed, there are a couple of new filters now, e.g. org.apache.spark.sql.sources.And. However, for a sql with condition "A AND B", I noticed PrunedFilteredScan.buildScan still gets an Array[Filte