Re: SPARK SQL- Parquet projection pushdown for nested data

2015-10-29 Thread Sadhan Sood
Thanks Michael, I will upvote this. On Thu, Oct 29, 2015 at 10:29 AM, Michael Armbrust wrote: > Yeah, this is unfortunate. It would be good to fix this, but its a > non-trivial change. > > Tracked here if you'd like to vote on the issue: > https://issues.apache.org/jira/browse/SPARK-4502 > > On

Re: SPARK SQL- Parquet projection pushdown for nested data

2015-10-29 Thread Michael Armbrust
Yeah, this is unfortunate. It would be good to fix this, but its a non-trivial change. Tracked here if you'd like to vote on the issue: https://issues.apache.org/jira/browse/SPARK-4502 On Thu, Oct 29, 2015 at 6:00 PM, Sadhan Sood wrote: > I noticed when querying struct data in spark sql, we ar

SPARK SQL- Parquet projection pushdown for nested data

2015-10-29 Thread Sadhan Sood
I noticed when querying struct data in spark sql, we are requesting the whole column from parquet files. Is this intended or is there some kind of config to control this behaviour? Wouldn't it be better to request just the struct field?