Re: Error while reading hive tables with tmp/hidden files inside partitions

2020-04-23 Thread Wenchen Fan
Yea, please report the bug on a supported Spark version like 2.4. On Thu, Apr 23, 2020 at 3:40 PM Dhrubajyoti Hati wrote: > FYI we are using Spark 2.2.0. Should the change be present in this spark > version? Wanted to check before opening a JIRA ticket? > > > > > *Regards,Dhrubajyoti Hati.* > >

Re: Error while reading hive tables with tmp/hidden files inside partitions

2020-04-23 Thread Dhrubajyoti Hati
FYI we are using Spark 2.2.0. Should the change be present in this spark version? Wanted to check before opening a JIRA ticket? *Regards,Dhrubajyoti Hati.* On Thu, Apr 23, 2020 at 10:12 AM Wenchen Fan wrote: > This looks like a bug that path filter doesn't work for hive table > reading. Can

Re: Error while reading hive tables with tmp/hidden files inside partitions

2020-04-22 Thread Wenchen Fan
This looks like a bug that path filter doesn't work for hive table reading. Can you open a JIRA ticket? On Thu, Apr 23, 2020 at 3:15 AM Dhrubajyoti Hati wrote: > Just wondering if any one could help me out on this. > > Thank you! > > > > > *Regards,Dhrubajyoti Hati.* > > > On Wed, Apr 22, 2020 a

Re: Error while reading hive tables with tmp/hidden files inside partitions

2020-04-22 Thread Dhrubajyoti Hati
Just wondering if any one could help me out on this. Thank you! *Regards,Dhrubajyoti Hati.* On Wed, Apr 22, 2020 at 7:15 PM Dhrubajyoti Hati wrote: > Hi, > > Is there any way to discard files starting with dot(.) or ending with .tmp > in the hive partition while reading from Hive table usi