etter solution for you.
>
> Michael
>
>
>
>
> On Jan 17, 2017, at 8:59 PM, Raju Bairishetti wrote:
>
> Tested on both 1.5.2 and 1.61.
>
> On Wed, Jan 18, 2017 at 12:52 PM, Michael Allman
> wrote:
>
>> What version of Spark are you running?
>>
&g
Tested on both 1.5.2 and 1.61.
On Wed, Jan 18, 2017 at 12:52 PM, Michael Allman
wrote:
> What version of Spark are you running?
>
> On Jan 17, 2017, at 8:42 PM, Raju Bairishetti wrote:
>
> describe dummy;
>
> OK
>
> sample string
>
> yea
*/month=9,
maprfs:/user/rajub/dummy/sample/year=2017/month=10,
maprfs:/user/rajub/dummy/sample/year=2017/month=11,
maprfs:/user/rajub/dummy/sample/year=2017/month=9
On Wed, Jan 18, 2017 at 12:25 PM, Michael Allman
wrote:
> Can you paste the actual query plan here, please?
>
> On Jan 17, 20
On Wed, Jan 18, 2017 at 11:13 AM, Michael Allman
wrote:
> What is the physical query plan after you set
> spark.sql.hive.convertMetastoreParquet
> to true?
>
Physical plan continas all the partition locations
>
> Michael
>
> On Jan 17, 2017, at 6:51 PM, Raju Bairis
see? If not, run "msck repair
> table .".
>
Yes. It is listing the partitions
> Cheers,
>
> Michael
>
>
> On Jan 17, 2017, at 12:02 AM, Raju Bairishetti wrote:
>
> Had a high level look into the code. Seems getHiveQlPartitions method
> from HiveMetastoreC
] =
client.getPartitionsByFilter(this, predicates)
lazy val allPartitions = table.getAllPartitions
But somehow getAllPartitions is getting called eventough after setting
metastorePartitionPruning to true.
Am I missing something or looking at wrong place?
On Tue, Jan 17, 2017 at 4:01 PM, Raju Bairishetti wrote
at 4:00 PM, Raju Bairishetti wrote:
> Had a high level look into the code. Seems getHiveQlPartitions method
> from HiveMetastoreCatalog is getting called irrespective of
> metastorePartitionPruning
> conf value.
>
> It should not fetch all partitions if we set metastorePart
