Hi Maciek,
I am pulling in Timo who might help you with this problem.
Cheers,
Till
On Tue, Dec 1, 2020 at 6:51 PM Maciek Próchniak <m...@touk.pl> wrote:
> Hello,
>
> I try to configure SQL Client to query partitioned ORC data on local
> filesystem. I have directory structure like that:
>
> /tmp/table1/startdate=2020-11-28
>
> /tmp/table1/startdate=2020-11-27
>
> etc.
>
>
> If I run SQL Client session and create table by hand:
>
> create table tst (column1 string, startdate string) partitioned by
> (startdate) with ('connector'='filesystem', 'format'='orc',
> 'path'='/tmp/table1');
>
> everything runs fine:
>
> explain select * from tst where startdate='2020-11-27'
>
> shows that only one partition in 'readPartitions'
>
>
> However, I struggle to configure table in .yaml config.
>
> I tried like this (after some struggle, as "partition.keys" setting
> doesn't seem to be documented...) :
>
> tables:
> - name: tst2
> type: source-table
> connector: filesystem
> path: "/tmp/table1"
> format: orc
> partition.keys:
> - name: startdate
> schema:
> - name: column1
> data-type: string
> - name: startdate
> data-type: string
>
> and it more or less works - queries are executed properly. However,
> partitions are not pruned:
>
> explain select * from tst2 where startdate='2020-11-27'
>
> show all partitions in 'readPartitions'
>
>
> Any idea what can be wrong? I'm using Flink 1.11.2
>
>
> thanks,
>
> maciek
>
>
>
