Hello,
I try to configure SQL Client to query partitioned ORC data on local
filesystem. I have directory structure like that:
/tmp/table1/startdate=2020-11-28
/tmp/table1/startdate=2020-11-27
etc.
If I run SQL Client session and create table by hand:
create table tst (column1 string, startdate string) partitioned by
(startdate) with ('connector'='filesystem', 'format'='orc',
'path'='/tmp/table1');
everything runs fine:
explain select * from tst where startdate='2020-11-27'
shows that only one partition in 'readPartitions'
However, I struggle to configure table in .yaml config.
I tried like this (after some struggle, as "partition.keys" setting
doesn't seem to be documented...) :
tables:
- name: tst2
type: source-table
connector: filesystem
path: "/tmp/table1"
format: orc
partition.keys:
- name: startdate
schema:
- name: column1
data-type: string
- name: startdate
data-type: string
and it more or less works - queries are executed properly. However,
partitions are not pruned:
explain select * from tst2 where startdate='2020-11-27'
show all partitions in 'readPartitions'
Any idea what can be wrong? I'm using Flink 1.11.2
thanks,
maciek