Hello,

I try to configure SQL Client to query partitioned ORC data on local filesystem. I have directory structure like that:

/tmp/table1/startdate=2020-11-28

/tmp/table1/startdate=2020-11-27

etc.


If I run SQL Client session and create table by hand:

create table tst (column1 string, startdate string) partitioned by (startdate) with ('connector'='filesystem', 'format'='orc', 'path'='/tmp/table1');

everything runs fine:

explain select * from tst where startdate='2020-11-27'

shows that only one partition in 'readPartitions'


However, I struggle to configure table in .yaml config.

I tried like this (after some struggle, as "partition.keys" setting doesn't seem to be documented...) :

tables:
  - name: tst2
    type: source-table
    connector: filesystem
    path: "/tmp/table1"
    format: orc
    partition.keys:
      - name: startdate
    schema:
      - name: column1
        data-type: string
      - name: startdate
        data-type: string

and it more or less works - queries are executed properly. However, partitions are not pruned:

explain select * from tst2 where startdate='2020-11-27'

show all partitions in 'readPartitions'


Any idea what can be wrong? I'm using Flink 1.11.2


thanks,

maciek


Reply via email to