[ https://issues.apache.org/jira/browse/FLINK-16860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jingsong Lee resolved FLINK-16860. ---------------------------------- Resolution: Fixed master: c8a23c74e618b752bbdc58dca62d997ddd303d40 release-1.10: 9d21a4a5539e6fe0b253def255d6641335da18df release-1.9: 78e2c0de6ef21e497e82de4a180759a960d11eab > TableException: Failed to push filter into table source! when upgrading flink > to 1.9.2 > -------------------------------------------------------------------------------------- > > Key: FLINK-16860 > URL: https://issues.apache.org/jira/browse/FLINK-16860 > Project: Flink > Issue Type: Bug > Components: Connectors / ORC, Table SQL / API > Affects Versions: 1.9.2, 1.10.0 > Environment: flink 1.8.2 > flink 1.9.2 > Reporter: Nikola > Assignee: Jingsong Lee > Priority: Major > Labels: pull-request-available > Fix For: 1.9.3, 1.10.1 > > Attachments: flink-1.8.2.txt, flink-1.9.2.txt > > Time Spent: 1h > Remaining Estimate: 0h > > We have a batch job which we currently have on a flink cluster running 1.8.2 > The job runs fine. We wanted to upgrade to flink 1.10, but that yielded > errors, so we started downgrading until we found that the issue is in flink > 1.9.2 > The job on 1.9.2 fails with: > {code:java} > Caused by: org.apache.flink.table.api.TableException: Failed to push filter > into table source! table source with pushdown capability must override and > change explainSource() API to explain the pushdown applied!{code} > Which is not happening on flink 1.8.2. You can check the logs for the exactly > same job, just running on different cluster versions: [^flink-1.8.2.txt] > [^flink-1.9.2.txt] > > I tried to narrow it down and it seems that this exception has been added in > FLINK-12399 and there was a small discussion regarding the exception: > [https://github.com/apache/flink/pull/8468#discussion_r329876088] > Our code looks something like this: > > {code:java} > String tempTableName = "tempTable"; > String sql = SqlBuilder.buildSql(tempTableName); > BatchTableEnvironment tableEnv = BatchTableEnvironment.create(env); > OrcTableSource orcTableSource = OrcTableSource.builder() > .path(hdfsFolder, true) > .forOrcSchema(ORC.getSchema()) > .withConfiguration(config) > .build(); > tableEnv.registerTableSource(tempTableName, orcTableSource); > Table tempTable = tableEnv.sqlQuery(sql); > return tableEnv.toDataSet(tempTable, Row.class); > {code} > Where the sql build is nothing more than > {code:java} > SELECT * FROM table WHERE id IN (1,2,3) AND mid IN(4,5,6){code} > -- This message was sent by Atlassian Jira (v8.3.4#803005)