Alexander Trushev created FLINK-24139:
-----------------------------------------
Summary: Push down more predicates through Join in stream mode
Key: FLINK-24139
URL: https://issues.apache.org/jira/browse/FLINK-24139
Project: Flink
Issue Type: Improvement
Components: Table SQL / Planner
Reporter: Alexander Trushev
Attachments: q13_after.json, q13_after.png, q13_after.txt,
q13_before.json, q13_before.png, q13_before.txt
h3. Context
Rule {{JoinDependentConditionDerivationRule}} is introduced in FLINK-12509.
This rule rewrites join condition in such way that more predicates can be
pushed down through join. For example,
# Source A = [a0, a1, a2], source B = [b0, b1]
#
{code:sql}
select * from A join B on a0 = b0 where (a1 = 0 and b1 = 0) or a2 = 0{code}
# {{JoinDependentConditionDerivationRule}} transforms condition ((a1 and b1)
or a2) to (((a1 and b1) or a2) and (a1 or a2))
# {{JoinConditionPushRule}} pushes (a1 or a2) to A source
It is a good optimization that can lead to performance improvement of query
execution.
Currently, {{JoinDependentConditionDerivationRule}} is used only in batch mode.
h3. Proposal
Enable {{JoinDependentConditionDerivationRule}} in stream mode.
h3. Benefit
Experiment based on [https://github.com/ververica/flink-sql-benchmark]
Cluster – 4 nodes each 2 slots
Dataset – tpcds_bin_orc_20
Before – 1.14.0-rc0
After – 1.14.0-rc0 + patched {{FlinkStreamProgram}} including
{{JoinDependentConditionDerivationRule}}
||TPC-DS 20 GB||Before||After||
|q13 stream mode|83 s|8 s|
Query plan, stream graph, dashboard visualization before and after the patch
are in the attachment
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
