[
https://issues.apache.org/jira/browse/FLINK-37160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhu Zhu updated FLINK-37160:
----------------------------
Description:
In Flink 2.0, we support the capability of adaptive broadcast join for batch
jobs, which can determine at runtime whether the small table can be broadcast
based on the actual input data size of the join operator. It then transforms
the operators that meet the condition to achieve better join performance.
We may need the following tests:
1. Test the case where `table.optimizer.adaptive-broadcast-join.strategy` is
set to 'auto', we may need to construct a case (e.g. a case where the source
includes a downstream filter operator) such that the join operator is not
optimized as a broadcast hash join during the table compilation phase but can
be dynamically optimized as a broadcast hash join at runtime.
2. Test the case where `table.optimizer.adaptive-broadcast-join.strategy` is
set to 'runtime_only', to verify that the join operator is not optimized into a
broadcast hash join during the table compilation phase, even if the small table
side meets the broadcast threshold, while also being able to dynamically
convert into a broadcast hash join at runtime.
3. Test the case where `table.optimizer.adaptive-broadcast-join.strategy` as
none, and verify that the join operator will not be optimized into a broadcast
join operator under any circumstances.
was:
In Flink 2.0, we support the capability of adaptive broadcast join for batch
jobs, which can determine at runtime whether the small table can be broadcast
based on the actual input data size of the join operator. It then transforms
the operators that meet the condition to achieve better join performance.
We may need the following tests:
1. Test the case where `table.optimizer.adaptive-broadcast-join.strategy` is
set to 'auto', we may need to construct a case (e.g. a case where the source
includes a downstream filter operator) such that the join operator is not
optimized as a broadcast hash join during the table compilation phase but can
be dynamically optimized as a broadcast hash join at runtime.
2. Test the case where `table.optimizer.adaptive-broadcast-join.strategy` is
set to 'runtime_only', to verify that the join operator is not optimized into a
broadcast hash join during the table compilation phase, even if the small table
side meets the broadcast threshold, while also being able to dynamically
convert into a broadcast hash join at runtime.
3. Test the case where `table.optimizer.adaptive-broadcast-join.strategy` as
none, and verify that the join operator will not be optimized into an adaptive
join operator under any circumstances.
> Cross-team verification for "Adaptive broadcast join for batch jobs"
> --------------------------------------------------------------------
>
> Key: FLINK-37160
> URL: https://issues.apache.org/jira/browse/FLINK-37160
> Project: Flink
> Issue Type: Sub-task
> Reporter: Junrui Lee
> Assignee: Lei Yang
> Priority: Blocker
> Fix For: 2.0.0
>
>
> In Flink 2.0, we support the capability of adaptive broadcast join for batch
> jobs, which can determine at runtime whether the small table can be broadcast
> based on the actual input data size of the join operator. It then transforms
> the operators that meet the condition to achieve better join performance.
> We may need the following tests:
> 1. Test the case where `table.optimizer.adaptive-broadcast-join.strategy` is
> set to 'auto', we may need to construct a case (e.g. a case where the source
> includes a downstream filter operator) such that the join operator is not
> optimized as a broadcast hash join during the table compilation phase but can
> be dynamically optimized as a broadcast hash join at runtime.
> 2. Test the case where `table.optimizer.adaptive-broadcast-join.strategy` is
> set to 'runtime_only', to verify that the join operator is not optimized into
> a broadcast hash join during the table compilation phase, even if the small
> table side meets the broadcast threshold, while also being able to
> dynamically convert into a broadcast hash join at runtime.
> 3. Test the case where `table.optimizer.adaptive-broadcast-join.strategy` as
> none, and verify that the join operator will not be optimized into a
> broadcast join operator under any circumstances.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
