Hi everyone, Look up join <https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/queries/joins/#lookup-join>[1] is commonly used feature in Flink SQL. We have received many optimization requirements on look up join. For example: 1. Enforces left side of lookup join do a hash partitioner to raise cache hint ratio 2. Solves the data skew problem after introduces hash lookup join 3. Enables mini-batch optimization to reduce RPC call
Next we will solve these problems one by one. Firstly, we would focus on point 1, and continue to discuss point 2 and point 3 later. There are many similar requirements from user mail list and JIRA about hash Lookup Join, for example: 1. FLINK-23687 <https://issues.apache.org/jira/browse/FLINK-23687> - Introduce partitioned lookup join to enforce input of LookupJoin to hash shuffle by lookup keys 2. FLINK-25396 <https://issues.apache.org/jira/browse/FLINK-25396> - lookupjoin source table for pre-partitioning 3. FLINK-25262 <https://issues.apache.org/jira/browse/FLINK-25262> - Support to send data to lookup table for KeyGroupStreamPartitioner way for SQL. In this FLIP, I would like to start a discussion about Hash Lookup Join. The core idea is introducing a 'USE_HASH' hint in query. This syntax is directly user-oriented and therefore requires careful design. There are two ways about how to propagate this hint to LookupJoin in optimizer. We need further discussion to do final decide. Anyway, the difference between the two solution is only about the internal implementation and has no impact on the user. For more detail on the proposal: https://cwiki.apache.org/confluence/display/FLINK/FLIP-204%3A+Introduce+Hash+Lookup+Join Looking forward to your feedback, thanks. Best, Jing Zhang [1] https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/queries/joins/#lookup-join
