Hi Rascal,
Yes, you got the right answer for these two questions. If you have
more questions about using Flink, welcome mailing to user support and
questions mailing list: u...@flink.apache.org / user...@flink.apache.org (
https://flink.apache.org/community.html#mailing-lists). This dev mailing
list is mainly for development related discussions.
Best,
Lincoln Lee
Rascal Wu <stayras...@gmail.com> 于2022年8月11日周四 13:18写道：
> Thanks Lincoln a lot for your quick response and clear explanation.
>
> It's clear now. So regarding "*For the retry support in FLIP-234, it only
> provide a 'lookup_miss' retry predicate which is to retry if can not find
> it*" , try to double confirm again:
> 1) if the dim record doesn't exist in dim table after retry finished, the
> output will be +[order1, id1, null], if the dim record coming in dim table
> before last retry, the output will be +[order1, id1, xxx], right? (assume
> left join here)
> 2) And if I expect the latest event (+[order1, id1, 12]) update the
> previous event (+[order1, id1, 10]), I should use regular join instead of
> lookup join, right?
>
> Lincoln Lee <lincoln.8...@gmail.com> 于2022年8月11日周四 12:01写道：
>
> > @Rascal Thanks for looking at this new feature! There is a lot of
> content
> > in these two flips and we will prepare a more detailed user documentation
> > before the 1.16 release.
> >
> > First of all, unlike regular join, the lookup join only triggers the
> lookup
> > action (access the dimension table) through the records of the stream
> > table, and the updates of the dimension table itself do not actively
> > trigger the updating results to
> > downstream.
> >
> > For your example, whether retry is enabled or not, only the first
> > "+[order1, id1, 10]" will sent to downstream (because only one order
> record
> > has come), the new updates of dimension table "10:02 -> (id1, 12)" will
> not
> > trigger updates.
> >
> > For the retry support in FLIP-234, it only provide a 'lookup_miss' retry
> > predicate which is to retry if can not find it (not always equivalent to
> > the complete join condition [1]), and not to trigger the retry if can
> find
> > a non-null value from the dimension table. If more complex value check
> > logic required, a viable way is to implement a custom
> `AsyncRetryPredicate`
> > in DataStream API (as FLIP-232 introduced).
> >
> > [1]: for different connector, the index-lookup capability maybe
> different,
> > e.g., HBase can lookup on rowkey only (by default, without secondary
> > index), while RDBMS can provided more powerful index-lookup capabilities
> > let's see the lookup join example from flink document(
> >
> >
> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/queries/joins/#lookup-join
> > )
> >
> >
> > if the query with a join condition "ON o.customer_id = c.id and
> c.country
> > =
> > 'US'"
> >
> > -- enrich each order with customer informationSELECT o.order_id,
> > o.total, c.country, c.zipFROM Orders AS o JOIN Customers FOR
> > SYSTEM_TIME AS OF o.proc_time AS c ON o.customer_id = c.id and
> > c.country = 'US';
> >
> > for the dimension table in mysql, all of the columns (id & country) can
> be
> > used as index-lookup condition
> >
> > -- Customers is backed by the JDBC connector and can be used for
> > lookup joinsCREATE TEMPORARY TABLE Customers ( id INT, name STRING,
> > country STRING, zip STRING) WITH ( 'connector' = 'jdbc', 'url' =
> > 'jdbc:mysql://mysqlhost:3306/customerdb', 'table-name' =
> > 'customers');
> >
> >
> >
> > while if the same table stored in HBase (with no secondary index), only
> the
> > 'id' column (rowkey in HBase) can be the index-lookup condition
> >
> > -- Customers is backed by the JDBC connector and can be used for
> > lookup joinsCREATE TEMPORARY TABLE Customers ( id INT, name STRING,
> > country STRING, zip STRING,
> > PRIMARY KEY (id) NOT ENFORCED) WITH ( 'connector' = 'hbase-xxx',
> > ...);
> >
> >
> > so, the 'lookup_miss' retry predicte may result differently in different
> > connectors.
> >
> > wish this can helps.
> >
> >
> > Best,
> > Lincoln Lee
> >
> >
> > Rascal Wu <stayras...@gmail.com> 于2022年8月11日周四 10:32写道：
> >
> > > Hi here,
> > >
> > > Sorry for digging up this old email.
> > >
> > > May I consult one question about what's the final behavior we expected
> > > after we enabled retryable mechanism. I viewed the FLIP-232 & FLIP-234,
> > but
> > > I didn't get a clear answer as lacking of some knowledge.
> > >
> > > Let me take a example there, let's assume we have a dim table(id,
> price),
> > > the record(id1) existed in dim table before 10:00, and then it was
> > updated
> > > in business system at 10:01, but delay updated in dim table at 10:02.
> the
> > > record timeline as follow:
> > > 10:00 -> (id1, 10)
> > > 10:00 -> the id1 records was been updated, but delayed persistency or
> > sync
> > > to dim table.
> > > 10:02 -> (id1, 12)
> > >
> > > And a Flink application processes an order record that will join the
> dim
> > > table at 10:01, so it will output an event +[order1, id1, 10].
> > > And if we enable retry mechanism no matter it's sync or async, does it
> > mean
> > > there will be two events sink to downstream:
> > > 1. retract event: -[order1, id1, 10]
> > > 2. new event: +[order1, id1, 12]
> > >
> > > does the above all behavior is what we expected to solve the delay
> update
> > > for dimension table?
> > >
> > >
> > > Lincoln Lee <lincoln.8...@gmail.com> 于2022年6月7日周二 12:19写道：
> > >
> > > > Hi everyone,
> > > >
> > > > I started a vote for this FLIP [1], please vote there or ask
> additional
> > > > questions here. [2]
> > > >
> > > > [1] https://lists.apache.org/thread/bb0kqjs8co3hhmtklmwptws4fc4rz810
> > > > [2] https://lists.apache.org/thread/9k1sl2519kh2n3yttwqc00p07xdfns3h
> > > >
> > > >
> > > > Best,
> > > > Lincoln Lee
> > > >
> > > >
> > > > Lincoln Lee <lincoln.8...@gmail.com> 于2022年6月2日周四 15:51写道：
> > > >
> > > > > Hi everyone,
> > > > >
> > > > > When reviewing the name of the hint option
> > 'miss-retry'='true|false', I
> > > > > feel the current name is not precise enough, it might be easier to
> > > > > understand by using the retry-predicate directly from flip-232,
> > > > > e.g. 'retry-predicate'='lookup-miss', which has the additional
> > benefit
> > > of
> > > > > extensibility(maybe more retry condition in the future).
> > > > >
> > > > > Jark & Jingsong, do you have any suggestions? If we agree with the
> > name
> > > > > 'retry-predicate' or other better candidate, I'll update the FLIP.
> > > > >
> > > > > Best,
> > > > > Lincoln Lee
> > > > >
> > > > >
> > > > > Lincoln Lee <lincoln.8...@gmail.com> 于2022年6月2日周四 11:23写道：
> > > > >
> > > > >> Hi everyone,
> > > > >>
> > > > >> I've updated the FLIP[1] based on this discussion thread that we
> > agree
> > > > to
> > > > >> have a single unified 'LOOKUP' hint and also a related part in
> > > > FLIP-221[2]
> > > > >> which is mainly for the necessity of the common table option
> > > > >> 'lookup.async'.
> > > > >>
> > > > >> The main updates are:
> > > > >> 1. the new unified 'LOOKUP' hint, make retry support both on sync
> > and
> > > > >> async lookup
> > > > >> 2. clarify the default choice of the planner for those connectors
> > > which
> > > > >> have both sync and async lookup capabilities, and how to deal with
> > the
> > > > >> query hint
> > > > >> 3. will add a followup issue to discuss whether to remove the
> > > > >> 'lookup.async' option in HBase connector.
> > > > >>
> > > > >> [1]
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-234%3A+Support+Retryable+Lookup+Join+To+Solve+Delayed+Updates+Issue+In+External+Systems
> > > > >> [2]
> > https://lists.apache.org/thread/1vokqdnnt01yycl7y1p74g556cc8yvtq
> > > > >>
> > > > >> Best,
> > > > >> Lincoln Lee
> > > > >>
> > > > >>
> > > > >> Lincoln Lee <lincoln.8...@gmail.com> 于2022年6月1日周三 16:03写道：
> > > > >>
> > > > >>> Hi Jingsong,
> > > > >>>
> > > > >>> There will be no change for connectors with only one capability
> > (sync
> > > > or
> > > > >>> async).
> > > > >>>
> > > > >>> Query hint works in a best effort manner, so if users specifies a
> > > hint
> > > > >>> with invalid option, the query plan keeps unchanged, e.g., use
> > > > >>> LOOKUP('table'='customer', 'async'='true'), but backend lookup
> > source
> > > > only
> > > > >>> implemented the sync lookup function, then the async lookup hint
> > > takes
> > > > no
> > > > >>> effect.
> > > > >>>
> > > > >>> For these connectors which can have both capabilities of async
> and
> > > sync
> > > > >>> lookup, our advice for the connector developer is implementing
> both
> > > > sync
> > > > >>> and async interfaces if both capabilities have suitable use
> cases,
> > > and
> > > > the
> > > > >>> planner can decide which capability is the preferable one based
> on
> > > cost
> > > > >>> model or maybe other mechanism (another use case is exactly what
> > > we're
> > > > >>> discussing here, users can give the query hint), otherwise choose
> > one
> > > > >>> interface to implement.
> > > > >>>
> > > > >>> Also, this should be clarified for the lookup function related
> > APIs.
> > > > >>>
> > > > >>> Best,
> > > > >>> Lincoln Lee
> > > > >>>
> > > > >>>
> > > > >>> Jingsong Li <jingsongl...@gmail.com> 于2022年6月1日周三 15:18写道：
> > > > >>>
> > > > >>>> Hi Lincoln,
> > > > >>>>
> > > > >>>> > It's better making decisions at the query level when a
> connector
> > > has
> > > > >>>> both
> > > > >>>> capabilities.
> > > > >>>>
> > > > >>>> Can you clarify the mechanism?
> > > > >>>> - only sync connector: What connector developers should do
> > > > >>>> - only async connector: What connector developers should do
> > > > >>>> - both async and sync connector: What connector developers
> should
> > do
> > > > >>>>
> > > > >>>> Best,
> > > > >>>> Jingsong
> > > > >>>>
> > > > >>>> On Wed, Jun 1, 2022 at 2:29 PM Lincoln Lee <
> > lincoln.8...@gmail.com>
> > > > >>>> wrote:
> > > > >>>>
> > > > >>>> > Hi Jingsong,
> > > > >>>> >
> > > > >>>> > Thanks for your feedback!
> > > > >>>> >
> > > > >>>> > Yes, the existing HBase connector use an option 'lookup.async'
> > to
> > > > >>>> control
> > > > >>>> > its lookup source implementations that exposed to the planner,
> > > > >>>> however it's
> > > > >>>> > a private option for the HBase connector only, so it will not
> > > affect
> > > > >>>> the
> > > > >>>> > common API.
> > > > >>>> >
> > > > >>>> > And as discussed in the mailing thread of FLIP-221[1], we got
> a
> > > > >>>> consensus
> > > > >>>> > that do not make it as a common option. It's better making
> > > decisions
> > > > >>>> at the
> > > > >>>> > query level when a connector has both capabilities.
> > > > >>>> >
> > > > >>>> > So if everything goes well, we should discuss it whether to
> > > > deprecate
> > > > >>>> > the 'lookup.async'
> > > > >>>> > or not for HBase connector after the hint been done.
> > > > >>>> >
> > > > >>>> > This will be mentioned in the Compatibility part of this
> > FLIP[2].
> > > > >>>> >
> > > > >>>> > WDYT?
> > > > >>>> >
> > > > >>>> > [1]:
> > > > https://lists.apache.org/thread/v76g8v1o9sjdho9kbzlgjyv38l2oynox
> > > > >>>> > [2]:
> > > > >>>> >
> > > > >>>> >
> > > > >>>>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-234%3A+Support+Retryable+Lookup+Join+To+Solve+Delayed+Updates+Issue+In+External+Systems?src=contextnavpagetreemode
> > > > >>>> >
> > > > >>>> > Best,
> > > > >>>> > Lincoln Lee
> > > > >>>> >
> > > > >>>> >
> > > > >>>> > Jingsong Li <jingsongl...@gmail.com> 于2022年6月1日周三 14:11写道：
> > > > >>>> >
> > > > >>>> > > Hi Lincoln,
> > > > >>>> > >
> > > > >>>> > > The unified lookup hint is what I want.
> > > > >>>> > >
> > > > >>>> > > And I like 'async'='true|false' option.
> > > > >>>> > >
> > > > >>>> > > But there is a compatibility issue, as I remember if async
> is
> > > > >>>> currently
> > > > >>>> > > controlled by connector, and this may also require some API
> > > > changes?
> > > > >>>> > >
> > > > >>>> > > We need to have a clear story for the connector combined
> with
> > > this
> > > > >>>> > option:
> > > > >>>> > > - only sync connector
> > > > >>>> > > - only async connector
> > > > >>>> > > - both async and sync connector
> > > > >>>> > >
> > > > >>>> > > Best,
> > > > >>>> > > Jingsong
> > > > >>>> > >
> > > > >>>> > > On Mon, May 30, 2022 at 3:55 PM Lincoln Lee <
> > > > lincoln.8...@gmail.com
> > > > >>>> >
> > > > >>>> > > wrote:
> > > > >>>> > >
> > > > >>>> > > > Thanks Jark for your quick response and the consensus!
> > > > >>>> > > >
> > > > >>>> > > > And I will update the FLIP after Jingsong or other
> > developers
> > > > >>>> confirm
> > > > >>>> > > that
> > > > >>>> > > > there is no problem.
> > > > >>>> > > >
> > > > >>>> > > > Best,
> > > > >>>> > > > Lincoln Lee
> > > > >>>> > > >
> > > > >>>> > > >
> > > > >>>> > > > Jark Wu <imj...@gmail.com> 于2022年5月30日周一 15:49写道：
> > > > >>>> > > >
> > > > >>>> > > > > Thanks for the update.
> > > > >>>> > > > >
> > > > >>>> > > > > The unified lookup hint looks good to me.
> > > > >>>> > > > > And thanks for explaining ALLOW_UNORDERED.
> > > > >>>> > > > >
> > > > >>>> > > > > Best,
> > > > >>>> > > > > Jark
> > > > >>>> > > > >
> > > > >>>> > > > > On Mon, 30 May 2022 at 15:31, Lincoln Lee <
> > > > >>>> lincoln.8...@gmail.com>
> > > > >>>> > > > wrote:
> > > > >>>> > > > >
> > > > >>>> > > > > > Hi Jark & Jingsong,
> > > > >>>> > > > > >
> > > > >>>> > > > > > Thanks for your feedback!
> > > > >>>> > > > > >
> > > > >>>> > > > > > 1.) support retry on sync lookup
> > > > >>>> > > > > > I also agree with supporting it, this will be useful
> for
> > > > >>>> connectors
> > > > >>>> > > > that
> > > > >>>> > > > > > don't have asynchronous lookup implementations and can
> > > also
> > > > >>>> solve
> > > > >>>> > the
> > > > >>>> > > > > ASYNC
> > > > >>>> > > > > > non-target problem to some extent(because the retrying
> > is
> > > > >>>> blocking
> > > > >>>> > > for
> > > > >>>> > > > > sync
> > > > >>>> > > > > > lookup, and may accumulate delay, but it maybe
> > acceptable
> > > > for
> > > > >>>> the
> > > > >>>> > > case
> > > > >>>> > > > > that
> > > > >>>> > > > > > most or all data want to do a delayed lookup).
> > > > >>>> > > > > >
> > > > >>>> > > > > > For the api perspective, we can do some unification.
> > Let's
> > > > >>>> think of
> > > > >>>> > > the
> > > > >>>> > > > > > whole user story for lookup join:
> > > > >>>> > > > > >
> > > > >>>> > > > > > ASYNC_LOOKUP vs SYNC_LOOKUP can share a common one:
> > > LOOKUP
> > > > by
> > > > >>>> > > > different
> > > > >>>> > > > > > hint option values: 'async'='true|false'
> > > > >>>> > > > > >
> > > > >>>> > > > > > ASYNC_LOOKUP_MISS_RETRY vs SYNC_LOOKUP_MISS_RETRY can
> > > share
> > > > >>>> the
> > > > >>>> > > > > > LOOKUP_MISS_RETRY with hint option:
> > > > 'miss-retry'='true|false'
> > > > >>>> > > > > >
> > > > >>>> > > > > > we can use one single hint LOOKUP with different hint
> > > > options
> > > > >>>> > > > > > ('async'='true|false', 'miss-retry'='true|false') to
> > cover
> > > > all
> > > > >>>> > > related
> > > > >>>> > > > > > functionalities.
> > > > >>>> > > > > > Compared to multiple hints with different subsets of
> > > > >>>> > functionality, a
> > > > >>>> > > > > > single hint may be easier for users to understand and
> > use,
> > > > and
> > > > >>>> > > specific
> > > > >>>> > > > > > parameters can be quickly found through documentation
> > > > >>>> > > > > >
> > > > >>>> > > > > > the support matrix will be:
> > > > >>>> > > > > > lookup support async retry
> > > > >>>> > > > > > sync w/o retry N N
> > > > >>>> > > > > > sync w/ retry N Y
> > > > >>>> > > > > > async w/o retry Y N
> > > > >>>> > > > > > async w/ retry Y Y
> > > > >>>> > > > > >
> > > > >>>> > > > > > and the available hint options for each mode:
> > > > >>>> > > > > > mode support hint options
> > > > >>>> > > > > > async async'='true'
> > > > >>>> > > > > > 'output-mode'='ordere|allow-unordered'
> > > > >>>> > > > > > 'capacity'='100'
> > > > >>>> > > > > > 'timeout'='180s'
> > > > >>>> > > > > > retry miss-retry'='true'
> > > > >>>> > > > > > 'retry-strategy'='fixed-delay'
> > > > >>>> > > > > > 'delay'='10s'
> > > > >>>> > > > > > 'max-attempts'='3'
> > > > >>>> > > > > >
> > > > >>>> > > > > >
> > > > >>>> > > > > > 2.) 'allow-unordered' vs 'unordered' for
> > > > >>>> > > > > > 'table.exec.async-lookup.output-mode'
> > > > >>>> > > > > > Yes, make it align with DataStream Api maybe more
> > > intuitive,
> > > > >>>> but
> > > > >>>> > > > there's
> > > > >>>> > > > > > some difference in table layer that makes the
> > > > >>>> 'allow-unordered'
> > > > >>>> > > > > meaningful:
> > > > >>>> > > > > > updates in the pipeline need to be processed in order,
> > > > >>>> > > ALLOW_UNORDERED
> > > > >>>> > > > > > means if users allow unordered result, it will attempt
> > to
> > > > use
> > > > >>>> > > > > > AsyncDataStream.OutputMode.UNORDERED when it does not
> > > affect
> > > > >>>> the
> > > > >>>> > > > > > correctness of the result, otherwise ORDERED will be
> > still
> > > > >>>> used.
> > > > >>>> > > > > >
> > > > >>>> > > > > > Another choice is that when the user specifies
> unordered
> > > > mode,
> > > > >>>> > > planner
> > > > >>>> > > > > > throws an error when it finds that it may affect
> > > > correctness.
> > > > >>>> But
> > > > >>>> > > this
> > > > >>>> > > > is
> > > > >>>> > > > > > not user-friendly and is not consistent with the
> > customary
> > > > >>>> > treatment
> > > > >>>> > > of
> > > > >>>> > > > > > invalid query hints(best effort).
> > > > >>>> > > > > >
> > > > >>>> > > > > > I opened a pr
> > https://github.com/apache/flink/pull/19759
> > > > for
> > > > >>>> the
> > > > >>>> > new
> > > > >>>> > > > > > option
> > > > >>>> > > > > > 'table.exec.async-lookup.output-mode' and also a
> > > discussion
> > > > on
> > > > >>>> > > > > FLINK-27625:
> > > > >>>> > > > > > add query hint 'ASYNC_LOOKUP' for async lookup
> > join(Since
> > > > the
> > > > >>>> > changes
> > > > >>>> > > > > were
> > > > >>>> > > > > > relatively minor, no new flip was created)
> > > > >>>> > > > > >
> > > > >>>> > > > > > If we can reach a consensus on the single unified
> hint,
> > > > e.g.,
> > > > >>>> > LOOKUP,
> > > > >>>> > > > > then
> > > > >>>> > > > > > FLINK-27625 can be covered.
> > > > >>>> > > > > >
> > > > >>>> > > > > > WDYT?
> > > > >>>> > > > > >
> > > > >>>> > > > > > Best,
> > > > >>>> > > > > > Lincoln Lee
> > > > >>>> > > > > >
> > > > >>>> > > > > >
> > > > >>>> > > > > > Jark Wu <imj...@gmail.com> 于2022年5月27日周五 21:04写道：
> > > > >>>> > > > > >
> > > > >>>> > > > > > > Hi Lincoln,
> > > > >>>> > > > > > >
> > > > >>>> > > > > > > Delayed Dim Join is a frequently requested feature,
> > it's
> > > > >>>> exciting
> > > > >>>> > > to
> > > > >>>> > > > > see
> > > > >>>> > > > > > > this feature is on the road.
> > > > >>>> > > > > > > The FLIP looks good to me in general. I only left
> some
> > > > minor
> > > > >>>> > > > comments.
> > > > >>>> > > > > > >
> > > > >>>> > > > > > > 1) support retry for sync lookup
> > > > >>>> > > > > > > I'm also fine with the idea proposed by Jingsong.
> But
> > > this
> > > > >>>> > doesn't
> > > > >>>> > > > > > conflict
> > > > >>>> > > > > > > with the FLIP and can
> > > > >>>> > > > > > > be future work. It would be great if we can
> determine
> > > the
> > > > >>>> APIs
> > > > >>>> > > first.
> > > > >>>> > > > > > >
> > > > >>>> > > > > > > 1) "allow-unordered" => "unordered"
> > > > >>>> > > > > > > I would prefer the "unordered" output mode rather
> than
> > > > >>>> > > > > "allow-unordered".
> > > > >>>> > > > > > > Because this fully aligns with the DataStream
> > behaviors
> > > > and
> > > > >>>> > avoids
> > > > >>>> > > > > > > confusion on the differences.
> > > > >>>> > > > > > > I understand the purpose that adding a "allow"
> prefix
> > > > here,
> > > > >>>> but I
> > > > >>>> > > > think
> > > > >>>> > > > > > the
> > > > >>>> > > > > > > semantic is fine to just
> > > > >>>> > > > > > > use "unordered" here. We didn't see any users
> confused
> > > > about
> > > > >>>> > > > > > > OutputMode#UNORDERED.
> > > > >>>> > > > > > >
> > > > >>>> > > > > > > Best,
> > > > >>>> > > > > > > Jark
> > > > >>>> > > > > > >
> > > > >>>> > > > > > >
> > > > >>>> > > > > > >
> > > > >>>> > > > > > > On Fri, 27 May 2022 at 12:58, Jingsong Li <
> > > > >>>> > jingsongl...@gmail.com>
> > > > >>>> > > > > > wrote:
> > > > >>>> > > > > > >
> > > > >>>> > > > > > > > Thanks Lincoln for your proposal.
> > > > >>>> > > > > > > >
> > > > >>>> > > > > > > > Take a look at `strategy: fixed-delay delay:
> > duration,
> > > > >>>> e.g.,
> > > > >>>> > 10s
> > > > >>>> > > > > > > > max-attempts: integer, e.g., 3`.
> > > > >>>> > > > > > > >
> > > > >>>> > > > > > > > Are these options only for async? It looks like
> > normal
> > > > >>>> lookups
> > > > >>>> > > work
> > > > >>>> > > > > > too?
> > > > >>>> > > > > > > >
> > > > >>>> > > > > > > > One thing is: most of the lookup functions seem to
> > be
> > > > >>>> > synchronous
> > > > >>>> > > > > now?
> > > > >>>> > > > > > > > There are not so many asynchronous ones?
> > > > >>>> > > > > > > >
> > > > >>>> > > > > > > > Best,
> > > > >>>> > > > > > > > Jingsong
> > > > >>>> > > > > > > >
> > > > >>>> > > > > > > > On Tue, May 24, 2022 at 11:48 AM Lincoln Lee <
> > > > >>>> > > > lincoln.8...@gmail.com
> > > > >>>> > > > > >
> > > > >>>> > > > > > > > wrote:
> > > > >>>> > > > > > > >
> > > > >>>> > > > > > > > > Hi all,
> > > > >>>> > > > > > > > >
> > > > >>>> > > > > > > > > Considering the new common table option
> > > > >>>> 'lookup.max-retries'
> > > > >>>> > > > > proposed
> > > > >>>> > > > > > > in
> > > > >>>> > > > > > > > > FLIP-221[1] which is commonly used for exception
> > > > >>>> handling in
> > > > >>>> > > > > > connector
> > > > >>>> > > > > > > > > implementation, we should clearly distinguish
> > > > >>>> > > ASYNC_LOOKUP_RETRY
> > > > >>>> > > > > from
> > > > >>>> > > > > > > it
> > > > >>>> > > > > > > > to
> > > > >>>> > > > > > > > > avoid confusing users.
> > > > >>>> > > > > > > > >
> > > > >>>> > > > > > > > > To do so, the name ASYNC_LOOKUP_RETRY can change
> > to
> > > > >>>> > > > > > > > > ASYNC_LOOKUP_MISS_RETRY, and as the name
> implies,
> > > > >>>> restrict
> > > > >>>> > it
> > > > >>>> > > to
> > > > >>>> > > > > > > support
> > > > >>>> > > > > > > > > retries only for lookup misses and no longer
> > include
> > > > >>>> > exceptions
> > > > >>>> > > > > (for
> > > > >>>> > > > > > > sql
> > > > >>>> > > > > > > > > connectors, let the connector implementer decide
> > how
> > > > to
> > > > >>>> > handle
> > > > >>>> > > > > > > exceptions
> > > > >>>> > > > > > > > > since there are various kinds of retryable
> > > exceptions
> > > > >>>> and can
> > > > >>>> > > not
> > > > >>>> > > > > > retry
> > > > >>>> > > > > > > > > ones).
> > > > >>>> > > > > > > > >
> > > > >>>> > > > > > > > > The FLIP[2] has been updated.
> > > > >>>> > > > > > > > >
> > > > >>>> > > > > > > > > [1]
> > > > >>>> > > > > > > > >
> > > > >>>> > > > > > > > >
> > > > >>>> > > > > > > >
> > > > >>>> > > > > > >
> > > > >>>> > > > > >
> > > > >>>> > > > >
> > > > >>>> > > >
> > > > >>>> > >
> > > > >>>> >
> > > > >>>>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-221%3A+Abstraction+for+lookup+source+cache+and+metric
> > > > >>>> > > > > > > > > [2]
> > > > >>>> > > > > > > > >
> > > > >>>> > > > > > > > >
> > > > >>>> > > > > > > >
> > > > >>>> > > > > > >
> > > > >>>> > > > > >
> > > > >>>> > > > >
> > > > >>>> > > >
> > > > >>>> > >
> > > > >>>> >
> > > > >>>>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-234%3A+Support+Retryable+Lookup+Join+To+Solve+Delayed+Updates+Issue+In+External+Systems
> > > > >>>> > > > > > > > >
> > > > >>>> > > > > > > > >
> > > > >>>> > > > > > > > > Best,
> > > > >>>> > > > > > > > > Lincoln Lee
> > > > >>>> > > > > > > > >
> > > > >>>> > > > > > > > >
> > > > >>>> > > > > > > > > Lincoln Lee <lincoln.8...@gmail.com>
> > 于2022年5月19日周四
> > > > >>>> 18:24写道：
> > > > >>>> > > > > > > > >
> > > > >>>> > > > > > > > > > Dear Flink developers,
> > > > >>>> > > > > > > > > >
> > > > >>>> > > > > > > > > > I would like to open a discussion on FLIP 234
> > [1]
> > > to
> > > > >>>> > support
> > > > >>>> > > > > > > retryable
> > > > >>>> > > > > > > > > > lookup join to solve delayed updates issue,
> as a
> > > > >>>> pre-work
> > > > >>>> > for
> > > > >>>> > > > > this
> > > > >>>> > > > > > > > > > solution, we proposed FLIP-232[2] which adds a
> > > > generic
> > > > >>>> > retry
> > > > >>>> > > > > > support
> > > > >>>> > > > > > > > for
> > > > >>>> > > > > > > > > > Async I/O.
> > > > >>>> > > > > > > > > > We prefer to offer this retry capability via
> > query
> > > > >>>> hints,
> > > > >>>> > > > similar
> > > > >>>> > > > > > to
> > > > >>>> > > > > > > > new
> > > > >>>> > > > > > > > > > join hints proposed in FLINK-27625[3] &
> > > FLIP-204[4].
> > > > >>>> > > > > > > > > >
> > > > >>>> > > > > > > > > > This feature is backwards compatible and
> > > > >>>> transparently to
> > > > >>>> > > > > > connectors.
> > > > >>>> > > > > > > > For
> > > > >>>> > > > > > > > > > existing connectors which implements
> > > > >>>> AsyncTableFunction,
> > > > >>>> > can
> > > > >>>> > > > > easily
> > > > >>>> > > > > > > > > enable
> > > > >>>> > > > > > > > > > async retry via the new join hint.
> > > > >>>> > > > > > > > > >
> > > > >>>> > > > > > > > > > [1]
> > > > >>>> > > > > > > > > >
> > > > >>>> > > > > > > > >
> > > > >>>> > > > > > > >
> > > > >>>> > > > > > >
> > > > >>>> > > > > >
> > > > >>>> > > > >
> > > > >>>> > > >
> > > > >>>> > >
> > > > >>>> >
> > > > >>>>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-234%3A+Support+Retryable+Lookup+Join+To+Solve+Delayed+Updates+Issue+In+External+Systems
> > > > >>>> > > > > > > > > > [2]
> > > > >>>> > > > > > > > > >
> > > > >>>> > > > > > > > >
> > > > >>>> > > > > > > >
> > > > >>>> > > > > > >
> > > > >>>> > > > > >
> > > > >>>> > > > >
> > > > >>>> > > >
> > > > >>>> > >
> > > > >>>> >
> > > > >>>>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=211883963
> > > > >>>> > > > > > > > > > [3]
> > > > >>>> > > > > >
> > > > >>>>
> https://lists.apache.org/thread/jm9kg33wk9z2bvo2b0g5bp3n5kfj6qv8
> > > > >>>> > > > > > > > > > [4]
> > > > >>>> > > > > > > > > >
> > > > >>>> > > > > > > > >
> > > > >>>> > > > > > > >
> > > > >>>> > > > > > >
> > > > >>>> > > > > >
> > > > >>>> > > > >
> > > > >>>> > > >
> > > > >>>> > >
> > > > >>>> >
> > > > >>>>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-204:+Introduce+Hash+Lookup+Join
> > > > >>>> > > > > > > > > >
> > > > >>>> > > > > > > > > > Best,
> > > > >>>> > > > > > > > > > Lincoln Lee
> > > > >>>> > > > > > > > > >
> > > > >>>> > > > > > > > >
> > > > >>>> > > > > > > >
> > > > >>>> > > > > > >
> > > > >>>> > > > > >
> > > > >>>> > > > >
> > > > >>>> > > >
> > > > >>>> > >
> > > > >>>> >
> > > > >>>>
> > > > >>>
> > > >
> > >
> >
>
