Hi, Jing,
Thanks for your explanation. I get your point now.
For the performance part, I think it's a good idea to run with returning a
big table case, the memory consumption
should be a point to be taken care about. Because in the ordered mode, the
head element in buffer may affect the
total memory consumption.
Thanks,
Aitozi.
Jing Ge <j...@ververica.com.invalid> 于2023年6月12日周一 20:28写道：
> Hi Aitozi,
>
> Which key will be used for lookup is not an issue, only one row will be
> required for each key in order to enrich it. True, it depends on the
> implementation whether multiple rows or single row for each key will be
> returned. However, for the lookup & enrichment scenario, one row/key is
> recommended, otherwise, like I mentioned previously, enrichment won't work.
>
> I am a little bit concerned about returning a big table for each key, since
> it will take the async call longer to return and need more memory. The
> performance tests should cover this scenario. This is not a blocking issue
> for this FLIP.
>
> Best regards,
> Jing
>
> On Sat, Jun 10, 2023 at 4:11 AM Aitozi <gjying1...@gmail.com> wrote:
>
> > Hi Jing,
> > I means the join key is not necessary to be the primary key or unique
> > index of the database.
> > In this situation, we may queried out multi rows for one join key. I
> think
> > that's why the
> > LookupFunction#lookup will return a collection of RowData.
> >
> > BTW, I think the behavior of lookup join will not affect the semantic of
> > the async udtf.
> > We use the Async TableFunction here and the table function can collect
> > multiple rows.
> >
> > Thanks,
> > Atiozi.
> >
> >
> >
> > Jing Ge <j...@ververica.com.invalid> 于2023年6月10日周六 00:15写道：
> >
> > > Hi Aitozi,
> > >
> > > The keyRow used in this case contains all keys[1].
> > >
> > > Best regards,
> > > Jing
> > >
> > > [1]
> > >
> > >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L49
> > >
> > >
> > > On Fri, Jun 9, 2023 at 3:42 PM Aitozi <gjying1...@gmail.com> wrote:
> > >
> > > > Hi Jing,
> > > >
> > > > The performance test is added to the FLIP.
> > > >
> > > > As I know, The lookup join can return multi rows, it depends on
> > > > whether the join key
> > > > is the primary key of the external database or not. The `lookup` [1]
> > will
> > > > return a collection of
> > > > joined result, and each of them will be collected
> > > >
> > > >
> > > > [1]:
> > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L52
> > > >
> > > >
> > > > Thanks,
> > > > Aitozi.
> > > >
> > > > Jing Ge <j...@ververica.com.invalid> 于2023年6月9日周五 17:05写道：
> > > >
> > > > > Hi Aitozi,
> > > > >
> > > > > Thanks for the feedback. Looking forward to the performance tests.
> > > > >
> > > > > Afaik, lookup returns one row for each key [1] [2]. Conceptually,
> the
> > > > > lookup function is used to enrich column(s) from the dimension
> table.
> > > If,
> > > > > for the given key, there will be more than one row, there will be
> no
> > > way
> > > > to
> > > > > know which row will be used to enrich the key.
> > > > >
> > > > > [1]
> > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L49
> > > > > [2]
> > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/TableFunction.java#L196
> > > > >
> > > > > Best regards,
> > > > > Jing
> > > > >
> > > > > On Fri, Jun 9, 2023 at 5:18 AM Aitozi <gjying1...@gmail.com>
> wrote:
> > > > >
> > > > > > Hi Jing
> > > > > > Thanks for your good questions. I have updated the example to
> > the
> > > > > FLIP.
> > > > > >
> > > > > > > Only one row for each lookup
> > > > > > lookup can also return multi rows, based on the query result. [1]
> > > > > >
> > > > > > [1]:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L56
> > > > > >
> > > > > > > If we use async calls with lateral join, my gut feeling is
> > > > > > that we might have many more async calls than lookup join. I am
> not
> > > > > really
> > > > > > sure if we will be facing potential issues in this case or not.
> > > > > >
> > > > > > IMO, the work pattern is similar to the lookup function, for each
> > row
> > > > > from
> > > > > > the left table,
> > > > > > it will evaluate the eval method once, so the async call numbers
> > will
> > > > not
> > > > > > change.
> > > > > > and the maximum calls in flight is limited by the Async operators
> > > > buffer
> > > > > > capacity
> > > > > > which will be controlled by the option.
> > > > > >
> > > > > > BTW, for the naming of these option, I updated the FLIP about
> this
> > > you
> > > > > can
> > > > > > refer to
> > > > > > the section of "ConfigOption" and "Rejected Alternatives"
> > > > > >
> > > > > > In the end, for the performance evaluation, I'd like to do some
> > tests
> > > > and
> > > > > > will update it to the FLIP doc
> > > > > >
> > > > > > Thanks,
> > > > > > Aitozi.
> > > > > >
> > > > > >
> > > > > > Jing Ge <j...@ververica.com.invalid> 于2023年6月9日周五 07:23写道：
> > > > > >
> > > > > > > Hi Aitozi,
> > > > > > >
> > > > > > > Thanks for the clarification. The code example looks
> > interesting. I
> > > > > would
> > > > > > > suggest adding them into the FLIP. The description with code
> > > examples
> > > > > > will
> > > > > > > help readers understand the motivation and how to use it.
> Afaiac,
> > > it
> > > > > is a
> > > > > > > valid feature for Flink users.
> > > > > > >
> > > > > > > As we knew, lookup join is based on temporal join, i.e. FOR
> > > > SYSTEM_TIME
> > > > > > AS
> > > > > > > OF which is also used in your code example. Temporal join
> > performs
> > > > the
> > > > > > > lookup based on the processing time match. Only one row for
> each
> > > > > > > lookup(afaiu, I need to check the source code to double
> confirm)
> > > will
> > > > > > > return for further enrichment. One the other hand, lateral join
> > > will
> > > > > have
> > > > > > > sub-queries correlated with every individual value of the
> > reference
> > > > > table
> > > > > > > from the preceding part of the query and each sub query will
> > return
> > > > > > > multiple rows. If we use async calls with lateral join, my gut
> > > > feeling
> > > > > is
> > > > > > > that we might have many more async calls than lookup join. I am
> > not
> > > > > > really
> > > > > > > sure if we will be facing potential issues in this case or not.
> > > > > Possible
> > > > > > > issues I can think of now e.g. too many PRC calls, too many
> async
> > > > calls
> > > > > > > processing, the sub query will return a table which might be
> > (too)
> > > > big,
> > > > > > and
> > > > > > > might cause performance issues. I would suggest preparing some
> > use
> > > > > cases
> > > > > > > and running some performance tests to check it. These are my
> > > concerns
> > > > > > about
> > > > > > > using async calls with lateral join and I'd like to share with
> > you,
> > > > > happy
> > > > > > > to discuss with you and hear different opinions, hopefully the
> > > > > > > discussion could help me understand it more deeply. Please
> > correct
> > > me
> > > > > if
> > > > > > I
> > > > > > > am wrong.
> > > > > > >
> > > > > > > Best regards,
> > > > > > > Jing
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Jun 8, 2023 at 7:22 AM Aitozi <gjying1...@gmail.com>
> > > wrote:
> > > > > > >
> > > > > > > > Hi Mason,
> > > > > > > > Thanks for your input. I think if we support the user
> > defined
> > > > > async
> > > > > > > > table function,
> > > > > > > > user will be able to use it to hold a batch data then handle
> it
> > > at
> > > > > one
> > > > > > > time
> > > > > > > > in the customized function.
> > > > > > > >
> > > > > > > > AsyncSink is meant for the sink operator. I have not figure
> out
> > > how
> > > > > to
> > > > > > > > integrate in this case.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Atiozi.
> > > > > > > >
> > > > > > > >
> > > > > > > > Mason Chen <mas.chen6...@gmail.com> 于2023年6月8日周四 12:40写道：
> > > > > > > >
> > > > > > > > > Hi Aitozi,
> > > > > > > > >
> > > > > > > > > I think it makes sense to make it easier for SQL users to
> > make
> > > > > RPCs.
> > > > > > Do
> > > > > > > > you
> > > > > > > > > think your proposal can extend to the ability to batch data
> > for
> > > > the
> > > > > > > RPC?
> > > > > > > > > This is also another common strategy to increase
> throughput.
> > > > Also,
> > > > > > have
> > > > > > > > you
> > > > > > > > > considered solving this a bit differently by leveraging
> > Flink's
> > > > > > > > AsyncSink?
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Mason
> > > > > > > > >
> > > > > > > > > On Mon, Jun 5, 2023 at 1:50 AM Aitozi <
> gjying1...@gmail.com>
> > > > > wrote:
> > > > > > > > >
> > > > > > > > > > One more thing for discussion:
> > > > > > > > > >
> > > > > > > > > > In our internal implementation, we reuse the option
> > > > > > > > > > `table.exec.async-lookup.buffer-capacity` and
> > > > > > > > > > `table.exec.async-lookup.timeout` to config
> > > > > > > > > > the async udtf. Do you think we should add two extra
> option
> > > to
> > > > > > > > > distinguish
> > > > > > > > > > from the lookup option such as
> > > > > > > > > >
> > > > > > > > > > `table.exec.async-udtf.buffer-capacity`
> > > > > > > > > > `table.exec.async-udtf.timeout`
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Aitozi.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Aitozi <gjying1...@gmail.com> 于2023年6月5日周一 12:20写道：
> > > > > > > > > >
> > > > > > > > > > > Hi Jing,
> > > > > > > > > > >
> > > > > > > > > > > > what is the difference between the RPC call or
> > query
> > > > you
> > > > > > > > > mentioned
> > > > > > > > > > > and the lookup in a very
> > > > > > > > > > > general way
> > > > > > > > > > >
> > > > > > > > > > > I think the RPC call or query service is quite similar
> to
> > > the
> > > > > > > lookup
> > > > > > > > > > join.
> > > > > > > > > > > But lookup join should work
> > > > > > > > > > > with `LookupTableSource`.
> > > > > > > > > > >
> > > > > > > > > > > Let's see how we can perform an async RPC call with
> > lookup
> > > > > join:
> > > > > > > > > > >
> > > > > > > > > > > (1) Implement an AsyncTableFunction with RPC call
> logic.
> > > > > > > > > > > (2) Implement a `LookupTableSource` connector run with
> > the
> > > > > async
> > > > > > > udtf
> > > > > > > > > > > defined in (1).
> > > > > > > > > > > (3) Then define a DDL of this look up table in SQL
> > > > > > > > > > >
> > > > > > > > > > > CREATE TEMPORARY TABLE Customers (
> > > > > > > > > > > id INT,
> > > > > > > > > > > name STRING,
> > > > > > > > > > > country STRING,
> > > > > > > > > > > zip STRING
> > > > > > > > > > > ) WITH (
> > > > > > > > > > > 'connector' = 'custom'
> > > > > > > > > > > );
> > > > > > > > > > >
> > > > > > > > > > > (4) Run with the query as below:
> > > > > > > > > > >
> > > > > > > > > > > SELECT o.order_id, o.total, c.country, c.zip
> > > > > > > > > > > FROM Orders AS o
> > > > > > > > > > > JOIN Customers FOR SYSTEM_TIME AS OF o.proc_time AS c
> > > > > > > > > > > ON o.customer_id = c.id;
> > > > > > > > > > >
> > > > > > > > > > > This example is from doc
> > > > > > > > > > > <
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/queries/joins/#lookup-join
> > > > > > > > > > >.You
> > > > > > > > > > > can image the look up process as an async RPC call
> > process.
> > > > > > > > > > >
> > > > > > > > > > > Let's see how we can perform an async RPC call with
> > lateral
> > > > > join:
> > > > > > > > > > >
> > > > > > > > > > > (1) Implement an AsyncTableFunction with RPC call
> logic.
> > > > > > > > > > > (2) Run query with
> > > > > > > > > > >
> > > > > > > > > > > Create function f1 as '...' ;
> > > > > > > > > > >
> > > > > > > > > > > SELECT o.order_id, o.total, c.country, c.zip FROM
> Orders
> > > > > lateral
> > > > > > > > table
> > > > > > > > > > > (f1(order_id)) as T(...);
> > > > > > > > > > >
> > > > > > > > > > > As you can see, the lateral join version is more simple
> > and
> > > > > > > intuitive
> > > > > > > > > to
> > > > > > > > > > > users. Users do not have to wrap a
> > > > > > > > > > > LookupTableSource for the purpose of using async udtf.
> > > > > > > > > > >
> > > > > > > > > > > In the end, We can also see the user defined async
> table
> > > > > function
> > > > > > > is
> > > > > > > > an
> > > > > > > > > > > enhancement of the current lateral table join
> > > > > > > > > > > which only supports sync lateral join now.
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > > Aitozi.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Jing Ge <j...@ververica.com.invalid> 于2023年6月2日周五
> > 19:37写道：
> > > > > > > > > > >
> > > > > > > > > > >> Hi Aitozi,
> > > > > > > > > > >>
> > > > > > > > > > >> Thanks for the update. Just out of curiosity, what is
> > the
> > > > > > > difference
> > > > > > > > > > >> between the RPC call or query you mentioned and the
> > lookup
> > > > in
> > > > > a
> > > > > > > very
> > > > > > > > > > >> general way? Since Lateral join is used in the FLIP.
> Is
> > > > there
> > > > > > any
> > > > > > > > > > special
> > > > > > > > > > >> thought for that? Sorry for asking so many questions.
> > The
> > > > FLIP
> > > > > > > > > contains
> > > > > > > > > > >> limited information to understand the motivation.
> > > > > > > > > > >>
> > > > > > > > > > >> Best regards,
> > > > > > > > > > >> Jing
> > > > > > > > > > >>
> > > > > > > > > > >> On Fri, Jun 2, 2023 at 3:48 AM Aitozi <
> > > gjying1...@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > > > > >>
> > > > > > > > > > >> > Hi Jing,
> > > > > > > > > > >> > I have updated the proposed changes to the FLIP.
> > > IMO,
> > > > > > lookup
> > > > > > > > has
> > > > > > > > > > its
> > > > > > > > > > >> > clear
> > > > > > > > > > >> > async call requirement is due to its IO heavy
> > operator.
> > > In
> > > > > our
> > > > > > > > > usage,
> > > > > > > > > > >> sql
> > > > > > > > > > >> > users have
> > > > > > > > > > >> > logic to do some RPC call or query the third-party
> > > service
> > > > > > which
> > > > > > > > is
> > > > > > > > > > >> also IO
> > > > > > > > > > >> > intensive.
> > > > > > > > > > >> > In these case, we'd like to leverage the async
> > function
> > > to
> > > > > > > improve
> > > > > > > > > the
> > > > > > > > > > >> > throughput.
> > > > > > > > > > >> >
> > > > > > > > > > >> > Thanks,
> > > > > > > > > > >> > Aitozi.
> > > > > > > > > > >> >
> > > > > > > > > > >> > Jing Ge <j...@ververica.com.invalid> 于2023年6月1日周四
> > > > 22:55写道：
> > > > > > > > > > >> >
> > > > > > > > > > >> > > Hi Aitozi,
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > Sorry for the late reply. Would you like to update
> > the
> > > > > > > proposed
> > > > > > > > > > >> changes
> > > > > > > > > > >> > > with more details into the FLIP too?
> > > > > > > > > > >> > > I got your point. It looks like a rational idea.
> > > > However,
> > > > > > > since
> > > > > > > > > > lookup
> > > > > > > > > > >> > has
> > > > > > > > > > >> > > its clear async call requirement, are there any
> real
> > > use
> > > > > > cases
> > > > > > > > > that
> > > > > > > > > > >> > > need this change? This will help us understand the
> > > > > > motivation.
> > > > > > > > > After
> > > > > > > > > > >> all,
> > > > > > > > > > >> > > lateral join and temporal lookup join[1] are quite
> > > > > > different.
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > Best regards,
> > > > > > > > > > >> > > Jing
> > > > > > > > > > >> > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > [1]
> > > > > > > > > > >> > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/d90a72da2fd601ca4e2a46700e91ec5b348de2ad/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/AsyncTableFunction.java#L54
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > On Wed, May 31, 2023 at 8:53 AM Aitozi <
> > > > > > gjying1...@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > > Hi Jing,
> > > > > > > > > > >> > > > What do you think about it? Can we move
> > forward
> > > > this
> > > > > > > > > feature?
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > Thanks,
> > > > > > > > > > >> > > > Aitozi.
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > Aitozi <gjying1...@gmail.com> 于2023年5月29日周一
> > > 09:56写道：
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > > Hi Jing,
> > > > > > > > > > >> > > > > > "Do you mean to support the
> > > AyncTableFunction
> > > > > > beyond
> > > > > > > > the
> > > > > > > > > > >> > > > > LookupTableSource?"
> > > > > > > > > > >> > > > > Yes, I mean to support the AyncTableFunction
> > > beyond
> > > > > the
> > > > > > > > > > >> > > > LookupTableSource.
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > > The "AsyncTableFunction" is the function with
> > > > ability
> > > > > to
> > > > > > > be
> > > > > > > > > > >> executed
> > > > > > > > > > >> > > > async
> > > > > > > > > > >> > > > > (with AsyncWaitOperator).
> > > > > > > > > > >> > > > > The async lookup join is a one of usage of
> this.
> > > So,
> > > > > we
> > > > > > > > don't
> > > > > > > > > > >> have to
> > > > > > > > > > >> > > > bind
> > > > > > > > > > >> > > > > the AyncTableFunction with LookupTableSource.
> > > > > > > > > > >> > > > > If User-defined AsyncTableFunction is
> supported,
> > > > user
> > > > > > can
> > > > > > > > > > directly
> > > > > > > > > > >> > use
> > > > > > > > > > >> > > > > lateral table syntax to perform async
> operation.
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > > > "It would be better if you could elaborate
> the
> > > > > > proposed
> > > > > > > > > > changes
> > > > > > > > > > >> wrt
> > > > > > > > > > >> > > the
> > > > > > > > > > >> > > > > CorrelatedCodeGenerator with more details"
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > > In the proposal, we use lateral table syntax
> to
> > > > > support
> > > > > > > the
> > > > > > > > > > async
> > > > > > > > > > >> > table
> > > > > > > > > > >> > > > > function. So the planner will also treat this
> > > > > statement
> > > > > > > to a
> > > > > > > > > > >> > > > > CommonExecCorrelate node. So the runtime code
> > > should
> > > > > be
> > > > > > > > > > generated
> > > > > > > > > > >> in
> > > > > > > > > > >> > > > > CorrelatedCodeGenerator.
> > > > > > > > > > >> > > > > In CorrelatedCodeGenerator, we will know the
> > > > > > > TableFunction's
> > > > > > > > > > Kind
> > > > > > > > > > >> of
> > > > > > > > > > >> > > > > `FunctionKind.Table` or
> > `FunctionKind.ASYNC_TABLE`
> > > > > > > > > > >> > > > > For `FunctionKind.ASYNC_TABLE` we can
> generate
> > a
> > > > > > > > > > >> AsyncWaitOperator
> > > > > > > > > > >> > to
> > > > > > > > > > >> > > > > execute the async table function.
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > > Thanks,
> > > > > > > > > > >> > > > > Aitozi.
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > > Jing Ge <j...@ververica.com.invalid>
> > > 于2023年5月29日周一
> > > > > > > 03:22写道：
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > >> Hi Aitozi,
> > > > > > > > > > >> > > > >>
> > > > > > > > > > >> > > > >> Thanks for the clarification. The naming
> > "Lookup"
> > > > > might
> > > > > > > > > suggest
> > > > > > > > > > >> > using
> > > > > > > > > > >> > > it
> > > > > > > > > > >> > > > >> for table look up. But conceptually what the
> > > eval()
> > > > > > > method
> > > > > > > > > will
> > > > > > > > > > >> do
> > > > > > > > > > >> > is
> > > > > > > > > > >> > > to
> > > > > > > > > > >> > > > >> get a collection of results(Row, RowData)
> from
> > > the
> > > > > > given
> > > > > > > > > keys.
> > > > > > > > > > >> How
> > > > > > > > > > >> > it
> > > > > > > > > > >> > > > will
> > > > > > > > > > >> > > > >> be done depends on the implementation, i.e.
> you
> > > can
> > > > > > > > implement
> > > > > > > > > > >> your
> > > > > > > > > > >> > own
> > > > > > > > > > >> > > > >> Source[1][2]. The example in the FLIP should
> be
> > > > able
> > > > > to
> > > > > > > be
> > > > > > > > > > >> handled
> > > > > > > > > > >> > in
> > > > > > > > > > >> > > > this
> > > > > > > > > > >> > > > >> way.
> > > > > > > > > > >> > > > >>
> > > > > > > > > > >> > > > >> Do you mean to support the AyncTableFunction
> > > beyond
> > > > > the
> > > > > > > > > > >> > > > LookupTableSource?
> > > > > > > > > > >> > > > >> It would be better if you could elaborate the
> > > > > proposed
> > > > > > > > > changes
> > > > > > > > > > >> wrt
> > > > > > > > > > >> > the
> > > > > > > > > > >> > > > >> CorrelatedCodeGenerator with more details.
> > > Thanks!
> > > > > > > > > > >> > > > >>
> > > > > > > > > > >> > > > >> Best regards,
> > > > > > > > > > >> > > > >> Jing
> > > > > > > > > > >> > > > >>
> > > > > > > > > > >> > > > >> [1]
> > > > > > > > > > >> > > > >>
> > > > > > > > > > >> > > > >>
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/678370b18e1b6c4a23e5ce08f8efd05675a0cc17/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/LookupTableSource.java#L64
> > > > > > > > > > >> > > > >> [2]
> > > > > > > > > > >> > > > >>
> > > > > > > > > > >> > > > >>
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/678370b18e1b6c4a23e5ce08f8efd05675a0cc17/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/AsyncTableFunctionProvider.java#L49
> > > > > > > > > > >> > > > >>
> > > > > > > > > > >> > > > >> On Sat, May 27, 2023 at 9:48 AM Aitozi <
> > > > > > > > gjying1...@gmail.com
> > > > > > > > > >
> > > > > > > > > > >> > wrote:
> > > > > > > > > > >> > > > >>
> > > > > > > > > > >> > > > >> > Hi Jing,
> > > > > > > > > > >> > > > >> > Thanks for your response. As stated in
> > the
> > > > > FLIP,
> > > > > > > the
> > > > > > > > > > >> purpose
> > > > > > > > > > >> > of
> > > > > > > > > > >> > > > this
> > > > > > > > > > >> > > > >> > FLIP is meant to support
> > > > > > > > > > >> > > > >> > user-defined async table function. As
> > described
> > > > in
> > > > > > > flink
> > > > > > > > > > >> document
> > > > > > > > > > >> > > [1]
> > > > > > > > > > >> > > > >> >
> > > > > > > > > > >> > > > >> > Async table functions are special functions
> > for
> > > > > table
> > > > > > > > > sources
> > > > > > > > > > >> that
> > > > > > > > > > >> > > > >> perform
> > > > > > > > > > >> > > > >> > > a lookup.
> > > > > > > > > > >> > > > >> > >
> > > > > > > > > > >> > > > >> >
> > > > > > > > > > >> > > > >> > So end user can not directly define and use
> > > async
> > > > > > table
> > > > > > > > > > >> function
> > > > > > > > > > >> > > now.
> > > > > > > > > > >> > > > An
> > > > > > > > > > >> > > > >> > user case is reported in [2]
> > > > > > > > > > >> > > > >> >
> > > > > > > > > > >> > > > >> > So, in conclusion, no new interface is
> > > > introduced,
> > > > > > but
> > > > > > > we
> > > > > > > > > > >> extend
> > > > > > > > > > >> > the
> > > > > > > > > > >> > > > >> > ability to support user-defined async table
> > > > > function.
> > > > > > > > > > >> > > > >> >
> > > > > > > > > > >> > > > >> > [1]:
> > > > > > > > > > >> > > > >> >
> > > > > > > > > > >> > > > >> >
> > > > > > > > > > >> > > > >>
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/functions/udfs/
> > > > > > > > > > >> > > > >> > [2]:
> > > > > > > > > > >> > >
> > > > > > > >
> > https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
> > > > > > > > > > >> > > > >> >
> > > > > > > > > > >> > > > >> > Thanks.
> > > > > > > > > > >> > > > >> > Aitozi.
> > > > > > > > > > >> > > > >> >
> > > > > > > > > > >> > > > >> >
> > > > > > > > > > >> > > > >> > Jing Ge <j...@ververica.com.invalid>
> > > > 于2023年5月27日周六
> > > > > > > > > 06:40写道：
> > > > > > > > > > >> > > > >> >
> > > > > > > > > > >> > > > >> > > Hi Aitozi,
> > > > > > > > > > >> > > > >> > >
> > > > > > > > > > >> > > > >> > > Thanks for your proposal. I am not quite
> > sure
> > > > if
> > > > > I
> > > > > > > > > > understood
> > > > > > > > > > >> > your
> > > > > > > > > > >> > > > >> > thoughts
> > > > > > > > > > >> > > > >> > > correctly. You described a special case
> > > > > > > implementation
> > > > > > > > of
> > > > > > > > > > the
> > > > > > > > > > >> > > > >> > > AsyncTableFunction with on public API
> > > changes.
> > > > > > Would
> > > > > > > > you
> > > > > > > > > > >> please
> > > > > > > > > > >> > > > >> elaborate
> > > > > > > > > > >> > > > >> > > your purpose of writing a FLIP according
> to
> > > the
> > > > > > FLIP
> > > > > > > > > > >> > > > documentation[1]?
> > > > > > > > > > >> > > > >> > > Thanks!
> > > > > > > > > > >> > > > >> > >
> > > > > > > > > > >> > > > >> > > [1]
> > > > > > > > > > >> > > > >> > >
> > > > > > > > > > >> > > > >> > >
> > > > > > > > > > >> > > > >> >
> > > > > > > > > > >> > > > >>
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
> > > > > > > > > > >> > > > >> > >
> > > > > > > > > > >> > > > >> > > Best regards,
> > > > > > > > > > >> > > > >> > > Jing
> > > > > > > > > > >> > > > >> > >
> > > > > > > > > > >> > > > >> > > On Wed, May 24, 2023 at 1:07 PM Aitozi <
> > > > > > > > > > gjying1...@gmail.com
> > > > > > > > > > >> >
> > > > > > > > > > >> > > > wrote:
> > > > > > > > > > >> > > > >> > >
> > > > > > > > > > >> > > > >> > > > May I ask for some feedback :D
> > > > > > > > > > >> > > > >> > > >
> > > > > > > > > > >> > > > >> > > > Thanks,
> > > > > > > > > > >> > > > >> > > > Aitozi
> > > > > > > > > > >> > > > >> > > >
> > > > > > > > > > >> > > > >> > > > Aitozi <gjying1...@gmail.com>
> > > 于2023年5月23日周二
> > > > > > > 19:14写道：
> > > > > > > > > > >> > > > >> > > > >
> > > > > > > > > > >> > > > >> > > > > Just catch an user case report from
> > > Giannis
> > > > > > > Polyzos
> > > > > > > > > for
> > > > > > > > > > >> this
> > > > > > > > > > >> > > > >> usage:
> > > > > > > > > > >> > > > >> > > > >
> > > > > > > > > > >> > > > >> > > > >
> > > > > > > > > > >> > > >
> > > > > > > > >
> > > https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
> > > > > > > > > > >> > > > >> > > > >
> > > > > > > > > > >> > > > >> > > > > Aitozi <gjying1...@gmail.com>
> > > > 于2023年5月23日周二
> > > > > > > > 17:45写道：
> > > > > > > > > > >> > > > >> > > > > >
> > > > > > > > > > >> > > > >> > > > > > Hi guys,
> > > > > > > > > > >> > > > >> > > > > > I want to bring up a discussion
> > > about
> > > > > > > adding
> > > > > > > > > > >> support
> > > > > > > > > > >> > of
> > > > > > > > > > >> > > > User
> > > > > > > > > > >> > > > >> > > > > > Defined AsyncTableFunction in
> Flink.
> > > > > > > > > > >> > > > >> > > > > > Currently, async table function are
> > > > special
> > > > > > > > > functions
> > > > > > > > > > >> for
> > > > > > > > > > >> > > > table
> > > > > > > > > > >> > > > >> > > source
> > > > > > > > > > >> > > > >> > > > > > to perform
> > > > > > > > > > >> > > > >> > > > > > async lookup. However, it's worth
> to
> > > > > support
> > > > > > > the
> > > > > > > > > user
> > > > > > > > > > >> > > defined
> > > > > > > > > > >> > > > >> async
> > > > > > > > > > >> > > > >> > > > > > table function.
> > > > > > > > > > >> > > > >> > > > > > Because, in this way, the end SQL
> > user
> > > > can
> > > > > > > > leverage
> > > > > > > > > > it
> > > > > > > > > > >> to
> > > > > > > > > > >> > > > >> perform
> > > > > > > > > > >> > > > >> > the
> > > > > > > > > > >> > > > >> > > > > > async operation
> > > > > > > > > > >> > > > >> > > > > > which is useful to maximum the
> system
> > > > > > > throughput
> > > > > > > > > > >> > especially
> > > > > > > > > > >> > > > for
> > > > > > > > > > >> > > > >> IO
> > > > > > > > > > >> > > > >> > > > > > bottleneck case.
> > > > > > > > > > >> > > > >> > > > > >
> > > > > > > > > > >> > > > >> > > > > > You can find some more detail in
> [1].
> > > > > > > > > > >> > > > >> > > > > >
> > > > > > > > > > >> > > > >> > > > > > Looking forward to feedback
> > > > > > > > > > >> > > > >> > > > > >
> > > > > > > > > > >> > > > >> > > > > >
> > > > > > > > > > >> > > > >> > > > > > [1]:
> > > > > > > > > > >> > > > >> > > >
> > > > > > > > > > >> > > > >> > >
> > > > > > > > > > >> > > > >> >
> > > > > > > > > > >> > > > >>
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/%5BFLIP-313%5D+Add+support+of+User+Defined+AsyncTableFunction
> > > > > > > > > > >> > > > >> > > > > >
> > > > > > > > > > >> > > > >> > > > > > Thanks,
> > > > > > > > > > >> > > > >> > > > > > Aitozi.
> > > > > > > > > > >> > > > >> > > >
> > > > > > > > > > >> > > > >> > >
> > > > > > > > > > >> > > > >> >
> > > > > > > > > > >> > > > >>
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
