Re: [DISCUSS] FLIP-425: Asynchronous Execution Model

[Yanfei Lei]

Hi,
Thanks for Lorenzo'reply and Zakelly's supplement.

@Lorenzo

> I think you got my question, and I did not realize that is not even allowed 
> to modify some externally scoped variable in a lambda. I guess the point is 
> that it is possible, but the user would really need to be willing to do it 
> and "shoot him/herself in the foot".

I mean there is such a limitation (modifying the non-final local local
variable) in Java but we can have other ways to get around the
limitation as mentioned by Zakelly.

> For example, If the user now would not be sure that elements end up being in 
> correct windows, I am afraid this would somewhat simply hinder the watermark 
> concept as a whole. What do you think?

Elements are divided into different windows according to event time,
while a Watermark(T) only suggests that all elements with event time
before T have arrived. The watermark is used to determine whether an
element is late, the advance of watermarks would make the time fire
and further trigger the fire of windows. As long as we ensure that the
window fires after corresponding elements processing, the behavior is
the same as the behavior under sync API.

Zakelly Lan  于2024年3月25日周一 11:44写道：
>
> Hi Lorenzo,
>
> Thanks for your comments!
>
> I think you got my question, and I did not realize that is not even allowed
> > to modify some externally scoped variable in a lambda.
> > I guess the point is that it is possible, but the user would really need
> > to be willing to do it and "shoot him/herself in the foot".
> >
> Well, I think what Yanfei means is you cannot capture local variables which
> are not final or effectively final. It is not allowed in Java and the
> compiler as well as the IDE will report this error.
> Users can use some final containers to avoid this, with the lambda
> capturing the final container and modifying the internal value of it. e.g.
>
> ```
> final AtomicInteger x = new AtomicInteger(0);
>
> void processElement(...) {
>state.value().then( val -> { x.addAndGet(val); } );
>...
>out.collect(x.get());
> }
> ```
>
> For example, If the user now would not be sure that elements end up being
> > in correct windows, I am afraid this would somewhat simply hinder the
> > watermark concept as a whole. What do you think?
> >
> Well, I'm afraid there is no problem like this. Typically, the element will
> be assigned to a window based on its event time, and they will only
> manipulate the state associated with the related window. So as long as we
> ensure that the window fires after corresponding elements processing, the
> behavior is correct. Whether the elements for the latter window are
> processed before or after former window fires  does not matter. Well this
> is true for most cases, except for the scenarios where the state is not
> divided into slices for different windows, while we haven't seen a concrete
> use case for it. That's where the discussion between Xintong and me comes
> from[1].
>
>
> Hope these answer your questions.
>
> [1] https://lists.apache.org/thread/986zxq1k9rv3vkbk39yw16g24o6h83mz
>
> Best,
> Zakelly
>
>
> On Fri, Mar 22, 2024 at 11:11 PM 
> wrote:
>
> > Thank you Yanfei for addressing all the questions!
> >
> > > I'm not sure if I understand your question. In Java, this
> > case(modifying the local local variable) is not allowed[1], but there
> > are ways to get around the limitation of lambda.
> > In this case, the modification to x may be concurrent, which needs to
> > be handled carefully.
> >
> > I think you got my question, and I did not realize that is not even
> > allowed to modify some externally scoped variable in a lambda.
> > I guess the point is that it is possible, but the user would really need
> > to be willing to do it and "shoot him/herself in the foot".
> >
> > > an implicit fact in sync
> > API is that "event timer fire" would execute before "the subsequent
> > records of watermark", but in out-of-order mode(async API), the
> > execution order between them is not guaranteed
> >
> > Got it, what I don't get exactly is what type of inconsistency/issue the
> > user could face.
> > For example, If the user now would not be sure that elements end up being
> > in correct windows, I am afraid this would somewhat simply hinder the
> > watermark concept as a whole. What do you think?
> >
> > Thank you.
> >
> > On Mar 21, 2024 at 14:27 +0100, Yanfei Lei , wrote:
> > > Thanks for your reading and valuable comments!
> > >
> > > > 1) About locking VS reference counting: I would like to clear out
> > which mechanism prevents what:
> > > The `KeyAccountingUnit` implements locking behavior on keys and
> > > ensures 2 state requests on the same key happen in order.
> > > Double-locking the same key does not result in deadlocks (thanks to
> > > the `previous == record` condition in your pseudo-code), so, the same
> > > callback chain can update/read multiple times the same piece of state.
> > > On the other side we have the reference counting mechanism th

Re: [DISCUSS] FLIP-399: Flink Connector Doris

[wudi]

Hi, Feng

Yes, if the StreamLoad transaction timeout is very short, you may encounter 
this situation.

The timeout for StreamLoad transactions is controlled by the 
streaming_label_keep_max_second parameter [1] in FE (Frontend), and the default 
value is 12 hours. Currently, it is a global transaction configuration and 
cannot be set separately for a specific transaction.

However, I understand the default 12-hour timeout should cover most cases 
unless you are restarting from a checkpoint that occurred a long time ago. What 
do you think?


[1] 
https://github.com/apache/doris/blob/master/fe/fe-common/src/main/java/org/apache/doris/common/Config.java#L163-L168


Brs
di.wu

> 2024年3月25日 11:45，Feng Jin  写道：
> 
> Hi Di
> 
> Thanks for your reply.
> 
> The timeout I'm referring to here is not the commit timeout, but rather the
> timeout for a single streamLoad transaction.
> 
> Let's say we have set the transaction timeout for StreamLoad to be 10
> minutes. Now, imagine there is a Flink job with two subtasks. Due to
> significant data skew and backpressure issues, subtask 0 and subtask 1 are
> processing at different speeds. Subtask 0 finishes processing this
> checkpoint first, while subtask 1 takes another 10 minutes to complete its
> processing. At this point, the job's checkpoint is done. However, since
> subtask 0 has been waiting for subtask 1 all along, its corresponding
> streamLoad transaction closes after more than 10 minutes have passed - by
> which time the server has already cleaned up this transaction, leading to a
> failed commit.
> Therefore, I would like to know if in such situations we can avoid this
> problem by setting a longer lifespan for transactions.
> 
> 
> Best,
> Feng
> 
> 
> On Fri, Mar 22, 2024 at 10:24 PM wudi <676366...@qq.com.invalid> wrote:
> 
>> Hi, Feng,
>> 
>> 1. Are you suggesting that when a commit gets stuck, we can interrupt the
>> commit request using a timeout parameter? Currently, there is no such
>> parameter. In my understanding, in a two-phase commit, checkpoint must be
>> enabled, so the commit timeout is essentially the checkpoint timeout.
>> Therefore, it seems unnecessary to add an additional parameter here. What
>> do you think?
>> 
>> 2. In addition to deleting checkpoints to re-consume data again, the
>> Connector also provides an option to ignore commit errors[1]. However, this
>> option is only used for error recovery scenarios, such as when a
>> transaction is cleared by the server but you want to reuse the upstream
>> offset from the checkpoint.
>> 
>> 3. Also, thank you for pointing out the issue with the parameter. It has
>> already been addressed[2], but the FLIP changes were overlooked. It has
>> been updated.
>> 
>> [1]
>> https://github.com/apache/doris-flink-connector/blob/master/flink-doris-connector/src/main/java/org/apache/doris/flink/sink/committer/DorisCommitter.java#L150-L160
>> [2]
>> https://github.com/apache/doris-flink-connector/blob/master/flink-doris-connector/src/main/java/org/apache/doris/flink/table/DorisConfigOptions.java#L89-L98
>> 
>> Brs
>> di.wu
>> 
>> 
>> 
>>> 2024年3月22日 18:28，Feng Jin  写道：
>>> 
>>> Hi Di,
>>> 
>>> Thank you for the update, as well as quickly implementing corresponding
>>> capabilities including filter push down and project push down.
>>> 
>>> Regarding the transaction timeout, I still have some doubts. I would like
>>> to confirm if we can control this timeout parameter in the connector,
>> such
>>> as setting it to 10 minutes or 1 hour.
>>> Also, when a transaction is cleared by the server, the commit operation
>> of
>>> the connector will fail, leading to job failure. In this case, can users
>>> only choose to delete the checkpoint and re-consume historical data?
>>> 
>>> There is also a small question regarding the parameters*: *
>>> *doris.request.connect.timeout.ms <
>> http://doris.request.connect.timeout.ms>*
>>> and d*oris.request.read.timeout.ms >> *,
>>> can we change them to Duration type and remove the "ms" suffix.?
>>> This way, all time parameters can be kept uniform in type as duration.
>>> 
>>> 
>>> Best,
>>> Feng
>>> 
>>> On Fri, Mar 22, 2024 at 4:46 PM wudi <676366...@qq.com.invalid> wrote:
>>> 
 Hi, Feng,
 Thank you, that's a great suggestion !
 
 I have already implemented FilterPushDown and removed that parameter on
 DorisDynamicTableSource[1], and also updated FLIP.
 
 Regarding the mention of [Doris also aborts transactions], it may not
>> have
 been described accurately. It mainly refers to the automatic expiration
>> of
 long-running transactions in Doris that have not been committed for a
 prolonged period.
 
 As for two-phase commit, when a commit fails, the checkpoint will also
 fail, and the job will be continuously retried.
 
 [1]
 
>> https://github.com/apache/doris-flink-connector/blob/master/flink-doris-connector/src/main/java/org/apache/doris/flink/table/DorisDynamicTableSo

[DISCUSS] Flink Website Menu Adjustment

[gongzhongqiang]

Hi everyone,

I'd like to start a discussion on adjusting the Flink website [1] menu to
improve accuracy and usability.While migrating Flink CDC documentation
to the website, I found outdated links, need to review and update menus
for the most relevant information for our users.


Proposal:

- Remove Paimon [2] from the "Getting Started" and "Documentation" menus:
Paimon [2] is now an independent top project of ASF. CC： jingsong lees

- Sort the projects in the subdirectory by the activity of the projects.
Here I list the number of releases for each project in the past year.

Flink Kubernetes Operator : 7
Flink CDC : 5
Flink ML  : 2
Flink Stateful Functions : 1


Expected Outcome :

- Menu "Getting Started"

Before:

With Flink

With Flink Stateful Functions

With Flink ML

With Flink Kubernetes Operator

With Paimon(incubating) (formerly Flink Table Store)

With Flink CDC

Training Course


After:

With Flink
With Flink Kubernetes Operator

With Flink CDC

With Flink ML

With Flink Stateful Functions

Training Course


- Menu "Documentation" will same with "Getting Started"


I look forward to hearing your thoughts and suggestions on this proposal.

[1] https://flink.apache.org/
[2] https://github.com/apache/incubator-paimon
[3] https://github.com/apache/flink-statefun



Best regards,

Zhongqiang Gong

Re: Support minibatch for TopNFunction

[Ron liu]

Hi, Roman

Thanks for your proposal, I intuitively feel that this optimization would
be very useful to reduce the amount of message amplification for TopN
operators. After briefly looking at your google docs, I have the following
questions:

1. Whether you can describe in detail the principle of solving the TopN
operator record amplification, similar to Minibatch Join[1], through the
figure of current Motivation part, I can not understand how you did it
2. TopN has currently multiple implementation functions, including
AppendOnlyFirstNFunction, AppendOnlyTopNFunction, FastTop1Function,
RetractableTopNFunction, UpdatableTopNFunction. Is it possible to elaborate
on which patterns the Minibatch optimization applies to?
3. Is it possible to provide the PoC code?
4. finally, we need a formal FLIP document on the wiki[2].

[1]
https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/tuning/#minibatch-regular-joins
[2]
https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals

Best,
Ron

Roman Boyko  于2024年3月24日周日 01:14写道：

> Hi Flink Community,
>
> I tried to describe my idea about minibatch for TopNFunction in this doc -
>
> https://docs.google.com/document/d/1YPHwxKfiGSUOUOa6bc68fIJHO_UojTwZEC29VVEa-Uk/edit?usp=sharing
>
> Looking forward to your feedback, thank you
>
> On Tue, 19 Mar 2024 at 12:24, Roman Boyko  wrote:
>
> > Hello Flink Community,
> >
> > The same problem with record amplification as described in FLIP-415:
> Introduce
> > a new join operator to support minibatch[1] exists for most of
> > implementations of AbstractTopNFunction. Especially when the rank is
> > provided to output. For example, when calculating Top100 with rank
> output,
> > every input record might produce 100 -U records and 100 +U records.
> >
> > According to my POC (which is similar to FLIP-415) the record
> > amplification could be significantly reduced by using input or output
> > buffer.
> >
> > What do you think if we implement such optimization for TopNFunctions?
> >
> > [1]
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-415
> > %3A+Introduce+a+new+join+operator+to+support+minibatch
> >
> > --
> > Best regards,
> > Roman Boyko
> > e.: ro.v.bo...@gmail.com
> > m.: +79059592443
> > telegram: @rboyko
> >
>
>
> --
> Best regards,
> Roman Boyko
> e.: ro.v.bo...@gmail.com
> m.: +79059592443
> telegram: @rboyko
>

Re: [DISCUSS] FLIP-437: Support ML Models in Flink SQL

[Timo Walther]

Hi Jark and Hao,

thanks for the information, Jark! Great that the Calcite community 
already fixed the problem for us. +1 to adopt the simplified syntax 
asap. Maybe even before we upgrade Calcite (i.e. copy over classes), if 
upgrading Calcite is too much work right now?


> Is `DESCRIPTOR` a must in the syntax?

Yes, we should still stick to the standard as much as possible and all 
vendors use DESCRIPTOR/COLUMNS for distinuishing columns vs. literal 
arguments. So the final syntax of this discussion would be:



SELECT f1, f2, label FROM
  ML_PREDICT(TABLE `my_data`, `classifier_model`, DESCRIPTOR(f1, f2))

SELECT * FROM
  ML_EVALUATE(TABLE `eval_data`, `classifier_model`, DESCRIPTOR(f1, f2))

Please double check if this is implementable with the current stack. I 
fear the parser or validator might not like the "identifier" argument?


Make sure that also these variations are supported:

SELECT f1, f2, label FROM
  ML_PREDICT(
TABLE `my_data`,
my_cat.my_db.classifier_model,
DESCRIPTOR(f1, f2))

SELECT f1, f2, label FROM
  ML_PREDICT(
input => TABLE `my_data`,
model => my_cat.my_db.classifier_model,
args => DESCRIPTOR(f1, f2))

It might be safer and more future proof to wrap a MODEL() function 
around it. This would be more in sync with the standard that actually 
still requires to put a TABLE() around the input argument:


ML_PREDICT(TABLE(`my_data`) PARTITIONED BY c1 ORDERED BY c1, )

So the safest option would be the long-term solution:

SELECT f1, f2, label FROM
  ML_PREDICT(
input => TABLE(my_data),
model => MODEL(my_cat.my_db.classifier_model),
args => DESCRIPTOR(f1, f2))

But I'm fine with this if others have a strong opinion:

SELECT f1, f2, label FROM
  ML_PREDICT(
input => TABLE `my_data`,
model => my_cat.my_db.classifier_model,
args => DESCRIPTOR(f1, f2))

Some feedback for the remainder of the FLIP:

1) Simplify catalog objects

I would suggest to drop:
CatalogModel.getModelKind()
CatalogModel.getModelTask()

A catalog object should fully resemble the DDL. And since the DDL puts 
those properties in the WITH clause, the catalog object should the same 
(i.e. put them into the `getModelOptions()`). Btw renaming this method 
to just `getOptions()` for consistency should be good as well. 
Internally, we can still provide enums for these frequently used 
classes. Similar to what we do in `FactoryUtil` for other frequently 
used options.


Remove `getDescription()` and `getDetailedDescription()`. They were a 
mistake for CatalogTable and should actually be deprecated. They got 
replaced by `getComment()` which is sufficient.


2) CREATE TEMPORARY MODEL is not supported.

This is an unnecessary restriction. We should support temporary versions 
of these catalog objects as well for consistency. Adding support for 
this should be straightforward.


3) DESCRIBE | DESC } MODEL [catalog_name.][database_name.]model_name

I would suggest we support `SHOW CREATE MODEL` instead. Similar to `SHOW 
CREATE TABLE`, this should show all properties. If we support `DESCRIBE 
MODEL` it should only list the input parameters similar to `DESCRIBE 
TABLE` only shows the columns (not the WITH clause).


Regards,
Timo


On 23.03.24 13:17, Ahmed Hamdy wrote:

Hi everyone,
+1 for this proposal, I believe it is very useful to the minimum, It would
be great even having  "ML_PREDICT" and "ML_EVALUATE" as built-in PTFs in
this FLIP as discussed.
IIUC this will be included in the 1.20 roadmap?
Best Regards
Ahmed Hamdy


On Fri, 22 Mar 2024 at 23:54, Hao Li  wrote:


Hi Timo and Jark,

I agree Oracle's syntax seems concise and more descriptive. For the
built-in `ML_PREDICT` and `ML_EVALUATE` functions I agree with Jark we can
support them as built-in PTF using `SqlTableFunction` for this FLIP. We can
have a different FLIP discussing user defined PTF and adopt that later for
model functions later. To summarize, the current proposed syntax is

SELECT f1, f2, label FROM TABLE(ML_PREDICT(TABLE `my_data`,
`classifier_model`, f1, f2))

SELECT * FROM TABLE(ML_EVALUATE(TABLE `eval_data`, `classifier_model`, f1,
f2))

Is `DESCRIPTOR` a must in the syntax? If so, it becomes

SELECT f1, f2, label FROM TABLE(ML_PREDICT(TABLE `my_data`,
`classifier_model`, DESCRIPTOR(f1), DESCRIPTOR(f2)))

SELECT * FROM TABLE(ML_EVALUATE(TABLE `eval_data`, `classifier_model`,
DESCRIPTOR(f1), DESCRIPTOR(f2)))

If Calcite supports dropping outer table keyword, it becomes

SELECT f1, f2, label FROM ML_PREDICT(TABLE `my_data`, `classifier_model`,
DESCRIPTOR(f1), DESCRIPTOR(f2))

SELECT * FROM ML_EVALUATE(TABLE `eval_data`, `classifier_model`,
DESCRIPTOR(
f1), DESCRIPTOR(f2))

Thanks,
Hao



On Fri, Mar 22, 2024 at 9:16 AM Jark Wu  wrote:


Sorry, I mean we can bump the Calcite version if needed in Flink 1.20.

On Fri, 22 Mar 2024 at 22:19, Jark Wu  wrote:


Hi Timo,

Introducing user-defined PTF is very useful in Flink, I'm +1 for this.
But I think the ML model FLIP is not blocked by this, because we
can intro

Re: [DISCUSS] FLIP-435: Introduce a New Dynamic Table for Simplifying Data Pipelines

[Timo Walther]

Hi Ron and Lincoln,

thanks for the quick response and the very insightful discussion.

> we might limit future opportunities to optimize queries
> through automatic materialization rewriting by allowing data
> modifications, thus losing the potential for such optimizations.

This argument makes a lot of sense to me. Due to the updates, the system 
is not in full control of the persisted data. However, the system is 
still in full control of the job that powers the refresh. So if the 
system manages all updating pipelines, it could still leverage automatic 
materialization rewriting but without leveraging the data at rest (only 
the data in flight).


> we are considering another candidate, Derived Table, the term 'derive'
> suggests a query, and 'table' retains modifiability. This approach
> would not disrupt our current concept of a dynamic table

I did some research on this term. The SQL standard uses the term 
"derived table" extensively (defined in section 4.17.3). Thus, a lot of 
vendors adopt this for simply referring to a table within a subclause:


https://dev.mysql.com/doc/refman/8.0/en/derived-tables.html

https://infocenter.sybase.com/help/topic/com.sybase.infocenter.dc32300.1600/doc/html/san1390612291252.html

https://www.c-sharpcorner.com/article/derived-tables-vs-common-table-expressions/

https://stackoverflow.com/questions/26529804/what-are-the-derived-tables-in-my-explain-statement

https://www.sqlservercentral.com/articles/sql-derived-tables

Esp. the latter example is interesting, SQL Server allows things like 
this on derived tables:


UPDATE T SET Name='Timo' FROM (SELECT * FROM Product) AS T

SELECT * FROM Product;

Btw also Snowflake's dynamic table state:

> Because the content of a dynamic table is fully determined
> by the given query, the content cannot be changed by using DML.
> You don’t insert, update, or delete the rows in a dynamic table.

So a new term makes a lot of sense.

How about using `UPDATING`?

CREATE UPDATING TABLE

This reflects that modifications can be made and from an 
English-language perspective you can PAUSE or RESUME the UPDATING.

Thus, a user can define UPDATING interval and mode?

Looking forward to your thoughts.

Regards,
Timo


On 25.03.24 07:09, Ron liu wrote:

Hi, Ahmed

Thanks for your feedback.

Regarding your question:


I want to iterate on Timo's comments regarding the confusion between

"Dynamic Table" and current Flink "Table". Should the refactoring of the
system happen in 2.0, should we rename it in this Flip ( as the suggestions
in the thread ) and address the holistic changes in a separate Flip for 2.0?

Lincoln proposed a new concept in reply to Timo: Derived Table, which is a
combination of Dynamic Table + Continuous Query, and the use of Derived
Table will not conflict with existing concepts, what do you think?


I feel confused with how it is further with other components, the

examples provided feel like a standalone ETL job, could you provide in the
FLIP an example where the table is further used in subsequent queries
(specially in batch mode).

Thanks for your suggestion, I added how to use Dynamic Table in FLIP user
story section, Dynamic Table can be referenced by downstream Dynamic Table
and can also support OLAP queries.

Best,
Ron

Ron liu  于2024年3月23日周六 10:35写道：


Hi, Feng

Thanks for your feedback.


Although currently we restrict users from modifying the query, I wonder

if
we can provide a better way to help users rebuild it without affecting
downstream OLAP queries.

Considering the problem of data consistency, so in the first step we are
strictly limited in semantics and do not support modify the query. This is
really a good problem, one of my ideas is to introduce a syntax similar to
SWAP [1], which supports exchanging two Dynamic Tables.


 From the documentation, the definitions SQL and job information are

stored in the Catalog. Does this mean that if a system needs to adapt to
Dynamic Tables, it also needs to store Flink's job information in the
corresponding system?
For example, does MySQL's Catalog need to store flink job information as
well?

Yes, currently we need to rely on Catalog to store refresh job information.


Users still need to consider how much memory is being used, how large

the concurrency is, which type of state backend is being used, and may need
to set TTL expiration.

Similar to the current practice, job parameters can be set via the Flink
conf or SET commands


When we submit a refresh command, can we help users detect if there are

any
running jobs and automatically stop them before executing the refresh
command? Then wait for it to complete before restarting the background
streaming job?

Purely from a technical implementation point of view, your proposal is
doable, but it would be more costly. Also I think data consistency itself
is the responsibility of the user, similar to how Regular Table is now also
the responsibility of the user, so it's consistent with its behavior and no
addition

Re: [DISCUSS] FLIP-399: Flink Connector Doris

[Feng Jin]

Hi Di

Thank you for your patience and explanation.

If this is a server-side configuration, we currently cannot modify it in
the client configuration. If Doris supports client-side configuration in
the future, we can reconsider whether to support it.

I currently have no other questions regarding this FLIP.  LGTM.


Best,
Feng

On Mon, Mar 25, 2024 at 3:42 PM wudi <676366...@qq.com.invalid> wrote:

> Hi, Feng
>
> Yes, if the StreamLoad transaction timeout is very short, you may
> encounter this situation.
>
> The timeout for StreamLoad transactions is controlled by the
> streaming_label_keep_max_second parameter [1] in FE (Frontend), and the
> default value is 12 hours. Currently, it is a global transaction
> configuration and cannot be set separately for a specific transaction.
>
> However, I understand the default 12-hour timeout should cover most cases
> unless you are restarting from a checkpoint that occurred a long time ago.
> What do you think?
>
>
> [1]
> https://github.com/apache/doris/blob/master/fe/fe-common/src/main/java/org/apache/doris/common/Config.java#L163-L168
>
>
> Brs
> di.wu
>
> > 2024年3月25日 11:45，Feng Jin  写道：
> >
> > Hi Di
> >
> > Thanks for your reply.
> >
> > The timeout I'm referring to here is not the commit timeout, but rather
> the
> > timeout for a single streamLoad transaction.
> >
> > Let's say we have set the transaction timeout for StreamLoad to be 10
> > minutes. Now, imagine there is a Flink job with two subtasks. Due to
> > significant data skew and backpressure issues, subtask 0 and subtask 1
> are
> > processing at different speeds. Subtask 0 finishes processing this
> > checkpoint first, while subtask 1 takes another 10 minutes to complete
> its
> > processing. At this point, the job's checkpoint is done. However, since
> > subtask 0 has been waiting for subtask 1 all along, its corresponding
> > streamLoad transaction closes after more than 10 minutes have passed - by
> > which time the server has already cleaned up this transaction, leading
> to a
> > failed commit.
> > Therefore, I would like to know if in such situations we can avoid this
> > problem by setting a longer lifespan for transactions.
> >
> >
> > Best,
> > Feng
> >
> >
> > On Fri, Mar 22, 2024 at 10:24 PM wudi <676366...@qq.com.invalid> wrote:
> >
> >> Hi, Feng,
> >>
> >> 1. Are you suggesting that when a commit gets stuck, we can interrupt
> the
> >> commit request using a timeout parameter? Currently, there is no such
> >> parameter. In my understanding, in a two-phase commit, checkpoint must
> be
> >> enabled, so the commit timeout is essentially the checkpoint timeout.
> >> Therefore, it seems unnecessary to add an additional parameter here.
> What
> >> do you think?
> >>
> >> 2. In addition to deleting checkpoints to re-consume data again, the
> >> Connector also provides an option to ignore commit errors[1]. However,
> this
> >> option is only used for error recovery scenarios, such as when a
> >> transaction is cleared by the server but you want to reuse the upstream
> >> offset from the checkpoint.
> >>
> >> 3. Also, thank you for pointing out the issue with the parameter. It has
> >> already been addressed[2], but the FLIP changes were overlooked. It has
> >> been updated.
> >>
> >> [1]
> >>
> https://github.com/apache/doris-flink-connector/blob/master/flink-doris-connector/src/main/java/org/apache/doris/flink/sink/committer/DorisCommitter.java#L150-L160
> >> [2]
> >>
> https://github.com/apache/doris-flink-connector/blob/master/flink-doris-connector/src/main/java/org/apache/doris/flink/table/DorisConfigOptions.java#L89-L98
> >>
> >> Brs
> >> di.wu
> >>
> >>
> >>
> >>> 2024年3月22日 18:28，Feng Jin  写道：
> >>>
> >>> Hi Di,
> >>>
> >>> Thank you for the update, as well as quickly implementing corresponding
> >>> capabilities including filter push down and project push down.
> >>>
> >>> Regarding the transaction timeout, I still have some doubts. I would
> like
> >>> to confirm if we can control this timeout parameter in the connector,
> >> such
> >>> as setting it to 10 minutes or 1 hour.
> >>> Also, when a transaction is cleared by the server, the commit operation
> >> of
> >>> the connector will fail, leading to job failure. In this case, can
> users
> >>> only choose to delete the checkpoint and re-consume historical data?
> >>>
> >>> There is also a small question regarding the parameters*: *
> >>> *doris.request.connect.timeout.ms <
> >> http://doris.request.connect.timeout.ms>*
> >>> and d*oris.request.read.timeout.ms <
> http://oris.request.read.timeout.ms
> >>> *,
> >>> can we change them to Duration type and remove the "ms" suffix.?
> >>> This way, all time parameters can be kept uniform in type as duration.
> >>>
> >>>
> >>> Best,
> >>> Feng
> >>>
> >>> On Fri, Mar 22, 2024 at 4:46 PM wudi <676366...@qq.com.invalid> wrote:
> >>>
>  Hi, Feng,
>  Thank you, that's a great suggestion !
> 
>  I have already implemented FilterPushDown and removed that parameter
> on
>  Dor

Re: [DISCUSS] Planning Flink 1.20

[Ufuk Celebi]

Hey all,

I'd like to join the release managers for 1.20 as well. I'm looking forward to 
getting more actively involved again.

Cheers,

Ufuk

On Sun, Mar 24, 2024, at 11:27 AM, Ahmed Hamdy wrote:
> +1 for the proposed timeline and release managers.
> Best Regards
> Ahmed Hamdy
> 
> 
> On Fri, 22 Mar 2024 at 07:41, Xintong Song  wrote:
> 
> > +1 for the proposed timeline and Weijie & Rui as the release managers.
> >
> > I think it would be welcomed if another 1-2 volunteers join as the release
> > managers, but that's not a must. We used to have only 1-2 release managers
> > for each release,
> >
> > Best,
> >
> > Xintong
> >
> >
> >
> > On Fri, Mar 22, 2024 at 2:55 PM Jark Wu  wrote:
> >
> > > Thanks for kicking this off.
> > >
> > > +1 for the volunteered release managers (Weijie Guo, Rui Fan) and the
> > > targeting date (feature freeze: June 15).
> > >
> > > Best,
> > > Jark
> > >
> > >
> > >
> > >
> > >
> > > On Fri, 22 Mar 2024 at 14:00, Rui Fan <1996fan...@gmail.com> wrote:
> > >
> > > > Thanks Leonard for this feedback and help!
> > > >
> > > > Best,
> > > > Rui
> > > >
> > > > On Fri, Mar 22, 2024 at 12:36 PM weijie guo  > >
> > > > wrote:
> > > >
> > > > > Thanks Leonard!
> > > > >
> > > > > > I'd like to help you if you need some help like permissions from
> > PMC
> > > > > side, please feel free to ping me.
> > > > >
> > > > > Nice to know. It'll help a lot!
> > > > >
> > > > > Best regards,
> > > > >
> > > > > Weijie
> > > > >
> > > > >
> > > > > Leonard Xu  于2024年3月22日周五 12:09写道：
> > > > >
> > > > >> +1 for the proposed release managers (Weijie Guo, Rui Fan), both the
> > > two
> > > > >> candidates are pretty active committers thus I believe they know the
> > > > >> community development process well. The recent releases have four
> > > > release
> > > > >> managers, and I am also looking forward to having other volunteers
> > > > >>  join the management of Flink 1.20.
> > > > >>
> > > > >> +1 for targeting date (feature freeze: June 15, 2024), referring to
> > > the
> > > > >> release cycle of recent versions, release cycle of 4 months makes
> > > sense
> > > > to
> > > > >> me.
> > > > >>
> > > > >>
> > > > >> I'd like to help you if you need some help like permissions from PMC
> > > > >> side, please feel free to ping me.
> > > > >>
> > > > >> Best,
> > > > >> Leonard
> > > > >>
> > > > >>
> > > > >> > 2024年3月19日 下午5:35，Rui Fan <1996fan...@gmail.com> 写道：
> > > > >> >
> > > > >> > Hi Weijie,
> > > > >> >
> > > > >> > Thanks for kicking off 1.20! I'd like to join you and participate
> > in
> > > > the
> > > > >> > 1.20 release.
> > > > >> >
> > > > >> > Best,
> > > > >> > Rui
> > > > >> >
> > > > >> > On Tue, Mar 19, 2024 at 5:30 PM weijie guo <
> > > guoweijieres...@gmail.com
> > > > >
> > > > >> > wrote:
> > > > >> >
> > > > >> >> Hi everyone,
> > > > >> >>
> > > > >> >> With the release announcement of Flink 1.19, it's a good time to
> > > kick
> > > > >> off
> > > > >> >> discussion of the next release 1.20.
> > > > >> >>
> > > > >> >>
> > > > >> >> - Release managers
> > > > >> >>
> > > > >> >>
> > > > >> >> I'd like to volunteer as one of the release managers this time.
> > It
> > > > has
> > > > >> been
> > > > >> >> good practice to have a team of release managers from different
> > > > >> >> backgrounds, so please raise you hand if you'd like to volunteer
> > > and
> > > > >> get
> > > > >> >> involved.
> > > > >> >>
> > > > >> >>
> > > > >> >>
> > > > >> >> - Timeline
> > > > >> >>
> > > > >> >>
> > > > >> >> Flink 1.19 has been released. With a target release cycle of 4
> > > > months,
> > > > >> >> we propose a feature freeze date of *June 15, 2024*.
> > > > >> >>
> > > > >> >>
> > > > >> >>
> > > > >> >> - Collecting features
> > > > >> >>
> > > > >> >>
> > > > >> >> As usual, we've created a wiki page[1] for collecting new
> > features
> > > in
> > > > >> 1.20.
> > > > >> >>
> > > > >> >>
> > > > >> >> In addition, we already have a number of FLIPs that have been
> > voted
> > > > or
> > > > >> are
> > > > >> >> in the process, including pre-works for version 2.0.
> > > > >> >>
> > > > >> >>
> > > > >> >> In the meantime, the release management team will be finalized in
> > > the
> > > > >> next
> > > > >> >> few days, and we'll continue to create Jira Boards and Sync
> > > meetings
> > > > >> >> to make it easy
> > > > >> >> for everyone to get an overview and track progress.
> > > > >> >>
> > > > >> >>
> > > > >> >>
> > > > >> >> Best regards,
> > > > >> >>
> > > > >> >> Weijie
> > > > >> >>
> > > > >> >>
> > > > >> >>
> > > > >> >> [1]
> > https://cwiki.apache.org/confluence/display/FLINK/1.20+Release
> > > > >> >>
> > > > >>
> > > > >>
> > > >
> > >
> >
> 

Re: [DISCUSS] Planning Flink 1.20

[Robert Metzger]

Hi, thanks for starting the discussion.

+1 for the proposed timeline and the three proposed release managers.

I'm happy to join the release managers group as well, as a backup for Ufuk
(unless there are objections about the number of release managers)

On Mon, Mar 25, 2024 at 11:04 AM Ufuk Celebi  wrote:

> Hey all,
>
> I'd like to join the release managers for 1.20 as well. I'm looking
> forward to getting more actively involved again.
>
> Cheers,
>
> Ufuk
>
> On Sun, Mar 24, 2024, at 11:27 AM, Ahmed Hamdy wrote:
> > +1 for the proposed timeline and release managers.
> > Best Regards
> > Ahmed Hamdy
> >
> >
> > On Fri, 22 Mar 2024 at 07:41, Xintong Song 
> wrote:
> >
> > > +1 for the proposed timeline and Weijie & Rui as the release managers.
> > >
> > > I think it would be welcomed if another 1-2 volunteers join as the
> release
> > > managers, but that's not a must. We used to have only 1-2 release
> managers
> > > for each release,
> > >
> > > Best,
> > >
> > > Xintong
> > >
> > >
> > >
> > > On Fri, Mar 22, 2024 at 2:55 PM Jark Wu  wrote:
> > >
> > > > Thanks for kicking this off.
> > > >
> > > > +1 for the volunteered release managers (Weijie Guo, Rui Fan) and the
> > > > targeting date (feature freeze: June 15).
> > > >
> > > > Best,
> > > > Jark
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Fri, 22 Mar 2024 at 14:00, Rui Fan <1996fan...@gmail.com> wrote:
> > > >
> > > > > Thanks Leonard for this feedback and help!
> > > > >
> > > > > Best,
> > > > > Rui
> > > > >
> > > > > On Fri, Mar 22, 2024 at 12:36 PM weijie guo <
> guoweijieres...@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > Thanks Leonard!
> > > > > >
> > > > > > > I'd like to help you if you need some help like permissions
> from
> > > PMC
> > > > > > side, please feel free to ping me.
> > > > > >
> > > > > > Nice to know. It'll help a lot!
> > > > > >
> > > > > > Best regards,
> > > > > >
> > > > > > Weijie
> > > > > >
> > > > > >
> > > > > > Leonard Xu  于2024年3月22日周五 12:09写道：
> > > > > >
> > > > > >> +1 for the proposed release managers (Weijie Guo, Rui Fan),
> both the
> > > > two
> > > > > >> candidates are pretty active committers thus I believe they
> know the
> > > > > >> community development process well. The recent releases have
> four
> > > > > release
> > > > > >> managers, and I am also looking forward to having other
> volunteers
> > > > > >>  join the management of Flink 1.20.
> > > > > >>
> > > > > >> +1 for targeting date (feature freeze: June 15, 2024),
> referring to
> > > > the
> > > > > >> release cycle of recent versions, release cycle of 4 months
> makes
> > > > sense
> > > > > to
> > > > > >> me.
> > > > > >>
> > > > > >>
> > > > > >> I'd like to help you if you need some help like permissions
> from PMC
> > > > > >> side, please feel free to ping me.
> > > > > >>
> > > > > >> Best,
> > > > > >> Leonard
> > > > > >>
> > > > > >>
> > > > > >> > 2024年3月19日 下午5:35，Rui Fan <1996fan...@gmail.com> 写道：
> > > > > >> >
> > > > > >> > Hi Weijie,
> > > > > >> >
> > > > > >> > Thanks for kicking off 1.20! I'd like to join you and
> participate
> > > in
> > > > > the
> > > > > >> > 1.20 release.
> > > > > >> >
> > > > > >> > Best,
> > > > > >> > Rui
> > > > > >> >
> > > > > >> > On Tue, Mar 19, 2024 at 5:30 PM weijie guo <
> > > > guoweijieres...@gmail.com
> > > > > >
> > > > > >> > wrote:
> > > > > >> >
> > > > > >> >> Hi everyone,
> > > > > >> >>
> > > > > >> >> With the release announcement of Flink 1.19, it's a good
> time to
> > > > kick
> > > > > >> off
> > > > > >> >> discussion of the next release 1.20.
> > > > > >> >>
> > > > > >> >>
> > > > > >> >> - Release managers
> > > > > >> >>
> > > > > >> >>
> > > > > >> >> I'd like to volunteer as one of the release managers this
> time.
> > > It
> > > > > has
> > > > > >> been
> > > > > >> >> good practice to have a team of release managers from
> different
> > > > > >> >> backgrounds, so please raise you hand if you'd like to
> volunteer
> > > > and
> > > > > >> get
> > > > > >> >> involved.
> > > > > >> >>
> > > > > >> >>
> > > > > >> >>
> > > > > >> >> - Timeline
> > > > > >> >>
> > > > > >> >>
> > > > > >> >> Flink 1.19 has been released. With a target release cycle of
> 4
> > > > > months,
> > > > > >> >> we propose a feature freeze date of *June 15, 2024*.
> > > > > >> >>
> > > > > >> >>
> > > > > >> >>
> > > > > >> >> - Collecting features
> > > > > >> >>
> > > > > >> >>
> > > > > >> >> As usual, we've created a wiki page[1] for collecting new
> > > features
> > > > in
> > > > > >> 1.20.
> > > > > >> >>
> > > > > >> >>
> > > > > >> >> In addition, we already have a number of FLIPs that have been
> > > voted
> > > > > or
> > > > > >> are
> > > > > >> >> in the process, including pre-works for version 2.0.
> > > > > >> >>
> > > > > >> >>
> > > > > >> >> In the meantime, the release management team will be
> finalized in
> > > > the
> > > > > >> next
> > > > > >> >> few days, and we'll continue to create Jira Boards and Sync
> > > > meetings
> > > > > >> >

[ANNOUNCE] Apache Flink Kubernetes Operator 1.8.0 released

[Maximilian Michels]

The Apache Flink community is very happy to announce the release of
the Apache Flink Kubernetes Operator version 1.8.0.

The Flink Kubernetes Operator allows users to manage their Apache
Flink applications on Kubernetes through all aspects of their
lifecycle.

Release highlights:
- Flink Autotuning automatically adjusts TaskManager memory
- Flink Autoscaling metrics and decision accuracy improved
- Improve standalone Flink Autoscaling
- Savepoint trigger nonce for savepoint-based restarts
- Operator stability improvements for cluster shutdown

Blog post: 
https://flink.apache.org/2024/03/21/apache-flink-kubernetes-operator-1.8.0-release-announcement/

The release is available for download at:
https://flink.apache.org/downloads.html

Maven artifacts for Flink Kubernetes Operator can be found at:
https://search.maven.org/artifact/org.apache.flink/flink-kubernetes-operator

Official Docker image for Flink Kubernetes Operator can be found at:
https://hub.docker.com/r/apache/flink-kubernetes-operator

The full release notes are available in Jira:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12353866&projectId=12315522

We would like to thank the Apache Flink community and its contributors
who made this release possible!

Cheers,
Max

Re: [DISCUSS] FLIP-435: Introduce a New Dynamic Table for Simplifying Data Pipelines

[Timo Walther]

After thinking about this more, this discussion boils down to whether 
this is a special table or a special materialized view. In both cases, 
we would need to add a special keyword:


Either

CREATE UPDATING TABLE

or

CREATE UPDATING MATERIALIZED VIEW

I still feel that the periodic refreshing behavior is closer to a MV. If 
we add a special keyword to MV, the optimizer would know that the data 
cannot be used for query optimizations.


I will ask more people for their opinion.

Regards,
Timo


On 25.03.24 10:45, Timo Walther wrote:

Hi Ron and Lincoln,

thanks for the quick response and the very insightful discussion.

 > we might limit future opportunities to optimize queries
 > through automatic materialization rewriting by allowing data
 > modifications, thus losing the potential for such optimizations.

This argument makes a lot of sense to me. Due to the updates, the system 
is not in full control of the persisted data. However, the system is 
still in full control of the job that powers the refresh. So if the 
system manages all updating pipelines, it could still leverage automatic 
materialization rewriting but without leveraging the data at rest (only 
the data in flight).


 > we are considering another candidate, Derived Table, the term 'derive'
 > suggests a query, and 'table' retains modifiability. This approach
 > would not disrupt our current concept of a dynamic table

I did some research on this term. The SQL standard uses the term 
"derived table" extensively (defined in section 4.17.3). Thus, a lot of 
vendors adopt this for simply referring to a table within a subclause:


https://dev.mysql.com/doc/refman/8.0/en/derived-tables.html

https://infocenter.sybase.com/help/topic/com.sybase.infocenter.dc32300.1600/doc/html/san1390612291252.html

https://www.c-sharpcorner.com/article/derived-tables-vs-common-table-expressions/

https://stackoverflow.com/questions/26529804/what-are-the-derived-tables-in-my-explain-statement

https://www.sqlservercentral.com/articles/sql-derived-tables

Esp. the latter example is interesting, SQL Server allows things like 
this on derived tables:


UPDATE T SET Name='Timo' FROM (SELECT * FROM Product) AS T

SELECT * FROM Product;

Btw also Snowflake's dynamic table state:

 > Because the content of a dynamic table is fully determined
 > by the given query, the content cannot be changed by using DML.
 > You don’t insert, update, or delete the rows in a dynamic table.

So a new term makes a lot of sense.

How about using `UPDATING`?

CREATE UPDATING TABLE

This reflects that modifications can be made and from an 
English-language perspective you can PAUSE or RESUME the UPDATING.

Thus, a user can define UPDATING interval and mode?

Looking forward to your thoughts.

Regards,
Timo


On 25.03.24 07:09, Ron liu wrote:

Hi, Ahmed

Thanks for your feedback.

Regarding your question:


I want to iterate on Timo's comments regarding the confusion between

"Dynamic Table" and current Flink "Table". Should the refactoring of the
system happen in 2.0, should we rename it in this Flip ( as the 
suggestions
in the thread ) and address the holistic changes in a separate Flip 
for 2.0?


Lincoln proposed a new concept in reply to Timo: Derived Table, which 
is a

combination of Dynamic Table + Continuous Query, and the use of Derived
Table will not conflict with existing concepts, what do you think?


I feel confused with how it is further with other components, the
examples provided feel like a standalone ETL job, could you provide in 
the

FLIP an example where the table is further used in subsequent queries
(specially in batch mode).

Thanks for your suggestion, I added how to use Dynamic Table in FLIP user
story section, Dynamic Table can be referenced by downstream Dynamic 
Table

and can also support OLAP queries.

Best,
Ron

Ron liu  于2024年3月23日周六 10:35写道：


Hi, Feng

Thanks for your feedback.


Although currently we restrict users from modifying the query, I wonder

if
we can provide a better way to help users rebuild it without affecting
downstream OLAP queries.

Considering the problem of data consistency, so in the first step we are
strictly limited in semantics and do not support modify the query. 
This is
really a good problem, one of my ideas is to introduce a syntax 
similar to

SWAP [1], which supports exchanging two Dynamic Tables.


 From the documentation, the definitions SQL and job information are

stored in the Catalog. Does this mean that if a system needs to adapt to
Dynamic Tables, it also needs to store Flink's job information in the
corresponding system?
For example, does MySQL's Catalog need to store flink job information as
well?

Yes, currently we need to rely on Catalog to store refresh job 
information.



Users still need to consider how much memory is being used, how large
the concurrency is, which type of state backend is being used, and 
may need

to set TTL expiration.

Similar to the current practice, job parameters can be set via the Flin

Re: [ANNOUNCE] Apache Flink Kubernetes Operator 1.8.0 released

[Rui Fan]

Congratulations! Thanks Max for the release and all involved for the great
work!

A gentle reminder to users: the maven artifact has just been released and
will take some time to complete.

Best,
Rui

On Mon, Mar 25, 2024 at 6:35 PM Maximilian Michels  wrote:

> The Apache Flink community is very happy to announce the release of
> the Apache Flink Kubernetes Operator version 1.8.0.
>
> The Flink Kubernetes Operator allows users to manage their Apache
> Flink applications on Kubernetes through all aspects of their
> lifecycle.
>
> Release highlights:
> - Flink Autotuning automatically adjusts TaskManager memory
> - Flink Autoscaling metrics and decision accuracy improved
> - Improve standalone Flink Autoscaling
> - Savepoint trigger nonce for savepoint-based restarts
> - Operator stability improvements for cluster shutdown
>
> Blog post:
> https://flink.apache.org/2024/03/21/apache-flink-kubernetes-operator-1.8.0-release-announcement/
>
> The release is available for download at:
> https://flink.apache.org/downloads.html
>
> Maven artifacts for Flink Kubernetes Operator can be found at:
>
> https://search.maven.org/artifact/org.apache.flink/flink-kubernetes-operator
>
> Official Docker image for Flink Kubernetes Operator can be found at:
> https://hub.docker.com/r/apache/flink-kubernetes-operator
>
> The full release notes are available in Jira:
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12353866&projectId=12315522
>
> We would like to thank the Apache Flink community and its contributors
> who made this release possible!
>
> Cheers,
> Max
>

Re: [ANNOUNCE] Apache Flink Kubernetes Operator 1.8.0 released

[Muhammet Orazov]

Great! Thanks Maximilian and everyone involved for the effort and 
release!


Best,
Muhammet

On 2024-03-25 10:35, Maximilian Michels wrote:

The Apache Flink community is very happy to announce the release of
the Apache Flink Kubernetes Operator version 1.8.0.

The Flink Kubernetes Operator allows users to manage their Apache
Flink applications on Kubernetes through all aspects of their
lifecycle.

Release highlights:
- Flink Autotuning automatically adjusts TaskManager memory
- Flink Autoscaling metrics and decision accuracy improved
- Improve standalone Flink Autoscaling
- Savepoint trigger nonce for savepoint-based restarts
- Operator stability improvements for cluster shutdown

Blog post: 
https://flink.apache.org/2024/03/21/apache-flink-kubernetes-operator-1.8.0-release-announcement/


The release is available for download at:
https://flink.apache.org/downloads.html

Maven artifacts for Flink Kubernetes Operator can be found at:
https://search.maven.org/artifact/org.apache.flink/flink-kubernetes-operator

Official Docker image for Flink Kubernetes Operator can be found at:
https://hub.docker.com/r/apache/flink-kubernetes-operator

The full release notes are available in Jira:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12353866&projectId=12315522

We would like to thank the Apache Flink community and its contributors
who made this release possible!

Cheers,
Max

Re: [DISCUSS] Flink Website Menu Adjustment

[Leonard Xu]

Thanks Zhongqiang for starting this discussion, updating documentation menus 
according to sub-projects' activities makes sense to me.  

+1 for the proposed menus:

> After:
> 
> With Flink
> With Flink Kubernetes Operator
> With Flink CDC
> With Flink ML
> With Flink Stateful Functions
> Training Course



Best,
Leonard

> 2024年3月25日 下午3:48，gongzhongqiang  写道：
> 
> Hi everyone,
> 
> I'd like to start a discussion on adjusting the Flink website [1] menu to
> improve accuracy and usability.While migrating Flink CDC documentation
> to the website, I found outdated links, need to review and update menus
> for the most relevant information for our users.
> 
> 
> Proposal:
> 
> - Remove Paimon [2] from the "Getting Started" and "Documentation" menus:
> Paimon [2] is now an independent top project of ASF. CC： jingsong lees
> 
> - Sort the projects in the subdirectory by the activity of the projects.
> Here I list the number of releases for each project in the past year.
> 
> Flink Kubernetes Operator : 7
> Flink CDC : 5
> Flink ML  : 2
> Flink Stateful Functions : 1
> 
> 
> Expected Outcome :
> 
> - Menu "Getting Started"
> 
> Before:
> 
> With Flink
> 
> With Flink Stateful Functions
> 
> With Flink ML
> 
> With Flink Kubernetes Operator
> 
> With Paimon(incubating) (formerly Flink Table Store)
> 
> With Flink CDC
> 
> Training Course
> 
> 
> After:
> 
> With Flink
> With Flink Kubernetes Operator
> 
> With Flink CDC
> 
> With Flink ML
> 
> With Flink Stateful Functions
> 
> Training Course
> 
> 
> - Menu "Documentation" will same with "Getting Started"
> 
> 
> I look forward to hearing your thoughts and suggestions on this proposal.
> 
> [1] https://flink.apache.org/
> [2] https://github.com/apache/incubator-paimon
> [3] https://github.com/apache/flink-statefun
> 
> 
> 
> Best regards,
> 
> Zhongqiang Gong

Re: [ANNOUNCE] Apache Flink Kubernetes Operator 1.8.0 released

[Leonard Xu]

Congratulations!  Thanks Maximilian for the release work and all involved.

Best,
Leonard



> 2024年3月25日 下午7:04，Muhammet Orazov  写道：
> 
> Great! Thanks Maximilian and everyone involved for the effort and release!
> 
> Best,
> Muhammet
> 
> On 2024-03-25 10:35, Maximilian Michels wrote:
>> The Apache Flink community is very happy to announce the release of
>> the Apache Flink Kubernetes Operator version 1.8.0.
>> The Flink Kubernetes Operator allows users to manage their Apache
>> Flink applications on Kubernetes through all aspects of their
>> lifecycle.
>> Release highlights:
>> - Flink Autotuning automatically adjusts TaskManager memory
>> - Flink Autoscaling metrics and decision accuracy improved
>> - Improve standalone Flink Autoscaling
>> - Savepoint trigger nonce for savepoint-based restarts
>> - Operator stability improvements for cluster shutdown
>> Blog post: 
>> https://flink.apache.org/2024/03/21/apache-flink-kubernetes-operator-1.8.0-release-announcement/
>> The release is available for download at:
>> https://flink.apache.org/downloads.html
>> Maven artifacts for Flink Kubernetes Operator can be found at:
>> https://search.maven.org/artifact/org.apache.flink/flink-kubernetes-operator
>> Official Docker image for Flink Kubernetes Operator can be found at:
>> https://hub.docker.com/r/apache/flink-kubernetes-operator
>> The full release notes are available in Jira:
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12353866&projectId=12315522
>> We would like to thank the Apache Flink community and its contributors
>> who made this release possible!
>> Cheers,
>> Max

Re: [DISCUSS] Planning Flink 1.20

[Leonard Xu]

Wow, happy to see Ufuk and Robert join the release managers group.

+1 for the release manager candidates(Weijie, Rui Fan,Ufuk and Robert) from my 
side.


Best,
Leonard



> 2024年3月25日 下午6:09，Robert Metzger  写道：
> 
> Hi, thanks for starting the discussion.
> 
> +1 for the proposed timeline and the three proposed release managers.
> 
> I'm happy to join the release managers group as well, as a backup for Ufuk
> (unless there are objections about the number of release managers)
> 
> On Mon, Mar 25, 2024 at 11:04 AM Ufuk Celebi  wrote:
> 
>> Hey all,
>> 
>> I'd like to join the release managers for 1.20 as well. I'm looking
>> forward to getting more actively involved again.
>> 
>> Cheers,
>> 
>> Ufuk
>> 
>> On Sun, Mar 24, 2024, at 11:27 AM, Ahmed Hamdy wrote:
>>> +1 for the proposed timeline and release managers.
>>> Best Regards
>>> Ahmed Hamdy
>>> 
>>> 
>>> On Fri, 22 Mar 2024 at 07:41, Xintong Song 
>> wrote:
>>> 
 +1 for the proposed timeline and Weijie & Rui as the release managers.
 
 I think it would be welcomed if another 1-2 volunteers join as the
>> release
 managers, but that's not a must. We used to have only 1-2 release
>> managers
 for each release,
 
 Best,
 
 Xintong
 
 
 
 On Fri, Mar 22, 2024 at 2:55 PM Jark Wu  wrote:
 
> Thanks for kicking this off.
> 
> +1 for the volunteered release managers (Weijie Guo, Rui Fan) and the
> targeting date (feature freeze: June 15).
> 
> Best,
> Jark
> 
> 
> 
> 
> 
> On Fri, 22 Mar 2024 at 14:00, Rui Fan <1996fan...@gmail.com> wrote:
> 
>> Thanks Leonard for this feedback and help!
>> 
>> Best,
>> Rui
>> 
>> On Fri, Mar 22, 2024 at 12:36 PM weijie guo <
>> guoweijieres...@gmail.com
> 
>> wrote:
>> 
>>> Thanks Leonard!
>>> 
 I'd like to help you if you need some help like permissions
>> from
 PMC
>>> side, please feel free to ping me.
>>> 
>>> Nice to know. It'll help a lot!
>>> 
>>> Best regards,
>>> 
>>> Weijie
>>> 
>>> 
>>> Leonard Xu  于2024年3月22日周五 12:09写道：
>>> 
 +1 for the proposed release managers (Weijie Guo, Rui Fan),
>> both the
> two
 candidates are pretty active committers thus I believe they
>> know the
 community development process well. The recent releases have
>> four
>> release
 managers, and I am also looking forward to having other
>> volunteers
 join the management of Flink 1.20.
 
 +1 for targeting date (feature freeze: June 15, 2024),
>> referring to
> the
 release cycle of recent versions, release cycle of 4 months
>> makes
> sense
>> to
 me.
 
 
 I'd like to help you if you need some help like permissions
>> from PMC
 side, please feel free to ping me.
 
 Best,
 Leonard
 
 
> 2024年3月19日 下午5:35，Rui Fan <1996fan...@gmail.com> 写道：
> 
> Hi Weijie,
> 
> Thanks for kicking off 1.20! I'd like to join you and
>> participate
 in
>> the
> 1.20 release.
> 
> Best,
> Rui
> 
> On Tue, Mar 19, 2024 at 5:30 PM weijie guo <
> guoweijieres...@gmail.com
>>> 
> wrote:
> 
>> Hi everyone,
>> 
>> With the release announcement of Flink 1.19, it's a good
>> time to
> kick
 off
>> discussion of the next release 1.20.
>> 
>> 
>> - Release managers
>> 
>> 
>> I'd like to volunteer as one of the release managers this
>> time.
 It
>> has
 been
>> good practice to have a team of release managers from
>> different
>> backgrounds, so please raise you hand if you'd like to
>> volunteer
> and
 get
>> involved.
>> 
>> 
>> 
>> - Timeline
>> 
>> 
>> Flink 1.19 has been released. With a target release cycle of
>> 4
>> months,
>> we propose a feature freeze date of *June 15, 2024*.
>> 
>> 
>> 
>> - Collecting features
>> 
>> 
>> As usual, we've created a wiki page[1] for collecting new
 features
> in
 1.20.
>> 
>> 
>> In addition, we already have a number of FLIPs that have been
 voted
>> or
 are
>> in the process, including pre-works for version 2.0.
>> 
>> 
>> In the meantime, the release management team will be
>> finalized in
> the
 next
>> few days, and we'll continue to create Jira Boards and Sync
> meetings
>> to make it easy
>> for everyone to get an overview and track progress.
>> 
>> 
>> 
>> Best regards,
>> 
>> Weijie
>> 
>>

Re: [DISCUSS] Planning Flink 1.20

[Xintong Song]

Thanks for joining, Ufuk & Robert. +1

Best,

Xintong



On Mon, Mar 25, 2024 at 7:54 PM Leonard Xu  wrote:

> Wow, happy to see Ufuk and Robert join the release managers group.
>
> +1 for the release manager candidates(Weijie, Rui Fan,Ufuk and Robert)
> from my side.
>
>
> Best,
> Leonard
>
>
>
> > 2024年3月25日 下午6:09，Robert Metzger  写道：
> >
> > Hi, thanks for starting the discussion.
> >
> > +1 for the proposed timeline and the three proposed release managers.
> >
> > I'm happy to join the release managers group as well, as a backup for
> Ufuk
> > (unless there are objections about the number of release managers)
> >
> > On Mon, Mar 25, 2024 at 11:04 AM Ufuk Celebi  wrote:
> >
> >> Hey all,
> >>
> >> I'd like to join the release managers for 1.20 as well. I'm looking
> >> forward to getting more actively involved again.
> >>
> >> Cheers,
> >>
> >> Ufuk
> >>
> >> On Sun, Mar 24, 2024, at 11:27 AM, Ahmed Hamdy wrote:
> >>> +1 for the proposed timeline and release managers.
> >>> Best Regards
> >>> Ahmed Hamdy
> >>>
> >>>
> >>> On Fri, 22 Mar 2024 at 07:41, Xintong Song 
> >> wrote:
> >>>
>  +1 for the proposed timeline and Weijie & Rui as the release managers.
> 
>  I think it would be welcomed if another 1-2 volunteers join as the
> >> release
>  managers, but that's not a must. We used to have only 1-2 release
> >> managers
>  for each release,
> 
>  Best,
> 
>  Xintong
> 
> 
> 
>  On Fri, Mar 22, 2024 at 2:55 PM Jark Wu  wrote:
> 
> > Thanks for kicking this off.
> >
> > +1 for the volunteered release managers (Weijie Guo, Rui Fan) and the
> > targeting date (feature freeze: June 15).
> >
> > Best,
> > Jark
> >
> >
> >
> >
> >
> > On Fri, 22 Mar 2024 at 14:00, Rui Fan <1996fan...@gmail.com> wrote:
> >
> >> Thanks Leonard for this feedback and help!
> >>
> >> Best,
> >> Rui
> >>
> >> On Fri, Mar 22, 2024 at 12:36 PM weijie guo <
> >> guoweijieres...@gmail.com
> >
> >> wrote:
> >>
> >>> Thanks Leonard!
> >>>
>  I'd like to help you if you need some help like permissions
> >> from
>  PMC
> >>> side, please feel free to ping me.
> >>>
> >>> Nice to know. It'll help a lot!
> >>>
> >>> Best regards,
> >>>
> >>> Weijie
> >>>
> >>>
> >>> Leonard Xu  于2024年3月22日周五 12:09写道：
> >>>
>  +1 for the proposed release managers (Weijie Guo, Rui Fan),
> >> both the
> > two
>  candidates are pretty active committers thus I believe they
> >> know the
>  community development process well. The recent releases have
> >> four
> >> release
>  managers, and I am also looking forward to having other
> >> volunteers
>  join the management of Flink 1.20.
> 
>  +1 for targeting date (feature freeze: June 15, 2024),
> >> referring to
> > the
>  release cycle of recent versions, release cycle of 4 months
> >> makes
> > sense
> >> to
>  me.
> 
> 
>  I'd like to help you if you need some help like permissions
> >> from PMC
>  side, please feel free to ping me.
> 
>  Best,
>  Leonard
> 
> 
> > 2024年3月19日 下午5:35，Rui Fan <1996fan...@gmail.com> 写道：
> >
> > Hi Weijie,
> >
> > Thanks for kicking off 1.20! I'd like to join you and
> >> participate
>  in
> >> the
> > 1.20 release.
> >
> > Best,
> > Rui
> >
> > On Tue, Mar 19, 2024 at 5:30 PM weijie guo <
> > guoweijieres...@gmail.com
> >>>
> > wrote:
> >
> >> Hi everyone,
> >>
> >> With the release announcement of Flink 1.19, it's a good
> >> time to
> > kick
>  off
> >> discussion of the next release 1.20.
> >>
> >>
> >> - Release managers
> >>
> >>
> >> I'd like to volunteer as one of the release managers this
> >> time.
>  It
> >> has
>  been
> >> good practice to have a team of release managers from
> >> different
> >> backgrounds, so please raise you hand if you'd like to
> >> volunteer
> > and
>  get
> >> involved.
> >>
> >>
> >>
> >> - Timeline
> >>
> >>
> >> Flink 1.19 has been released. With a target release cycle of
> >> 4
> >> months,
> >> we propose a feature freeze date of *June 15, 2024*.
> >>
> >>
> >>
> >> - Collecting features
> >>
> >>
> >> As usual, we've created a wiki page[1] for collecting new
>  features
> > in
>  1.20.
> >>
> >>
> >> In addition, we already have a number of FLIPs that have been
>  voted
> >> or
>  are
> >> in the process, including pre-works for

Re: [DISCUSS] Planning Flink 1.20

[Márton Balassi]

Thanks for kicking this off, folks.

+1 for the timeline and the release manager candidates (Weijie, Rui
Fan,Ufuk/Robert).

On Mon, Mar 25, 2024 at 12:54 PM Leonard Xu  wrote:

> Wow, happy to see Ufuk and Robert join the release managers group.
>
> +1 for the release manager candidates(Weijie, Rui Fan,Ufuk and Robert)
> from my side.
>
>
> Best,
> Leonard
>
>
>
> > 2024年3月25日 下午6:09，Robert Metzger  写道：
> >
> > Hi, thanks for starting the discussion.
> >
> > +1 for the proposed timeline and the three proposed release managers.
> >
> > I'm happy to join the release managers group as well, as a backup for
> Ufuk
> > (unless there are objections about the number of release managers)
> >
> > On Mon, Mar 25, 2024 at 11:04 AM Ufuk Celebi  wrote:
> >
> >> Hey all,
> >>
> >> I'd like to join the release managers for 1.20 as well. I'm looking
> >> forward to getting more actively involved again.
> >>
> >> Cheers,
> >>
> >> Ufuk
> >>
> >> On Sun, Mar 24, 2024, at 11:27 AM, Ahmed Hamdy wrote:
> >>> +1 for the proposed timeline and release managers.
> >>> Best Regards
> >>> Ahmed Hamdy
> >>>
> >>>
> >>> On Fri, 22 Mar 2024 at 07:41, Xintong Song 
> >> wrote:
> >>>
>  +1 for the proposed timeline and Weijie & Rui as the release managers.
> 
>  I think it would be welcomed if another 1-2 volunteers join as the
> >> release
>  managers, but that's not a must. We used to have only 1-2 release
> >> managers
>  for each release,
> 
>  Best,
> 
>  Xintong
> 
> 
> 
>  On Fri, Mar 22, 2024 at 2:55 PM Jark Wu  wrote:
> 
> > Thanks for kicking this off.
> >
> > +1 for the volunteered release managers (Weijie Guo, Rui Fan) and the
> > targeting date (feature freeze: June 15).
> >
> > Best,
> > Jark
> >
> >
> >
> >
> >
> > On Fri, 22 Mar 2024 at 14:00, Rui Fan <1996fan...@gmail.com> wrote:
> >
> >> Thanks Leonard for this feedback and help!
> >>
> >> Best,
> >> Rui
> >>
> >> On Fri, Mar 22, 2024 at 12:36 PM weijie guo <
> >> guoweijieres...@gmail.com
> >
> >> wrote:
> >>
> >>> Thanks Leonard!
> >>>
>  I'd like to help you if you need some help like permissions
> >> from
>  PMC
> >>> side, please feel free to ping me.
> >>>
> >>> Nice to know. It'll help a lot!
> >>>
> >>> Best regards,
> >>>
> >>> Weijie
> >>>
> >>>
> >>> Leonard Xu  于2024年3月22日周五 12:09写道：
> >>>
>  +1 for the proposed release managers (Weijie Guo, Rui Fan),
> >> both the
> > two
>  candidates are pretty active committers thus I believe they
> >> know the
>  community development process well. The recent releases have
> >> four
> >> release
>  managers, and I am also looking forward to having other
> >> volunteers
>  join the management of Flink 1.20.
> 
>  +1 for targeting date (feature freeze: June 15, 2024),
> >> referring to
> > the
>  release cycle of recent versions, release cycle of 4 months
> >> makes
> > sense
> >> to
>  me.
> 
> 
>  I'd like to help you if you need some help like permissions
> >> from PMC
>  side, please feel free to ping me.
> 
>  Best,
>  Leonard
> 
> 
> > 2024年3月19日 下午5:35，Rui Fan <1996fan...@gmail.com> 写道：
> >
> > Hi Weijie,
> >
> > Thanks for kicking off 1.20! I'd like to join you and
> >> participate
>  in
> >> the
> > 1.20 release.
> >
> > Best,
> > Rui
> >
> > On Tue, Mar 19, 2024 at 5:30 PM weijie guo <
> > guoweijieres...@gmail.com
> >>>
> > wrote:
> >
> >> Hi everyone,
> >>
> >> With the release announcement of Flink 1.19, it's a good
> >> time to
> > kick
>  off
> >> discussion of the next release 1.20.
> >>
> >>
> >> - Release managers
> >>
> >>
> >> I'd like to volunteer as one of the release managers this
> >> time.
>  It
> >> has
>  been
> >> good practice to have a team of release managers from
> >> different
> >> backgrounds, so please raise you hand if you'd like to
> >> volunteer
> > and
>  get
> >> involved.
> >>
> >>
> >>
> >> - Timeline
> >>
> >>
> >> Flink 1.19 has been released. With a target release cycle of
> >> 4
> >> months,
> >> we propose a feature freeze date of *June 15, 2024*.
> >>
> >>
> >>
> >> - Collecting features
> >>
> >>
> >> As usual, we've created a wiki page[1] for collecting new
>  features
> > in
>  1.20.
> >>
> >>
> >> In addition, we already have a number of FLIPs that have been
>  voted
> >> or

[RESULT][VOTE] FLIP-439: Externalize Kudu Connector from Bahir

[Ferenc Csaky]

Hi everyone,

I'm happy to announce that FLIP-439: Externalize Kudu Connector from Bahir [1]
has been accepted with 12 approving votes, 6 of which are binding: [2]:

- Mate Czagany (non-binding)
- Gyula Fóra (binding)
- Gabor Somogyi (binding)
- Mátyás Őrhidi (binding)
- Hang Ruan (non-binding)
- Samrat Deb (non-binding)
- Gong Zhongqiang(non-binding)
- Martijn Visser (binding)
- Leonard Xu Jin (binding)
- Marton Balassi (binding)
- Jeyhun Karimov (non-binding)
- Yuepeng Pan (non-binding)

There are no disapproving votes. Thanks to everyone who participated in the
discussion and voting.

Best,
Ferenc

[1] 
https://cwiki.apache.org/confluence/display/FLINK/FLIP-439%3A+Externalize+Kudu+Connector+from+Bahir
[2] https://lists.apache.org/thread/cjv0lq7x3m98dbhjk4p2wjps5rk2l9kj

[jira] [Created] (FLINK-34928) FLIP-439: Externalize Kudu Connector from Bahir

[Ferenc Csaky (Jira)]

Ferenc Csaky created FLINK-34928:


 Summary: FLIP-439: Externalize Kudu Connector from Bahir
 Key: FLINK-34928
 URL: https://issues.apache.org/jira/browse/FLINK-34928
 Project: Flink
  Issue Type: Improvement
Reporter: Ferenc Csaky


Umbrella issue for: 
https://cwiki.apache.org/confluence/display/FLINK/FLIP-439%3A+Externalize+Kudu+Connector+from+Bahir



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-34929) Create "flink-connector-kudu" repository

[Ferenc Csaky (Jira)]

Ferenc Csaky created FLINK-34929:


 Summary: Create "flink-connector-kudu" repository
 Key: FLINK-34929
 URL: https://issues.apache.org/jira/browse/FLINK-34929
 Project: Flink
  Issue Type: Sub-task
Reporter: Ferenc Csaky


We should create a "flink-connector-kudu" repositry under the "apache" GitHub 
organization.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-34930) Move existing Kudu connector code from Bahir repo to dedicated repo

[Ferenc Csaky (Jira)]

Ferenc Csaky created FLINK-34930:


 Summary: Move existing Kudu connector code from Bahir repo to 
dedicated repo
 Key: FLINK-34930
 URL: https://issues.apache.org/jira/browse/FLINK-34930
 Project: Flink
  Issue Type: Sub-task
Reporter: Ferenc Csaky






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-34931) Update Kudu connector DataStream Source/Sink implementation

[Ferenc Csaky (Jira)]

Ferenc Csaky created FLINK-34931:


 Summary: Update Kudu connector DataStream Source/Sink 
implementation
 Key: FLINK-34931
 URL: https://issues.apache.org/jira/browse/FLINK-34931
 Project: Flink
  Issue Type: Sub-task
Reporter: Ferenc Csaky


Update the DataSource API classes to use the current interfaces.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Flink Website Menu Adjustment

[Gyula Fóra]

+1 for the proposal

Gyula

On Mon, Mar 25, 2024 at 12:49 PM Leonard Xu  wrote:

> Thanks Zhongqiang for starting this discussion, updating documentation
> menus according to sub-projects' activities makes sense to me.
>
> +1 for the proposed menus:
>
> > After:
> >
> > With Flink
> > With Flink Kubernetes Operator
> > With Flink CDC
> > With Flink ML
> > With Flink Stateful Functions
> > Training Course
>
>
>
> Best,
> Leonard
>
> > 2024年3月25日 下午3:48，gongzhongqiang  写道：
> >
> > Hi everyone,
> >
> > I'd like to start a discussion on adjusting the Flink website [1] menu to
> > improve accuracy and usability.While migrating Flink CDC documentation
> > to the website, I found outdated links, need to review and update menus
> > for the most relevant information for our users.
> >
> >
> > Proposal:
> >
> > - Remove Paimon [2] from the "Getting Started" and "Documentation" menus:
> > Paimon [2] is now an independent top project of ASF. CC： jingsong lees
> >
> > - Sort the projects in the subdirectory by the activity of the projects.
> > Here I list the number of releases for each project in the past year.
> >
> > Flink Kubernetes Operator : 7
> > Flink CDC : 5
> > Flink ML  : 2
> > Flink Stateful Functions : 1
> >
> >
> > Expected Outcome :
> >
> > - Menu "Getting Started"
> >
> > Before:
> >
> > With Flink
> >
> > With Flink Stateful Functions
> >
> > With Flink ML
> >
> > With Flink Kubernetes Operator
> >
> > With Paimon(incubating) (formerly Flink Table Store)
> >
> > With Flink CDC
> >
> > Training Course
> >
> >
> > After:
> >
> > With Flink
> > With Flink Kubernetes Operator
> >
> > With Flink CDC
> >
> > With Flink ML
> >
> > With Flink Stateful Functions
> >
> > Training Course
> >
> >
> > - Menu "Documentation" will same with "Getting Started"
> >
> >
> > I look forward to hearing your thoughts and suggestions on this proposal.
> >
> > [1] https://flink.apache.org/
> > [2] https://github.com/apache/incubator-paimon
> > [3] https://github.com/apache/flink-statefun
> >
> >
> >
> > Best regards,
> >
> > Zhongqiang Gong
>
>

Re: [DISCUSS] Flink Website Menu Adjustment

[Ferenc Csaky]

Suggested changes makes sense, +1 for the proposed menus and order.

Best,
Ferenc




On Monday, March 25th, 2024 at 14:50, Gyula Fóra  wrote:

> 
> 
> +1 for the proposal
> 
> Gyula
> 
> On Mon, Mar 25, 2024 at 12:49 PM Leonard Xu xbjt...@gmail.com wrote:
> 
> > Thanks Zhongqiang for starting this discussion, updating documentation
> > menus according to sub-projects' activities makes sense to me.
> > 
> > +1 for the proposed menus:
> > 
> > > After:
> > > 
> > > With Flink
> > > With Flink Kubernetes Operator
> > > With Flink CDC
> > > With Flink ML
> > > With Flink Stateful Functions
> > > Training Course
> > 
> > Best,
> > Leonard
> > 
> > > 2024年3月25日 下午3:48，gongzhongqiang gongzhongqi...@apache.org 写道：
> > > 
> > > Hi everyone,
> > > 
> > > I'd like to start a discussion on adjusting the Flink website [1] menu to
> > > improve accuracy and usability.While migrating Flink CDC documentation
> > > to the website, I found outdated links, need to review and update menus
> > > for the most relevant information for our users.
> > > 
> > > Proposal:
> > > 
> > > - Remove Paimon [2] from the "Getting Started" and "Documentation" menus:
> > > Paimon [2] is now an independent top project of ASF. CC： jingsong lees
> > > 
> > > - Sort the projects in the subdirectory by the activity of the projects.
> > > Here I list the number of releases for each project in the past year.
> > > 
> > > Flink Kubernetes Operator : 7
> > > Flink CDC : 5
> > > Flink ML : 2
> > > Flink Stateful Functions : 1
> > > 
> > > Expected Outcome :
> > > 
> > > - Menu "Getting Started"
> > > 
> > > Before:
> > > 
> > > With Flink
> > > 
> > > With Flink Stateful Functions
> > > 
> > > With Flink ML
> > > 
> > > With Flink Kubernetes Operator
> > > 
> > > With Paimon(incubating) (formerly Flink Table Store)
> > > 
> > > With Flink CDC
> > > 
> > > Training Course
> > > 
> > > After:
> > > 
> > > With Flink
> > > With Flink Kubernetes Operator
> > > 
> > > With Flink CDC
> > > 
> > > With Flink ML
> > > 
> > > With Flink Stateful Functions
> > > 
> > > Training Course
> > > 
> > > - Menu "Documentation" will same with "Getting Started"
> > > 
> > > I look forward to hearing your thoughts and suggestions on this proposal.
> > > 
> > > [1] https://flink.apache.org/
> > > [2] https://github.com/apache/incubator-paimon
> > > [3] https://github.com/apache/flink-statefun
> > > 
> > > Best regards,
> > > 
> > > Zhongqiang Gong

[jira] [Created] (FLINK-34932) Translate concepts of Flink-Kubernetes-Operator documentation

[Caican Cai (Jira)]

Caican Cai created FLINK-34932:
--

 Summary: Translate concepts of Flink-Kubernetes-Operator 
documentation
 Key: FLINK-34932
 URL: https://issues.apache.org/jira/browse/FLINK-34932
 Project: Flink
  Issue Type: Sub-task
  Components: chinese-translation
Affects Versions: 1.19.0
Reporter: Caican Cai
 Fix For: 2.0.0






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Flink Website Menu Adjustment

[Jingsong Li]

+1 for the proposal

On Mon, Mar 25, 2024 at 10:01 PM Ferenc Csaky
 wrote:
>
> Suggested changes makes sense, +1 for the proposed menus and order.
>
> Best,
> Ferenc
>
>
>
>
> On Monday, March 25th, 2024 at 14:50, Gyula Fóra  wrote:
>
> >
> >
> > +1 for the proposal
> >
> > Gyula
> >
> > On Mon, Mar 25, 2024 at 12:49 PM Leonard Xu xbjt...@gmail.com wrote:
> >
> > > Thanks Zhongqiang for starting this discussion, updating documentation
> > > menus according to sub-projects' activities makes sense to me.
> > >
> > > +1 for the proposed menus:
> > >
> > > > After:
> > > >
> > > > With Flink
> > > > With Flink Kubernetes Operator
> > > > With Flink CDC
> > > > With Flink ML
> > > > With Flink Stateful Functions
> > > > Training Course
> > >
> > > Best,
> > > Leonard
> > >
> > > > 2024年3月25日 下午3:48，gongzhongqiang gongzhongqi...@apache.org 写道：
> > > >
> > > > Hi everyone,
> > > >
> > > > I'd like to start a discussion on adjusting the Flink website [1] menu 
> > > > to
> > > > improve accuracy and usability.While migrating Flink CDC documentation
> > > > to the website, I found outdated links, need to review and update menus
> > > > for the most relevant information for our users.
> > > >
> > > > Proposal:
> > > >
> > > > - Remove Paimon [2] from the "Getting Started" and "Documentation" 
> > > > menus:
> > > > Paimon [2] is now an independent top project of ASF. CC： jingsong lees
> > > >
> > > > - Sort the projects in the subdirectory by the activity of the projects.
> > > > Here I list the number of releases for each project in the past year.
> > > >
> > > > Flink Kubernetes Operator : 7
> > > > Flink CDC : 5
> > > > Flink ML : 2
> > > > Flink Stateful Functions : 1
> > > >
> > > > Expected Outcome :
> > > >
> > > > - Menu "Getting Started"
> > > >
> > > > Before:
> > > >
> > > > With Flink
> > > >
> > > > With Flink Stateful Functions
> > > >
> > > > With Flink ML
> > > >
> > > > With Flink Kubernetes Operator
> > > >
> > > > With Paimon(incubating) (formerly Flink Table Store)
> > > >
> > > > With Flink CDC
> > > >
> > > > Training Course
> > > >
> > > > After:
> > > >
> > > > With Flink
> > > > With Flink Kubernetes Operator
> > > >
> > > > With Flink CDC
> > > >
> > > > With Flink ML
> > > >
> > > > With Flink Stateful Functions
> > > >
> > > > Training Course
> > > >
> > > > - Menu "Documentation" will same with "Getting Started"
> > > >
> > > > I look forward to hearing your thoughts and suggestions on this 
> > > > proposal.
> > > >
> > > > [1] https://flink.apache.org/
> > > > [2] https://github.com/apache/incubator-paimon
> > > > [3] https://github.com/apache/flink-statefun
> > > >
> > > > Best regards,
> > > >
> > > > Zhongqiang Gong

Re: [DISCUSS] Flink Website Menu Adjustment

[Rui Fan]

Thanks Zhongqiang for driving this discussion, +1 for the proposal

Best,
Rui

Jingsong Li 于2024年3月25日 周一22:47写道：

> +1 for the proposal
>
> On Mon, Mar 25, 2024 at 10:01 PM Ferenc Csaky
>  wrote:
> >
> > Suggested changes makes sense, +1 for the proposed menus and order.
> >
> > Best,
> > Ferenc
> >
> >
> >
> >
> > On Monday, March 25th, 2024 at 14:50, Gyula Fóra 
> wrote:
> >
> > >
> > >
> > > +1 for the proposal
> > >
> > > Gyula
> > >
> > > On Mon, Mar 25, 2024 at 12:49 PM Leonard Xu xbjt...@gmail.com wrote:
> > >
> > > > Thanks Zhongqiang for starting this discussion, updating
> documentation
> > > > menus according to sub-projects' activities makes sense to me.
> > > >
> > > > +1 for the proposed menus:
> > > >
> > > > > After:
> > > > >
> > > > > With Flink
> > > > > With Flink Kubernetes Operator
> > > > > With Flink CDC
> > > > > With Flink ML
> > > > > With Flink Stateful Functions
> > > > > Training Course
> > > >
> > > > Best,
> > > > Leonard
> > > >
> > > > > 2024年3月25日 下午3:48，gongzhongqiang gongzhongqi...@apache.org 写道：
> > > > >
> > > > > Hi everyone,
> > > > >
> > > > > I'd like to start a discussion on adjusting the Flink website [1]
> menu to
> > > > > improve accuracy and usability.While migrating Flink CDC
> documentation
> > > > > to the website, I found outdated links, need to review and update
> menus
> > > > > for the most relevant information for our users.
> > > > >
> > > > > Proposal:
> > > > >
> > > > > - Remove Paimon [2] from the "Getting Started" and "Documentation"
> menus:
> > > > > Paimon [2] is now an independent top project of ASF. CC： jingsong
> lees
> > > > >
> > > > > - Sort the projects in the subdirectory by the activity of the
> projects.
> > > > > Here I list the number of releases for each project in the past
> year.
> > > > >
> > > > > Flink Kubernetes Operator : 7
> > > > > Flink CDC : 5
> > > > > Flink ML : 2
> > > > > Flink Stateful Functions : 1
> > > > >
> > > > > Expected Outcome :
> > > > >
> > > > > - Menu "Getting Started"
> > > > >
> > > > > Before:
> > > > >
> > > > > With Flink
> > > > >
> > > > > With Flink Stateful Functions
> > > > >
> > > > > With Flink ML
> > > > >
> > > > > With Flink Kubernetes Operator
> > > > >
> > > > > With Paimon(incubating) (formerly Flink Table Store)
> > > > >
> > > > > With Flink CDC
> > > > >
> > > > > Training Course
> > > > >
> > > > > After:
> > > > >
> > > > > With Flink
> > > > > With Flink Kubernetes Operator
> > > > >
> > > > > With Flink CDC
> > > > >
> > > > > With Flink ML
> > > > >
> > > > > With Flink Stateful Functions
> > > > >
> > > > > Training Course
> > > > >
> > > > > - Menu "Documentation" will same with "Getting Started"
> > > > >
> > > > > I look forward to hearing your thoughts and suggestions on this
> proposal.
> > > > >
> > > > > [1] https://flink.apache.org/
> > > > > [2] https://github.com/apache/incubator-paimon
> > > > > [3] https://github.com/apache/flink-statefun
> > > > >
> > > > > Best regards,
> > > > >
> > > > > Zhongqiang Gong
>

Re: [DISCUSS] Planning Flink 1.20

[Rui Fan]

Welcome Ufuk and Robert join the release manager group!

Best,
Rui

Márton Balassi 于2024年3月25日 周一20:04写道：

> Thanks for kicking this off, folks.
>
> +1 for the timeline and the release manager candidates (Weijie, Rui
> Fan,Ufuk/Robert).
>
> On Mon, Mar 25, 2024 at 12:54 PM Leonard Xu  wrote:
>
> > Wow, happy to see Ufuk and Robert join the release managers group.
> >
> > +1 for the release manager candidates(Weijie, Rui Fan,Ufuk and Robert)
> > from my side.
> >
> >
> > Best,
> > Leonard
> >
> >
> >
> > > 2024年3月25日 下午6:09，Robert Metzger  写道：
> > >
> > > Hi, thanks for starting the discussion.
> > >
> > > +1 for the proposed timeline and the three proposed release managers.
> > >
> > > I'm happy to join the release managers group as well, as a backup for
> > Ufuk
> > > (unless there are objections about the number of release managers)
> > >
> > > On Mon, Mar 25, 2024 at 11:04 AM Ufuk Celebi  wrote:
> > >
> > >> Hey all,
> > >>
> > >> I'd like to join the release managers for 1.20 as well. I'm looking
> > >> forward to getting more actively involved again.
> > >>
> > >> Cheers,
> > >>
> > >> Ufuk
> > >>
> > >> On Sun, Mar 24, 2024, at 11:27 AM, Ahmed Hamdy wrote:
> > >>> +1 for the proposed timeline and release managers.
> > >>> Best Regards
> > >>> Ahmed Hamdy
> > >>>
> > >>>
> > >>> On Fri, 22 Mar 2024 at 07:41, Xintong Song 
> > >> wrote:
> > >>>
> >  +1 for the proposed timeline and Weijie & Rui as the release
> managers.
> > 
> >  I think it would be welcomed if another 1-2 volunteers join as the
> > >> release
> >  managers, but that's not a must. We used to have only 1-2 release
> > >> managers
> >  for each release,
> > 
> >  Best,
> > 
> >  Xintong
> > 
> > 
> > 
> >  On Fri, Mar 22, 2024 at 2:55 PM Jark Wu  wrote:
> > 
> > > Thanks for kicking this off.
> > >
> > > +1 for the volunteered release managers (Weijie Guo, Rui Fan) and
> the
> > > targeting date (feature freeze: June 15).
> > >
> > > Best,
> > > Jark
> > >
> > >
> > >
> > >
> > >
> > > On Fri, 22 Mar 2024 at 14:00, Rui Fan <1996fan...@gmail.com>
> wrote:
> > >
> > >> Thanks Leonard for this feedback and help!
> > >>
> > >> Best,
> > >> Rui
> > >>
> > >> On Fri, Mar 22, 2024 at 12:36 PM weijie guo <
> > >> guoweijieres...@gmail.com
> > >
> > >> wrote:
> > >>
> > >>> Thanks Leonard!
> > >>>
> >  I'd like to help you if you need some help like permissions
> > >> from
> >  PMC
> > >>> side, please feel free to ping me.
> > >>>
> > >>> Nice to know. It'll help a lot!
> > >>>
> > >>> Best regards,
> > >>>
> > >>> Weijie
> > >>>
> > >>>
> > >>> Leonard Xu  于2024年3月22日周五 12:09写道：
> > >>>
> >  +1 for the proposed release managers (Weijie Guo, Rui Fan),
> > >> both the
> > > two
> >  candidates are pretty active committers thus I believe they
> > >> know the
> >  community development process well. The recent releases have
> > >> four
> > >> release
> >  managers, and I am also looking forward to having other
> > >> volunteers
> >  join the management of Flink 1.20.
> > 
> >  +1 for targeting date (feature freeze: June 15, 2024),
> > >> referring to
> > > the
> >  release cycle of recent versions, release cycle of 4 months
> > >> makes
> > > sense
> > >> to
> >  me.
> > 
> > 
> >  I'd like to help you if you need some help like permissions
> > >> from PMC
> >  side, please feel free to ping me.
> > 
> >  Best,
> >  Leonard
> > 
> > 
> > > 2024年3月19日 下午5:35，Rui Fan <1996fan...@gmail.com> 写道：
> > >
> > > Hi Weijie,
> > >
> > > Thanks for kicking off 1.20! I'd like to join you and
> > >> participate
> >  in
> > >> the
> > > 1.20 release.
> > >
> > > Best,
> > > Rui
> > >
> > > On Tue, Mar 19, 2024 at 5:30 PM weijie guo <
> > > guoweijieres...@gmail.com
> > >>>
> > > wrote:
> > >
> > >> Hi everyone,
> > >>
> > >> With the release announcement of Flink 1.19, it's a good
> > >> time to
> > > kick
> >  off
> > >> discussion of the next release 1.20.
> > >>
> > >>
> > >> - Release managers
> > >>
> > >>
> > >> I'd like to volunteer as one of the release managers this
> > >> time.
> >  It
> > >> has
> >  been
> > >> good practice to have a team of release managers from
> > >> different
> > >> backgrounds, so please raise you hand if you'd like to
> > >> volunteer
> > > and
> >  get
> > >> involved.
> > >>
> > >>
> > >>
> > >> - Timeline
> > >>
> > >>
> > >> Flink 1.19 has 

Re: [DISCUSS] Flink Website Menu Adjustment

[Yanquan Lv]

+1 for this proposal.

gongzhongqiang  于2024年3月25日周一 15:49写道：

> Hi everyone,
>
> I'd like to start a discussion on adjusting the Flink website [1] menu to
> improve accuracy and usability.While migrating Flink CDC documentation
> to the website, I found outdated links, need to review and update menus
> for the most relevant information for our users.
>
>
> Proposal:
>
> - Remove Paimon [2] from the "Getting Started" and "Documentation" menus:
> Paimon [2] is now an independent top project of ASF. CC： jingsong lees
>
> - Sort the projects in the subdirectory by the activity of the projects.
> Here I list the number of releases for each project in the past year.
>
> Flink Kubernetes Operator : 7
> Flink CDC : 5
> Flink ML  : 2
> Flink Stateful Functions : 1
>
>
> Expected Outcome :
>
> - Menu "Getting Started"
>
> Before:
>
> With Flink
>
> With Flink Stateful Functions
>
> With Flink ML
>
> With Flink Kubernetes Operator
>
> With Paimon(incubating) (formerly Flink Table Store)
>
> With Flink CDC
>
> Training Course
>
>
> After:
>
> With Flink
> With Flink Kubernetes Operator
>
> With Flink CDC
>
> With Flink ML
>
> With Flink Stateful Functions
>
> Training Course
>
>
> - Menu "Documentation" will same with "Getting Started"
>
>
> I look forward to hearing your thoughts and suggestions on this proposal.
>
> [1] https://flink.apache.org/
> [2] https://github.com/apache/incubator-paimon
> [3] https://github.com/apache/flink-statefun
>
>
>
> Best regards,
>
> Zhongqiang Gong
>

Re: [DISCUSS] Flink Website Menu Adjustment

[Martijn Visser]

Hi Zhongqiang Gong,

Are you suggesting to continuously update the menu based on the number of
releases, or just this one time? I wouldn't be in favor of continuously
updating: returning customers expect a certain order in the menu, and I
don't see a lot of value in continuously changing that. I do think that the
order that you have currently proposed is better then the one we have right
now, so I would +1 a one-time update but not a continuously updating order.

Best regards,

Martijn

On Mon, Mar 25, 2024 at 4:15 PM Yanquan Lv  wrote:

> +1 for this proposal.
>
> gongzhongqiang  于2024年3月25日周一 15:49写道：
>
> > Hi everyone,
> >
> > I'd like to start a discussion on adjusting the Flink website [1] menu to
> > improve accuracy and usability.While migrating Flink CDC documentation
> > to the website, I found outdated links, need to review and update menus
> > for the most relevant information for our users.
> >
> >
> > Proposal:
> >
> > - Remove Paimon [2] from the "Getting Started" and "Documentation" menus:
> > Paimon [2] is now an independent top project of ASF. CC： jingsong lees
> >
> > - Sort the projects in the subdirectory by the activity of the projects.
> > Here I list the number of releases for each project in the past year.
> >
> > Flink Kubernetes Operator : 7
> > Flink CDC : 5
> > Flink ML  : 2
> > Flink Stateful Functions : 1
> >
> >
> > Expected Outcome :
> >
> > - Menu "Getting Started"
> >
> > Before:
> >
> > With Flink
> >
> > With Flink Stateful Functions
> >
> > With Flink ML
> >
> > With Flink Kubernetes Operator
> >
> > With Paimon(incubating) (formerly Flink Table Store)
> >
> > With Flink CDC
> >
> > Training Course
> >
> >
> > After:
> >
> > With Flink
> > With Flink Kubernetes Operator
> >
> > With Flink CDC
> >
> > With Flink ML
> >
> > With Flink Stateful Functions
> >
> > Training Course
> >
> >
> > - Menu "Documentation" will same with "Getting Started"
> >
> >
> > I look forward to hearing your thoughts and suggestions on this proposal.
> >
> > [1] https://flink.apache.org/
> > [2] https://github.com/apache/incubator-paimon
> > [3] https://github.com/apache/flink-statefun
> >
> >
> >
> > Best regards,
> >
> > Zhongqiang Gong
> >
>

[jira] [Created] (FLINK-34933) JobMasterServiceLeadershipRunnerTest#testResultFutureCompletionOfOutdatedLeaderIsIgnored isn't implemented properly

[Matthias Pohl (Jira)]

Matthias Pohl created FLINK-34933:
-

 Summary: 
JobMasterServiceLeadershipRunnerTest#testResultFutureCompletionOfOutdatedLeaderIsIgnored
 isn't implemented properly
 Key: FLINK-34933
 URL: https://issues.apache.org/jira/browse/FLINK-34933
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Coordination
Affects Versions: 1.18.1, 1.19.0, 1.17.2, 1.20.0
Reporter: Matthias Pohl


{{testResultFutureCompletionOfOutdatedLeaderIsIgnored}} doesn't test the 
desired behavior: The {{TestingJobMasterService#closeAsync()}} callback throws 
an {{UnsupportedOperationException}} by default which prevents the test from 
properly finalizing the leadership revocation.

The test is still passing because the test checks implicitly for this error. 
Instead, we should verify that the runner's resultFuture doesn't complete until 
the runner is closed.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Support minibatch for TopNFunction

[Roman Boyko]

Hi Ron,
Thank you so much for your reply!

1. I added the description to Motivation part of my document [1]
2. I suppose to inject this functionality to AbstractTopNFunction, thus it
will work for all its implementations. It doesn't depend of implementation
(either it would be AppendOnlyTopNFunction or RetractableTopNFunction,
except maybe FastTop1Function), the most effect it would have for functions
with:
 - topN functions without no-ranking optimization [2]
 - high value of N (top1 has less possibilities for optimization here)
 - frequent input records which are placed to the top 1 position
3. I will do it in a week - I need to fix and recheck some parts
4. Unfortunately I don't have permissions to Flink confluence and according
to contribution guide I first expressed my idea as google doc. I would be
happy to transform this idea to FLIP.

[1]
https://docs.google.com/document/d/1YPHwxKfiGSUOUOa6bc68fIJHO_UojTwZEC29VVEa-Uk

[2]
https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/queries/topn/#no-ranking-output-optimization

--
Best regards,
Roman Boyko
e.: ro.v.bo...@gmail.com
m.: +79059592443
telegram: @rboyko

On Mon, 25 Mar 2024 at 15:12, Ron liu  wrote:

> Hi, Roman
>
> Thanks for your proposal, I intuitively feel that this optimization would
> be very useful to reduce the amount of message amplification for TopN
> operators. After briefly looking at your google docs, I have the following
> questions:
>
> 1. Whether you can describe in detail the principle of solving the TopN
> operator record amplification, similar to Minibatch Join[1], through the
> figure of current Motivation part, I can not understand how you did it
> 2. TopN has currently multiple implementation functions, including
> AppendOnlyFirstNFunction, AppendOnlyTopNFunction, FastTop1Function,
> RetractableTopNFunction, UpdatableTopNFunction. Is it possible to elaborate
> on which patterns the Minibatch optimization applies to?
> 3. Is it possible to provide the PoC code?
> 4. finally, we need a formal FLIP document on the wiki[2].
>
> [1]
>
> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/tuning/#minibatch-regular-joins
> [2]
>
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
>
> Best,
> Ron
>
> Roman Boyko  于2024年3月24日周日 01:14写道：
>
> > Hi Flink Community,
> >
> > I tried to describe my idea about minibatch for TopNFunction in this doc
> -
> >
> >
> https://docs.google.com/document/d/1YPHwxKfiGSUOUOa6bc68fIJHO_UojTwZEC29VVEa-Uk/edit?usp=sharing
> >
> > Looking forward to your feedback, thank you
> >
> > On Tue, 19 Mar 2024 at 12:24, Roman Boyko  wrote:
> >
> > > Hello Flink Community,
> > >
> > > The same problem with record amplification as described in FLIP-415:
> > Introduce
> > > a new join operator to support minibatch[1] exists for most of
> > > implementations of AbstractTopNFunction. Especially when the rank is
> > > provided to output. For example, when calculating Top100 with rank
> > output,
> > > every input record might produce 100 -U records and 100 +U records.
> > >
> > > According to my POC (which is similar to FLIP-415) the record
> > > amplification could be significantly reduced by using input or output
> > > buffer.
> > >
> > > What do you think if we implement such optimization for TopNFunctions?
> > >
> > > [1]
> > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-415
> > > %3A+Introduce+a+new+join+operator+to+support+minibatch
> > >
> > > --
> > > Best regards,
> > > Roman Boyko
> > > e.: ro.v.bo...@gmail.com
> > > m.: +79059592443
> > > telegram: @rboyko
> > >
> >
> >
> > --
> > Best regards,
> > Roman Boyko
> > e.: ro.v.bo...@gmail.com
> > m.: +79059592443
> > telegram: @rboyko
> >
>

Re: [DISCUSS] FLIP-437: Support ML Models in Flink SQL

[Hao Li]

Hi Ahmed,

Looks like the feature freeze time for 1.20 release is June 15th. We can
definitely get the model DDL into 1.20. For predict and evaluate functions,
if we can't get into the 1.20 release, we can get them into the 1.21
release for sure.

Thanks,
Hao



On Mon, Mar 25, 2024 at 1:25 AM Timo Walther  wrote:

> Hi Jark and Hao,
>
> thanks for the information, Jark! Great that the Calcite community
> already fixed the problem for us. +1 to adopt the simplified syntax
> asap. Maybe even before we upgrade Calcite (i.e. copy over classes), if
> upgrading Calcite is too much work right now?
>
>  > Is `DESCRIPTOR` a must in the syntax?
>
> Yes, we should still stick to the standard as much as possible and all
> vendors use DESCRIPTOR/COLUMNS for distinuishing columns vs. literal
> arguments. So the final syntax of this discussion would be:
>
>
> SELECT f1, f2, label FROM
>ML_PREDICT(TABLE `my_data`, `classifier_model`, DESCRIPTOR(f1, f2))
>
> SELECT * FROM
>ML_EVALUATE(TABLE `eval_data`, `classifier_model`, DESCRIPTOR(f1, f2))
>
> Please double check if this is implementable with the current stack. I
> fear the parser or validator might not like the "identifier" argument?
>
> Make sure that also these variations are supported:
>
> SELECT f1, f2, label FROM
>ML_PREDICT(
>  TABLE `my_data`,
>  my_cat.my_db.classifier_model,
>  DESCRIPTOR(f1, f2))
>
> SELECT f1, f2, label FROM
>ML_PREDICT(
>  input => TABLE `my_data`,
>  model => my_cat.my_db.classifier_model,
>  args => DESCRIPTOR(f1, f2))
>
> It might be safer and more future proof to wrap a MODEL() function
> around it. This would be more in sync with the standard that actually
> still requires to put a TABLE() around the input argument:
>
> ML_PREDICT(TABLE(`my_data`) PARTITIONED BY c1 ORDERED BY c1, )
>
> So the safest option would be the long-term solution:
>
> SELECT f1, f2, label FROM
>ML_PREDICT(
>  input => TABLE(my_data),
>  model => MODEL(my_cat.my_db.classifier_model),
>  args => DESCRIPTOR(f1, f2))
>
> But I'm fine with this if others have a strong opinion:
>
> SELECT f1, f2, label FROM
>ML_PREDICT(
>  input => TABLE `my_data`,
>  model => my_cat.my_db.classifier_model,
>  args => DESCRIPTOR(f1, f2))
>
> Some feedback for the remainder of the FLIP:
>
> 1) Simplify catalog objects
>
> I would suggest to drop:
> CatalogModel.getModelKind()
> CatalogModel.getModelTask()
>
> A catalog object should fully resemble the DDL. And since the DDL puts
> those properties in the WITH clause, the catalog object should the same
> (i.e. put them into the `getModelOptions()`). Btw renaming this method
> to just `getOptions()` for consistency should be good as well.
> Internally, we can still provide enums for these frequently used
> classes. Similar to what we do in `FactoryUtil` for other frequently
> used options.
>
> Remove `getDescription()` and `getDetailedDescription()`. They were a
> mistake for CatalogTable and should actually be deprecated. They got
> replaced by `getComment()` which is sufficient.
>
> 2) CREATE TEMPORARY MODEL is not supported.
>
> This is an unnecessary restriction. We should support temporary versions
> of these catalog objects as well for consistency. Adding support for
> this should be straightforward.
>
> 3) DESCRIBE | DESC } MODEL [catalog_name.][database_name.]model_name
>
> I would suggest we support `SHOW CREATE MODEL` instead. Similar to `SHOW
> CREATE TABLE`, this should show all properties. If we support `DESCRIBE
> MODEL` it should only list the input parameters similar to `DESCRIBE
> TABLE` only shows the columns (not the WITH clause).
>
> Regards,
> Timo
>
>
> On 23.03.24 13:17, Ahmed Hamdy wrote:
> > Hi everyone,
> > +1 for this proposal, I believe it is very useful to the minimum, It
> would
> > be great even having  "ML_PREDICT" and "ML_EVALUATE" as built-in PTFs in
> > this FLIP as discussed.
> > IIUC this will be included in the 1.20 roadmap?
> > Best Regards
> > Ahmed Hamdy
> >
> >
> > On Fri, 22 Mar 2024 at 23:54, Hao Li  wrote:
> >
> >> Hi Timo and Jark,
> >>
> >> I agree Oracle's syntax seems concise and more descriptive. For the
> >> built-in `ML_PREDICT` and `ML_EVALUATE` functions I agree with Jark we
> can
> >> support them as built-in PTF using `SqlTableFunction` for this FLIP. We
> can
> >> have a different FLIP discussing user defined PTF and adopt that later
> for
> >> model functions later. To summarize, the current proposed syntax is
> >>
> >> SELECT f1, f2, label FROM TABLE(ML_PREDICT(TABLE `my_data`,
> >> `classifier_model`, f1, f2))
> >>
> >> SELECT * FROM TABLE(ML_EVALUATE(TABLE `eval_data`, `classifier_model`,
> f1,
> >> f2))
> >>
> >> Is `DESCRIPTOR` a must in the syntax? If so, it becomes
> >>
> >> SELECT f1, f2, label FROM TABLE(ML_PREDICT(TABLE `my_data`,
> >> `classifier_model`, DESCRIPTOR(f1), DESCRIPTOR(f2)))
> >>
> >> SELECT * FROM TABLE(ML_EVALUATE(TABLE `eval_data`, `classifier_model`,
> >> DESCRIPTOR(f1), DESCRI

Re: [DISCUSS] FLIP-437: Support ML Models in Flink SQL

[Hao Li]

Hi Timo,

> Please double check if this is implementable with the current stack. I
fear the parser or validator might not like the "identifier" argument?

I checked this, currently the validator throws an exception trying to get
the full qualifier name for `classifier_model`. But since
`SqlValidatorImpl` is implemented in Flink, we should be able to fix this.
The only caveator is if not full model path is provided,
the qualifier is interpreted as a column. We should be able to special
handle this by rewriting the `ml_predict` function to add the catalog and
database name in `FlinkCalciteSqlValidator` though.

> SELECT f1, f2, label FROM
   ML_PREDICT(
 TABLE `my_data`,
 my_cat.my_db.classifier_model,
 DESCRIPTOR(f1, f2))

SELECT f1, f2, label FROM
   ML_PREDICT(
 input => TABLE `my_data`,
 model => my_cat.my_db.classifier_model,
 args => DESCRIPTOR(f1, f2))

I verified these can be parsed. The problem is in validator for qualifier
as mentioned above.

> So the safest option would be the long-term solution:

SELECT f1, f2, label FROM
   ML_PREDICT(
 input => TABLE(my_data),
 model => MODEL(my_cat.my_db.classifier_model),
 args => DESCRIPTOR(f1, f2))

`TABLE(my_data)` and `MODEL(my_cat.my_db.classifier_model)` doesn't work
since `TABLE` and `MODEL` are already key words in calcite used by `CREATE
TABLE`, `CREATE MODEL`. Changing to `model_name(...)` works and will be
treated as a function.

So I think

SELECT f1, f2, label FROM
   ML_PREDICT(
 input => TABLE `my_data`,
 model => my_cat.my_db.classifier_model,
 args => DESCRIPTOR(f1, f2))
should be fine for now.

For the syntax part:
1). Sounds good. We can drop model task and model kind from the definition.
They can be deduced from the options.

2). Sure. We can add temporary model

3). Make sense. We can use `show create model ` to display all
information and `describe model ` to show input/output schema

Thanks,
Hao

On Mon, Mar 25, 2024 at 3:21 PM Hao Li  wrote:

> Hi Ahmed,
>
> Looks like the feature freeze time for 1.20 release is June 15th. We can
> definitely get the model DDL into 1.20. For predict and evaluate functions,
> if we can't get into the 1.20 release, we can get them into the 1.21
> release for sure.
>
> Thanks,
> Hao
>
>
>
> On Mon, Mar 25, 2024 at 1:25 AM Timo Walther  wrote:
>
>> Hi Jark and Hao,
>>
>> thanks for the information, Jark! Great that the Calcite community
>> already fixed the problem for us. +1 to adopt the simplified syntax
>> asap. Maybe even before we upgrade Calcite (i.e. copy over classes), if
>> upgrading Calcite is too much work right now?
>>
>>  > Is `DESCRIPTOR` a must in the syntax?
>>
>> Yes, we should still stick to the standard as much as possible and all
>> vendors use DESCRIPTOR/COLUMNS for distinuishing columns vs. literal
>> arguments. So the final syntax of this discussion would be:
>>
>>
>> SELECT f1, f2, label FROM
>>ML_PREDICT(TABLE `my_data`, `classifier_model`, DESCRIPTOR(f1, f2))
>>
>> SELECT * FROM
>>ML_EVALUATE(TABLE `eval_data`, `classifier_model`, DESCRIPTOR(f1, f2))
>>
>> Please double check if this is implementable with the current stack. I
>> fear the parser or validator might not like the "identifier" argument?
>>
>> Make sure that also these variations are supported:
>>
>> SELECT f1, f2, label FROM
>>ML_PREDICT(
>>  TABLE `my_data`,
>>  my_cat.my_db.classifier_model,
>>  DESCRIPTOR(f1, f2))
>>
>> SELECT f1, f2, label FROM
>>ML_PREDICT(
>>  input => TABLE `my_data`,
>>  model => my_cat.my_db.classifier_model,
>>  args => DESCRIPTOR(f1, f2))
>>
>> It might be safer and more future proof to wrap a MODEL() function
>> around it. This would be more in sync with the standard that actually
>> still requires to put a TABLE() around the input argument:
>>
>> ML_PREDICT(TABLE(`my_data`) PARTITIONED BY c1 ORDERED BY c1, )
>>
>> So the safest option would be the long-term solution:
>>
>> SELECT f1, f2, label FROM
>>ML_PREDICT(
>>  input => TABLE(my_data),
>>  model => MODEL(my_cat.my_db.classifier_model),
>>  args => DESCRIPTOR(f1, f2))
>>
>> But I'm fine with this if others have a strong opinion:
>>
>> SELECT f1, f2, label FROM
>>ML_PREDICT(
>>  input => TABLE `my_data`,
>>  model => my_cat.my_db.classifier_model,
>>  args => DESCRIPTOR(f1, f2))
>>
>> Some feedback for the remainder of the FLIP:
>>
>> 1) Simplify catalog objects
>>
>> I would suggest to drop:
>> CatalogModel.getModelKind()
>> CatalogModel.getModelTask()
>>
>> A catalog object should fully resemble the DDL. And since the DDL puts
>> those properties in the WITH clause, the catalog object should the same
>> (i.e. put them into the `getModelOptions()`). Btw renaming this method
>> to just `getOptions()` for consistency should be good as well.
>> Internally, we can still provide enums for these frequently used
>> classes. Similar to what we do in `FactoryUtil` for other frequently
>> used options.
>>
>> Remove `getDes

Re: [VOTE] FLIP-433: State Access on DataStream API V2

[Yunfeng Zhou]

+1 (non-binding)

Best,
Yunfeng

On Wed, Mar 20, 2024 at 8:29 PM weijie guo  wrote:
>
> Hi everyone,
>
>
> Thanks for all the feedback about the FLIP-433: State Access on
> DataStream API V2 [1]. The discussion thread is here [2].
>
>
> The vote will be open for at least 72 hours unless there is an
> objection or insufficient votes.
>
>
> Best regards,
>
> Weijie
>
>
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-433%3A+State+Access+on+DataStream+API+V2
>
> [2] https://lists.apache.org/thread/8ghzqkvt99p4k7hjqxzwhqny7zb7xrwo

Re: [DISCUSS] Planning Flink 1.20

[weijie guo]

Thanks for joining, Ufuk & Robert :)

Best regards,

Weijie


Rui Fan <1996fan...@gmail.com> 于2024年3月25日周一 23:06写道：

> Welcome Ufuk and Robert join the release manager group!
>
> Best,
> Rui
>
> Márton Balassi 于2024年3月25日 周一20:04写道：
>
> > Thanks for kicking this off, folks.
> >
> > +1 for the timeline and the release manager candidates (Weijie, Rui
> > Fan,Ufuk/Robert).
> >
> > On Mon, Mar 25, 2024 at 12:54 PM Leonard Xu  wrote:
> >
> > > Wow, happy to see Ufuk and Robert join the release managers group.
> > >
> > > +1 for the release manager candidates(Weijie, Rui Fan,Ufuk and Robert)
> > > from my side.
> > >
> > >
> > > Best,
> > > Leonard
> > >
> > >
> > >
> > > > 2024年3月25日 下午6:09，Robert Metzger  写道：
> > > >
> > > > Hi, thanks for starting the discussion.
> > > >
> > > > +1 for the proposed timeline and the three proposed release managers.
> > > >
> > > > I'm happy to join the release managers group as well, as a backup for
> > > Ufuk
> > > > (unless there are objections about the number of release managers)
> > > >
> > > > On Mon, Mar 25, 2024 at 11:04 AM Ufuk Celebi  wrote:
> > > >
> > > >> Hey all,
> > > >>
> > > >> I'd like to join the release managers for 1.20 as well. I'm looking
> > > >> forward to getting more actively involved again.
> > > >>
> > > >> Cheers,
> > > >>
> > > >> Ufuk
> > > >>
> > > >> On Sun, Mar 24, 2024, at 11:27 AM, Ahmed Hamdy wrote:
> > > >>> +1 for the proposed timeline and release managers.
> > > >>> Best Regards
> > > >>> Ahmed Hamdy
> > > >>>
> > > >>>
> > > >>> On Fri, 22 Mar 2024 at 07:41, Xintong Song 
> > > >> wrote:
> > > >>>
> > >  +1 for the proposed timeline and Weijie & Rui as the release
> > managers.
> > > 
> > >  I think it would be welcomed if another 1-2 volunteers join as the
> > > >> release
> > >  managers, but that's not a must. We used to have only 1-2 release
> > > >> managers
> > >  for each release,
> > > 
> > >  Best,
> > > 
> > >  Xintong
> > > 
> > > 
> > > 
> > >  On Fri, Mar 22, 2024 at 2:55 PM Jark Wu  wrote:
> > > 
> > > > Thanks for kicking this off.
> > > >
> > > > +1 for the volunteered release managers (Weijie Guo, Rui Fan) and
> > the
> > > > targeting date (feature freeze: June 15).
> > > >
> > > > Best,
> > > > Jark
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Fri, 22 Mar 2024 at 14:00, Rui Fan <1996fan...@gmail.com>
> > wrote:
> > > >
> > > >> Thanks Leonard for this feedback and help!
> > > >>
> > > >> Best,
> > > >> Rui
> > > >>
> > > >> On Fri, Mar 22, 2024 at 12:36 PM weijie guo <
> > > >> guoweijieres...@gmail.com
> > > >
> > > >> wrote:
> > > >>
> > > >>> Thanks Leonard!
> > > >>>
> > >  I'd like to help you if you need some help like permissions
> > > >> from
> > >  PMC
> > > >>> side, please feel free to ping me.
> > > >>>
> > > >>> Nice to know. It'll help a lot!
> > > >>>
> > > >>> Best regards,
> > > >>>
> > > >>> Weijie
> > > >>>
> > > >>>
> > > >>> Leonard Xu  于2024年3月22日周五 12:09写道：
> > > >>>
> > >  +1 for the proposed release managers (Weijie Guo, Rui Fan),
> > > >> both the
> > > > two
> > >  candidates are pretty active committers thus I believe they
> > > >> know the
> > >  community development process well. The recent releases have
> > > >> four
> > > >> release
> > >  managers, and I am also looking forward to having other
> > > >> volunteers
> > >  join the management of Flink 1.20.
> > > 
> > >  +1 for targeting date (feature freeze: June 15, 2024),
> > > >> referring to
> > > > the
> > >  release cycle of recent versions, release cycle of 4 months
> > > >> makes
> > > > sense
> > > >> to
> > >  me.
> > > 
> > > 
> > >  I'd like to help you if you need some help like permissions
> > > >> from PMC
> > >  side, please feel free to ping me.
> > > 
> > >  Best,
> > >  Leonard
> > > 
> > > 
> > > > 2024年3月19日 下午5:35，Rui Fan <1996fan...@gmail.com> 写道：
> > > >
> > > > Hi Weijie,
> > > >
> > > > Thanks for kicking off 1.20! I'd like to join you and
> > > >> participate
> > >  in
> > > >> the
> > > > 1.20 release.
> > > >
> > > > Best,
> > > > Rui
> > > >
> > > > On Tue, Mar 19, 2024 at 5:30 PM weijie guo <
> > > > guoweijieres...@gmail.com
> > > >>>
> > > > wrote:
> > > >
> > > >> Hi everyone,
> > > >>
> > > >> With the release announcement of Flink 1.19, it's a good
> > > >> time to
> > > > kick
> > >  off
> > > >> discussion of the next release 1.20.
> > > >>
> > > >>
> > > >> - Release managers
> > > >>
> > > >>
> > > >> I'd like to 

[jira] [Created] (FLINK-34934) Translation Flink-Kubernetes-Operator document framework construction

[Caican Cai (Jira)]

Caican Cai created FLINK-34934:
--

 Summary: Translation Flink-Kubernetes-Operator document framework 
construction
 Key: FLINK-34934
 URL: https://issues.apache.org/jira/browse/FLINK-34934
 Project: Flink
  Issue Type: Sub-task
  Components: Kubernetes Operator
Affects Versions: 1.9.0
Reporter: Caican Cai
 Fix For: 1.9.0


Build the Flink-Kubernetes-Operator document translation framework, and 
subsequently translate English documents one by one.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[RESULT][VOTE] FLIP-433: State Access on DataStream API V2

[weijie guo]

Hi devs,


I'm happy to announce that FLIP-433: State Access on DataStream API V2
[1] has been accepted with 9 approving votes (4 binding) [2]:


- Xintong Song (binding)

- Hangxiang Yu (binding)

- Rui Fan (binding)

- Gyula Fóra (binding)

- Zakelly Lan (non-binding)

- Jinzhong Li (non-binding)

- Ahmed Hamdy (non-binding)

- Samrat Deb (non-binding)

- Yunfeng Zhou (non-binding)


There are no disapproving votes. Thanks to everyone who participated
in the discussion and voting.


Best regards,

Weijie


[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-433%3A+State+Access+on+DataStream+API+V2

[2] https://lists.apache.org/thread/hwxwfd8fcqo6olg5p9w41jv6rflzw53o

Re: [DISCUSS] FLIP-435: Introduce a New Dynamic Table for Simplifying Data Pipelines

[Ron liu]

Hi, Timo

Thanks for your quick response, and your suggestion.

Yes, this discussion has turned into confirming whether it's a special
table or a special MV.

1. The key problem with MVs is that they don't support modification, so I
prefer it to be a special table. Although the periodic refresh behavior is
more characteristic of an MV, since we are already a special table,
supporting periodic refresh behavior is quite natural, similar to Snowflake
dynamic tables.

2. Regarding the keyword UPDATING, since the current Regular Table is a
Dynamic Table, which implies support for updating through Continuous Query,
I think it is redundant to add the keyword UPDATING. In addition, UPDATING
can not reflect the Continuous Query part, can not express the purpose we
want to simplify the data pipeline through Dynamic Table + Continuous Query.

3. From the perspective of the SQL standard definition, I can understand
your concerns about Derived Table, but is it possible to make a slight
adjustment to meet our needs? Additionally, as Lincoln mentioned, the
Google Looker platform has introduced Persistent Derived Table, and there
are precedents in the industry; could Derived Table be a candidate?

Of course, look forward to your better suggestions.

Best,
Ron



Timo Walther  于2024年3月25日周一 18:49写道：

> After thinking about this more, this discussion boils down to whether
> this is a special table or a special materialized view. In both cases,
> we would need to add a special keyword:
>
> Either
>
> CREATE UPDATING TABLE
>
> or
>
> CREATE UPDATING MATERIALIZED VIEW
>
> I still feel that the periodic refreshing behavior is closer to a MV. If
> we add a special keyword to MV, the optimizer would know that the data
> cannot be used for query optimizations.
>
> I will ask more people for their opinion.
>
> Regards,
> Timo
>
>
> On 25.03.24 10:45, Timo Walther wrote:
> > Hi Ron and Lincoln,
> >
> > thanks for the quick response and the very insightful discussion.
> >
> >  > we might limit future opportunities to optimize queries
> >  > through automatic materialization rewriting by allowing data
> >  > modifications, thus losing the potential for such optimizations.
> >
> > This argument makes a lot of sense to me. Due to the updates, the system
> > is not in full control of the persisted data. However, the system is
> > still in full control of the job that powers the refresh. So if the
> > system manages all updating pipelines, it could still leverage automatic
> > materialization rewriting but without leveraging the data at rest (only
> > the data in flight).
> >
> >  > we are considering another candidate, Derived Table, the term 'derive'
> >  > suggests a query, and 'table' retains modifiability. This approach
> >  > would not disrupt our current concept of a dynamic table
> >
> > I did some research on this term. The SQL standard uses the term
> > "derived table" extensively (defined in section 4.17.3). Thus, a lot of
> > vendors adopt this for simply referring to a table within a subclause:
> >
> > https://dev.mysql.com/doc/refman/8.0/en/derived-tables.html
> >
> >
> https://infocenter.sybase.com/help/topic/com.sybase.infocenter.dc32300.1600/doc/html/san1390612291252.html
> >
> >
> https://www.c-sharpcorner.com/article/derived-tables-vs-common-table-expressions/
> >
> >
> https://stackoverflow.com/questions/26529804/what-are-the-derived-tables-in-my-explain-statement
> >
> > https://www.sqlservercentral.com/articles/sql-derived-tables
> >
> > Esp. the latter example is interesting, SQL Server allows things like
> > this on derived tables:
> >
> > UPDATE T SET Name='Timo' FROM (SELECT * FROM Product) AS T
> >
> > SELECT * FROM Product;
> >
> > Btw also Snowflake's dynamic table state:
> >
> >  > Because the content of a dynamic table is fully determined
> >  > by the given query, the content cannot be changed by using DML.
> >  > You don’t insert, update, or delete the rows in a dynamic table.
> >
> > So a new term makes a lot of sense.
> >
> > How about using `UPDATING`?
> >
> > CREATE UPDATING TABLE
> >
> > This reflects that modifications can be made and from an
> > English-language perspective you can PAUSE or RESUME the UPDATING.
> > Thus, a user can define UPDATING interval and mode?
> >
> > Looking forward to your thoughts.
> >
> > Regards,
> > Timo
> >
> >
> > On 25.03.24 07:09, Ron liu wrote:
> >> Hi, Ahmed
> >>
> >> Thanks for your feedback.
> >>
> >> Regarding your question:
> >>
> >>> I want to iterate on Timo's comments regarding the confusion between
> >> "Dynamic Table" and current Flink "Table". Should the refactoring of the
> >> system happen in 2.0, should we rename it in this Flip ( as the
> >> suggestions
> >> in the thread ) and address the holistic changes in a separate Flip
> >> for 2.0?
> >>
> >> Lincoln proposed a new concept in reply to Timo: Derived Table, which
> >> is a
> >> combination of Dynamic Table + Continuous Query, and the use of Derived
> >> Table will not conflict with existing concep

Re: [DISCUSS] Flink Website Menu Adjustment

[gongzhongqiang]

Hi Martijn,

Thank you for your feedback.

I agree with your point that we should make a one-time update to the menu,
rather than continuously updating it. This will be done unless some
sub-projects are moved or archived.

Best regards,

Zhongqiang Gong


Martijn Visser  于2024年3月25日周一 23:35写道：

> Hi Zhongqiang Gong,
>
> Are you suggesting to continuously update the menu based on the number of
> releases, or just this one time? I wouldn't be in favor of continuously
> updating: returning customers expect a certain order in the menu, and I
> don't see a lot of value in continuously changing that. I do think that the
> order that you have currently proposed is better then the one we have right
> now, so I would +1 a one-time update but not a continuously updating order.
>
> Best regards,
>
> Martijn
>
> On Mon, Mar 25, 2024 at 4:15 PM Yanquan Lv  wrote:
>
> > +1 for this proposal.
> >
> > gongzhongqiang  于2024年3月25日周一 15:49写道：
> >
> > > Hi everyone,
> > >
> > > I'd like to start a discussion on adjusting the Flink website [1] menu
> to
> > > improve accuracy and usability.While migrating Flink CDC documentation
> > > to the website, I found outdated links, need to review and update menus
> > > for the most relevant information for our users.
> > >
> > >
> > > Proposal:
> > >
> > > - Remove Paimon [2] from the "Getting Started" and "Documentation"
> menus:
> > > Paimon [2] is now an independent top project of ASF. CC： jingsong lees
> > >
> > > - Sort the projects in the subdirectory by the activity of the
> projects.
> > > Here I list the number of releases for each project in the past year.
> > >
> > > Flink Kubernetes Operator : 7
> > > Flink CDC : 5
> > > Flink ML  : 2
> > > Flink Stateful Functions : 1
> > >
> > >
> > > Expected Outcome :
> > >
> > > - Menu "Getting Started"
> > >
> > > Before:
> > >
> > > With Flink
> > >
> > > With Flink Stateful Functions
> > >
> > > With Flink ML
> > >
> > > With Flink Kubernetes Operator
> > >
> > > With Paimon(incubating) (formerly Flink Table Store)
> > >
> > > With Flink CDC
> > >
> > > Training Course
> > >
> > >
> > > After:
> > >
> > > With Flink
> > > With Flink Kubernetes Operator
> > >
> > > With Flink CDC
> > >
> > > With Flink ML
> > >
> > > With Flink Stateful Functions
> > >
> > > Training Course
> > >
> > >
> > > - Menu "Documentation" will same with "Getting Started"
> > >
> > >
> > > I look forward to hearing your thoughts and suggestions on this
> proposal.
> > >
> > > [1] https://flink.apache.org/
> > > [2] https://github.com/apache/incubator-paimon
> > > [3] https://github.com/apache/flink-statefun
> > >
> > >
> > >
> > > Best regards,
> > >
> > > Zhongqiang Gong
> > >
> >
>

Re: [DISCUSS] Flink Website Menu Adjustment

[Hongshun Wang]

+1 for the proposal

Best Regards,
Hongshun Wang

On Tue, Mar 26, 2024 at 11:37 AM gongzhongqiang 
wrote:

> Hi Martijn,
>
> Thank you for your feedback.
>
> I agree with your point that we should make a one-time update to the menu,
> rather than continuously updating it. This will be done unless some
> sub-projects are moved or archived.
>
> Best regards,
>
> Zhongqiang Gong
>
>
> Martijn Visser  于2024年3月25日周一 23:35写道：
>
> > Hi Zhongqiang Gong,
> >
> > Are you suggesting to continuously update the menu based on the number of
> > releases, or just this one time? I wouldn't be in favor of continuously
> > updating: returning customers expect a certain order in the menu, and I
> > don't see a lot of value in continuously changing that. I do think that
> the
> > order that you have currently proposed is better then the one we have
> right
> > now, so I would +1 a one-time update but not a continuously updating
> order.
> >
> > Best regards,
> >
> > Martijn
> >
> > On Mon, Mar 25, 2024 at 4:15 PM Yanquan Lv  wrote:
> >
> > > +1 for this proposal.
> > >
> > > gongzhongqiang  于2024年3月25日周一 15:49写道：
> > >
> > > > Hi everyone,
> > > >
> > > > I'd like to start a discussion on adjusting the Flink website [1]
> menu
> > to
> > > > improve accuracy and usability.While migrating Flink CDC
> documentation
> > > > to the website, I found outdated links, need to review and update
> menus
> > > > for the most relevant information for our users.
> > > >
> > > >
> > > > Proposal:
> > > >
> > > > - Remove Paimon [2] from the "Getting Started" and "Documentation"
> > menus:
> > > > Paimon [2] is now an independent top project of ASF. CC： jingsong
> lees
> > > >
> > > > - Sort the projects in the subdirectory by the activity of the
> > projects.
> > > > Here I list the number of releases for each project in the past year.
> > > >
> > > > Flink Kubernetes Operator : 7
> > > > Flink CDC : 5
> > > > Flink ML  : 2
> > > > Flink Stateful Functions : 1
> > > >
> > > >
> > > > Expected Outcome :
> > > >
> > > > - Menu "Getting Started"
> > > >
> > > > Before:
> > > >
> > > > With Flink
> > > >
> > > > With Flink Stateful Functions
> > > >
> > > > With Flink ML
> > > >
> > > > With Flink Kubernetes Operator
> > > >
> > > > With Paimon(incubating) (formerly Flink Table Store)
> > > >
> > > > With Flink CDC
> > > >
> > > > Training Course
> > > >
> > > >
> > > > After:
> > > >
> > > > With Flink
> > > > With Flink Kubernetes Operator
> > > >
> > > > With Flink CDC
> > > >
> > > > With Flink ML
> > > >
> > > > With Flink Stateful Functions
> > > >
> > > > Training Course
> > > >
> > > >
> > > > - Menu "Documentation" will same with "Getting Started"
> > > >
> > > >
> > > > I look forward to hearing your thoughts and suggestions on this
> > proposal.
> > > >
> > > > [1] https://flink.apache.org/
> > > > [2] https://github.com/apache/incubator-paimon
> > > > [3] https://github.com/apache/flink-statefun
> > > >
> > > >
> > > >
> > > > Best regards,
> > > >
> > > > Zhongqiang Gong
> > > >
> > >
> >
>

[jira] [Created] (FLINK-34935) TIMESTAMP_LTZ type Unsupported when using JdbcCatalog to read from Postgres

[Du Yuzhou (Jira)]

Du Yuzhou created FLINK-34935:
-

 Summary: TIMESTAMP_LTZ type Unsupported when using JdbcCatalog to 
read from Postgres
 Key: FLINK-34935
 URL: https://issues.apache.org/jira/browse/FLINK-34935
 Project: Flink
  Issue Type: Bug
  Components: Connectors / JDBC
Affects Versions: 1.18.1
 Environment: flink: 1.17.0
flink-connector-jdbc: 3.1.0-1.17
postgres: 14.5
java: 11
Reporter: Du Yuzhou


When I use JdbcCatalog to select from postgres table, it throw Exception:
{code:java}
Exception in thread "main" java.lang.UnsupportedOperationException: Unsupported 
type:TIMESTAMP_LTZ(6)
    at 
org.apache.flink.connector.jdbc.converter.AbstractJdbcRowConverter.createInternalConverter(AbstractJdbcRowConverter.java:186)
    at 
org.apache.flink.connector.jdbc.internal.converter.PostgresRowConverter.createPrimitiveConverter(PostgresRowConverter.java:99)
    at 
org.apache.flink.connector.jdbc.internal.converter.PostgresRowConverter.createInternalConverter(PostgresRowConverter.java:58)
    at 
org.apache.flink.connector.jdbc.converter.AbstractJdbcRowConverter.createNullableInternalConverter(AbstractJdbcRowConverter.java:118)
    at 
org.apache.flink.connector.jdbc.converter.AbstractJdbcRowConverter.(AbstractJdbcRowConverter.java:68)
    at 
org.apache.flink.connector.jdbc.internal.converter.PostgresRowConverter.(PostgresRowConverter.java:47)
    at 
org.apache.flink.connector.jdbc.dialect.psql.PostgresDialect.getRowConverter(PostgresDialect.java:50)
    at 
org.apache.flink.connector.jdbc.table.JdbcDynamicTableSource.getScanRuntimeProvider(JdbcDynamicTableSource.java:182)
    at 
org.apache.flink.table.planner.connectors.DynamicSourceUtils.validateScanSource(DynamicSourceUtils.java:466)
    at 
org.apache.flink.table.planner.connectors.DynamicSourceUtils.prepareDynamicSource(DynamicSourceUtils.java:161)
    at 
org.apache.flink.table.planner.connectors.DynamicSourceUtils.convertSourceToRel(DynamicSourceUtils.java:125)
    at 
org.apache.flink.table.planner.plan.schema.CatalogSourceTable.toRel(CatalogSourceTable.java:118)
    at 
org.apache.calcite.rel.core.RelFactories$TableScanFactoryImpl.createScan(RelFactories.java:481)
    at org.apache.calcite.tools.RelBuilder.scan(RelBuilder.java:1757)
    at 
org.apache.flink.table.planner.plan.QueryOperationConverter$SingleRelVisitor.visit(QueryOperationConverter.java:366)
    at 
org.apache.flink.table.planner.plan.QueryOperationConverter$SingleRelVisitor.visit(QueryOperationConverter.java:158)
    at 
org.apache.flink.table.operations.SourceQueryOperation.accept(SourceQueryOperation.java:86)
    at 
org.apache.flink.table.planner.plan.QueryOperationConverter.defaultMethod(QueryOperationConverter.java:155)
    at 
org.apache.flink.table.planner.plan.QueryOperationConverter.defaultMethod(QueryOperationConverter.java:135)
    at 
org.apache.flink.table.operations.utils.QueryOperationDefaultVisitor.visit(QueryOperationDefaultVisitor.java:92)
    at 
org.apache.flink.table.operations.SourceQueryOperation.accept(SourceQueryOperation.java:86)
    at 
org.apache.flink.table.planner.calcite.FlinkRelBuilder.queryOperation(FlinkRelBuilder.java:261)
    at 
org.apache.flink.table.planner.catalog.QueryOperationCatalogViewTable.convertToRel(QueryOperationCatalogViewTable.java:80)
    at 
org.apache.flink.table.planner.plan.schema.ExpandingPreparingTable.expand(ExpandingPreparingTable.java:70)
    at 
org.apache.flink.table.planner.plan.schema.ExpandingPreparingTable.toRel(ExpandingPreparingTable.java:57)
    at 
org.apache.calcite.sql2rel.SqlToRelConverter.toRel(SqlToRelConverter.java:3743)
    at 
org.apache.calcite.sql2rel.SqlToRelConverter.convertIdentifier(SqlToRelConverter.java:2666)
    at 
org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2233)
    at 
org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2147)
    at 
org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2092)
    at 
org.apache.calcite.sql2rel.SqlToRelConverter.convertSelectImpl(SqlToRelConverter.java:700)
    at 
org.apache.calcite.sql2rel.SqlToRelConverter.convertSelect(SqlToRelConverter.java:686)
    at 
org.apache.calcite.sql2rel.SqlToRelConverter.convertQueryRecursive(SqlToRelConverter.java:3589)
    at 
org.apache.calcite.sql2rel.SqlToRelConverter.convertQuery(SqlToRelConverter.java:599)
    at 
org.apache.flink.table.planner.calcite.FlinkPlannerImpl.org$apache$flink$table$planner$calcite$FlinkPlannerImpl$$rel(FlinkPlannerImpl.scala:216)
    at 
org.apache.flink.table.planner.calcite.FlinkPlannerImpl.rel(FlinkPlannerImpl.scala:192)
    at 
org.apache.flink.table.planner.operations.SqlToOperationConverter.toQueryOperation(SqlToOperationConverter.java:1580)
    at 
org.apache.flink.table.planner.operations.SqlToOperationConverter.convertSqlQuery(SqlToOperationConverter.java:1285)
    at 
org.apache.flink.ta

Re: [DISCUSS] Flink Website Menu Adjustment

[Shawn Huang]

+1 for the proposal

Best,
Shawn Huang


Hongshun Wang  于2024年3月26日周二 11:56写道：

> +1 for the proposal
>
> Best Regards,
> Hongshun Wang
>
> On Tue, Mar 26, 2024 at 11:37 AM gongzhongqiang  >
> wrote:
>
> > Hi Martijn,
> >
> > Thank you for your feedback.
> >
> > I agree with your point that we should make a one-time update to the
> menu,
> > rather than continuously updating it. This will be done unless some
> > sub-projects are moved or archived.
> >
> > Best regards,
> >
> > Zhongqiang Gong
> >
> >
> > Martijn Visser  于2024年3月25日周一 23:35写道：
> >
> > > Hi Zhongqiang Gong,
> > >
> > > Are you suggesting to continuously update the menu based on the number
> of
> > > releases, or just this one time? I wouldn't be in favor of continuously
> > > updating: returning customers expect a certain order in the menu, and I
> > > don't see a lot of value in continuously changing that. I do think that
> > the
> > > order that you have currently proposed is better then the one we have
> > right
> > > now, so I would +1 a one-time update but not a continuously updating
> > order.
> > >
> > > Best regards,
> > >
> > > Martijn
> > >
> > > On Mon, Mar 25, 2024 at 4:15 PM Yanquan Lv 
> wrote:
> > >
> > > > +1 for this proposal.
> > > >
> > > > gongzhongqiang  于2024年3月25日周一 15:49写道：
> > > >
> > > > > Hi everyone,
> > > > >
> > > > > I'd like to start a discussion on adjusting the Flink website [1]
> > menu
> > > to
> > > > > improve accuracy and usability.While migrating Flink CDC
> > documentation
> > > > > to the website, I found outdated links, need to review and update
> > menus
> > > > > for the most relevant information for our users.
> > > > >
> > > > >
> > > > > Proposal:
> > > > >
> > > > > - Remove Paimon [2] from the "Getting Started" and "Documentation"
> > > menus:
> > > > > Paimon [2] is now an independent top project of ASF. CC： jingsong
> > lees
> > > > >
> > > > > - Sort the projects in the subdirectory by the activity of the
> > > projects.
> > > > > Here I list the number of releases for each project in the past
> year.
> > > > >
> > > > > Flink Kubernetes Operator : 7
> > > > > Flink CDC : 5
> > > > > Flink ML  : 2
> > > > > Flink Stateful Functions : 1
> > > > >
> > > > >
> > > > > Expected Outcome :
> > > > >
> > > > > - Menu "Getting Started"
> > > > >
> > > > > Before:
> > > > >
> > > > > With Flink
> > > > >
> > > > > With Flink Stateful Functions
> > > > >
> > > > > With Flink ML
> > > > >
> > > > > With Flink Kubernetes Operator
> > > > >
> > > > > With Paimon(incubating) (formerly Flink Table Store)
> > > > >
> > > > > With Flink CDC
> > > > >
> > > > > Training Course
> > > > >
> > > > >
> > > > > After:
> > > > >
> > > > > With Flink
> > > > > With Flink Kubernetes Operator
> > > > >
> > > > > With Flink CDC
> > > > >
> > > > > With Flink ML
> > > > >
> > > > > With Flink Stateful Functions
> > > > >
> > > > > Training Course
> > > > >
> > > > >
> > > > > - Menu "Documentation" will same with "Getting Started"
> > > > >
> > > > >
> > > > > I look forward to hearing your thoughts and suggestions on this
> > > proposal.
> > > > >
> > > > > [1] https://flink.apache.org/
> > > > > [2] https://github.com/apache/incubator-paimon
> > > > > [3] https://github.com/apache/flink-statefun
> > > > >
> > > > >
> > > > >
> > > > > Best regards,
> > > > >
> > > > > Zhongqiang Gong
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Flink Website Menu Adjustment

[Hangxiang Yu]

Thanks Zhongqiang for driving this.
+1 for the proposal.

On Tue, Mar 26, 2024 at 1:36 PM Shawn Huang  wrote:

> +1 for the proposal
>
> Best,
> Shawn Huang
>
>
> Hongshun Wang  于2024年3月26日周二 11:56写道：
>
> > +1 for the proposal
> >
> > Best Regards,
> > Hongshun Wang
> >
> > On Tue, Mar 26, 2024 at 11:37 AM gongzhongqiang <
> gongzhongqi...@apache.org
> > >
> > wrote:
> >
> > > Hi Martijn,
> > >
> > > Thank you for your feedback.
> > >
> > > I agree with your point that we should make a one-time update to the
> > menu,
> > > rather than continuously updating it. This will be done unless some
> > > sub-projects are moved or archived.
> > >
> > > Best regards,
> > >
> > > Zhongqiang Gong
> > >
> > >
> > > Martijn Visser  于2024年3月25日周一 23:35写道：
> > >
> > > > Hi Zhongqiang Gong,
> > > >
> > > > Are you suggesting to continuously update the menu based on the
> number
> > of
> > > > releases, or just this one time? I wouldn't be in favor of
> continuously
> > > > updating: returning customers expect a certain order in the menu,
> and I
> > > > don't see a lot of value in continuously changing that. I do think
> that
> > > the
> > > > order that you have currently proposed is better then the one we have
> > > right
> > > > now, so I would +1 a one-time update but not a continuously updating
> > > order.
> > > >
> > > > Best regards,
> > > >
> > > > Martijn
> > > >
> > > > On Mon, Mar 25, 2024 at 4:15 PM Yanquan Lv 
> > wrote:
> > > >
> > > > > +1 for this proposal.
> > > > >
> > > > > gongzhongqiang  于2024年3月25日周一 15:49写道：
> > > > >
> > > > > > Hi everyone,
> > > > > >
> > > > > > I'd like to start a discussion on adjusting the Flink website [1]
> > > menu
> > > > to
> > > > > > improve accuracy and usability.While migrating Flink CDC
> > > documentation
> > > > > > to the website, I found outdated links, need to review and update
> > > menus
> > > > > > for the most relevant information for our users.
> > > > > >
> > > > > >
> > > > > > Proposal:
> > > > > >
> > > > > > - Remove Paimon [2] from the "Getting Started" and
> "Documentation"
> > > > menus:
> > > > > > Paimon [2] is now an independent top project of ASF. CC： jingsong
> > > lees
> > > > > >
> > > > > > - Sort the projects in the subdirectory by the activity of the
> > > > projects.
> > > > > > Here I list the number of releases for each project in the past
> > year.
> > > > > >
> > > > > > Flink Kubernetes Operator : 7
> > > > > > Flink CDC : 5
> > > > > > Flink ML  : 2
> > > > > > Flink Stateful Functions : 1
> > > > > >
> > > > > >
> > > > > > Expected Outcome :
> > > > > >
> > > > > > - Menu "Getting Started"
> > > > > >
> > > > > > Before:
> > > > > >
> > > > > > With Flink
> > > > > >
> > > > > > With Flink Stateful Functions
> > > > > >
> > > > > > With Flink ML
> > > > > >
> > > > > > With Flink Kubernetes Operator
> > > > > >
> > > > > > With Paimon(incubating) (formerly Flink Table Store)
> > > > > >
> > > > > > With Flink CDC
> > > > > >
> > > > > > Training Course
> > > > > >
> > > > > >
> > > > > > After:
> > > > > >
> > > > > > With Flink
> > > > > > With Flink Kubernetes Operator
> > > > > >
> > > > > > With Flink CDC
> > > > > >
> > > > > > With Flink ML
> > > > > >
> > > > > > With Flink Stateful Functions
> > > > > >
> > > > > > Training Course
> > > > > >
> > > > > >
> > > > > > - Menu "Documentation" will same with "Getting Started"
> > > > > >
> > > > > >
> > > > > > I look forward to hearing your thoughts and suggestions on this
> > > > proposal.
> > > > > >
> > > > > > [1] https://flink.apache.org/
> > > > > > [2] https://github.com/apache/incubator-paimon
> > > > > > [3] https://github.com/apache/flink-statefun
> > > > > >
> > > > > >
> > > > > >
> > > > > > Best regards,
> > > > > >
> > > > > > Zhongqiang Gong
> > > > > >
> > > > >
> > > >
> > >
> >
>


-- 
Best,
Hangxiang.

Re: [DISCUSS] Flink Website Menu Adjustment

[Hang Ruan]

+1 for the proposal.

Best,
Hang

Hangxiang Yu  于2024年3月26日周二 13:40写道：

> Thanks Zhongqiang for driving this.
> +1 for the proposal.
>
> On Tue, Mar 26, 2024 at 1:36 PM Shawn Huang  wrote:
>
> > +1 for the proposal
> >
> > Best,
> > Shawn Huang
> >
> >
> > Hongshun Wang  于2024年3月26日周二 11:56写道：
> >
> > > +1 for the proposal
> > >
> > > Best Regards,
> > > Hongshun Wang
> > >
> > > On Tue, Mar 26, 2024 at 11:37 AM gongzhongqiang <
> > gongzhongqi...@apache.org
> > > >
> > > wrote:
> > >
> > > > Hi Martijn,
> > > >
> > > > Thank you for your feedback.
> > > >
> > > > I agree with your point that we should make a one-time update to the
> > > menu,
> > > > rather than continuously updating it. This will be done unless some
> > > > sub-projects are moved or archived.
> > > >
> > > > Best regards,
> > > >
> > > > Zhongqiang Gong
> > > >
> > > >
> > > > Martijn Visser  于2024年3月25日周一 23:35写道：
> > > >
> > > > > Hi Zhongqiang Gong,
> > > > >
> > > > > Are you suggesting to continuously update the menu based on the
> > number
> > > of
> > > > > releases, or just this one time? I wouldn't be in favor of
> > continuously
> > > > > updating: returning customers expect a certain order in the menu,
> > and I
> > > > > don't see a lot of value in continuously changing that. I do think
> > that
> > > > the
> > > > > order that you have currently proposed is better then the one we
> have
> > > > right
> > > > > now, so I would +1 a one-time update but not a continuously
> updating
> > > > order.
> > > > >
> > > > > Best regards,
> > > > >
> > > > > Martijn
> > > > >
> > > > > On Mon, Mar 25, 2024 at 4:15 PM Yanquan Lv 
> > > wrote:
> > > > >
> > > > > > +1 for this proposal.
> > > > > >
> > > > > > gongzhongqiang  于2024年3月25日周一
> 15:49写道：
> > > > > >
> > > > > > > Hi everyone,
> > > > > > >
> > > > > > > I'd like to start a discussion on adjusting the Flink website
> [1]
> > > > menu
> > > > > to
> > > > > > > improve accuracy and usability.While migrating Flink CDC
> > > > documentation
> > > > > > > to the website, I found outdated links, need to review and
> update
> > > > menus
> > > > > > > for the most relevant information for our users.
> > > > > > >
> > > > > > >
> > > > > > > Proposal:
> > > > > > >
> > > > > > > - Remove Paimon [2] from the "Getting Started" and
> > "Documentation"
> > > > > menus:
> > > > > > > Paimon [2] is now an independent top project of ASF. CC：
> jingsong
> > > > lees
> > > > > > >
> > > > > > > - Sort the projects in the subdirectory by the activity of the
> > > > > projects.
> > > > > > > Here I list the number of releases for each project in the past
> > > year.
> > > > > > >
> > > > > > > Flink Kubernetes Operator : 7
> > > > > > > Flink CDC : 5
> > > > > > > Flink ML  : 2
> > > > > > > Flink Stateful Functions : 1
> > > > > > >
> > > > > > >
> > > > > > > Expected Outcome :
> > > > > > >
> > > > > > > - Menu "Getting Started"
> > > > > > >
> > > > > > > Before:
> > > > > > >
> > > > > > > With Flink
> > > > > > >
> > > > > > > With Flink Stateful Functions
> > > > > > >
> > > > > > > With Flink ML
> > > > > > >
> > > > > > > With Flink Kubernetes Operator
> > > > > > >
> > > > > > > With Paimon(incubating) (formerly Flink Table Store)
> > > > > > >
> > > > > > > With Flink CDC
> > > > > > >
> > > > > > > Training Course
> > > > > > >
> > > > > > >
> > > > > > > After:
> > > > > > >
> > > > > > > With Flink
> > > > > > > With Flink Kubernetes Operator
> > > > > > >
> > > > > > > With Flink CDC
> > > > > > >
> > > > > > > With Flink ML
> > > > > > >
> > > > > > > With Flink Stateful Functions
> > > > > > >
> > > > > > > Training Course
> > > > > > >
> > > > > > >
> > > > > > > - Menu "Documentation" will same with "Getting Started"
> > > > > > >
> > > > > > >
> > > > > > > I look forward to hearing your thoughts and suggestions on this
> > > > > proposal.
> > > > > > >
> > > > > > > [1] https://flink.apache.org/
> > > > > > > [2] https://github.com/apache/incubator-paimon
> > > > > > > [3] https://github.com/apache/flink-statefun
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Best regards,
> > > > > > >
> > > > > > > Zhongqiang Gong
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
>
> --
> Best,
> Hangxiang.
>

[jira] [Created] (FLINK-34936) Register shared state files to FileMergingSnapshotManager

[Jinzhong Li (Jira)]

Jinzhong Li created FLINK-34936:
---

 Summary: Register shared state files to FileMergingSnapshotManager
 Key: FLINK-34936
 URL: https://issues.apache.org/jira/browse/FLINK-34936
 Project: Flink
  Issue Type: New Feature
Reporter: Jinzhong Li
 Fix For: 1.20.0


The shared state files should be registered into the 
FileMergingSnapshotManager, so that these files can be properly cleaned up  
when checkpoint aborted/subsumed.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)