Re: [ANNOUNCE] New Apache Flink PMC Member - Jingsong Lee

[Qingsheng Ren]

Congratulations, Jingsong! 

Best, 
Qingsheng

> On Jun 13, 2022, at 14:58, Becket Qin  wrote:
> 
> Hi all,
> 
> I'm very happy to announce that Jingsong Lee has joined the Flink PMC!
> 
> Jingsong became a Flink committer in Feb 2020 and has been continuously
> contributing to the project since then, mainly in Flink SQL. He has been
> quite active in the mailing list, fixing bugs, helping verifying releases,
> reviewing patches and FLIPs. Jingsong is also devoted to pushing Flink SQL
> to new use cases. He spent a lot of time in implementing the Flink
> connectors for Apache Iceberg. Jingsong is also the primary driver behind
> the effort of flink-table-store, which aims to provide a stream-batch
> unified storage for Flink dynamic tables.
> 
> Congratulations and welcome, Jingsong!
> 
> Cheers,
> 
> Jiangjie (Becket) Qin
> (On behalf of the Apache Flink PMC)

Re: [VOTE] FLIP-228: Support Within between events in CEP Pattern

[md peng]

+1(not-binding)

Best,
Mingde Peng

Martijn Visser  于2022年6月13日周一 14:48写道：

> +1 (binding)
>
> Looking forward :)
>
> Thanks, Martijn
>
> Op ma 13 jun. 2022 om 04:16 schreef Nicholas :
>
> > Hi everyone，
> >
> >
> >
> >
> > Thanks for feedback for FLIP-228: Support Within between events in CEP
> > Pattern[1] on the discussion thread[2]. I'd like to start a VOTE thread
> for
> > FLIP-228.
> >
> > The vote will be open for at least 72 hours unless there is an objection
> > or not enough votes.
> >
> >
> >
> >
> > Regards,
> >
> > Nicholas Jiang
> >
> >
> >
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-228%3A+Support+Within+between+events+in+CEP+Pattern
> >
> > [2] https://lists.apache.org/thread/p60ctx213f8921rgklk5f0b6xfrs8ksz
>

Re: [ANNOUNCE] New Apache Flink PMC Member - Jingsong Lee

[Nicholas Jiang]

Congratulations, Jingsong!

Regards,
Nicholas Jiang

On 2022/06/13 07:01:11 Qingsheng Ren wrote:
> Congratulations, Jingsong! 
> 
> Best, 
> Qingsheng
> 
> > On Jun 13, 2022, at 14:58, Becket Qin  wrote:
> > 
> > Hi all,
> > 
> > I'm very happy to announce that Jingsong Lee has joined the Flink PMC!
> > 
> > Jingsong became a Flink committer in Feb 2020 and has been continuously
> > contributing to the project since then, mainly in Flink SQL. He has been
> > quite active in the mailing list, fixing bugs, helping verifying releases,
> > reviewing patches and FLIPs. Jingsong is also devoted to pushing Flink SQL
> > to new use cases. He spent a lot of time in implementing the Flink
> > connectors for Apache Iceberg. Jingsong is also the primary driver behind
> > the effort of flink-table-store, which aims to provide a stream-batch
> > unified storage for Flink dynamic tables.
> > 
> > Congratulations and welcome, Jingsong!
> > 
> > Cheers,
> > 
> > Jiangjie (Becket) Qin
> > (On behalf of the Apache Flink PMC)
> 
> 

Re: [ANNOUNCE] New Apache Flink PMC Member - Jingsong Lee

[Ingo Bürk]

Congrats, Jingsong!

On 13.06.22 08:58, Becket Qin wrote:

Hi all,

I'm very happy to announce that Jingsong Lee has joined the Flink PMC!

Jingsong became a Flink committer in Feb 2020 and has been continuously
contributing to the project since then, mainly in Flink SQL. He has been
quite active in the mailing list, fixing bugs, helping verifying releases,
reviewing patches and FLIPs. Jingsong is also devoted to pushing Flink SQL
to new use cases. He spent a lot of time in implementing the Flink
connectors for Apache Iceberg. Jingsong is also the primary driver behind
the effort of flink-table-store, which aims to provide a stream-batch
unified storage for Flink dynamic tables.

Congratulations and welcome, Jingsong!

Cheers,

Jiangjie (Becket) Qin
(On behalf of the Apache Flink PMC)

Re: [ANNOUNCE] New Apache Flink PMC Member - Jingsong Lee

[LuNing Wang]

Congratulations, Jingsong!

Best,
LuNing Wang

Ingo Bürk  于2022年6月13日周一 15:36写道：

> Congrats, Jingsong!
>
> On 13.06.22 08:58, Becket Qin wrote:
> > Hi all,
> >
> > I'm very happy to announce that Jingsong Lee has joined the Flink PMC!
> >
> > Jingsong became a Flink committer in Feb 2020 and has been continuously
> > contributing to the project since then, mainly in Flink SQL. He has been
> > quite active in the mailing list, fixing bugs, helping verifying
> releases,
> > reviewing patches and FLIPs. Jingsong is also devoted to pushing Flink
> SQL
> > to new use cases. He spent a lot of time in implementing the Flink
> > connectors for Apache Iceberg. Jingsong is also the primary driver behind
> > the effort of flink-table-store, which aims to provide a stream-batch
> > unified storage for Flink dynamic tables.
> >
> > Congratulations and welcome, Jingsong!
> >
> > Cheers,
> >
> > Jiangjie (Becket) Qin
> > (On behalf of the Apache Flink PMC)
> >
>

Re: [ANNOUNCE] New Apache Flink PMC Member - Jingsong Lee

[Jing Ge]

Congrats, Jingsong! Very well deserved!

Best regards,
Jing

On Mon, Jun 13, 2022 at 9:39 AM LuNing Wang  wrote:

> Congratulations, Jingsong!
>
> Best,
> LuNing Wang
>
> Ingo Bürk  于2022年6月13日周一 15:36写道：
>
> > Congrats, Jingsong!
> >
> > On 13.06.22 08:58, Becket Qin wrote:
> > > Hi all,
> > >
> > > I'm very happy to announce that Jingsong Lee has joined the Flink PMC!
> > >
> > > Jingsong became a Flink committer in Feb 2020 and has been continuously
> > > contributing to the project since then, mainly in Flink SQL. He has
> been
> > > quite active in the mailing list, fixing bugs, helping verifying
> > releases,
> > > reviewing patches and FLIPs. Jingsong is also devoted to pushing Flink
> > SQL
> > > to new use cases. He spent a lot of time in implementing the Flink
> > > connectors for Apache Iceberg. Jingsong is also the primary driver
> behind
> > > the effort of flink-table-store, which aims to provide a stream-batch
> > > unified storage for Flink dynamic tables.
> > >
> > > Congratulations and welcome, Jingsong!
> > >
> > > Cheers,
> > >
> > > Jiangjie (Becket) Qin
> > > (On behalf of the Apache Flink PMC)
> > >
> >
>

Re: [ANNOUNCE] New Apache Flink PMC Member - Jingsong Lee

[Rui Fan]

Congratulations, Jingsong!

Best,
Rui Fan

On Mon, Jun 13, 2022 at 3:40 PM LuNing Wang  wrote:

> Congratulations, Jingsong!
>
> Best,
> LuNing Wang
>
> Ingo Bürk  于2022年6月13日周一 15:36写道：
>
> > Congrats, Jingsong!
> >
> > On 13.06.22 08:58, Becket Qin wrote:
> > > Hi all,
> > >
> > > I'm very happy to announce that Jingsong Lee has joined the Flink PMC!
> > >
> > > Jingsong became a Flink committer in Feb 2020 and has been continuously
> > > contributing to the project since then, mainly in Flink SQL. He has
> been
> > > quite active in the mailing list, fixing bugs, helping verifying
> > releases,
> > > reviewing patches and FLIPs. Jingsong is also devoted to pushing Flink
> > SQL
> > > to new use cases. He spent a lot of time in implementing the Flink
> > > connectors for Apache Iceberg. Jingsong is also the primary driver
> behind
> > > the effort of flink-table-store, which aims to provide a stream-batch
> > > unified storage for Flink dynamic tables.
> > >
> > > Congratulations and welcome, Jingsong!
> > >
> > > Cheers,
> > >
> > > Jiangjie (Becket) Qin
> > > (On behalf of the Apache Flink PMC)
> > >
> >
>

Re: [VOTE] FLIP-228: Support Within between events in CEP Pattern

[Rui Fan]

Thanks for Nicholas driving this.

+1(non-binding)

Best,
Rui Fan

On Mon, Jun 13, 2022 at 3:05 PM md peng  wrote:

> +1(not-binding)
>
> Best,
> Mingde Peng
>
> Martijn Visser  于2022年6月13日周一 14:48写道：
>
> > +1 (binding)
> >
> > Looking forward :)
> >
> > Thanks, Martijn
> >
> > Op ma 13 jun. 2022 om 04:16 schreef Nicholas :
> >
> > > Hi everyone，
> > >
> > >
> > >
> > >
> > > Thanks for feedback for FLIP-228: Support Within between events in CEP
> > > Pattern[1] on the discussion thread[2]. I'd like to start a VOTE thread
> > for
> > > FLIP-228.
> > >
> > > The vote will be open for at least 72 hours unless there is an
> objection
> > > or not enough votes.
> > >
> > >
> > >
> > >
> > > Regards,
> > >
> > > Nicholas Jiang
> > >
> > >
> > >
> > >
> > > [1]
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-228%3A+Support+Within+between+events+in+CEP+Pattern
> > >
> > > [2] https://lists.apache.org/thread/p60ctx213f8921rgklk5f0b6xfrs8ksz
> >
>

Re: [ANNOUNCE] New Apache Flink PMC Member - Jingsong Lee

[Benchao Li]

Congratulations, Jingsong!  Well deserved.

Rui Fan <1996fan...@gmail.com> 于2022年6月13日周一 15:53写道：

> Congratulations, Jingsong!
>
> Best,
> Rui Fan
>
> On Mon, Jun 13, 2022 at 3:40 PM LuNing Wang  wrote:
>
> > Congratulations, Jingsong!
> >
> > Best,
> > LuNing Wang
> >
> > Ingo Bürk  于2022年6月13日周一 15:36写道：
> >
> > > Congrats, Jingsong!
> > >
> > > On 13.06.22 08:58, Becket Qin wrote:
> > > > Hi all,
> > > >
> > > > I'm very happy to announce that Jingsong Lee has joined the Flink
> PMC!
> > > >
> > > > Jingsong became a Flink committer in Feb 2020 and has been
> continuously
> > > > contributing to the project since then, mainly in Flink SQL. He has
> > been
> > > > quite active in the mailing list, fixing bugs, helping verifying
> > > releases,
> > > > reviewing patches and FLIPs. Jingsong is also devoted to pushing
> Flink
> > > SQL
> > > > to new use cases. He spent a lot of time in implementing the Flink
> > > > connectors for Apache Iceberg. Jingsong is also the primary driver
> > behind
> > > > the effort of flink-table-store, which aims to provide a stream-batch
> > > > unified storage for Flink dynamic tables.
> > > >
> > > > Congratulations and welcome, Jingsong!
> > > >
> > > > Cheers,
> > > >
> > > > Jiangjie (Becket) Qin
> > > > (On behalf of the Apache Flink PMC)
> > > >
> > >
> >
>


-- 

Best,
Benchao Li

Re: [ANNOUNCE] New Apache Flink PMC Member - Jingsong Lee

[Lijie Wang]

Congratulations, Jingsong!

Best,
Lijie

Rui Fan <1996fan...@gmail.com> 于2022年6月13日周一 15:53写道：

> Congratulations, Jingsong!
>
> Best,
> Rui Fan
>
> On Mon, Jun 13, 2022 at 3:40 PM LuNing Wang  wrote:
>
> > Congratulations, Jingsong!
> >
> > Best,
> > LuNing Wang
> >
> > Ingo Bürk  于2022年6月13日周一 15:36写道：
> >
> > > Congrats, Jingsong!
> > >
> > > On 13.06.22 08:58, Becket Qin wrote:
> > > > Hi all,
> > > >
> > > > I'm very happy to announce that Jingsong Lee has joined the Flink
> PMC!
> > > >
> > > > Jingsong became a Flink committer in Feb 2020 and has been
> continuously
> > > > contributing to the project since then, mainly in Flink SQL. He has
> > been
> > > > quite active in the mailing list, fixing bugs, helping verifying
> > > releases,
> > > > reviewing patches and FLIPs. Jingsong is also devoted to pushing
> Flink
> > > SQL
> > > > to new use cases. He spent a lot of time in implementing the Flink
> > > > connectors for Apache Iceberg. Jingsong is also the primary driver
> > behind
> > > > the effort of flink-table-store, which aims to provide a stream-batch
> > > > unified storage for Flink dynamic tables.
> > > >
> > > > Congratulations and welcome, Jingsong!
> > > >
> > > > Cheers,
> > > >
> > > > Jiangjie (Becket) Qin
> > > > (On behalf of the Apache Flink PMC)
> > > >
> > >
> >
>

Re: [ANNOUNCE] New Apache Flink PMC Member - Jingsong Lee

[Martijn Visser]

Like everyone has mentioned, this is very well deserved. Congratulations!

Op ma 13 jun. 2022 om 09:57 schreef Benchao Li :

> Congratulations, Jingsong!  Well deserved.
>
> Rui Fan <1996fan...@gmail.com> 于2022年6月13日周一 15:53写道：
>
> > Congratulations, Jingsong!
> >
> > Best,
> > Rui Fan
> >
> > On Mon, Jun 13, 2022 at 3:40 PM LuNing Wang 
> wrote:
> >
> > > Congratulations, Jingsong!
> > >
> > > Best,
> > > LuNing Wang
> > >
> > > Ingo Bürk  于2022年6月13日周一 15:36写道：
> > >
> > > > Congrats, Jingsong!
> > > >
> > > > On 13.06.22 08:58, Becket Qin wrote:
> > > > > Hi all,
> > > > >
> > > > > I'm very happy to announce that Jingsong Lee has joined the Flink
> > PMC!
> > > > >
> > > > > Jingsong became a Flink committer in Feb 2020 and has been
> > continuously
> > > > > contributing to the project since then, mainly in Flink SQL. He has
> > > been
> > > > > quite active in the mailing list, fixing bugs, helping verifying
> > > > releases,
> > > > > reviewing patches and FLIPs. Jingsong is also devoted to pushing
> > Flink
> > > > SQL
> > > > > to new use cases. He spent a lot of time in implementing the Flink
> > > > > connectors for Apache Iceberg. Jingsong is also the primary driver
> > > behind
> > > > > the effort of flink-table-store, which aims to provide a
> stream-batch
> > > > > unified storage for Flink dynamic tables.
> > > > >
> > > > > Congratulations and welcome, Jingsong!
> > > > >
> > > > > Cheers,
> > > > >
> > > > > Jiangjie (Becket) Qin
> > > > > (On behalf of the Apache Flink PMC)
> > > > >
> > > >
> > >
> >
>
>
> --
>
> Best,
> Benchao Li
>

Re: [VOTE] FLIP-228: Support Within between events in CEP Pattern

[Dian Fu]

+1 (binding)

Regards,
Dian

On Mon, Jun 13, 2022 at 3:55 PM Rui Fan <1996fan...@gmail.com> wrote:

> Thanks for Nicholas driving this.
>
> +1(non-binding)
>
> Best,
> Rui Fan
>
> On Mon, Jun 13, 2022 at 3:05 PM md peng  wrote:
>
> > +1(not-binding)
> >
> > Best,
> > Mingde Peng
> >
> > Martijn Visser  于2022年6月13日周一 14:48写道：
> >
> > > +1 (binding)
> > >
> > > Looking forward :)
> > >
> > > Thanks, Martijn
> > >
> > > Op ma 13 jun. 2022 om 04:16 schreef Nicholas :
> > >
> > > > Hi everyone，
> > > >
> > > >
> > > >
> > > >
> > > > Thanks for feedback for FLIP-228: Support Within between events in
> CEP
> > > > Pattern[1] on the discussion thread[2]. I'd like to start a VOTE
> thread
> > > for
> > > > FLIP-228.
> > > >
> > > > The vote will be open for at least 72 hours unless there is an
> > objection
> > > > or not enough votes.
> > > >
> > > >
> > > >
> > > >
> > > > Regards,
> > > >
> > > > Nicholas Jiang
> > > >
> > > >
> > > >
> > > >
> > > > [1]
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-228%3A+Support+Within+between+events+in+CEP+Pattern
> > > >
> > > > [2] https://lists.apache.org/thread/p60ctx213f8921rgklk5f0b6xfrs8ksz
> > >
> >
>

[jira] [Created] (FLINK-28016) Support Maven 3.3+

[Chesnay Schepler (Jira)]

Chesnay Schepler created FLINK-28016:


 Summary: Support Maven 3.3+
 Key: FLINK-28016
 URL: https://issues.apache.org/jira/browse/FLINK-28016
 Project: Flink
  Issue Type: Technical Debt
  Components: Build System
Reporter: Chesnay Schepler
Assignee: Chesnay Schepler
 Fix For: 1.16.0






--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Re: [ANNOUNCE] New Apache Flink PMC Member - Jingsong Lee

[Jiangang Liu]

Congratulations, Jingsong!

Best,
Jiangang Liu

Martijn Visser  于2022年6月13日周一 16:06写道：

> Like everyone has mentioned, this is very well deserved. Congratulations!
>
> Op ma 13 jun. 2022 om 09:57 schreef Benchao Li :
>
> > Congratulations, Jingsong!  Well deserved.
> >
> > Rui Fan <1996fan...@gmail.com> 于2022年6月13日周一 15:53写道：
> >
> > > Congratulations, Jingsong!
> > >
> > > Best,
> > > Rui Fan
> > >
> > > On Mon, Jun 13, 2022 at 3:40 PM LuNing Wang 
> > wrote:
> > >
> > > > Congratulations, Jingsong!
> > > >
> > > > Best,
> > > > LuNing Wang
> > > >
> > > > Ingo Bürk  于2022年6月13日周一 15:36写道：
> > > >
> > > > > Congrats, Jingsong!
> > > > >
> > > > > On 13.06.22 08:58, Becket Qin wrote:
> > > > > > Hi all,
> > > > > >
> > > > > > I'm very happy to announce that Jingsong Lee has joined the Flink
> > > PMC!
> > > > > >
> > > > > > Jingsong became a Flink committer in Feb 2020 and has been
> > > continuously
> > > > > > contributing to the project since then, mainly in Flink SQL. He
> has
> > > > been
> > > > > > quite active in the mailing list, fixing bugs, helping verifying
> > > > > releases,
> > > > > > reviewing patches and FLIPs. Jingsong is also devoted to pushing
> > > Flink
> > > > > SQL
> > > > > > to new use cases. He spent a lot of time in implementing the
> Flink
> > > > > > connectors for Apache Iceberg. Jingsong is also the primary
> driver
> > > > behind
> > > > > > the effort of flink-table-store, which aims to provide a
> > stream-batch
> > > > > > unified storage for Flink dynamic tables.
> > > > > >
> > > > > > Congratulations and welcome, Jingsong!
> > > > > >
> > > > > > Cheers,
> > > > > >
> > > > > > Jiangjie (Becket) Qin
> > > > > > (On behalf of the Apache Flink PMC)
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> > --
> >
> > Best,
> > Benchao Li
> >
>

[jira] [Created] (FLINK-28017) Introduce bucket-key to table store

[Jingsong Lee (Jira)]

Jingsong Lee created FLINK-28017:


 Summary: Introduce bucket-key to table store
 Key: FLINK-28017
 URL: https://issues.apache.org/jira/browse/FLINK-28017
 Project: Flink
  Issue Type: New Feature
  Components: Table Store
Reporter: Jingsong Lee
 Fix For: table-store-0.2.0


Specifies the table store distribution policy. Data is assigned to each bucket 
according to the hash value of bucket-key.
 * It is primary key when table has primary key. The user can specify a bucket 
key, it should be part of primary keys.
 * It is all fields when table has no primary key.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Re:Re: Re: [DISCUSS] FLIP-218: Support SELECT clause in CREATE TABLE(CTAS)

[Mang Zhang]

Hi Jing,



Thanks for your reply!
I recently updated the FLIP documentation, when you have time, you can look at 
it again.
About Atomicity and Isolation, the main discussion point is on batch mode, to 
be more specific is Hive/Spark.
My point of consideration is that according to the needs of the actual business 
scenario and the current implementation in spark, 
I decided to keep an approximation with the spark implementation first.
In our company, spark sql is currently the main engine for offline computing, 
so I currently choose the Spark DataSource v1 solution.



--

Best regards,
Mang Zhang





At 2022-06-04 03:35:59, "Jing Ge"  wrote:
>Hi Mang,
>
>Thanks for driving this! Finally there is a discussion about CTAS. It is
>one of the most important features.
>
>I agree with Jark that it would be good to be able to take care of both
>atomicity and isolation, which lead to Spark DataSource v2. Would you like
>to help me understand the connection between the reason and your decision
>to pick Spark DataSource V1?
>
>*Reasons:*
>
>   - *Streaming mode requires the table to be created first, downstream
>   jobs can consume in real time. *(both Spark DataSource V1 and V2)
>   - *In most cases, Streaming jobs do not need to be cleaned up even if
>   the job fails. * (both Spark DataSource V1 and V2)
>   - *Flink has a rich connector ecosystem, and the capabilities provided
>   by external storage systems are different, Flink needs to behave
>   consistently. *(how will it lead to Spark DataSource V1)
>   - *Batch jobs try to ensure final atomicity. *(means to choose Spark
>   DataSource V2 right?)
>
>
>I think there are some differences between Spark DataSource V1 and V2, e.g.
>when will the sind table be visible. Whether the result will be written to
>a temporary directory or directly to the sink table, etc.
>
>It would be great if you could update the reasons that led to your
>decision. thanks!
>
>Best regards,
>Jing
>
>On Tue, May 31, 2022 at 8:34 AM Yun Gao 
>wrote:
>
>> Hi,
>>
>> Regarding the drop operation, with some offline discussion with Dalong and
>> Zhu,
>> we think that listening in the client side might be problematic since it
>> would exit
>> after submitting the jobs in detached mode, thus the operation might need
>> to
>> be in the JobMaster side.
>>
>> For the listener interface, currently JobListener only resides in the
>> client side
>> and contains unsuitable methods like onJobSubmitted for this scenario, and
>> the internal JobStatusListener is designed to be used inside JM and is not
>> serializable, thus we tend to add a new interface JobStatusHook,
>> which could be attached to the JobGraph and executed in the JobMaster.
>> The interface will also be marked as Internal.
>>
>> Best,
>> Yun
>>
>>
>> --
>> From:Mang Zhang 
>> Send Time:2022 May 25 (Wed.) 10:24
>> To:dev 
>> Subject:Re:Re: [DISCUSS] FLIP-218: Support SELECT clause in CREATE
>> TABLE(CTAS)
>>
>> Hi, Martijn
>> Thanks for your reply!
>> I looked at the SQL standard, CTAS is part of the SQL standard.
>> Feature T172 is "AS subquery clause in table definition".
>>
>>
>>
>> --
>>
>> Best regards,
>> Mang Zhang
>>
>>
>>
>>
>>
>> At 2022-05-04 21:49:00, "Martijn Visser"  wrote:
>> >Hi everyone,
>> >
>> >Can we identify if this proposed syntax is part of the SQL standard?
>> >
>> >Best regards,
>> >
>> >Martijn Visser
>> >https://twitter.com/MartijnVisser82
>> >https://github.com/MartijnVisser
>> >
>> >
>> >On Fri, 29 Apr 2022 at 11:19, yuxia  wrote:
>> >
>> >> Thanks for for driving this work, it's to be a useful feature.
>> >> About the flip-218, I have some questions.
>> >>
>> >> 1: Does our CTAS syntax support specify target table's schema including
>> >> column name and data type? I think it maybe a useful fature in case we
>> want
>> >> to change the data types in target table instead of always copy the
>> source
>> >> table's schema. It'll be more flexible with this feature.
>> >> Btw, MySQL's "CREATE TABLE ... SELECT Statement"[1] support this
>> feature.
>> >>
>> >> 2: Seems it'll requre sink to implement an public interface to drop
>> table,
>> >> so what's the interface will look like?
>> >>
>> >> [1] https://dev.mysql.com/doc/refman/8.0/en/create-table-select.html
>> >>
>> >> Best regards,
>> >> Yuxia
>> >>
>> >> - 原始邮件 -
>> >> 发件人: "Mang Zhang" 
>> >> 收件人: "dev" 
>> >> 发送时间: 星期四, 2022年 4 月 28日 下午 4:57:24
>> >> 主题: [DISCUSS] FLIP-218: Support SELECT clause in CREATE TABLE(CTAS)
>> >>
>> >> Hi, everyone
>> >>
>> >>
>> >> I would like to open a discussion for support select clause in CREATE
>> >> TABLE(CTAS),
>> >> With the development of business and the enhancement of flink sql
>> >> capabilities, queries become more and more complex.
>> >> Now the user needs to use the Create Table statement to create the
>> target
>> >> table first, and then execute the insert statement.
>> >> However, the target table may have many columns, which will bring 

Re: [ANNOUNCE] New Apache Flink PMC Member - Jingsong Lee

[Jark Wu]

Congrats, Jingsong!

Cheers,
Jark

On Mon, 13 Jun 2022 at 16:16, Jiangang Liu 
wrote:

> Congratulations, Jingsong!
>
> Best,
> Jiangang Liu
>
> Martijn Visser  于2022年6月13日周一 16:06写道：
>
> > Like everyone has mentioned, this is very well deserved. Congratulations!
> >
> > Op ma 13 jun. 2022 om 09:57 schreef Benchao Li :
> >
> > > Congratulations, Jingsong!  Well deserved.
> > >
> > > Rui Fan <1996fan...@gmail.com> 于2022年6月13日周一 15:53写道：
> > >
> > > > Congratulations, Jingsong!
> > > >
> > > > Best,
> > > > Rui Fan
> > > >
> > > > On Mon, Jun 13, 2022 at 3:40 PM LuNing Wang 
> > > wrote:
> > > >
> > > > > Congratulations, Jingsong!
> > > > >
> > > > > Best,
> > > > > LuNing Wang
> > > > >
> > > > > Ingo Bürk  于2022年6月13日周一 15:36写道：
> > > > >
> > > > > > Congrats, Jingsong!
> > > > > >
> > > > > > On 13.06.22 08:58, Becket Qin wrote:
> > > > > > > Hi all,
> > > > > > >
> > > > > > > I'm very happy to announce that Jingsong Lee has joined the
> Flink
> > > > PMC!
> > > > > > >
> > > > > > > Jingsong became a Flink committer in Feb 2020 and has been
> > > > continuously
> > > > > > > contributing to the project since then, mainly in Flink SQL. He
> > has
> > > > > been
> > > > > > > quite active in the mailing list, fixing bugs, helping
> verifying
> > > > > > releases,
> > > > > > > reviewing patches and FLIPs. Jingsong is also devoted to
> pushing
> > > > Flink
> > > > > > SQL
> > > > > > > to new use cases. He spent a lot of time in implementing the
> > Flink
> > > > > > > connectors for Apache Iceberg. Jingsong is also the primary
> > driver
> > > > > behind
> > > > > > > the effort of flink-table-store, which aims to provide a
> > > stream-batch
> > > > > > > unified storage for Flink dynamic tables.
> > > > > > >
> > > > > > > Congratulations and welcome, Jingsong!
> > > > > > >
> > > > > > > Cheers,
> > > > > > >
> > > > > > > Jiangjie (Becket) Qin
> > > > > > > (On behalf of the Apache Flink PMC)
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > --
> > >
> > > Best,
> > > Benchao Li
> > >
> >
>

Re: [ANNOUNCE] New Apache Flink PMC Member - Jingsong Lee

[Lincoln Lee]

Congratulations, Jingsong!

Best,
Lincoln Lee


Jark Wu  于2022年6月13日周一 16:29写道：

> Congrats, Jingsong!
>
> Cheers,
> Jark
>
> On Mon, 13 Jun 2022 at 16:16, Jiangang Liu 
> wrote:
>
> > Congratulations, Jingsong!
> >
> > Best,
> > Jiangang Liu
> >
> > Martijn Visser  于2022年6月13日周一 16:06写道：
> >
> > > Like everyone has mentioned, this is very well deserved.
> Congratulations!
> > >
> > > Op ma 13 jun. 2022 om 09:57 schreef Benchao Li :
> > >
> > > > Congratulations, Jingsong!  Well deserved.
> > > >
> > > > Rui Fan <1996fan...@gmail.com> 于2022年6月13日周一 15:53写道：
> > > >
> > > > > Congratulations, Jingsong!
> > > > >
> > > > > Best,
> > > > > Rui Fan
> > > > >
> > > > > On Mon, Jun 13, 2022 at 3:40 PM LuNing Wang  >
> > > > wrote:
> > > > >
> > > > > > Congratulations, Jingsong!
> > > > > >
> > > > > > Best,
> > > > > > LuNing Wang
> > > > > >
> > > > > > Ingo Bürk  于2022年6月13日周一 15:36写道：
> > > > > >
> > > > > > > Congrats, Jingsong!
> > > > > > >
> > > > > > > On 13.06.22 08:58, Becket Qin wrote:
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > I'm very happy to announce that Jingsong Lee has joined the
> > Flink
> > > > > PMC!
> > > > > > > >
> > > > > > > > Jingsong became a Flink committer in Feb 2020 and has been
> > > > > continuously
> > > > > > > > contributing to the project since then, mainly in Flink SQL.
> He
> > > has
> > > > > > been
> > > > > > > > quite active in the mailing list, fixing bugs, helping
> > verifying
> > > > > > > releases,
> > > > > > > > reviewing patches and FLIPs. Jingsong is also devoted to
> > pushing
> > > > > Flink
> > > > > > > SQL
> > > > > > > > to new use cases. He spent a lot of time in implementing the
> > > Flink
> > > > > > > > connectors for Apache Iceberg. Jingsong is also the primary
> > > driver
> > > > > > behind
> > > > > > > > the effort of flink-table-store, which aims to provide a
> > > > stream-batch
> > > > > > > > unified storage for Flink dynamic tables.
> > > > > > > >
> > > > > > > > Congratulations and welcome, Jingsong!
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > >
> > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > (On behalf of the Apache Flink PMC)
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Best,
> > > > Benchao Li
> > > >
> > >
> >
>

[jira] [Created] (FLINK-28019) State staled errorRetractableTopNFunction

[lincoln lee (Jira)]

lincoln lee created FLINK-28019:
---

 Summary: State staled errorRetractableTopNFunction
 Key: FLINK-28019
 URL: https://issues.apache.org/jira/browse/FLINK-28019
 Project: Flink
  Issue Type: Bug
Reporter: lincoln lee






--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Re: [ANNOUNCE] New Apache Flink PMC Member - Jingsong Lee

[Paul Lam]

Congrats, Jingsong! Well deserved!

Best,
Paul Lam

> 2022年6月13日 16:31，Lincoln Lee  写道：
> 
> Congratulations, Jingsong!
> 
> Best,
> Lincoln Lee
> 
> 
> Jark Wu  于2022年6月13日周一 16:29写道：
> 
>> Congrats, Jingsong!
>> 
>> Cheers,
>> Jark
>> 
>> On Mon, 13 Jun 2022 at 16:16, Jiangang Liu 
>> wrote:
>> 
>>> Congratulations, Jingsong!
>>> 
>>> Best,
>>> Jiangang Liu
>>> 
>>> Martijn Visser  于2022年6月13日周一 16:06写道：
>>> 
 Like everyone has mentioned, this is very well deserved.
>> Congratulations!
 
 Op ma 13 jun. 2022 om 09:57 schreef Benchao Li :
 
> Congratulations, Jingsong!  Well deserved.
> 
> Rui Fan <1996fan...@gmail.com> 于2022年6月13日周一 15:53写道：
> 
>> Congratulations, Jingsong!
>> 
>> Best,
>> Rui Fan
>> 
>> On Mon, Jun 13, 2022 at 3:40 PM LuNing Wang >> 
> wrote:
>> 
>>> Congratulations, Jingsong!
>>> 
>>> Best,
>>> LuNing Wang
>>> 
>>> Ingo Bürk  于2022年6月13日周一 15:36写道：
>>> 
 Congrats, Jingsong!
 
 On 13.06.22 08:58, Becket Qin wrote:
> Hi all,
> 
> I'm very happy to announce that Jingsong Lee has joined the
>>> Flink
>> PMC!
> 
> Jingsong became a Flink committer in Feb 2020 and has been
>> continuously
> contributing to the project since then, mainly in Flink SQL.
>> He
 has
>>> been
> quite active in the mailing list, fixing bugs, helping
>>> verifying
 releases,
> reviewing patches and FLIPs. Jingsong is also devoted to
>>> pushing
>> Flink
 SQL
> to new use cases. He spent a lot of time in implementing the
 Flink
> connectors for Apache Iceberg. Jingsong is also the primary
 driver
>>> behind
> the effort of flink-table-store, which aims to provide a
> stream-batch
> unified storage for Flink dynamic tables.
> 
> Congratulations and welcome, Jingsong!
> 
> Cheers,
> 
> Jiangjie (Becket) Qin
> (On behalf of the Apache Flink PMC)
> 
 
>>> 
>> 
> 
> 
> --
> 
> Best,
> Benchao Li
> 
 
>>> 
>> 

[jira] [Created] (FLINK-28018) the start index to create empty splits in BinaryInputFormat#createInputSplits is inappropriate

[zl (Jira)]

zl created FLINK-28018:
--

 Summary: the start index to create empty splits in 
BinaryInputFormat#createInputSplits is inappropriate
 Key: FLINK-28018
 URL: https://issues.apache.org/jira/browse/FLINK-28018
 Project: Flink
  Issue Type: Bug
  Components: API / Core
Reporter: zl


when the number of created split is smaller than the minimum desired number of 
file splits, 
[BinaryInputFormat.java#L150|https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/api/common/io/BinaryInputFormat.java#L150]
 use `{_}*files.size()*{_}` as the start index to create empty splits. That is 
inappropriate, the start index should be `{_}*inputSplits.size()*{_}`.  



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Re: [DISCUSS] FLIP-222: Support full query lifecycle statements in SQL client

[Paul Lam]

Hi team,

I think we’ve reached consensus on the FLIP, thus I’m starting a vote thread.

Thank you all for the your advice in the discussion!

Best,
Paul Lam

> 2022年6月10日 02:26，Jing Ge  写道：
> 
> Hi Paul,
> 
> Fired a ticket: https://issues.apache.org/jira/browse/FLINK-27977 
>  for savepoints 
> housekeeping.
> 
> Best regards,
> Jing
> 
> On Thu, Jun 9, 2022 at 10:37 AM Martijn Visser  > wrote:
> Hi Paul,
> 
> That's a fair point, but I still think we should not offer that capability 
> via the CLI either. But that's a different discussion :)
> 
> Thanks,
> 
> Martijn
> 
> Op do 9 jun. 2022 om 10:08 schreef Paul Lam  >:
> Hi Martijn,
> 
> I think the `DROP SAVEPOINT` statement would not conflict with NO_CLAIM mode, 
> since the statement is triggered by users instead of Flink runtime.
> 
> We’re simply providing a tool for user to cleanup the savepoints, just like 
> `bin/flink savepoint -d :savepointPath` in Flink CLI [1].
> 
> [1] 
> https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/ops/state/savepoints/#disposing-savepoints
>  
> 
> 
> Best,
> Paul Lam
> 
>> 2022年6月9日 15:41，Martijn Visser > > 写道：
>> 
>> Hi all,
>> 
>> I would not include a DROP SAVEPOINT syntax. With the recently introduced 
>> CLAIM/NO CLAIM mode, I would argue that we've just clarified snapshot 
>> ownership and if you have a savepoint established "with NO_CLAIM it creates 
>> its own copy and leaves the existing one up to the user." [1] We shouldn't 
>> then again make it fuzzy by making it possible that Flink can remove 
>> snapshots. 
>> 
>> Best regards,
>> 
>> Martijn
>> 
>> [1] 
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-193%3A+Snapshots+ownership
>>  
>> 
>> Op do 9 jun. 2022 om 09:27 schreef Paul Lam > >:
>> Hi team,
>> 
>> It's great to see our opinions are finally converging!
>> 
>>> `STOP JOB  [WITH SAVEPOINT] [WITH DRAIN] `
>> 
>> 
>> LGTM. Adding it to the FLIP.
>> 
>> To Jark,
>> 
>>> We can simplify the statement to "CREATE SAVEPOINT FOR JOB ”
>> 
>> Good point. The default savepoint dir should be enough for most cases.
>> 
>> To Jing,
>> 
>>> DROP SAVEPOINT ALL
>> 
>> I think it’s valid to have such a statement, but I have two concerns:
>> `ALL` is already an SQL keyword, thus it may cause ambiguity.
>> Flink CLI and REST API doesn’t provided the corresponding functionalities, 
>> and we’d better keep them aligned.
>> How about making this statement as follow-up tasks which should touch REST 
>> API and Flink CLI?
>> 
>> Best,
>> Paul Lam
>> 
>>> 2022年6月9日 11:53，godfrey he >> > 写道：
>>> 
>>> Hi all,
>>> 
>>> Regarding `PIPELINE`, it comes from flink-core module, see
>>> `PipelineOptions` class for more details.
>>> `JOBS` is a more generic concept than `PIPELINES`. I'm also be fine with 
>>> `JOBS`.
>>> 
>>> +1 to discuss JOBTREE in other FLIP.
>>> 
>>> +1 to `STOP JOB  [WITH SAVEPOINT] [WITH DRAIN] `
>>> 
>>> +1 to `CREATE SAVEPOINT FOR JOB ` and `DROP SAVEPOINT 
>>> `
>>> 
>>> Best,
>>> Godfrey
>>> 
>>> Jing Ge mailto:j...@ververica.com>> 于2022年6月9日周四 
>>> 01:48写道：
 
 Hi Paul, Hi Jark,
 
 Re JOBTREE, agree that it is out of the scope of this FLIP
 
 Re `RELEASE SAVEPOINT ALL', if the community prefers 'DROP' then 'DROP 
 SAVEPOINT ALL' housekeeping. WDYT?
 
 Best regards,
 Jing
 
 
 On Wed, Jun 8, 2022 at 2:54 PM Jark Wu >>> > wrote:
> 
> Hi Jing,
> 
> Regarding JOBTREE (job lineage), I agree with Paul that this is out of 
> the scope
> of this FLIP and can be discussed in another FLIP.
> 
> Job lineage is a big topic that may involve many problems:
> 1) how to collect and report job entities, attributes, and lineages?
> 2) how to integrate with data catalogs, e.g. Apache Atlas, DataHub?
> 3) how does Flink SQL CLI/Gateway know the lineage information and show 
> jobtree?
> 4) ...
> 
> Best,
> Jark
> 
> On Wed, 8 Jun 2022 at 20:44, Jark Wu  > wrote:
>> 
>> Hi Paul,
>> 
>> I'm fine with using JOBS. The only concern is that this may conflict 
>> with displaying more detailed
>> information for query (e.g. query content, plan) in the future, e.g. 
>> SHOW QUERIES EXTENDED in ksqldb[1].
>> This is not a big problem as we can introduce SHOW QUERIES in the future 
>> if necessary.
>> 
>>> STOP JOBS  (with options `table.job.stop-with-savepoint` and 
>>> `table.job.stop-with-drain`)
>> What about STOP JOB  [WITH SAVEPOINT] [WITH DRAIN] ?
>> It might be trivial and error-prone 

Re:Re: Re: [DISCUSS] FLIP-218: Support SELECT clause in CREATE TABLE(CTAS)

[Mang Zhang]

Hi Yun,
Thanks for your reply!
Through offline communication with Dalong, I updated the JobStatusHook part to 
FLIP, looking forward to your feedback.



--

Best regards,
Mang Zhang





At 2022-05-31 14:34:25, "Yun Gao"  wrote:
>Hi, 
>
>Regarding the drop operation, with some offline discussion with Dalong and Zhu,
>we think that listening in the client side might be problematic since it would 
>exit
>after submitting the jobs in detached mode, thus the operation might need to
>be in the JobMaster side. 
>
>For the listener interface, currently JobListener only resides in the client 
>side
>and contains unsuitable methods like onJobSubmitted for this scenario, and 
>the internal JobStatusListener is designed to be used inside JM and is not 
>serializable, thus we tend to add a new interface JobStatusHook, 
>which could be attached to the JobGraph and executed in the JobMaster. 
>The interface will also be marked as Internal. 
>
>Best,
>Yun
>
>
>--
>From:Mang Zhang 
>Send Time:2022 May 25 (Wed.) 10:24
>To:dev 
>Subject:Re:Re: [DISCUSS] FLIP-218: Support SELECT clause in CREATE TABLE(CTAS)
>
>Hi, Martijn
>Thanks for your reply!
>I looked at the SQL standard, CTAS is part of the SQL standard.
>Feature T172 is "AS subquery clause in table definition".
>
>
>
>--
>
>Best regards,
>Mang Zhang
>
>
>
>
>
>At 2022-05-04 21:49:00, "Martijn Visser"  wrote:
>>Hi everyone,
>>
>>Can we identify if this proposed syntax is part of the SQL standard?
>>
>>Best regards,
>>
>>Martijn Visser
>>https://twitter.com/MartijnVisser82
>>https://github.com/MartijnVisser
>>
>>
>>On Fri, 29 Apr 2022 at 11:19, yuxia  wrote:
>>
>>> Thanks for for driving this work, it's to be a useful feature.
>>> About the flip-218, I have some questions.
>>>
>>> 1: Does our CTAS syntax support specify target table's schema including
>>> column name and data type? I think it maybe a useful fature in case we want
>>> to change the data types in target table instead of always copy the source
>>> table's schema. It'll be more flexible with this feature.
>>> Btw, MySQL's "CREATE TABLE ... SELECT Statement"[1] support this feature.
>>>
>>> 2: Seems it'll requre sink to implement an public interface to drop table,
>>> so what's the interface will look like?
>>>
>>> [1] https://dev.mysql.com/doc/refman/8.0/en/create-table-select.html
>>>
>>> Best regards,
>>> Yuxia
>>>
>>> - 原始邮件 -
>>> 发件人: "Mang Zhang" 
>>> 收件人: "dev" 
>>> 发送时间: 星期四, 2022年 4 月 28日 下午 4:57:24
>>> 主题: [DISCUSS] FLIP-218: Support SELECT clause in CREATE TABLE(CTAS)
>>>
>>> Hi, everyone
>>>
>>>
>>> I would like to open a discussion for support select clause in CREATE
>>> TABLE(CTAS),
>>> With the development of business and the enhancement of flink sql
>>> capabilities, queries become more and more complex.
>>> Now the user needs to use the Create Table statement to create the target
>>> table first, and then execute the insert statement.
>>> However, the target table may have many columns, which will bring a lot of
>>> work outside the business logic to the user.
>>> At the same time, ensure that the schema of the created target table is
>>> consistent with the schema of the query result.
>>> Using a CTAS syntax like Hive/Spark can greatly facilitate the user.
>>>
>>>
>>>
>>> You can find more details in FLIP-218[1]. Looking forward to your feedback.
>>>
>>>
>>>
>>> [1]
>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-218%3A+Support+SELECT+clause+in+CREATE+TABLE(CTAS)
>>>
>>>
>>>
>>>
>>> --
>>>
>>> Best regards,
>>> Mang Zhang
>>>
>

Re: [ANNOUNCE] New Apache Flink PMC Member - Jingsong Lee

[Shuo Cheng]

Congratulations, Jingsong!

On 6/13/22, Paul Lam  wrote:
> Congrats, Jingsong! Well deserved!
>
> Best,
> Paul Lam
>
>> 2022年6月13日 16:31，Lincoln Lee  写道：
>>
>> Congratulations, Jingsong!
>>
>> Best,
>> Lincoln Lee
>>
>>
>> Jark Wu  于2022年6月13日周一 16:29写道：
>>
>>> Congrats, Jingsong!
>>>
>>> Cheers,
>>> Jark
>>>
>>> On Mon, 13 Jun 2022 at 16:16, Jiangang Liu 
>>> wrote:
>>>
 Congratulations, Jingsong!

 Best,
 Jiangang Liu

 Martijn Visser  于2022年6月13日周一 16:06写道：

> Like everyone has mentioned, this is very well deserved.
>>> Congratulations!
>
> Op ma 13 jun. 2022 om 09:57 schreef Benchao Li :
>
>> Congratulations, Jingsong!  Well deserved.
>>
>> Rui Fan <1996fan...@gmail.com> 于2022年6月13日周一 15:53写道：
>>
>>> Congratulations, Jingsong!
>>>
>>> Best,
>>> Rui Fan
>>>
>>> On Mon, Jun 13, 2022 at 3:40 PM LuNing Wang >>>
>> wrote:
>>>
 Congratulations, Jingsong!

 Best,
 LuNing Wang

 Ingo Bürk  于2022年6月13日周一 15:36写道：

> Congrats, Jingsong!
>
> On 13.06.22 08:58, Becket Qin wrote:
>> Hi all,
>>
>> I'm very happy to announce that Jingsong Lee has joined the
 Flink
>>> PMC!
>>
>> Jingsong became a Flink committer in Feb 2020 and has been
>>> continuously
>> contributing to the project since then, mainly in Flink SQL.
>>> He
> has
 been
>> quite active in the mailing list, fixing bugs, helping
 verifying
> releases,
>> reviewing patches and FLIPs. Jingsong is also devoted to
 pushing
>>> Flink
> SQL
>> to new use cases. He spent a lot of time in implementing the
> Flink
>> connectors for Apache Iceberg. Jingsong is also the primary
> driver
 behind
>> the effort of flink-table-store, which aims to provide a
>> stream-batch
>> unified storage for Flink dynamic tables.
>>
>> Congratulations and welcome, Jingsong!
>>
>> Cheers,
>>
>> Jiangjie (Becket) Qin
>> (On behalf of the Apache Flink PMC)
>>
>

>>>
>>
>>
>> --
>>
>> Best,
>> Benchao Li
>>
>

>>>
>
>

[jira] [Created] (FLINK-28020) Refactor Flink connector for table store with FileStoreTable

[Caizhi Weng (Jira)]

Caizhi Weng created FLINK-28020:
---

 Summary: Refactor Flink connector for table store with 
FileStoreTable
 Key: FLINK-28020
 URL: https://issues.apache.org/jira/browse/FLINK-28020
 Project: Flink
  Issue Type: Sub-task
  Components: Table Store
Affects Versions: table-store-0.2.0
Reporter: Caizhi Weng


We've prepared {{FileStoreTable}} for {{RowData}} reading and writing, so it's 
time to refactor Flink connector for table store with {{FileStoreTable}}.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[VOTE] FLIP-222: Support full job lifecycle statements in SQL client

[Paul Lam]

Hi team,

I'd like to start a vote for FLIP-222: Support full job lifecycle statements in 
SQL client [1], 
which is discussed on the thread [2]. 

The vote will be open for at least 72 hours unless there is an objection or not 
enough votes.

Thanks everyone who has participated in the discussion!

[1] 
https://cwiki.apache.org/confluence/display/FLINK/FLIP-222%3A+Support+full+job+lifecycle+statements+in+SQL+client
[2] https://lists.apache.org/thread/qkvh9p5w9b12s7ykh3l7lv7m9dbgnf1g

Best,
Paul Lam

Re: [VOTE] FLIP-222: Support full job lifecycle statements in SQL client

[Jing Ge]

+1 (not binding)

Thanks Paul for your effort!

Best regards,
Jing

On Mon, Jun 13, 2022 at 11:11 AM Paul Lam  wrote:

> Hi team,
>
> I'd like to start a vote for FLIP-222: Support full job lifecycle
> statements in SQL client [1],
> which is discussed on the thread [2].
>
> The vote will be open for at least 72 hours unless there is an objection
> or not enough votes.
>
> Thanks everyone who has participated in the discussion!
>
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-222%3A+Support+full+job+lifecycle+statements+in+SQL+client
> [2] https://lists.apache.org/thread/qkvh9p5w9b12s7ykh3l7lv7m9dbgnf1g
>
> Best,
> Paul Lam
>
>

Re: [ANNOUNCE] New Apache Flink PMC Member - Jingsong Lee

[godfrey he]

Congratulations, Jingsong!

Best,
Godfrey

Shuo Cheng  于2022年6月13日周一 16:43写道：
>
> Congratulations, Jingsong!
>
> On 6/13/22, Paul Lam  wrote:
> > Congrats, Jingsong! Well deserved!
> >
> > Best,
> > Paul Lam
> >
> >> 2022年6月13日 16:31，Lincoln Lee  写道：
> >>
> >> Congratulations, Jingsong!
> >>
> >> Best,
> >> Lincoln Lee
> >>
> >>
> >> Jark Wu  于2022年6月13日周一 16:29写道：
> >>
> >>> Congrats, Jingsong!
> >>>
> >>> Cheers,
> >>> Jark
> >>>
> >>> On Mon, 13 Jun 2022 at 16:16, Jiangang Liu 
> >>> wrote:
> >>>
>  Congratulations, Jingsong!
> 
>  Best,
>  Jiangang Liu
> 
>  Martijn Visser  于2022年6月13日周一 16:06写道：
> 
> > Like everyone has mentioned, this is very well deserved.
> >>> Congratulations!
> >
> > Op ma 13 jun. 2022 om 09:57 schreef Benchao Li :
> >
> >> Congratulations, Jingsong!  Well deserved.
> >>
> >> Rui Fan <1996fan...@gmail.com> 于2022年6月13日周一 15:53写道：
> >>
> >>> Congratulations, Jingsong!
> >>>
> >>> Best,
> >>> Rui Fan
> >>>
> >>> On Mon, Jun 13, 2022 at 3:40 PM LuNing Wang  
> >> wrote:
> >>>
>  Congratulations, Jingsong!
> 
>  Best,
>  LuNing Wang
> 
>  Ingo Bürk  于2022年6月13日周一 15:36写道：
> 
> > Congrats, Jingsong!
> >
> > On 13.06.22 08:58, Becket Qin wrote:
> >> Hi all,
> >>
> >> I'm very happy to announce that Jingsong Lee has joined the
>  Flink
> >>> PMC!
> >>
> >> Jingsong became a Flink committer in Feb 2020 and has been
> >>> continuously
> >> contributing to the project since then, mainly in Flink SQL.
> >>> He
> > has
>  been
> >> quite active in the mailing list, fixing bugs, helping
>  verifying
> > releases,
> >> reviewing patches and FLIPs. Jingsong is also devoted to
>  pushing
> >>> Flink
> > SQL
> >> to new use cases. He spent a lot of time in implementing the
> > Flink
> >> connectors for Apache Iceberg. Jingsong is also the primary
> > driver
>  behind
> >> the effort of flink-table-store, which aims to provide a
> >> stream-batch
> >> unified storage for Flink dynamic tables.
> >>
> >> Congratulations and welcome, Jingsong!
> >>
> >> Cheers,
> >>
> >> Jiangjie (Becket) Qin
> >> (On behalf of the Apache Flink PMC)
> >>
> >
> 
> >>>
> >>
> >>
> >> --
> >>
> >> Best,
> >> Benchao Li
> >>
> >
> 
> >>>
> >
> >

Re: [ANNOUNCE] New Apache Flink PMC Member - Jingsong Lee

[Jane Chan]

Congratulations, Jingsong!

Best,
Jane Chan

On Mon, Jun 13, 2022 at 4:43 PM Shuo Cheng  wrote:

> Congratulations, Jingsong!
>
> On 6/13/22, Paul Lam  wrote:
> > Congrats, Jingsong! Well deserved!
> >
> > Best,
> > Paul Lam
> >
> >> 2022年6月13日 16:31，Lincoln Lee  写道：
> >>
> >> Congratulations, Jingsong!
> >>
> >> Best,
> >> Lincoln Lee
> >>
> >>
> >> Jark Wu  于2022年6月13日周一 16:29写道：
> >>
> >>> Congrats, Jingsong!
> >>>
> >>> Cheers,
> >>> Jark
> >>>
> >>> On Mon, 13 Jun 2022 at 16:16, Jiangang Liu 
> >>> wrote:
> >>>
>  Congratulations, Jingsong!
> 
>  Best,
>  Jiangang Liu
> 
>  Martijn Visser  于2022年6月13日周一 16:06写道：
> 
> > Like everyone has mentioned, this is very well deserved.
> >>> Congratulations!
> >
> > Op ma 13 jun. 2022 om 09:57 schreef Benchao Li  >:
> >
> >> Congratulations, Jingsong!  Well deserved.
> >>
> >> Rui Fan <1996fan...@gmail.com> 于2022年6月13日周一 15:53写道：
> >>
> >>> Congratulations, Jingsong!
> >>>
> >>> Best,
> >>> Rui Fan
> >>>
> >>> On Mon, Jun 13, 2022 at 3:40 PM LuNing Wang  
> >> wrote:
> >>>
>  Congratulations, Jingsong!
> 
>  Best,
>  LuNing Wang
> 
>  Ingo Bürk  于2022年6月13日周一 15:36写道：
> 
> > Congrats, Jingsong!
> >
> > On 13.06.22 08:58, Becket Qin wrote:
> >> Hi all,
> >>
> >> I'm very happy to announce that Jingsong Lee has joined the
>  Flink
> >>> PMC!
> >>
> >> Jingsong became a Flink committer in Feb 2020 and has been
> >>> continuously
> >> contributing to the project since then, mainly in Flink SQL.
> >>> He
> > has
>  been
> >> quite active in the mailing list, fixing bugs, helping
>  verifying
> > releases,
> >> reviewing patches and FLIPs. Jingsong is also devoted to
>  pushing
> >>> Flink
> > SQL
> >> to new use cases. He spent a lot of time in implementing the
> > Flink
> >> connectors for Apache Iceberg. Jingsong is also the primary
> > driver
>  behind
> >> the effort of flink-table-store, which aims to provide a
> >> stream-batch
> >> unified storage for Flink dynamic tables.
> >>
> >> Congratulations and welcome, Jingsong!
> >>
> >> Cheers,
> >>
> >> Jiangjie (Becket) Qin
> >> (On behalf of the Apache Flink PMC)
> >>
> >
> 
> >>>
> >>
> >>
> >> --
> >>
> >> Best,
> >> Benchao Li
> >>
> >
> 
> >>>
> >
> >
>

[jira] [Created] (FLINK-28021) Add FLIP-33 metrics to FileSystem connector

[Martijn Visser (Jira)]

Martijn Visser created FLINK-28021:
--

 Summary: Add FLIP-33 metrics to FileSystem connector
 Key: FLINK-28021
 URL: https://issues.apache.org/jira/browse/FLINK-28021
 Project: Flink
  Issue Type: Improvement
  Components: Connectors / FileSystem
Reporter: Martijn Visser


Both the current FileSource and FileSink have no metrics implemented. They 
should have the FLIP-33 metrics implemented. 



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Re: [ANNOUNCE] New Apache Flink PMC Member - Jingsong Lee

[Xingbo Huang]

Congratulations, Jingsong!

Best,
Xingbo

Jane Chan  于2022年6月13日周一 17:23写道：

> Congratulations, Jingsong!
>
> Best,
> Jane Chan
>
> On Mon, Jun 13, 2022 at 4:43 PM Shuo Cheng  wrote:
>
> > Congratulations, Jingsong!
> >
> > On 6/13/22, Paul Lam  wrote:
> > > Congrats, Jingsong! Well deserved!
> > >
> > > Best,
> > > Paul Lam
> > >
> > >> 2022年6月13日 16:31，Lincoln Lee  写道：
> > >>
> > >> Congratulations, Jingsong!
> > >>
> > >> Best,
> > >> Lincoln Lee
> > >>
> > >>
> > >> Jark Wu  于2022年6月13日周一 16:29写道：
> > >>
> > >>> Congrats, Jingsong!
> > >>>
> > >>> Cheers,
> > >>> Jark
> > >>>
> > >>> On Mon, 13 Jun 2022 at 16:16, Jiangang Liu <
> liujiangangp...@gmail.com>
> > >>> wrote:
> > >>>
> >  Congratulations, Jingsong!
> > 
> >  Best,
> >  Jiangang Liu
> > 
> >  Martijn Visser  于2022年6月13日周一 16:06写道：
> > 
> > > Like everyone has mentioned, this is very well deserved.
> > >>> Congratulations!
> > >
> > > Op ma 13 jun. 2022 om 09:57 schreef Benchao Li <
> libenc...@apache.org
> > >:
> > >
> > >> Congratulations, Jingsong!  Well deserved.
> > >>
> > >> Rui Fan <1996fan...@gmail.com> 于2022年6月13日周一 15:53写道：
> > >>
> > >>> Congratulations, Jingsong!
> > >>>
> > >>> Best,
> > >>> Rui Fan
> > >>>
> > >>> On Mon, Jun 13, 2022 at 3:40 PM LuNing Wang <
> wang4lun...@gmail.com
> > 
> > >> wrote:
> > >>>
> >  Congratulations, Jingsong!
> > 
> >  Best,
> >  LuNing Wang
> > 
> >  Ingo Bürk  于2022年6月13日周一 15:36写道：
> > 
> > > Congrats, Jingsong!
> > >
> > > On 13.06.22 08:58, Becket Qin wrote:
> > >> Hi all,
> > >>
> > >> I'm very happy to announce that Jingsong Lee has joined the
> >  Flink
> > >>> PMC!
> > >>
> > >> Jingsong became a Flink committer in Feb 2020 and has been
> > >>> continuously
> > >> contributing to the project since then, mainly in Flink SQL.
> > >>> He
> > > has
> >  been
> > >> quite active in the mailing list, fixing bugs, helping
> >  verifying
> > > releases,
> > >> reviewing patches and FLIPs. Jingsong is also devoted to
> >  pushing
> > >>> Flink
> > > SQL
> > >> to new use cases. He spent a lot of time in implementing the
> > > Flink
> > >> connectors for Apache Iceberg. Jingsong is also the primary
> > > driver
> >  behind
> > >> the effort of flink-table-store, which aims to provide a
> > >> stream-batch
> > >> unified storage for Flink dynamic tables.
> > >>
> > >> Congratulations and welcome, Jingsong!
> > >>
> > >> Cheers,
> > >>
> > >> Jiangjie (Becket) Qin
> > >> (On behalf of the Apache Flink PMC)
> > >>
> > >
> > 
> > >>>
> > >>
> > >>
> > >> --
> > >>
> > >> Best,
> > >> Benchao Li
> > >>
> > >
> > 
> > >>>
> > >
> > >
> >
>

Re: [VOTE] FLIP-228: Support Within between events in CEP Pattern

[Xingbo Huang]

+1 (binding)

Best,
Xingbo

Dian Fu  于2022年6月13日周一 16:08写道：

> +1 (binding)
>
> Regards,
> Dian
>
> On Mon, Jun 13, 2022 at 3:55 PM Rui Fan <1996fan...@gmail.com> wrote:
>
> > Thanks for Nicholas driving this.
> >
> > +1(non-binding)
> >
> > Best,
> > Rui Fan
> >
> > On Mon, Jun 13, 2022 at 3:05 PM md peng  wrote:
> >
> > > +1(not-binding)
> > >
> > > Best,
> > > Mingde Peng
> > >
> > > Martijn Visser  于2022年6月13日周一 14:48写道：
> > >
> > > > +1 (binding)
> > > >
> > > > Looking forward :)
> > > >
> > > > Thanks, Martijn
> > > >
> > > > Op ma 13 jun. 2022 om 04:16 schreef Nicholas :
> > > >
> > > > > Hi everyone，
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Thanks for feedback for FLIP-228: Support Within between events in
> > CEP
> > > > > Pattern[1] on the discussion thread[2]. I'd like to start a VOTE
> > thread
> > > > for
> > > > > FLIP-228.
> > > > >
> > > > > The vote will be open for at least 72 hours unless there is an
> > > objection
> > > > > or not enough votes.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Regards,
> > > > >
> > > > > Nicholas Jiang
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > [1]
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-228%3A+Support+Within+between+events+in+CEP+Pattern
> > > > >
> > > > > [2]
> https://lists.apache.org/thread/p60ctx213f8921rgklk5f0b6xfrs8ksz
> > > >
> > >
> >
>

[DISCUSS ] Make state.backend.incremental as true by default

[Lihe Ma]

Hi, Everyone,

I would like to open a discussion on setting incremental checkpoint as default 
behavior.

Currently, the configuration of state.backend.incremental is set as false by 
default. Incremental checkpoint has been adopted widely in industry community 
for many years , and it is also well-tested from the feedback in the community 
discussion. Incremental checkpointing is more light-weighted: shorter 
checkpoint duration, less uploaded data and less resource consumption.

In terms of backward compatibility, enable incremental checkpointing would not 
make any data loss no matter restoring from a full checkpoint/savepoint or an 
incremental checkpoint.

FLIP-193 (Snapshot ownership)[1] has been released in 1.15, incremental 
checkpoint no longer depends on a previous restored checkpoint in default 
NO_CLAIM mode, which makes the checkpoint lineage much cleaner, it is a good 
chance to change the configuration state.backend.incremental to true as default.

Thus, based on the above discussion, I suggest to make 
state.backend.incremental as true by default. What do you think of this 
proposal?

[1] 
https://cwiki.apache.org/confluence/display/FLINK/FLIP-193%3A+Snapshots+ownership

Best regards,
Lihe Ma

Re: [DISCUSS ] Make state.backend.incremental as true by default

[Martijn Visser]

Hi Lihe,

What happens if we enable incremental checkpoints by default while the used
memory backend is HashMapStateBackend, which doesn't support incremental
checkpoints?

Best regards,

Martijn

Op ma 13 jun. 2022 om 11:59 schreef Lihe Ma :

> Hi, Everyone,
>
> I would like to open a discussion on setting incremental checkpoint as
> default behavior.
>
> Currently, the configuration of state.backend.incremental is set as false
> by default. Incremental checkpoint has been adopted widely in industry
> community for many years , and it is also well-tested from the feedback in
> the community discussion. Incremental checkpointing is more light-weighted:
> shorter checkpoint duration, less uploaded data and less resource
> consumption.
>
> In terms of backward compatibility, enable incremental checkpointing would
> not make any data loss no matter restoring from a full checkpoint/savepoint
> or an incremental checkpoint.
>
> FLIP-193 (Snapshot ownership)[1] has been released in 1.15, incremental
> checkpoint no longer depends on a previous restored checkpoint in default
> NO_CLAIM mode, which makes the checkpoint lineage much cleaner, it is a
> good chance to change the configuration state.backend.incremental to true
> as default.
>
> Thus, based on the above discussion, I suggest to make
> state.backend.incremental as true by default. What do you think of this
> proposal?
>
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-193%3A+Snapshots+ownership
>
> Best regards,
> Lihe Ma
>

Re: [ANNOUNCE] New Apache Flink PMC Member - Jingsong Lee

[Yun Tang]

Congratulations, Jingsong! Well deserved.

Best
Yun Tang

From: Xingbo Huang 
Sent: Monday, June 13, 2022 17:39
To: dev 
Subject: Re: [ANNOUNCE] New Apache Flink PMC Member - Jingsong Lee

Congratulations, Jingsong!

Best,
Xingbo

Jane Chan  于2022年6月13日周一 17:23写道：

> Congratulations, Jingsong!
>
> Best,
> Jane Chan
>
> On Mon, Jun 13, 2022 at 4:43 PM Shuo Cheng  wrote:
>
> > Congratulations, Jingsong!
> >
> > On 6/13/22, Paul Lam  wrote:
> > > Congrats, Jingsong! Well deserved!
> > >
> > > Best,
> > > Paul Lam
> > >
> > >> 2022年6月13日 16:31，Lincoln Lee  写道：
> > >>
> > >> Congratulations, Jingsong!
> > >>
> > >> Best,
> > >> Lincoln Lee
> > >>
> > >>
> > >> Jark Wu  于2022年6月13日周一 16:29写道：
> > >>
> > >>> Congrats, Jingsong!
> > >>>
> > >>> Cheers,
> > >>> Jark
> > >>>
> > >>> On Mon, 13 Jun 2022 at 16:16, Jiangang Liu <
> liujiangangp...@gmail.com>
> > >>> wrote:
> > >>>
> >  Congratulations, Jingsong!
> > 
> >  Best,
> >  Jiangang Liu
> > 
> >  Martijn Visser  于2022年6月13日周一 16:06写道：
> > 
> > > Like everyone has mentioned, this is very well deserved.
> > >>> Congratulations!
> > >
> > > Op ma 13 jun. 2022 om 09:57 schreef Benchao Li <
> libenc...@apache.org
> > >:
> > >
> > >> Congratulations, Jingsong!  Well deserved.
> > >>
> > >> Rui Fan <1996fan...@gmail.com> 于2022年6月13日周一 15:53写道：
> > >>
> > >>> Congratulations, Jingsong!
> > >>>
> > >>> Best,
> > >>> Rui Fan
> > >>>
> > >>> On Mon, Jun 13, 2022 at 3:40 PM LuNing Wang <
> wang4lun...@gmail.com
> > 
> > >> wrote:
> > >>>
> >  Congratulations, Jingsong!
> > 
> >  Best,
> >  LuNing Wang
> > 
> >  Ingo Bürk  于2022年6月13日周一 15:36写道：
> > 
> > > Congrats, Jingsong!
> > >
> > > On 13.06.22 08:58, Becket Qin wrote:
> > >> Hi all,
> > >>
> > >> I'm very happy to announce that Jingsong Lee has joined the
> >  Flink
> > >>> PMC!
> > >>
> > >> Jingsong became a Flink committer in Feb 2020 and has been
> > >>> continuously
> > >> contributing to the project since then, mainly in Flink SQL.
> > >>> He
> > > has
> >  been
> > >> quite active in the mailing list, fixing bugs, helping
> >  verifying
> > > releases,
> > >> reviewing patches and FLIPs. Jingsong is also devoted to
> >  pushing
> > >>> Flink
> > > SQL
> > >> to new use cases. He spent a lot of time in implementing the
> > > Flink
> > >> connectors for Apache Iceberg. Jingsong is also the primary
> > > driver
> >  behind
> > >> the effort of flink-table-store, which aims to provide a
> > >> stream-batch
> > >> unified storage for Flink dynamic tables.
> > >>
> > >> Congratulations and welcome, Jingsong!
> > >>
> > >> Cheers,
> > >>
> > >> Jiangjie (Becket) Qin
> > >> (On behalf of the Apache Flink PMC)
> > >>
> > >
> > 
> > >>>
> > >>
> > >>
> > >> --
> > >>
> > >> Best,
> > >> Benchao Li
> > >>
> > >
> > 
> > >>>
> > >
> > >
> >
>

Re: [DISCUSS ] Make state.backend.incremental as true by default

[Yun Tang]

Strongly +1 for making incremental checkpoints as default. Many users have ever 
been asking why this configuration is not enabled by default.

BTW, from my knowledge, nothing would happen for HashMapStateBackend, which 
does not support incremental checkpoint yet, when enabling incremental 
checkpoints.


Best
Yun Tang

From: Martijn Visser 
Sent: Monday, June 13, 2022 18:05
To: dev@flink.apache.org 
Subject: Re: [DISCUSS ] Make state.backend.incremental as true by default

Hi Lihe,

What happens if we enable incremental checkpoints by default while the used
memory backend is HashMapStateBackend, which doesn't support incremental
checkpoints?

Best regards,

Martijn

Op ma 13 jun. 2022 om 11:59 schreef Lihe Ma :

> Hi, Everyone,
>
> I would like to open a discussion on setting incremental checkpoint as
> default behavior.
>
> Currently, the configuration of state.backend.incremental is set as false
> by default. Incremental checkpoint has been adopted widely in industry
> community for many years , and it is also well-tested from the feedback in
> the community discussion. Incremental checkpointing is more light-weighted:
> shorter checkpoint duration, less uploaded data and less resource
> consumption.
>
> In terms of backward compatibility, enable incremental checkpointing would
> not make any data loss no matter restoring from a full checkpoint/savepoint
> or an incremental checkpoint.
>
> FLIP-193 (Snapshot ownership)[1] has been released in 1.15, incremental
> checkpoint no longer depends on a previous restored checkpoint in default
> NO_CLAIM mode, which makes the checkpoint lineage much cleaner, it is a
> good chance to change the configuration state.backend.incremental to true
> as default.
>
> Thus, based on the above discussion, I suggest to make
> state.backend.incremental as true by default. What do you think of this
> proposal?
>
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-193%3A+Snapshots+ownership
>
> Best regards,
> Lihe Ma
>

Re: [DISCUSS ] Make state.backend.incremental as true by default

[Jing Ge]

+1

Glad to see the kickoff of this discussion. Thanks Lihe for driving this!

We have actually already discussed it internally a few months ago. After
considering some corner cases, all agreed on enabling the incremental
checkpoint as default.

Best regards,
Jing

On Mon, Jun 13, 2022 at 12:17 PM Yun Tang  wrote:

> Strongly +1 for making incremental checkpoints as default. Many users have
> ever been asking why this configuration is not enabled by default.
>
> BTW, from my knowledge, nothing would happen for HashMapStateBackend,
> which does not support incremental checkpoint yet, when enabling
> incremental checkpoints.
>
>
> Best
> Yun Tang
> 
> From: Martijn Visser 
> Sent: Monday, June 13, 2022 18:05
> To: dev@flink.apache.org 
> Subject: Re: [DISCUSS ] Make state.backend.incremental as true by default
>
> Hi Lihe,
>
> What happens if we enable incremental checkpoints by default while the used
> memory backend is HashMapStateBackend, which doesn't support incremental
> checkpoints?
>
> Best regards,
>
> Martijn
>
> Op ma 13 jun. 2022 om 11:59 schreef Lihe Ma :
>
> > Hi, Everyone,
> >
> > I would like to open a discussion on setting incremental checkpoint as
> > default behavior.
> >
> > Currently, the configuration of state.backend.incremental is set as false
> > by default. Incremental checkpoint has been adopted widely in industry
> > community for many years , and it is also well-tested from the feedback
> in
> > the community discussion. Incremental checkpointing is more
> light-weighted:
> > shorter checkpoint duration, less uploaded data and less resource
> > consumption.
> >
> > In terms of backward compatibility, enable incremental checkpointing
> would
> > not make any data loss no matter restoring from a full
> checkpoint/savepoint
> > or an incremental checkpoint.
> >
> > FLIP-193 (Snapshot ownership)[1] has been released in 1.15, incremental
> > checkpoint no longer depends on a previous restored checkpoint in default
> > NO_CLAIM mode, which makes the checkpoint lineage much cleaner, it is a
> > good chance to change the configuration state.backend.incremental to true
> > as default.
> >
> > Thus, based on the above discussion, I suggest to make
> > state.backend.incremental as true by default. What do you think of this
> > proposal?
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-193%3A+Snapshots+ownership
> >
> > Best regards,
> > Lihe Ma
> >
>

Re: [ANNOUNCE] New Apache Flink PMC Member - Jingsong Lee

[yuxia]

Congratulations, Jingsong!


Best regards,
Yuxia

- 原始邮件 -
发件人: "Yun Tang" 
收件人: "dev" 
发送时间: 星期一, 2022年 6 月 13日 下午 6:12:24
主题: Re: [ANNOUNCE] New Apache Flink PMC Member - Jingsong Lee

Congratulations, Jingsong! Well deserved.

Best
Yun Tang

From: Xingbo Huang 
Sent: Monday, June 13, 2022 17:39
To: dev 
Subject: Re: [ANNOUNCE] New Apache Flink PMC Member - Jingsong Lee

Congratulations, Jingsong!

Best,
Xingbo

Jane Chan  于2022年6月13日周一 17:23写道：

> Congratulations, Jingsong!
>
> Best,
> Jane Chan
>
> On Mon, Jun 13, 2022 at 4:43 PM Shuo Cheng  wrote:
>
> > Congratulations, Jingsong!
> >
> > On 6/13/22, Paul Lam  wrote:
> > > Congrats, Jingsong! Well deserved!
> > >
> > > Best,
> > > Paul Lam
> > >
> > >> 2022年6月13日 16:31，Lincoln Lee  写道：
> > >>
> > >> Congratulations, Jingsong!
> > >>
> > >> Best,
> > >> Lincoln Lee
> > >>
> > >>
> > >> Jark Wu  于2022年6月13日周一 16:29写道：
> > >>
> > >>> Congrats, Jingsong!
> > >>>
> > >>> Cheers,
> > >>> Jark
> > >>>
> > >>> On Mon, 13 Jun 2022 at 16:16, Jiangang Liu <
> liujiangangp...@gmail.com>
> > >>> wrote:
> > >>>
> >  Congratulations, Jingsong!
> > 
> >  Best,
> >  Jiangang Liu
> > 
> >  Martijn Visser  于2022年6月13日周一 16:06写道：
> > 
> > > Like everyone has mentioned, this is very well deserved.
> > >>> Congratulations!
> > >
> > > Op ma 13 jun. 2022 om 09:57 schreef Benchao Li <
> libenc...@apache.org
> > >:
> > >
> > >> Congratulations, Jingsong!  Well deserved.
> > >>
> > >> Rui Fan <1996fan...@gmail.com> 于2022年6月13日周一 15:53写道：
> > >>
> > >>> Congratulations, Jingsong!
> > >>>
> > >>> Best,
> > >>> Rui Fan
> > >>>
> > >>> On Mon, Jun 13, 2022 at 3:40 PM LuNing Wang <
> wang4lun...@gmail.com
> > 
> > >> wrote:
> > >>>
> >  Congratulations, Jingsong!
> > 
> >  Best,
> >  LuNing Wang
> > 
> >  Ingo Bürk  于2022年6月13日周一 15:36写道：
> > 
> > > Congrats, Jingsong!
> > >
> > > On 13.06.22 08:58, Becket Qin wrote:
> > >> Hi all,
> > >>
> > >> I'm very happy to announce that Jingsong Lee has joined the
> >  Flink
> > >>> PMC!
> > >>
> > >> Jingsong became a Flink committer in Feb 2020 and has been
> > >>> continuously
> > >> contributing to the project since then, mainly in Flink SQL.
> > >>> He
> > > has
> >  been
> > >> quite active in the mailing list, fixing bugs, helping
> >  verifying
> > > releases,
> > >> reviewing patches and FLIPs. Jingsong is also devoted to
> >  pushing
> > >>> Flink
> > > SQL
> > >> to new use cases. He spent a lot of time in implementing the
> > > Flink
> > >> connectors for Apache Iceberg. Jingsong is also the primary
> > > driver
> >  behind
> > >> the effort of flink-table-store, which aims to provide a
> > >> stream-batch
> > >> unified storage for Flink dynamic tables.
> > >>
> > >> Congratulations and welcome, Jingsong!
> > >>
> > >> Cheers,
> > >>
> > >> Jiangjie (Becket) Qin
> > >> (On behalf of the Apache Flink PMC)
> > >>
> > >
> > 
> > >>>
> > >>
> > >>
> > >> --
> > >>
> > >> Best,
> > >> Benchao Li
> > >>
> > >
> > 
> > >>>
> > >
> > >
> >
>

Re: [DISCUSS ] Make state.backend.incremental as true by default

[Alexander Fedulov]

+1

>From my experience, it is actually hard to come up with use cases where
incremental checkpoints should explicitly not be enabled with the RocksDB
state backend. If the state is so small that the full snapshots do not
have any negative impact, one should consider using HashMapStateBackend
anyway.

Best,
Alexander Fedulov


On Mon, Jun 13, 2022 at 12:26 PM Jing Ge  wrote:

> +1
>
> Glad to see the kickoff of this discussion. Thanks Lihe for driving this!
>
> We have actually already discussed it internally a few months ago. After
> considering some corner cases, all agreed on enabling the incremental
> checkpoint as default.
>
> Best regards,
> Jing
>
> On Mon, Jun 13, 2022 at 12:17 PM Yun Tang  wrote:
>
> > Strongly +1 for making incremental checkpoints as default. Many users
> have
> > ever been asking why this configuration is not enabled by default.
> >
> > BTW, from my knowledge, nothing would happen for HashMapStateBackend,
> > which does not support incremental checkpoint yet, when enabling
> > incremental checkpoints.
> >
> >
> > Best
> > Yun Tang
> > 
> > From: Martijn Visser 
> > Sent: Monday, June 13, 2022 18:05
> > To: dev@flink.apache.org 
> > Subject: Re: [DISCUSS ] Make state.backend.incremental as true by default
> >
> > Hi Lihe,
> >
> > What happens if we enable incremental checkpoints by default while the
> used
> > memory backend is HashMapStateBackend, which doesn't support incremental
> > checkpoints?
> >
> > Best regards,
> >
> > Martijn
> >
> > Op ma 13 jun. 2022 om 11:59 schreef Lihe Ma :
> >
> > > Hi, Everyone,
> > >
> > > I would like to open a discussion on setting incremental checkpoint as
> > > default behavior.
> > >
> > > Currently, the configuration of state.backend.incremental is set as
> false
> > > by default. Incremental checkpoint has been adopted widely in industry
> > > community for many years , and it is also well-tested from the feedback
> > in
> > > the community discussion. Incremental checkpointing is more
> > light-weighted:
> > > shorter checkpoint duration, less uploaded data and less resource
> > > consumption.
> > >
> > > In terms of backward compatibility, enable incremental checkpointing
> > would
> > > not make any data loss no matter restoring from a full
> > checkpoint/savepoint
> > > or an incremental checkpoint.
> > >
> > > FLIP-193 (Snapshot ownership)[1] has been released in 1.15, incremental
> > > checkpoint no longer depends on a previous restored checkpoint in
> default
> > > NO_CLAIM mode, which makes the checkpoint lineage much cleaner, it is a
> > > good chance to change the configuration state.backend.incremental to
> true
> > > as default.
> > >
> > > Thus, based on the above discussion, I suggest to make
> > > state.backend.incremental as true by default. What do you think of this
> > > proposal?
> > >
> > > [1]
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-193%3A+Snapshots+ownership
> > >
> > > Best regards,
> > > Lihe Ma
> > >
> >
>

Re: [ANNOUNCE] New Apache Flink PMC Member - Jingsong Lee

[??????]

Congratulations and well deserved, Jingsong!


Best regards,
Shouwei
--  --
??: 
   "dev"

Re: [DISCUSS ] Make state.backend.incremental as true by default

[Martijn Visser]

> BTW, from my knowledge, nothing would happen for HashMapStateBackend,
which does not support incremental checkpoint yet, when enabling
incremental checkpoints.

Thanks Yun, if no errors would occur then definitely +1 to enable it by
default

Op ma 13 jun. 2022 om 12:42 schreef Alexander Fedulov <
alexan...@ververica.com>:

> +1
>
> From my experience, it is actually hard to come up with use cases where
> incremental checkpoints should explicitly not be enabled with the RocksDB
> state backend. If the state is so small that the full snapshots do not
> have any negative impact, one should consider using HashMapStateBackend
> anyway.
>
> Best,
> Alexander Fedulov
>
>
> On Mon, Jun 13, 2022 at 12:26 PM Jing Ge  wrote:
>
> > +1
> >
> > Glad to see the kickoff of this discussion. Thanks Lihe for driving this!
> >
> > We have actually already discussed it internally a few months ago. After
> > considering some corner cases, all agreed on enabling the incremental
> > checkpoint as default.
> >
> > Best regards,
> > Jing
> >
> > On Mon, Jun 13, 2022 at 12:17 PM Yun Tang  wrote:
> >
> > > Strongly +1 for making incremental checkpoints as default. Many users
> > have
> > > ever been asking why this configuration is not enabled by default.
> > >
> > > BTW, from my knowledge, nothing would happen for HashMapStateBackend,
> > > which does not support incremental checkpoint yet, when enabling
> > > incremental checkpoints.
> > >
> > >
> > > Best
> > > Yun Tang
> > > 
> > > From: Martijn Visser 
> > > Sent: Monday, June 13, 2022 18:05
> > > To: dev@flink.apache.org 
> > > Subject: Re: [DISCUSS ] Make state.backend.incremental as true by
> default
> > >
> > > Hi Lihe,
> > >
> > > What happens if we enable incremental checkpoints by default while the
> > used
> > > memory backend is HashMapStateBackend, which doesn't support
> incremental
> > > checkpoints?
> > >
> > > Best regards,
> > >
> > > Martijn
> > >
> > > Op ma 13 jun. 2022 om 11:59 schreef Lihe Ma :
> > >
> > > > Hi, Everyone,
> > > >
> > > > I would like to open a discussion on setting incremental checkpoint
> as
> > > > default behavior.
> > > >
> > > > Currently, the configuration of state.backend.incremental is set as
> > false
> > > > by default. Incremental checkpoint has been adopted widely in
> industry
> > > > community for many years , and it is also well-tested from the
> feedback
> > > in
> > > > the community discussion. Incremental checkpointing is more
> > > light-weighted:
> > > > shorter checkpoint duration, less uploaded data and less resource
> > > > consumption.
> > > >
> > > > In terms of backward compatibility, enable incremental checkpointing
> > > would
> > > > not make any data loss no matter restoring from a full
> > > checkpoint/savepoint
> > > > or an incremental checkpoint.
> > > >
> > > > FLIP-193 (Snapshot ownership)[1] has been released in 1.15,
> incremental
> > > > checkpoint no longer depends on a previous restored checkpoint in
> > default
> > > > NO_CLAIM mode, which makes the checkpoint lineage much cleaner, it
> is a
> > > > good chance to change the configuration state.backend.incremental to
> > true
> > > > as default.
> > > >
> > > > Thus, based on the above discussion, I suggest to make
> > > > state.backend.incremental as true by default. What do you think of
> this
> > > > proposal?
> > > >
> > > > [1]
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-193%3A+Snapshots+ownership
> > > >
> > > > Best regards,
> > > > Lihe Ma
> > > >
> > >
> >
>

Re: [DISCUSS ] Make state.backend.incremental as true by default

[Rui Fan]

Strongly +1

Best,
Rui Fan

On Mon, Jun 13, 2022 at 7:35 PM Martijn Visser 
wrote:

> > BTW, from my knowledge, nothing would happen for HashMapStateBackend,
> which does not support incremental checkpoint yet, when enabling
> incremental checkpoints.
>
> Thanks Yun, if no errors would occur then definitely +1 to enable it by
> default
>
> Op ma 13 jun. 2022 om 12:42 schreef Alexander Fedulov <
> alexan...@ververica.com>:
>
> > +1
> >
> > From my experience, it is actually hard to come up with use cases where
> > incremental checkpoints should explicitly not be enabled with the RocksDB
> > state backend. If the state is so small that the full snapshots do not
> > have any negative impact, one should consider using HashMapStateBackend
> > anyway.
> >
> > Best,
> > Alexander Fedulov
> >
> >
> > On Mon, Jun 13, 2022 at 12:26 PM Jing Ge  wrote:
> >
> > > +1
> > >
> > > Glad to see the kickoff of this discussion. Thanks Lihe for driving
> this!
> > >
> > > We have actually already discussed it internally a few months ago.
> After
> > > considering some corner cases, all agreed on enabling the incremental
> > > checkpoint as default.
> > >
> > > Best regards,
> > > Jing
> > >
> > > On Mon, Jun 13, 2022 at 12:17 PM Yun Tang  wrote:
> > >
> > > > Strongly +1 for making incremental checkpoints as default. Many users
> > > have
> > > > ever been asking why this configuration is not enabled by default.
> > > >
> > > > BTW, from my knowledge, nothing would happen for HashMapStateBackend,
> > > > which does not support incremental checkpoint yet, when enabling
> > > > incremental checkpoints.
> > > >
> > > >
> > > > Best
> > > > Yun Tang
> > > > 
> > > > From: Martijn Visser 
> > > > Sent: Monday, June 13, 2022 18:05
> > > > To: dev@flink.apache.org 
> > > > Subject: Re: [DISCUSS ] Make state.backend.incremental as true by
> > default
> > > >
> > > > Hi Lihe,
> > > >
> > > > What happens if we enable incremental checkpoints by default while
> the
> > > used
> > > > memory backend is HashMapStateBackend, which doesn't support
> > incremental
> > > > checkpoints?
> > > >
> > > > Best regards,
> > > >
> > > > Martijn
> > > >
> > > > Op ma 13 jun. 2022 om 11:59 schreef Lihe Ma :
> > > >
> > > > > Hi, Everyone,
> > > > >
> > > > > I would like to open a discussion on setting incremental checkpoint
> > as
> > > > > default behavior.
> > > > >
> > > > > Currently, the configuration of state.backend.incremental is set as
> > > false
> > > > > by default. Incremental checkpoint has been adopted widely in
> > industry
> > > > > community for many years , and it is also well-tested from the
> > feedback
> > > > in
> > > > > the community discussion. Incremental checkpointing is more
> > > > light-weighted:
> > > > > shorter checkpoint duration, less uploaded data and less resource
> > > > > consumption.
> > > > >
> > > > > In terms of backward compatibility, enable incremental
> checkpointing
> > > > would
> > > > > not make any data loss no matter restoring from a full
> > > > checkpoint/savepoint
> > > > > or an incremental checkpoint.
> > > > >
> > > > > FLIP-193 (Snapshot ownership)[1] has been released in 1.15,
> > incremental
> > > > > checkpoint no longer depends on a previous restored checkpoint in
> > > default
> > > > > NO_CLAIM mode, which makes the checkpoint lineage much cleaner, it
> > is a
> > > > > good chance to change the configuration state.backend.incremental
> to
> > > true
> > > > > as default.
> > > > >
> > > > > Thus, based on the above discussion, I suggest to make
> > > > > state.backend.incremental as true by default. What do you think of
> > this
> > > > > proposal?
> > > > >
> > > > > [1]
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-193%3A+Snapshots+ownership
> > > > >
> > > > > Best regards,
> > > > > Lihe Ma
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] FLIP-222: Support full job lifecycle statements in SQL client

[Jark Wu]

+1 (binding)

nit: the stop job syntax should be
STOP JOB '' [WITH SAVEPOINT] [WITH DRAIN]
which means savepoint and drain can be enabled at the same time.

Best,
Jark

On Mon, 13 Jun 2022 at 17:20, Jing Ge  wrote:

> +1 (not binding)
>
> Thanks Paul for your effort!
>
> Best regards,
> Jing
>
> On Mon, Jun 13, 2022 at 11:11 AM Paul Lam  wrote:
>
> > Hi team,
> >
> > I'd like to start a vote for FLIP-222: Support full job lifecycle
> > statements in SQL client [1],
> > which is discussed on the thread [2].
> >
> > The vote will be open for at least 72 hours unless there is an objection
> > or not enough votes.
> >
> > Thanks everyone who has participated in the discussion!
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-222%3A+Support+full+job+lifecycle+statements+in+SQL+client
> > [2] https://lists.apache.org/thread/qkvh9p5w9b12s7ykh3l7lv7m9dbgnf1g
> >
> > Best,
> > Paul Lam
> >
> >
>

Re: [VOTE] FLIP-228: Support Within between events in CEP Pattern

[Utopia]

+1

Best  regards
Utopia
2022年6月13日 +0800 17:42 Xingbo Huang ，写道：
> +1 (binding)
>
> Best,
> Xingbo
>
> Dian Fu  于2022年6月13日周一 16:08写道：
>
> > +1 (binding)
> >
> > Regards,
> > Dian
> >
> > On Mon, Jun 13, 2022 at 3:55 PM Rui Fan <1996fan...@gmail.com> wrote:
> >
> > > Thanks for Nicholas driving this.
> > >
> > > +1(non-binding)
> > >
> > > Best,
> > > Rui Fan
> > >
> > > On Mon, Jun 13, 2022 at 3:05 PM md peng  wrote:
> > >
> > > > +1(not-binding)
> > > >
> > > > Best,
> > > > Mingde Peng
> > > >
> > > > Martijn Visser  于2022年6月13日周一 14:48写道：
> > > >
> > > > > +1 (binding)
> > > > >
> > > > > Looking forward :)
> > > > >
> > > > > Thanks, Martijn
> > > > >
> > > > > Op ma 13 jun. 2022 om 04:16 schreef Nicholas :
> > > > >
> > > > > > Hi everyone，
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > Thanks for feedback for FLIP-228: Support Within between events in
> > > CEP
> > > > > > Pattern[1] on the discussion thread[2]. I'd like to start a VOTE
> > > thread
> > > > > for
> > > > > > FLIP-228.
> > > > > >
> > > > > > The vote will be open for at least 72 hours unless there is an
> > > > objection
> > > > > > or not enough votes.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > Regards,
> > > > > >
> > > > > > Nicholas Jiang
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > [1]
> > > > > >
> > > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-228%3A+Support+Within+between+events+in+CEP+Pattern
> > > > > >
> > > > > > [2]
> > https://lists.apache.org/thread/p60ctx213f8921rgklk5f0b6xfrs8ksz
> > > > >
> > > >
> > >
> >

Re: [VOTE] FLIP-228: Support Within between events in CEP Pattern

[Utopia]

+1 (binding)

Best  regards
Utopia
2022年6月13日 +0800 10:16 dev@flink.apache.org，写道：
>
> Looking

[jira] [Created] (FLINK-28022) Support Google Cloud PubSub connector in Python DataStream API

[pengmd (Jira)]

pengmd created FLINK-28022:
--

 Summary: Support Google Cloud PubSub connector in Python 
DataStream API
 Key: FLINK-28022
 URL: https://issues.apache.org/jira/browse/FLINK-28022
 Project: Flink
  Issue Type: Improvement
  Components: API / Python, Connectors / Google Cloud PubSub
Reporter: pengmd
 Fix For: 1.16.0


Support Google Cloud PubSub connector in Python DataStream API



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Re: [VOTE] FLIP-222: Support full job lifecycle statements in SQL client

[Martijn Visser]

+1 (binding)

Op ma 13 jun. 2022 om 13:52 schreef Jark Wu :

> +1 (binding)
>
> nit: the stop job syntax should be
> STOP JOB '' [WITH SAVEPOINT] [WITH DRAIN]
> which means savepoint and drain can be enabled at the same time.
>
> Best,
> Jark
>
> On Mon, 13 Jun 2022 at 17:20, Jing Ge  wrote:
>
> > +1 (not binding)
> >
> > Thanks Paul for your effort!
> >
> > Best regards,
> > Jing
> >
> > On Mon, Jun 13, 2022 at 11:11 AM Paul Lam  wrote:
> >
> > > Hi team,
> > >
> > > I'd like to start a vote for FLIP-222: Support full job lifecycle
> > > statements in SQL client [1],
> > > which is discussed on the thread [2].
> > >
> > > The vote will be open for at least 72 hours unless there is an
> objection
> > > or not enough votes.
> > >
> > > Thanks everyone who has participated in the discussion!
> > >
> > > [1]
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-222%3A+Support+full+job+lifecycle+statements+in+SQL+client
> > > [2] https://lists.apache.org/thread/qkvh9p5w9b12s7ykh3l7lv7m9dbgnf1g
> > >
> > > Best,
> > > Paul Lam
> > >
> > >
> >
>

[jira] [Created] (FLINK-28023) Translate "Native Kubernetes" page into Chinese

[linqichen (Jira)]

linqichen created FLINK-28023:
-

 Summary: Translate "Native Kubernetes" page into Chinese
 Key: FLINK-28023
 URL: https://issues.apache.org/jira/browse/FLINK-28023
 Project: Flink
  Issue Type: Improvement
  Components: Deployment / Kubernetes
Affects Versions: 1.14.4
Reporter: linqichen






--
This message was sent by Atlassian Jira
(v8.20.7#820007)

RE: [DISCUSS ] Make state.backend.incremental as true by default

[kui yuan]

Strongly +1

On 2022/06/13 09:59:17 Lihe Ma wrote:
> Hi, Everyone,
>
> I would like to open a discussion on setting incremental checkpoint as
default behavior.
>
> Currently, the configuration of state.backend.incremental is set as false
by default. Incremental checkpoint has been adopted widely in industry
community for many years , and it is also well-tested from the feedback in
the community discussion. Incremental checkpointing is more light-weighted:
shorter checkpoint duration, less uploaded data and less resource
consumption.
>
> In terms of backward compatibility, enable incremental checkpointing
would not make any data loss no matter restoring from a full
checkpoint/savepoint or an incremental checkpoint.
>
> FLIP-193 (Snapshot ownership)[1] has been released in 1.15, incremental
checkpoint no longer depends on a previous restored checkpoint in default
NO_CLAIM mode, which makes the checkpoint lineage much cleaner, it is a
good chance to change the configuration state.backend.incremental to true
as default.
>
> Thus, based on the above discussion, I suggest to make
state.backend.incremental as true by default. What do you think of this
proposal?
>
> [1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-193%3A+Snapshots+ownership
>
> Best regards,
> Lihe Ma
>

[jira] [Created] (FLINK-28024) Azure worker stops operating due to log files becoming too big

[Matthias Pohl (Jira)]

Matthias Pohl created FLINK-28024:
-

 Summary: Azure worker stops operating due to log files becoming 
too big
 Key: FLINK-28024
 URL: https://issues.apache.org/jira/browse/FLINK-28024
 Project: Flink
  Issue Type: Bug
  Components: Build System / Azure Pipelines
Affects Versions: 1.16.0
Reporter: Matthias Pohl


We observed several situations already where log files reached a file size of 
over 120G. This caused the worker's disk usage to reach 100% resulting in the 
worker machine to go "offline", i.e. not being available to pick up new tasks.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Created] (FLINK-28025) Document the change of serviceAccount in upgrading doc of k8s opeator

[Biao Geng (Jira)]

Biao Geng created FLINK-28025:
-

 Summary: Document the change of serviceAccount in upgrading doc of 
k8s opeator
 Key: FLINK-28025
 URL: https://issues.apache.org/jira/browse/FLINK-28025
 Project: Flink
  Issue Type: Improvement
  Components: Kubernetes Operator
Reporter: Biao Geng


Since 1.0.0, we require users to specify serviceAccount and add corresponding 
validation.

According to the experience of [~czchen] , we had better document such change 
in the upgrading doc.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Created] (FLINK-28026) Add benchmark module for flink table store

[Aiden Gong (Jira)]

Aiden Gong created FLINK-28026:
--

 Summary: Add benchmark module for flink table store
 Key: FLINK-28026
 URL: https://issues.apache.org/jira/browse/FLINK-28026
 Project: Flink
  Issue Type: New Feature
  Components: Table Store
Reporter: Aiden Gong
 Fix For: table-store-0.2.0


Add benchmark module for flink table store.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Re: [DISCUSS] FLIP-240: Introduce "ANALYZE TABLE" Syntax

[cao zou]

Hi godfrey, thanks for your detail explanation.
After explaining and glancing over the FLIP-231, I think it is
really need, +1 for this and looking forward to it.

best
zoucao

godfrey he  于2022年6月13日周一 14:43写道：

> Hi Ingo,
>
> The semantics does not distinguish batch and streaming,
> It works for both batch and streaming, but the result of
> unbounded sources is meaningless.
> Currently, I throw exception for streaming mode,
> and we can support streaming mode with bounded source
> in the future.
>
> Best,
> Godfrey
>
> Ingo Bürk  于2022年6月13日周一 14:17写道：
> >
> > Hi Godfrey,
> >
> > thank you for the explanation. A SELECT is definitely more generic and
> > will work for all connectors automatically. As such I think it's a good
> > baseline solution regardless.
> >
> > We can also think about allowing connector-specific optimizations in the
> > future, but I do like your idea of letting the optimizer rules perform a
> > lot of the work here already by leveraging existing optimizations.
> > Similarly things like non-null counts of non-nullable columns would (or
> > at least could) be handled by the optimizer rules already.
> >
> > So as far as that point goes, +1 to the generic approach.
> >
> > One more point, though: In general we should avoid supporting features
> > only in specific modes as it breaks the unification promise. Given that
> > ANALYZE is a manual and completely optional operation I'm OK with doing
> > that here in principle. However, I wonder what will happen in the
> > streaming / unbounded case. Do you plan to throw an error? Or do we
> > complete the command as successful but without doing anything?
> >
> >
> > Best
> > Ingo
> >
> > On 13.06.22 05:50, godfrey he wrote:
> > > Hi Ingo,
> > >
> > > Thanks for the inputs.
> > >
> > > I think converting `ANALYZE TABLE` to `SELECT` statement is
> > > more generic approach. Because query plan optimization is more generic,
> > >   we can provide more optimization rules to optimize not only `SELECT`
> statement
> > > converted from `ANALYZE TABLE` but also the `SELECT` statement written
> by users.
> > >
> > >> JDBC connector can get a row count estimate without performing a
> > >> SELECT COUNT(1)
> > > To optimize such cases, we can implement a rule to push aggregate into
> > > table source.
> > > Currently, there is a similar rule: SupportsAggregatePushDown, which
> > > supports only pushing
> > > local aggregate into source now.
> > >
> > >
> > > Best,
> > > Godfrey
> > >
> > > Ingo Bürk  于2022年6月10日周五 17:15写道：
> > >>
> > >> Hi Godfrey,
> > >>
> > >> compared to the solution proposed in the FLIP (using a SELECT
> > >> statement), I wonder if you have considered adding APIs to catalogs /
> > >> connectors to perform this task as an alternative?
> > >> I could imagine that for many connectors, statistics could be
> > >> implemented in a less expensive way by leveraging the underlying
> system
> > >> (e.g. a JDBC connector can get a row count estimate without
> performing a
> > >> SELECT COUNT(1)).
> > >>
> > >>
> > >> Best
> > >> Ingo
> > >>
> > >>
> > >> On 10.06.22 09:53, godfrey he wrote:
> > >>> Hi all,
> > >>>
> > >>> I would like to open a discussion on FLIP-240:  Introduce "ANALYZE
> > >>> TABLE" Syntax.
> > >>>
> > >>> As FLIP-231 mentioned, statistics are one of the most important
> inputs
> > >>> to the optimizer. Accurate and complete statistics allows the
> > >>> optimizer to be more powerful. "ANALYZE TABLE" syntax is a very
> common
> > >>> but effective approach to gather statistics, which is already
> > >>> introduced by many compute engines and databases.
> > >>>
> > >>> The main purpose of  discussion is to introduce "ANALYZE TABLE"
> syntax
> > >>> for Flink sql.
> > >>>
> > >>> You can find more details in FLIP-240 document[1]. Looking forward to
> > >>> your feedback.
> > >>>
> > >>> [1]
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=217386481
> > >>> [2] POC: https://github.com/godfreyhe/flink/tree/FLIP-240
> > >>>
> > >>>
> > >>> Best,
> > >>> Godfrey
>

RE: [VOTE] Update Flink's Scala 2.12 support from 2.12.7 to 2.12.16

[Nick]

I think upgrading to 2.12.15 would be the correct choice, as 2.12.16 has a
known regression for projects compiled with Java & Scala, which Flink does:
https://github.com/scala/scala/releases/tag/v2.12.16

This regression will be addressed in a few months via 2.12.17, so for now
2.12.15 should be good to use.

On 2022/06/10 10:32:23 Martijn Visser wrote:
> Hi everyone,
>
> As previously discussed [1] I would like to open a vote thread for
updating
> Flink from Scala 2.12.7 to Scala 2.12.16 and accept that there will be a
> binary incompatibility when this upgrade is executed.
>
> For this vote, I'm following the FLIP criteria since I think that best
fits
> with this proposal. That means that the vote will be open for at least 72
> hours unless there is an objection or not enough votes. Any committer can
> provide a binding vote.
>
> Best regards,
>
> Martijn Visser
> https://twitter.com/MartijnVisser82
> https://github.com/MartijnVisser
>
> [1] https://lists.apache.org/thread/9ft0tbhj45fplt5j3gtb7jzbt4tdr6rh
>

Flink running same task on different Task Manager

[Great Info]

Hi,
I have one flink job which has two tasks
Task1- Source some static data over https and keep it in memory, this keeps
refreshing it every 1 hour
Task2- Process some real-time events from Kafka and uses static data to
validate something and transform, then forward to other Kafka topic.

so far, everything was running on the same Task manager(same node), but due
to some recent scaling requirements need to enable partitioning on Task2
and that will make some partitions run on other task managers. but other
task managers don't have the static data

is there a way to run Task1 on all the task managers? I don't want to
enable broadcasting since it is a little huge and also I can not persist
data in DB due to data regulations.

Re: [VOTE] Update Flink's Scala 2.12 support from 2.12.7 to 2.12.16

[Chesnay Schepler]

@Nick Have you checked whether this really affects Flink`?
Asking since the regressions doesn't categorically affect every mixed 
code base.


On 13/06/2022 16:50, Nick wrote:

I think upgrading to 2.12.15 would be the correct choice, as 2.12.16 has a
known regression for projects compiled with Java & Scala, which Flink does:
https://github.com/scala/scala/releases/tag/v2.12.16

This regression will be addressed in a few months via 2.12.17, so for now
2.12.15 should be good to use.

On 2022/06/10 10:32:23 Martijn Visser wrote:

Hi everyone,

As previously discussed [1] I would like to open a vote thread for

updating

Flink from Scala 2.12.7 to Scala 2.12.16 and accept that there will be a
binary incompatibility when this upgrade is executed.

For this vote, I'm following the FLIP criteria since I think that best

fits

with this proposal. That means that the vote will be open for at least 72
hours unless there is an objection or not enough votes. Any committer can
provide a binding vote.

Best regards,

Martijn Visser
https://twitter.com/MartijnVisser82
https://github.com/MartijnVisser

[1] https://lists.apache.org/thread/9ft0tbhj45fplt5j3gtb7jzbt4tdr6rh

[jira] [Created] (FLINK-28027) Initialise Async Sink maximum number of in flight messages to low number for rate limiting strategy

[Zichen Liu (Jira)]

Zichen Liu created FLINK-28027:
--

 Summary: Initialise Async Sink maximum number of in flight 
messages to low number for rate limiting strategy
 Key: FLINK-28027
 URL: https://issues.apache.org/jira/browse/FLINK-28027
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Common, Connectors / Kinesis
Affects Versions: 1.15.0
Reporter: Zichen Liu
 Fix For: 1.16.0


*Background*

In the AsyncSinkWriter, we implement a rate limiting strategy.

The initial value for the maximum number of in flight messages is set extremely 
high ({{{}maxBatchSize * maxInFlightRequests{}}}).

However, in accordance with the AIMD strategy, the TCP implementation for 
congestion control has found a small value to start with [is 
better]([https://en.wikipedia.org/wiki/TCP_congestion_control#Slow_start]).

*Suggestion*

A better default might be:
 * maxBatchSize
 * maxBatchSize / parallelism



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Created] (FLINK-28028) Common secure credential protection mechanism in Flink SQL

[Jing Ge (Jira)]

Jing Ge created FLINK-28028:
---

 Summary: Common secure credential protection mechanism in Flink SQL
 Key: FLINK-28028
 URL: https://issues.apache.org/jira/browse/FLINK-28028
 Project: Flink
  Issue Type: Improvement
Reporter: Jing Ge


Currently, the most common way to use credential is to use:

{{CREATE TABLE ...}}

{{WITH ( ...}}

{{'username' = 'user'}}

{{'password' = 'pass'}}

{{)}}

The credential could then be read by calling SHOW CREATE TABLE . 

We should provide a more strong way to protect the credential. 



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Created] (FLINK-28029) Checkpoint always hangs when running some jobs

[Pauli Gandhi (Jira)]

Pauli Gandhi created FLINK-28029:


 Summary: Checkpoint always hangs when running some jobs
 Key: FLINK-28029
 URL: https://issues.apache.org/jira/browse/FLINK-28029
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Checkpointing
Affects Versions: 1.14.3
Reporter: Pauli Gandhi


We have noticed that Flink jobs hangs and eventually times out after 2 hours 
every time at the first checkpoint after it completes 15/23 acknowledgments 
(65%).  There is no cpu activity but yet there are number of tasks reporting 
100% back pressure.  It is peculiar to this job and slight modifications to 
this job.  We have created many Flink jobs in the past and never encountered 
the issue.  

Here are the things we tried to narrow down the problem
 * The job runs fine if checkpointing is disabled.
 * Increasing the number of task managers and parallelism to 2 seems to help 
the job complete.  However, it stalled again when we sent a larger data set.
 * Increased taskmanager memory from 4 GB to 16 GB and cpu from 1 to 4 but 
didn't help.
 * Sometimes restarting the job manager helps but at other times not.
 * Breaking up the job into smaller parts helps the job to finish.
 * Analyzed the the thread dump and it appears all threads are either in 
sleeping or wait state.

Here are the environment details
 * Flink version 1.14.3
 * Running Kubernetes
 * Using RocksDB state backend.
 * Checkpoint storage is S3 storage using the Presto library
 * Exactly Once Semantics with unaligned checkpoints enabled.
 * Checkpoint timeout 2 hours
 * Maximum concurrent checkpoints is 1
 * Taskmanager CPU: 4, Slots: 1, Process Size: 12 GB
 * Using Kafka for input and output

I have attached the task manager logs, thread dump, and screen shots of the job 
graph and stalled checkpoint.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Created] (FLINK-28030) Checkpoint always hangs when running some jobs

[Pauli Gandhi (Jira)]

Pauli Gandhi created FLINK-28030:


 Summary: Checkpoint always hangs when running some jobs
 Key: FLINK-28030
 URL: https://issues.apache.org/jira/browse/FLINK-28030
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Checkpointing
Affects Versions: 1.14.3
Reporter: Pauli Gandhi


We have noticed that Flink jobs hangs and eventually times out after 2 hours 
every time at the first checkpoint after it completes 15/23 acknowledgments 
(65%).  There is no cpu activity but yet there are number of tasks reporting 
100% back pressure.  It is peculiar to this job and slight modifications to 
this job.  We have created many Flink jobs in the past and never encountered 
the issue.  

Here are the things we tried to narrow down the problem
 * The job runs fine if checkpointing is disabled.
 * Increasing the number of task managers and parallelism to 2 seems to help 
the job complete.  However, it stalled again when we sent a larger data set.
 * Increased taskmanager memory from 4 GB to 16 GB and cpu from 1 to 4 but 
didn't help.
 * Sometimes restarting the job manager helps but at other times not.
 * Breaking up the job into smaller parts helps the job to finish.
 * Analyzed the the thread dump and it appears all threads are either in 
sleeping or wait state.

Here are the environment details
 * Flink version 1.14.3
 * Running Kubernetes
 * Using RocksDB state backend.
 * Checkpoint storage is S3 storage using the Presto library
 * Exactly Once Semantics with unaligned checkpoints enabled.
 * Checkpoint timeout 2 hours
 * Maximum concurrent checkpoints is 1
 * Taskmanager CPU: 4, Slots: 1, Process Size: 12 GB
 * Using Kafka for input and output

I have attached the task manager logs, thread dump, and screen shots of the job 
graph and stalled checkpoint.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Created] (FLINK-28031) Checkpoint always hangs when running some jobs

[Pauli Gandhi (Jira)]

Pauli Gandhi created FLINK-28031:


 Summary: Checkpoint always hangs when running some jobs
 Key: FLINK-28031
 URL: https://issues.apache.org/jira/browse/FLINK-28031
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Checkpointing
Affects Versions: 1.14.3
Reporter: Pauli Gandhi


We have noticed that Flink jobs hangs and eventually times out after 2 hours 
every time at the first checkpoint after it completes 15/23 acknowledgments 
(65%).  There is no cpu activity but yet there are number of tasks reporting 
100% back pressure.  It is peculiar to this job and slight modifications to 
this job.  We have created many Flink jobs in the past and never encountered 
the issue.  

Here are the things we tried to narrow down the problem
 * The job runs fine if checkpointing is disabled.
 * Increasing the number of task managers and parallelism to 2 seems to help 
the job complete.  However, it stalled again when we sent a larger data set.
 * Increased taskmanager memory from 4 GB to 16 GB and cpu from 1 to 4 but 
didn't help.
 * Sometimes restarting the job manager helps but at other times not.
 * Breaking up the job into smaller parts helps the job to finish.
 * Analyzed the the thread dump and it appears all threads are either in 
sleeping or wait state.

Here are the environment details
 * Flink version 1.14.3
 * Running Kubernetes
 * Using RocksDB state backend.
 * Checkpoint storage is S3 storage using the Presto library
 * Exactly Once Semantics with unaligned checkpoints enabled.
 * Checkpoint timeout 2 hours
 * Maximum concurrent checkpoints is 1
 * Taskmanager CPU: 4, Slots: 1, Process Size: 12 GB
 * Using Kafka for input and output

I have attached the task manager logs, thread dump, and screen shots of the job 
graph and stalled checkpoint.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Created] (FLINK-28032) Flink checkpointing hangs with some jobs

[Pauli Gandhi (Jira)]

Pauli Gandhi created FLINK-28032:


 Summary: Flink checkpointing hangs with some jobs
 Key: FLINK-28032
 URL: https://issues.apache.org/jira/browse/FLINK-28032
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Checkpointing
Affects Versions: 1.14.3
 Environment: Here are the environment details
 * Flink version 1.14.3
 * Running Kubernetes
 * Using RocksDB state backend.
 * Checkpoint storage is S3 storage using the Presto library
 * Exactly Once Semantics with unaligned checkpoints enabled.
 * Checkpoint timeout 2 hours
 * Maximum concurrent checkpoints is 1
 * Taskmanager CPU: 4, Slots: 1, Process Size: 12 GB
 * Using Kafka for input and output
Reporter: Pauli Gandhi
 Attachments: checkpoint snapshot.png, jobgraph.png, 
taskmanager_10.112.55.143_6122-969889_log, 
taskmanager_10.112.55.143_6122-969889_thread_dump

We have noticed that Flink jobs hangs and eventually times out after 2 hours 
every time at the first checkpoint after it completes 15/23(65%) 
acknowledgments.  There is no cpu/record processing activity but yet there are 
a number of tasks reporting 100% back pressure.  It is peculiar to this job and 
slight modifications to this job.  We have created many Flink jobs in the past 
and never encountered the issue.  

Here are the things we tried to narrow down the problem
 * The job runs fine if checkpointing is disabled.
 * Increasing the number of task managers and parallelism to 2 seems to help 
the job complete.  However, it stalled again when we sent a larger data set.
 * Increased taskmanager memory from 4 GB to 16 GB and cpu from 1 to 4 but 
didn't help.
 * Sometimes restarting the job manager helps but at other times not.
 * Breaking up the job into smaller parts helps the job to finish.
 * Analyzed the the thread dump and it appears all threads are either in 
sleeping or wait state.

I have attached the task manager logs, thread dump, and screen shots of the job 
graph and stalled checkpoint.

Your help in resolving this issue is greatly appreciated.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Re:Re: [DISCUSS] FLIP-240: Introduce "ANALYZE TABLE" Syntax

[华宗]

退订

















At 2022-06-13 22:44:24, "cao zou"  wrote:
>Hi godfrey, thanks for your detail explanation.
>After explaining and glancing over the FLIP-231, I think it is
>really need, +1 for this and looking forward to it.
>
>best
>zoucao
>
>godfrey he  于2022年6月13日周一 14:43写道：
>
>> Hi Ingo,
>>
>> The semantics does not distinguish batch and streaming,
>> It works for both batch and streaming, but the result of
>> unbounded sources is meaningless.
>> Currently, I throw exception for streaming mode,
>> and we can support streaming mode with bounded source
>> in the future.
>>
>> Best,
>> Godfrey
>>
>> Ingo Bürk  于2022年6月13日周一 14:17写道：
>> >
>> > Hi Godfrey,
>> >
>> > thank you for the explanation. A SELECT is definitely more generic and
>> > will work for all connectors automatically. As such I think it's a good
>> > baseline solution regardless.
>> >
>> > We can also think about allowing connector-specific optimizations in the
>> > future, but I do like your idea of letting the optimizer rules perform a
>> > lot of the work here already by leveraging existing optimizations.
>> > Similarly things like non-null counts of non-nullable columns would (or
>> > at least could) be handled by the optimizer rules already.
>> >
>> > So as far as that point goes, +1 to the generic approach.
>> >
>> > One more point, though: In general we should avoid supporting features
>> > only in specific modes as it breaks the unification promise. Given that
>> > ANALYZE is a manual and completely optional operation I'm OK with doing
>> > that here in principle. However, I wonder what will happen in the
>> > streaming / unbounded case. Do you plan to throw an error? Or do we
>> > complete the command as successful but without doing anything?
>> >
>> >
>> > Best
>> > Ingo
>> >
>> > On 13.06.22 05:50, godfrey he wrote:
>> > > Hi Ingo,
>> > >
>> > > Thanks for the inputs.
>> > >
>> > > I think converting `ANALYZE TABLE` to `SELECT` statement is
>> > > more generic approach. Because query plan optimization is more generic,
>> > >   we can provide more optimization rules to optimize not only `SELECT`
>> statement
>> > > converted from `ANALYZE TABLE` but also the `SELECT` statement written
>> by users.
>> > >
>> > >> JDBC connector can get a row count estimate without performing a
>> > >> SELECT COUNT(1)
>> > > To optimize such cases, we can implement a rule to push aggregate into
>> > > table source.
>> > > Currently, there is a similar rule: SupportsAggregatePushDown, which
>> > > supports only pushing
>> > > local aggregate into source now.
>> > >
>> > >
>> > > Best,
>> > > Godfrey
>> > >
>> > > Ingo Bürk  于2022年6月10日周五 17:15写道：
>> > >>
>> > >> Hi Godfrey,
>> > >>
>> > >> compared to the solution proposed in the FLIP (using a SELECT
>> > >> statement), I wonder if you have considered adding APIs to catalogs /
>> > >> connectors to perform this task as an alternative?
>> > >> I could imagine that for many connectors, statistics could be
>> > >> implemented in a less expensive way by leveraging the underlying
>> system
>> > >> (e.g. a JDBC connector can get a row count estimate without
>> performing a
>> > >> SELECT COUNT(1)).
>> > >>
>> > >>
>> > >> Best
>> > >> Ingo
>> > >>
>> > >>
>> > >> On 10.06.22 09:53, godfrey he wrote:
>> > >>> Hi all,
>> > >>>
>> > >>> I would like to open a discussion on FLIP-240:  Introduce "ANALYZE
>> > >>> TABLE" Syntax.
>> > >>>
>> > >>> As FLIP-231 mentioned, statistics are one of the most important
>> inputs
>> > >>> to the optimizer. Accurate and complete statistics allows the
>> > >>> optimizer to be more powerful. "ANALYZE TABLE" syntax is a very
>> common
>> > >>> but effective approach to gather statistics, which is already
>> > >>> introduced by many compute engines and databases.
>> > >>>
>> > >>> The main purpose of  discussion is to introduce "ANALYZE TABLE"
>> syntax
>> > >>> for Flink sql.
>> > >>>
>> > >>> You can find more details in FLIP-240 document[1]. Looking forward to
>> > >>> your feedback.
>> > >>>
>> > >>> [1]
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=217386481
>> > >>> [2] POC: https://github.com/godfreyhe/flink/tree/FLIP-240
>> > >>>
>> > >>>
>> > >>> Best,
>> > >>> Godfrey
>>

[jira] [Created] (FLINK-28033) find and output new min watermark mybe wrong when in multichannel

[YeAble (Jira)]

YeAble created FLINK-28033:
--

 Summary: find and output new min watermark mybe wrong when in 
multichannel
 Key: FLINK-28033
 URL: https://issues.apache.org/jira/browse/FLINK-28033
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Task
Affects Versions: 1.15.0
Reporter: YeAble


File: StatusWatermarkValue.java

Method:  findAndOutputNewMinWatermarkAcrossAlignedChannels
{code:java}
//代码占位符
long newMinWatermark = Long.MAX_VALUE;
boolean hasAlignedChannels = false;

// determine new overall watermark by considering only watermark-aligned 
channels across all
// channels
for (InputChannelStatus channelStatus : channelStatuses) {
if (channelStatus.isWatermarkAligned) {
hasAlignedChannels = true;
newMinWatermark = Math.min(channelStatus.watermark, newMinWatermark);
}
}

// we acknowledge and output the new overall watermark if it really is 
aggregated
// from some remaining aligned channel, and is also larger than the last output 
watermark
if (hasAlignedChannels && newMinWatermark > lastOutputWatermark) {
lastOutputWatermark = newMinWatermark;
output.emitWatermark(new Watermark(lastOutputWatermark));
} {code}
 channelStatus's initalized watermark is Long.MIN_VALUE. when one 
channelStatus's watermark is changed,but other channelStatus's is not changed, 
the newMinWatermark is always Long.MIN_VALUE and output not emitwatermark。 



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Created] (FLINK-28034) ClassCastException occurred in creating a checkpoint with merge windows

[Takayuki Eimizu (Jira)]

Takayuki Eimizu created FLINK-28034:
---

 Summary: ClassCastException occurred in creating a checkpoint with 
merge windows 
 Key: FLINK-28034
 URL: https://issues.apache.org/jira/browse/FLINK-28034
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Checkpointing, Runtime / State Backends
Affects Versions: 1.15.0
Reporter: Takayuki Eimizu


h1. Summary

In Flink 1.15.0, the combination of following functions always occur 
ClassCastException.
 - Session Window
 - Checkpoint
 - Keyed State

The following repository provides minimal source code that can combine these 
features to reproduce the exception.

[https://github.com/t-eimizu/flink-checkpoint-with-merging-window]

 
h1. Description
h2. How the Exception Occurred
 
In the process window function of the session window, we must use 
`context.globalState()`
instead of `context.windowState()`. If you use `context.windowState()` in this 
situation, Flink throws `UnsupportedOperationException`.
 
So we have to do following:
 
{code:java}
   stPreviousValue = context.globalState().getState(desc4PreviousValue); 
{code}
 
Then stPreviousValue will have the following fields:
||Field Name||Value||
|currentNamespace|VoidNamespace|
|namespaceSerializer|TimeWindow$serializer|
As a result, when flink create checkpoint on this job, ClassCastException 
occurs.
{code:java}
2022-06-14 11:04:57,212 INFO  
org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable [] - 
ProcessingData -> Sink: PrintData (1/1)#0 - asynchronous part of checkpoint 1 
could not be completed. java.util.concurrent.ExecutionException: 
java.lang.ClassCastException: class 
org.apache.flink.runtime.state.VoidNamespace cannot be cast to class 
org.apache.flink.streaming.api.windowing.windows.TimeWindow 
(org.apache.flink.runtime.state.VoidNamespace and 
org.apache.flink.streaming.api.windowing.windows.TimeWindow are in unnamed 
module of loader 'app') at 
java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:?] at 
java.util.concurrent.FutureTask.get(FutureTask.java:191) ~[?:?] at 
org.apache.flink.util.concurrent.FutureUtils.runIfNotDoneAndGet(FutureUtils.java:645)
 ~[flink-core-1.15.0.jar:1.15.0] at 
org.apache.flink.streaming.api.operators.OperatorSnapshotFinalizer.(OperatorSnapshotFinalizer.java:54)
 ~[flink-streaming-java-1.15.0.jar:1.15.0] at 
org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable.finalizeNonFinishedSnapshots(AsyncCheckpointRunnable.java:191)
 ~[flink-streaming-java-1.15.0.jar:1.15.0] at 
org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable.run(AsyncCheckpointRunnable.java:124)
 [flink-streaming-java-1.15.0.jar:1.15.0] at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) 
[?:?] at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) 
[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: 
java.lang.ClassCastException: class 
org.apache.flink.runtime.state.VoidNamespace cannot be cast to class 
org.apache.flink.streaming.api.windowing.windows.TimeWindow 
(org.apache.flink.runtime.state.VoidNamespace and 
org.apache.flink.streaming.api.windowing.windows.TimeWindow are in unnamed 
module of loader 'app') at 
org.apache.flink.streaming.api.windowing.windows.TimeWindow$Serializer.serialize(TimeWindow.java:130)
 ~[flink-streaming-java-1.15.0.jar:1.15.0] at 
org.apache.flink.runtime.state.heap.CopyOnWriteStateMapSnapshot.writeState(CopyOnWriteStateMapSnapshot.java:145)
 ~[flink-runtime-1.15.0.jar:1.15.0] at 
org.apache.flink.runtime.state.heap.AbstractStateTableSnapshot.writeStateInKeyGroup(AbstractStateTableSnapshot.java:116)
 ~[flink-runtime-1.15.0.jar:1.15.0] at 
org.apache.flink.runtime.state.heap.CopyOnWriteStateTableSnapshot.writeStateInKeyGroup(CopyOnWriteStateTableSnapshot.java:38)
 ~[flink-runtime-1.15.0.jar:1.15.0] at 
org.apache.flink.runtime.state.heap.HeapSnapshotStrategy.lambda$asyncSnapshot$3(HeapSnapshotStrategy.java:172)
 ~[flink-runtime-1.15.0.jar:1.15.0] at 
org.apache.flink.runtime.state.SnapshotStrategyRunner$1.callInternal(SnapshotStrategyRunner.java:91)
 ~[flink-runtime-1.15.0.jar:1.15.0] at 
org.apache.flink.runtime.state.SnapshotStrategyRunner$1.callInternal(SnapshotStrategyRunner.java:88)
 ~[flink-runtime-1.15.0.jar:1.15.0] at 
org.apache.flink.runtime.state.AsyncSnapshotCallable.call(AsyncSnapshotCallable.java:78)
 ~[flink-runtime-1.15.0.jar:1.15.0] at 
java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?] at 
org.apache.flink.util.concurrent.FutureUtils.runIfNotDoneAndGet(FutureUtils.java:642)
 ~[flink-core-1.15.0.jar:1.15.0] ... 6 more  {code}
h2.  workaround
Turn off the checkpoint function.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Re: [DISCUSS] Deprecate SourceFunction APIs

[Becket Qin]

In general, I'll give a big +1 to deprecating the SourceFunction.

That said, it is indeed worth looking into what might be missing or less
easy to implement with FLIP-27 Source compared with the SourceFunction.
Maybe we can just compile a list of things to do in order to fully
deprecate the SourceFunction. As far as I am aware of, there are two things
that need to be taken care of：

1. A simple high level API, as Jing mentioned, that makes simple cases that
do not involve the split enumerator easier. Ideally this should be as
simple as SourceFunction, if not simpler. Off the top of my head, I think a
default no-op split enumerator will just do the work. And the data
generator of FLIP-238 could be an implementation using this high level API.

2. FLIP-208, which allows users to stop the job upon receiving a record in
the stream.

Is there anything else that we have heard from the users / connector
developers that needs some attention?

Thanks,

Jiangjie (Becket) Qin



On Fri, Jun 10, 2022 at 3:25 PM David Anderson  wrote:

> +1 for deprecating SourceFunction from me as well. And a big THANK YOU to
> Alex Fedulov for bringing forward FLIP-238.
>
> David
>
> On Fri, Jun 10, 2022 at 4:03 AM Lijie Wang 
> wrote:
>
> > Hi all,
> >
> > Sorry for my mistake. The `StreamExecutionEnvironment#readFiles` and can
> be
> > easily replaced by `FileSource#forRecordStreamFormat/forBulkFileFormat`.
> I
> > have no other concerns.
> >
> >  +1 to deprecate SourceFunction and deprecate the methods (in
> > StreamExecutionEnvironment) based on SourceFunction .
> >
> > Best,
> > Lijie
> >
> > Konstantin Knauf  于2022年6月10日周五 05:11写道：
> >
> > > Hi everyone,
> > >
> > > thank you Jing for redirecting the discussion back to the topic at
> hand.
> > I
> > > agree with all of your points.
> > >
> > > +1 to deprecate SourceFunction
> > >
> > > Is there really no replacement for the
> > StreamExecutionEnvironment#readXXX.
> > > There is already a FLIP-27 based FileSource, right? What's missing to
> > > recommend using that as opposed to the the readXXX methods?
> > >
> > > Cheers,
> > >
> > > Konstantin
> > >
> > > Am Do., 9. Juni 2022 um 20:11 Uhr schrieb Alexander Fedulov <
> > > alexan...@ververica.com>:
> > >
> > > > Hi all,
> > > >
> > > > It seems that there is some understandable cautiousness with regard
> to
> > > > deprecating methods and subclasses that do not have alternatives just
> > > yet.
> > > >
> > > > We should probably first agree if it is in general OK for Flink to
> use
> > > > @Deprecated
> > > > annotation for parts of the code that do not have alternatives. In
> that
> > > > case,
> > > > we could add a comment along the lines of:
> > > > "This implementation is based on a deprecated SourceFunction API that
> > > > will gradually be phased out from Flink. No direct substitute exists
> at
> > > the
> > > > moment.
> > > > If you want to have a more future-proof solution, consider helping
> the
> > > > project by
> > > > contributing an implementation based on the new Source API."
> > > >
> > > > This should clearly communicate the message that usage of these
> > > > methods/classes
> > > > is discouraged and at the same time promote contributions for
> > addressing
> > > > the gap.
> > > > What do you think?
> > > >
> > > > Best,
> > > > Alexander Fedulov
> > > >
> > > >
> > > > On Thu, Jun 9, 2022 at 6:27 PM Ingo Bürk 
> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > these APIs don't expose the underlying source directly, so I don't
> > > think
> > > > > we need to worry about deprecating them as well. There's also
> nothing
> > > > > inherently wrong with using a deprecated API internally, though
> even
> > > > > just for the experience of using our own new APIs I would
> personally
> > > say
> > > > > that they should be migrated to the new Source API. It's hard to
> > reason
> > > > > that users must migrate to a new API if we don't do it internally
> as
> > > > well.
> > > > >
> > > > >
> > > > > Best
> > > > > Ingo
> > > > >
> > > > > On 09.06.22 15:41, Lijie Wang wrote:
> > > > > >   Hi Martijn,
> > > > > >
> > > > > > I don't mean it's a blocker. Just a information. And I'm also +1
> > for
> > > > > this.
> > > > > >
> > > > > > Put it another way: should we migrate the `#readFile(...)` to new
> > API
> > > > or
> > > > > > provide a similar method "readxxx“ based on the new Source API?
> > > > > >
> > > > > > And if we don't migrate it, does it mean that the
> `#readFile(...)`
> > > > should
> > > > > > also be marked as deprecated?
> > > > > >
> > > > > > Best,
> > > > > > Lijie
> > > > > >
> > > > > > Martijn Visser  于2022年6月9日周四 21:03写道：
> > > > > >
> > > > > >> Hi Lijie,
> > > > > >>
> > > > > >> I don't see any problem with deprecating those methods at this
> > > moment,
> > > > > as
> > > > > >> long as we don't remove them until the replacements are
> available.
> > > > > Besides
> > > > > >> that, are we sure there are no replacements already, especially
> > with
> > > > the
> > > > > >> new Fil

[jira] [Created] (FLINK-28035) Enable scan reading different bucket num for rescale bucket

[Jane Chan (Jira)]

Jane Chan created FLINK-28035:
-

 Summary: Enable scan reading different bucket num for rescale 
bucket
 Key: FLINK-28035
 URL: https://issues.apache.org/jira/browse/FLINK-28035
 Project: Flink
  Issue Type: Sub-task
  Components: Table Store
Affects Versions: table-store-0.2.0
Reporter: Jane Chan
 Fix For: table-store-0.2.0






--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Re: [ANNOUNCE] New Apache Flink PMC Member - Jingsong Lee

[Leonard Xu]

Congratulations, Jingsong!


Best,
Leonard

> 2022年6月13日 下午6:52，刘首维  写道：
> 
> Congratulations and well deserved, Jingsong!
> 
> 
> Best regards,
> Shouwei
> -- 原始邮件 --
> 发件人:  
>   "dev"   
>   >;
> 发送时间: 2022年6月13日(星期一) 晚上6:09
> 收件人: "dev"mailto:dev@flink.apache.org>>;
> 
> 主题: Re: [ANNOUNCE] New Apache Flink PMC Member - Jingsong Lee
> 
> 
> 
> Congratulations, Jingsong!
> 
> 
> Best regards,
> Yuxia
> 
> - 原始邮件 -
> 发件人: "Yun Tang"  收件人: "dev"  发送时间: 星期一, 2022年 6 月 13日 下午 6:12:24
> 主题: Re: [ANNOUNCE] New Apache Flink PMC Member - Jingsong Lee
> 
> Congratulations, Jingsong! Well deserved.
> 
> Best
> Yun Tang
> 
> From: Xingbo Huang  Sent: Monday, June 13, 2022 17:39
> To: dev  Subject: Re: [ANNOUNCE] New Apache Flink PMC Member - Jingsong Lee
> 
> Congratulations, Jingsong!
> 
> Best,
> Xingbo
> 
> Jane Chan  
> > Congratulations, Jingsong!
> >
> > Best,
> > Jane Chan
> >
> > On Mon, Jun 13, 2022 at 4:43 PM Shuo Cheng  > wrote:
> >
> > > Congratulations, Jingsong!
> > >
> > > On 6/13/22, Paul Lam  > wrote:
> > > > Congrats, Jingsong! Well deserved!
> > > >
> > > > Best,
> > > > Paul Lam
> > > >
> > > >> 2022年6月13日 16:31，Lincoln Lee  > 写道：
> > > >>
> > > >> Congratulations, Jingsong!
> > > >>
> > > >> Best,
> > > >> Lincoln Lee
> > > >>
> > > >>
> > > >> Jark Wu mailto:imj...@gmail.com>> 
> 于2022年6月13日周一 16:29写道：
> > > >>
> > > >>> Congrats, Jingsong!
> > > >>>
> > > >>> Cheers,
> > > >>> Jark
> > > >>>
> > > >>> On Mon, 13 Jun 2022 at 16:16, Jiangang Liu <
> > liujiangangp...@gmail.com >
> > > >>> wrote:
> > > >>>
> > >  Congratulations, Jingsong!
> > > 
> > >  Best,
> > >  Jiangang Liu
> > > 
> > >  Martijn Visser  > 于2022年6月13日周一 16:06写道：
> > > 
> > > > Like everyone has mentioned, this is very well 
> deserved.
> > > >>> Congratulations!
> > > >
> > > > Op ma 13 jun. 2022 om 09:57 schreef Benchao Li 
> <
> > libenc...@apache.org 
> > > >:
> > > >
> > > >> Congratulations, Jingsong!  Well 
> deserved.
> > > >>
> > > >> Rui Fan <1996fan...@gmail.com 
> > 于2022年6月13日周一 15:53写道：
> > > >>
> > > >>> Congratulations, Jingsong!
> > > >>>
> > > >>> Best,
> > > >>> Rui Fan
> > > >>>
> > > >>> On Mon, Jun 13, 2022 at 3:40 PM LuNing 
> Wang <
> > wang4lun...@gmail.com 
> > > 
> > > >> wrote:
> > > >>>
> > >  Congratulations, Jingsong!
> > > 
> > >  Best,
> > >  LuNing Wang
> > > 
> > >  Ingo Bürk  > 于2022年6月13日周一 15:36写道：
> > > 
> > > > Congrats, Jingsong!
> > > >
> > > > On 13.06.22 08:58, Becket Qin 
> wrote:
> > > >> Hi all,
> > > >>
> > > >> I'm very happy to announce 
> that Jingsong Lee has joined the
> > >  Flink
> > > >>> PMC!
> > > >>
> > > >> Jingsong became a Flink 
> committer in Feb 2020 and has been
> > > >>> continuously
> > > >> contributing to the 
> project since then, mainly in Flink SQL.
> > > >>> He
> > > > has
> > >  been
> > > >> quite active in the 
> mailing list, fixing bugs, helping
> > >  verifying
> > > > releases,
> > > >> reviewing patches and 
> FLIPs. Jingsong is also devoted to
> > >  pushing
> > > >>> Flink
> > > > SQL
> > > >> to new use cases. He spent 
> a lot of time in implementing the
> > > > Flink
> > > >> connectors for Apache 
> Iceberg. Jingsong is also the primary
> > > > driver
> > >  behind
> > > >> the effort of 
> flink-table-store, which aims to provide a
> > > >> stream-batch
> > > >> unified storage for Flink 
> dynamic tables.
> > > >>
> > > >> Congratulations and 
> welcome, Jingsong!
> > > >>
> > > >> Cheers,
> > > >>
> > > >> Jiangjie (Becket) Qin
> > > >> (On behalf of the Apache 
> Flink PMC)
> > > >>
> > > >
> > > 
> > > >>>
> > > >>
> > > >>
> > > >> --
> > > >>
> > > >> Best,
> > > >> Benchao Li
> > > >>
> > > >
> > > 
> > > >>>
> > > >
> > > >
> > >
> >

Re: [ANNOUNCE] New Apache Flink PMC Member - Jingsong Lee

[Jing Zhang]

Congratulations, Jingsong!

Best,
Jing Zhang

Leonard Xu  于2022年6月14日周二 10:54写道：

> Congratulations, Jingsong!
>
>
> Best,
> Leonard
>
> > 2022年6月13日 下午6:52，刘首维  写道：
> >
> > Congratulations and well deserved, Jingsong!
> >
> >
> > Best regards,
> > Shouwei
> > -- 原始邮件 --
> > 发件人:
> "dev"
>>;
> > 发送时间: 2022年6月13日(星期一) 晚上6:09
> > 收件人: "dev"mailto:dev@flink.apache.org>>;
> >
> > 主题: Re: [ANNOUNCE] New Apache Flink PMC Member - Jingsong Lee
> >
> >
> >
> > Congratulations, Jingsong!
> >
> >
> > Best regards,
> > Yuxia
> >
> > - 原始邮件 -
> > 发件人: "Yun Tang"  > 收件人: "dev"  > 发送时间: 星期一, 2022年 6 月 13日 下午 6:12:24
> > 主题: Re: [ANNOUNCE] New Apache Flink PMC Member - Jingsong Lee
> >
> > Congratulations, Jingsong! Well deserved.
> >
> > Best
> > Yun Tang
> > 
> > From: Xingbo Huang  > Sent: Monday, June 13, 2022 17:39
> > To: dev  > Subject: Re: [ANNOUNCE] New Apache Flink PMC Member - Jingsong Lee
> >
> > Congratulations, Jingsong!
> >
> > Best,
> > Xingbo
> >
> > Jane Chan  >
> > > Congratulations, Jingsong!
> > >
> > > Best,
> > > Jane Chan
> > >
> > > On Mon, Jun 13, 2022 at 4:43 PM Shuo Cheng  > wrote:
> > >
> > > > Congratulations, Jingsong!
> > > >
> > > > On 6/13/22, Paul Lam  paullin3...@gmail.com>> wrote:
> > > > > Congrats, Jingsong! Well deserved!
> > > > >
> > > > > Best,
> > > > > Paul Lam
> > > > >
> > > > >> 2022年6月13日 16:31，Lincoln Lee  > 写道：
> > > > >>
> > > > >> Congratulations, Jingsong!
> > > > >>
> > > > >> Best,
> > > > >> Lincoln Lee
> > > > >>
> > > > >>
> > > > >> Jark Wu mailto:imj...@gmail.com>>
> 于2022年6月13日周一 16:29写道：
> > > > >>
> > > > >>> Congrats, Jingsong!
> > > > >>>
> > > > >>> Cheers,
> > > > >>> Jark
> > > > >>>
> > > > >>> On Mon, 13 Jun 2022 at 16:16, Jiangang Liu <
> > > liujiangangp...@gmail.com >
> > > > >>> wrote:
> > > > >>>
> > > >  Congratulations, Jingsong!
> > > > 
> > > >  Best,
> > > >  Jiangang Liu
> > > > 
> > > >  Martijn Visser  > 于2022年6月13日周一 16:06写道：
> > > > 
> > > > > Like everyone has mentioned, this is very
> well deserved.
> > > > >>> Congratulations!
> > > > >
> > > > > Op ma 13 jun. 2022 om 09:57 schreef
> Benchao Li <
> > > libenc...@apache.org 
> > > > >:
> > > > >
> > > > >> Congratulations, Jingsong!  Well
> deserved.
> > > > >>
> > > > >> Rui Fan <1996fan...@gmail.com
> > 于2022年6月13日周一 15:53写道：
> > > > >>
> > > > >>> Congratulations, Jingsong!
> > > > >>>
> > > > >>> Best,
> > > > >>> Rui Fan
> > > > >>>
> > > > >>> On Mon, Jun 13, 2022 at 3:40 PM
> LuNing Wang <
> > > wang4lun...@gmail.com 
> > > > 
> > > > >> wrote:
> > > > >>>
> > > >  Congratulations, Jingsong!
> > > > 
> > > >  Best,
> > > >  LuNing Wang
> > > > 
> > > >  Ingo Bürk <
> airbla...@apache.org > 于2022年6月13日周一
> 15:36写道：
> > > > 
> > > > > Congrats, Jingsong!
> > > > >
> > > > > On 13.06.22 08:58, Becket
> Qin wrote:
> > > > >> Hi all,
> > > > >>
> > > > >> I'm very happy to
> announce that Jingsong Lee has joined the
> > > >  Flink
> > > > >>> PMC!
> > > > >>
> > > > >> Jingsong became a
> Flink committer in Feb 2020 and has been
> > > > >>> continuously
> > > > >> contributing to the
> project since then, mainly in Flink SQL.
> > > > >>> He
> > > > > has
> > > >  been
> > > > >> quite active in the
> mailing list, fixing bugs, helping
> > > >  verifying
> > > > > releases,
> > > > >> reviewing patches and
> FLIPs. Jingsong is also devoted to
> > > >  pushing
> > > > >>> Flink
> > > > > SQL
> > > > >> to new use cases. He
> spent a lot of time in implementing the
> > > > > Flink
> > > > >> connectors for Apache
> Iceberg. Jingsong is also the primary
> > > > > driver
> > > >  behind
> > > > >> the effort of
> flink-table-store, which aims to provide a
> > > > >> stream-batch
> > > > >> unified storage for
> Flink dynamic tables.
> > > > >>
> > > > >> Congratulations and
> welcome, Jingsong!
> > > > >>
> > > > >> Cheers,
> > > > >>
> > > > >> Jiangjie (Becket) Qin
> > > > >> (On behalf of the
> Apache Flink PMC)
> > > > >>
> > > > >
> > > > 
> > > > >>>
> > > > >>
> > > > >>
> > > > >> --
> > > > >>
> > > > >> Best,
> > > > >> Benchao Li
> > > > >>
> > > > >
> > > > 
> > > > >>>
> > > > >

Re: [VOTE] Release 1.14.5, release candidate #1

[Xingbo Huang]

+1 (non-binding)

- verify signatures and checksums
- no binaries found in source archive
- build from source
- verify python wheel package contents
- pip install apache-flink-libraries and apache-flink wheel packages
- run the examples from Python Table API tutorial

Best,
Xingbo

Konstantin Knauf  于2022年6月12日周日 00:16写道：

> +1 (binding)
>
> * checked signatures and checksums of binaries -> OK
> * skimmed over changes since 1.14.4 -> no dependency updates without NOTICE
> updates
> * reviewed release blog post and -> requested changes
>
> Thanks for preparing this rc!
>
> Konstantin
>
>
> Am Fr., 10. Juni 2022 um 11:58 Uhr schrieb Xingbo Huang <
> hxbks...@gmail.com
> >:
>
> > Hi everyone,
> >
> > Please review and vote on the release candidate #1 for the version
> 1.14.5,
> > as follows:
> > [ ] +1, Approve the release
> > [ ] -1, Do not approve the release (please provide specific comments)
> >
> > The complete staging area is available for your review, which includes:
> > * JIRA release notes [1],
> > * the official Apache source release and binary convenience releases to
> be
> > deployed to dist.apache.org [2], which are signed with the key with
> > fingerprint 3C2C9FFB59DF9F3E [3],
> > * all artifacts to be deployed to the Maven Central Repository [4],
> > * source code tag "release-1.14.5-rc1" [5],
> > * website pull request listing the new release and adding announcement
> blog
> > post [6].
> >
> > The vote will be open for at least 72 hours. It is adopted by majority
> > approval, with at least 3 PMC affirmative votes.
> >
> > Best,
> > Xingbo
> >
> > [1]
> >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12351388
> > [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.14.5-rc1/
> > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > [4]
> > https://repository.apache.org/content/repositories/orgapacheflink-1509/
> > [5] https://github.com/apache/flink/tree/release-1.14.5-rc1
> > [6] https://github.com/apache/flink-web/pull/550
> >
>
>
> --
> https://twitter.com/snntrable
> https://github.com/knaufk
>

[jira] [Created] (FLINK-28036) Use correct writable type to creaet ConstantObjectInspector

[luoyuxia (Jira)]

luoyuxia created FLINK-28036:


 Summary: Use correct writable type to creaet 
ConstantObjectInspector
 Key: FLINK-28036
 URL: https://issues.apache.org/jira/browse/FLINK-28036
 Project: Flink
  Issue Type: Sub-task
Reporter: luoyuxia






--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Created] (FLINK-28037) Flink SQL Upsert-Kafka can not support Flink1.14.x

[Jiangfei Liu (Jira)]

Jiangfei Liu created FLINK-28037:


 Summary: Flink SQL Upsert-Kafka can not support Flink1.14.x
 Key: FLINK-28037
 URL: https://issues.apache.org/jira/browse/FLINK-28037
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Kafka
Affects Versions: 1.14.4, 1.14.3, 1.14.2, 1.14.0
 Environment: Flink Version: 1.14.0 1.14.2 1.14.3 1.14.4
Reporter: Jiangfei Liu
 Attachments: kafka-sql.png, kafka-sql2.png

in Flink 1.14.x，flink sql upsert-kafka sink can not write data into kafka topic 
with sink buffer flush config，eg 
h5. sink.buffer-flush.max-rows
h5. sink.buffer-flush.interval

in Flink1.13.x，flink sql upsert-kafka sink can write data into kafka topic with 
sink buffer lush config



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Re: Re: [DISCUSS] FLIP-240: Introduce "ANALYZE TABLE" Syntax

[Jing Ge]

Hi 华宗

退订请发送任意消息至dev-unsubscr...@flink.apache.org
In order to unsubscribe, please send an email to
dev-unsubscr...@flink.apache.org

Thanks

Best regards,
Jing


On Tue, Jun 14, 2022 at 2:05 AM 华宗  wrote:

> 退订
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> At 2022-06-13 22:44:24, "cao zou"  wrote:
> >Hi godfrey, thanks for your detail explanation.
> >After explaining and glancing over the FLIP-231, I think it is
> >really need, +1 for this and looking forward to it.
> >
> >best
> >zoucao
> >
> >godfrey he  于2022年6月13日周一 14:43写道：
> >
> >> Hi Ingo,
> >>
> >> The semantics does not distinguish batch and streaming,
> >> It works for both batch and streaming, but the result of
> >> unbounded sources is meaningless.
> >> Currently, I throw exception for streaming mode,
> >> and we can support streaming mode with bounded source
> >> in the future.
> >>
> >> Best,
> >> Godfrey
> >>
> >> Ingo Bürk  于2022年6月13日周一 14:17写道：
> >> >
> >> > Hi Godfrey,
> >> >
> >> > thank you for the explanation. A SELECT is definitely more generic and
> >> > will work for all connectors automatically. As such I think it's a
> good
> >> > baseline solution regardless.
> >> >
> >> > We can also think about allowing connector-specific optimizations in
> the
> >> > future, but I do like your idea of letting the optimizer rules
> perform a
> >> > lot of the work here already by leveraging existing optimizations.
> >> > Similarly things like non-null counts of non-nullable columns would
> (or
> >> > at least could) be handled by the optimizer rules already.
> >> >
> >> > So as far as that point goes, +1 to the generic approach.
> >> >
> >> > One more point, though: In general we should avoid supporting features
> >> > only in specific modes as it breaks the unification promise. Given
> that
> >> > ANALYZE is a manual and completely optional operation I'm OK with
> doing
> >> > that here in principle. However, I wonder what will happen in the
> >> > streaming / unbounded case. Do you plan to throw an error? Or do we
> >> > complete the command as successful but without doing anything?
> >> >
> >> >
> >> > Best
> >> > Ingo
> >> >
> >> > On 13.06.22 05:50, godfrey he wrote:
> >> > > Hi Ingo,
> >> > >
> >> > > Thanks for the inputs.
> >> > >
> >> > > I think converting `ANALYZE TABLE` to `SELECT` statement is
> >> > > more generic approach. Because query plan optimization is more
> generic,
> >> > >   we can provide more optimization rules to optimize not only
> `SELECT`
> >> statement
> >> > > converted from `ANALYZE TABLE` but also the `SELECT` statement
> written
> >> by users.
> >> > >
> >> > >> JDBC connector can get a row count estimate without performing a
> >> > >> SELECT COUNT(1)
> >> > > To optimize such cases, we can implement a rule to push aggregate
> into
> >> > > table source.
> >> > > Currently, there is a similar rule: SupportsAggregatePushDown, which
> >> > > supports only pushing
> >> > > local aggregate into source now.
> >> > >
> >> > >
> >> > > Best,
> >> > > Godfrey
> >> > >
> >> > > Ingo Bürk  于2022年6月10日周五 17:15写道：
> >> > >>
> >> > >> Hi Godfrey,
> >> > >>
> >> > >> compared to the solution proposed in the FLIP (using a SELECT
> >> > >> statement), I wonder if you have considered adding APIs to
> catalogs /
> >> > >> connectors to perform this task as an alternative?
> >> > >> I could imagine that for many connectors, statistics could be
> >> > >> implemented in a less expensive way by leveraging the underlying
> >> system
> >> > >> (e.g. a JDBC connector can get a row count estimate without
> >> performing a
> >> > >> SELECT COUNT(1)).
> >> > >>
> >> > >>
> >> > >> Best
> >> > >> Ingo
> >> > >>
> >> > >>
> >> > >> On 10.06.22 09:53, godfrey he wrote:
> >> > >>> Hi all,
> >> > >>>
> >> > >>> I would like to open a discussion on FLIP-240:  Introduce "ANALYZE
> >> > >>> TABLE" Syntax.
> >> > >>>
> >> > >>> As FLIP-231 mentioned, statistics are one of the most important
> >> inputs
> >> > >>> to the optimizer. Accurate and complete statistics allows the
> >> > >>> optimizer to be more powerful. "ANALYZE TABLE" syntax is a very
> >> common
> >> > >>> but effective approach to gather statistics, which is already
> >> > >>> introduced by many compute engines and databases.
> >> > >>>
> >> > >>> The main purpose of  discussion is to introduce "ANALYZE TABLE"
> >> syntax
> >> > >>> for Flink sql.
> >> > >>>
> >> > >>> You can find more details in FLIP-240 document[1]. Looking
> forward to
> >> > >>> your feedback.
> >> > >>>
> >> > >>> [1]
> >>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=217386481
> >> > >>> [2] POC: https://github.com/godfreyhe/flink/tree/FLIP-240
> >> > >>>
> >> > >>>
> >> > >>> Best,
> >> > >>> Godfrey
> >>
>

Re: [DISCUSS ] Make state.backend.incremental as true by default

[Jing Ge]

Hi,

Just want to let you know that your email is not shown in the main
discussion thread. I am not really sure why this is happening. You might
want to check if it works with the subject starting with "Re" instead of
"RE", which is the only difference I found.

Best regards,
Jing


On Mon, Jun 13, 2022 at 3:10 PM kui yuan  wrote:

> Strongly +1
>
> On 2022/06/13 09:59:17 Lihe Ma wrote:
> > Hi, Everyone,
> >
> > I would like to open a discussion on setting incremental checkpoint as
> default behavior.
> >
> > Currently, the configuration of state.backend.incremental is set as false
> by default. Incremental checkpoint has been adopted widely in industry
> community for many years , and it is also well-tested from the feedback in
> the community discussion. Incremental checkpointing is more light-weighted:
> shorter checkpoint duration, less uploaded data and less resource
> consumption.
> >
> > In terms of backward compatibility, enable incremental checkpointing
> would not make any data loss no matter restoring from a full
> checkpoint/savepoint or an incremental checkpoint.
> >
> > FLIP-193 (Snapshot ownership)[1] has been released in 1.15, incremental
> checkpoint no longer depends on a previous restored checkpoint in default
> NO_CLAIM mode, which makes the checkpoint lineage much cleaner, it is a
> good chance to change the configuration state.backend.incremental to true
> as default.
> >
> > Thus, based on the above discussion, I suggest to make
> state.backend.incremental as true by default. What do you think of this
> proposal?
> >
> > [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-193%3A+Snapshots+ownership
> >
> > Best regards,
> > Lihe Ma
> >
>

[jira] [Created] (FLINK-28038) RocksDB rescaling improvement & rescaling benchmark - umbrella ticket

[Yanfei Lei (Jira)]

Yanfei Lei created FLINK-28038:
--

 Summary: RocksDB rescaling improvement & rescaling benchmark - 
umbrella ticket
 Key: FLINK-28038
 URL: https://issues.apache.org/jira/browse/FLINK-28038
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / State Backends
Affects Versions: 1.16.0
Reporter: Yanfei Lei


Placeholder umbrella ticket for RocksDB rescaling improvement.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Created] (FLINK-28039) Flink Table Store Ecosystem: Compute Engine Readers

[Jingsong Lee (Jira)]

Jingsong Lee created FLINK-28039:


 Summary: Flink Table Store Ecosystem: Compute Engine Readers
 Key: FLINK-28039
 URL: https://issues.apache.org/jira/browse/FLINK-28039
 Project: Flink
  Issue Type: New Feature
  Components: Table Store
Reporter: Jingsong Lee
 Fix For: table-store-0.2.0


After some refactor, we have a stable connector interfaces, we can develop 
compute engine connectors in a controlled cost.

The most classic scenario is that write by Flink and read by other engines.

We can have Readers for Apache Hive, Apache Spark and Trino.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Created] (FLINK-28040) Introduce Trino reader for table store

[Jingsong Lee (Jira)]

Jingsong Lee created FLINK-28040:


 Summary: Introduce Trino reader for table store
 Key: FLINK-28040
 URL: https://issues.apache.org/jira/browse/FLINK-28040
 Project: Flink
  Issue Type: Sub-task
  Components: Table Store
Reporter: Jingsong Lee
 Fix For: table-store-0.2.0


Can refer to FLINK-27947 to write a Trino reader.

See https://trino.io/



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Re: [VOTE] FLIP-228: Support Within between events in CEP Pattern

[yue ma]

Thanks for Nicholas driving this.
+1

Nicholas  于2022年6月13日周一 10:16写道：

> Hi everyone，
>
>
>
>
> Thanks for feedback for FLIP-228: Support Within between events in CEP
> Pattern[1] on the discussion thread[2]. I'd like to start a VOTE thread for
> FLIP-228.
>
> The vote will be open for at least 72 hours unless there is an objection
> or not enough votes.
>
>
>
>
> Regards,
>
> Nicholas Jiang
>
>
>
>
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-228%3A+Support+Within+between+events+in+CEP+Pattern
>
> [2] https://lists.apache.org/thread/p60ctx213f8921rgklk5f0b6xfrs8ksz