Re: [Discuss] FLIP-366: Support standard YAML for FLINK configuration

[Jane Chan]

Hi, Junrui,

Sorry for the late reply. The update looks good to me and thanks for your
effort!

Best,
Jane

On Fri, Sep 22, 2023 at 2:25 PM Yuxin Tan  wrote:

> Hi, Junrui
>
> +1 for the proposal.
> Thanks for your effort.
>
> Best,
> Yuxin
>
>
> Samrat Deb  于2023年9月22日周五 13:23写道：
>
> > Hello Junrui,
> >
> > +1 for the proposal.
> >
> >
> > Bests,
> > Samrat
> >
> > On Fri, Sep 22, 2023 at 10:18 AM Shammon FY  wrote:
> >
> > > +1 for the proposal, thanks for driving.
> > >
> > > Bet,
> > > Shammon FY
> > >
> > > On Fri, Sep 22, 2023 at 12:41 PM Yangze Guo 
> wrote:
> > >
> > > > Thanks for driving this, +1 for the proposal.
> > > >
> > > > Best,
> > > > Yangze Guo
> > > >
> > > >
> > > > On Fri, Sep 22, 2023 at 11:59 AM Lijie Wang <
> wangdachui9...@gmail.com>
> > > > wrote:
> > > > >
> > > > > Hi Junrui,
> > > > >
> > > > > +1 for this proposal, thanks for driving.
> > > > >
> > > > > Best,
> > > > > Lijie
> > > > >
> > > > > ConradJam  于2023年9月22日周五 10:07写道：
> > > > >
> > > > > > +1 Support for standard YAML format facilitates specification
> > > > > >
> > > > > > Jing Ge  于2023年9月22日周五 02:23写道：
> > > > > >
> > > > > > > Hi Junrui,
> > > > > > >
> > > > > > > +1 for following the standard. Thanks for your effort!
> > > > > > >
> > > > > > > Best regards,
> > > > > > > Jing
> > > > > > >
> > > > > > > On Thu, Sep 21, 2023 at 5:09 AM Junrui Lee <
> jrlee@gmail.com>
> > > > wrote:
> > > > > > >
> > > > > > > > Hi Jane,
> > > > > > > >
> > > > > > > > Thank you for your valuable feedback and suggestions.
> > > > > > > > I agree with your point about differentiating between
> > > > > > "flink-config.yaml"
> > > > > > > > and "flink-conf.yaml" to determine the standard syntax at a
> > > glance.
> > > > > > > >
> > > > > > > > While I understand your suggestion of using
> > > > "flink-conf-default.yaml"
> > > > > > to
> > > > > > > > represent the default YAML file for Flink 1.x, I have been
> > > > considering
> > > > > > > > the option of using "flink-configuration.yaml" as the file
> name
> > > > for the
> > > > > > > > new configuration file.
> > > > > > > > This name "flink-configuration.yaml" provides a clear
> > distinction
> > > > > > between
> > > > > > > > the new and old configuration files based on their names, and
> > it
> > > > does
> > > > > > not
> > > > > > > > introduce any additional semantics. Moreover, this name
> > > > > > > > "flink-configuration.yaml" can continue to be used in future
> > > > versions
> > > > > > > > FLINK-2.0.
> > > > > > > >
> > > > > > > > WDYT? If we can reach a consensus on this, I will update the
> > FLIP
> > > > > > > > documentation
> > > > > > > > accordingly.
> > > > > > > >
> > > > > > > > Best regards,
> > > > > > > > Junrui
> > > > > > > >
> > > > > > > > Jane Chan  于2023年9月20日周三 23:38写道：
> > > > > > > >
> > > > > > > > > Hi Junrui,
> > > > > > > > >
> > > > > > > > > Thanks for driving this FLIP. +1 for adoption of the
> standard
> > > > YAML
> > > > > > > > syntax.
> > > > > > > > > I just have one minor suggestion. It's a little bit
> > challenging
> > > > to
> > > > > > > > > differentiate between `flink-config.yaml` and
> > `flink-conf.yaml`
> > > > to
> > > > > > > > > determine which one uses the standard syntax at a glance.
> How
> > > > about
> > > > > > > > > using `flink-conf-default.yaml` to represent the default
> yaml
> > > > file
> > > > > > for
> > > > > > > > > Flink 1.x?
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Jane
> > > > > > > > >
> > > > > > > > > On Wed, Sep 20, 2023 at 11:06 AM Junrui Lee <
> > > jrlee@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi devs,
> > > > > > > > > >
> > > > > > > > > > I would like to start a discussion about FLIP-366:
> > > > > > > > > > Support standard YAML for FLINK configuration[1]
> > > > > > > > > >
> > > > > > > > > > The current flink-conf.yaml parser in FLINK is not a
> > standard
> > > > YAML
> > > > > > > > > parser,
> > > > > > > > > > which has some shortcomings.
> > > > > > > > > > Firstly, it does not support nested structure
> configuration
> > > > items
> > > > > > and
> > > > > > > > > only
> > > > > > > > > > supports key-value pairs, resulting in poor readability.
> > > > Secondly,
> > > > > > if
> > > > > > > > the
> > > > > > > > > > value is a collection type, such as a List or Map, users
> > are
> > > > > > required
> > > > > > > > to
> > > > > > > > > > write the value in a FLINK-specific pattern, which is
> > > > inconvenient
> > > > > > to
> > > > > > > > > use.
> > > > > > > > > > Additionally, the parser of FLINK has some differences in
> > > > syntax
> > > > > > > > compared
> > > > > > > > > > to the standard YAML parser, such as the syntax for
> parsing
> > > > > > comments
> > > > > > > > and
> > > > > > > > > > null values. These inconsistencies can cause confusion
> for
> > > > users,
> > > > > > as
> > > > > > > > seen
> > > > > > > > > > in FLINK-15358 and FLINK-32740.
> > > > > > > > > >
> > > > > > > > > > By 

Re: [DISCUSS] FLIP-367: Support Setting Parallelism for Table/SQL Sources

[Jane Chan]

Hi Zhanghao,

Thanks for the update; +1 for the proposal!

Best,
Jane

On Fri, Sep 22, 2023 at 2:13 PM Chen Zhanghao 
wrote:

> Hi Jane,
>
> Thanks for the suggestions and totally agree with them. I've updated the
> FLIP with the following two changes:
>
> 1. ​Rename WrapperTransformation to SourceTransformationWrapper that wraps
> a SourceTransformation only. Note that we do not plan to support the legacy
> LegacySourceTransformation.
> 2. Choosing the partitioner after the source will be based on the
> changelog mode of the source + the existence of the primary key in source
> schema. If the source will produce update/delete message but a primary key
> does not exist, an exception will be thrown.
>
> Best,
> Zhanghao Chen
> 
> 发件人: Jane Chan 
> 发送时间: 2023年9月20日 15:13
> 收件人: dev@flink.apache.org 
> 主题: Re: [DISCUSS] FLIP-367: Support Setting Parallelism for Table/SQL
> Sources
>
> Hi Zhanghao,
>
> Thanks for the update. The FLIP now looks good to me in general, and I have
> two minor comments.
>
> 1. Compared with other subclasses like `CacheTransformation` or
> `PartitionTransformation`, the name  `WrapperTransformation` seems too
> general. What about `SourceTransformationWrapper`, which is more specific
> and descriptive, WDYT?
>
> 2.
>
> > When the source generates update and delete data (determined by checking
> > the existence of a primary key in the source schema), the source will use
> > hash partitioner to send data.
>
>
> It might not be sufficient to determine whether the source is a CDC source
> solely based on checking the existence of the primary key. It's better to
> check the changelog mode of the source. On the other hand, adding the hash
> partitioner requires the CDC source table to declare the primary key in the
> DDL. Therefore, it is preferable to explain this restriction in the FLIP
> and doc and throw a meaningful exception when users want to configure a
> different parallelism for a CDC source but forget to declare the primary
> key constraint.
>
> Best,
> Jane
>
> On Wed, Sep 20, 2023 at 9:20 AM Benchao Li  wrote:
>
> > Thank you for the update, the FLIP now looks good to me.
> >
> > Chen Zhanghao  于2023年9月19日周二 22:50写道：
> > >
> > > Thanks to everyone for the valuable inputs, we learnt a lot during the
> > discussion. We've updated the FLIP in three main aspects based on the
> > discussion here:
> > >
> > > - Add a new subsection on keeping downstream operators' parallelism
> > unchanged by wrapping the source transformation in a phantom
> transformation.
> > > - Add a new subsection on how to deal with changelog messages, simply
> > put, build a hash partitioner based on the primary key when a source
> > generates update/delete data.
> > > - Update the non-goals section to remove the possibly misleading
> > statement that setting parallelism for individual operators lacks public
> > interest and state that we leave it for future work due to its extra
> > complexity.
> > >
> > > Looking forward to your suggestions.
> > >
> > > Best,
> > > Zhanghao Chen
> > > 
> > > 发件人: Feng Jin 
> > > 发送时间: 2023年9月17日 0:56
> > > 收件人: dev@flink.apache.org 
> > > 主题: Re: [DISCUSS] FLIP-367: Support Setting Parallelism for Table/SQL
> > Sources
> > >
> > > Hi, Zhanghao
> > >
> > > Thank you for proposing this FLIP, it is a very meaningful feature.
> > >
> > > I agree that currently we may only consider the parallelism setting of
> > the
> > > source itself. If we consider the parallelism setting of other
> operators,
> > > it may make the entire design more complex.
> > >
> > > Regarding the situation where the parallelism of the source is
> different
> > > from that of downstream tasks, I did not find a more detailed
> description
> > > in FLIP.
> > >
> > > By default, if the parallelism between two operators is different, the
> > > rebalance partitioner will be used.
> > > But in the SQL scenario, I believe that we should keep the behavior of
> > > parallelism setting consistent with that of the sink.
> > >
> > > 1. When the source only generates insert-only data, if there is a
> > mismatch
> > > in parallelism between the source and downstream operators, rebalance
> is
> > > used by default.
> > >
> > > 2. When the source generates update and delete data, we should require
> > the
> > > source to configure a primary key and then build a hash partitioner
> based
> > > on that primary key.
> > >
> > > WDYT ？
> > >
> > >
> > > Best,
> > > Feng
> > >
> > >
> > > On Sat, Sep 16, 2023 at 5:58 PM Jane Chan 
> wrote:
> > >
> > > > Hi Zhanghao,
> > > >
> > > > Thanks for the explanation.
> > > >
> > > > For Q1, I think the key lies in determining the boundary where the
> > chain
> > > > should be broken. However, this boundary is ultimately determined by
> > the
> > > > specific requirements of each user query.
> > > >
> > > > The most straightforward approach is breaking the chain after the
> > source
> > > > operator, even though it involv

回复: [DISCUSS] FLIP-367: Support Setting Parallelism for Table/SQL Sources

[Chen Zhanghao]

Thanks to everyone who participated in the discussion here. If no further 
questions/concerns are raised, we'll start voting next Monday afternoon (GMT+8).

Best,
Zhanghao Chen

发件人: Jane Chan 
发送时间: 2023年9月22日 15:35
收件人: dev@flink.apache.org 
主题: Re: [DISCUSS] FLIP-367: Support Setting Parallelism for Table/SQL Sources

Hi Zhanghao,

Thanks for the update; +1 for the proposal!

Best,
Jane

On Fri, Sep 22, 2023 at 2:13 PM Chen Zhanghao 
wrote:

> Hi Jane,
>
> Thanks for the suggestions and totally agree with them. I've updated the
> FLIP with the following two changes:
>
> 1. ​Rename WrapperTransformation to SourceTransformationWrapper that wraps
> a SourceTransformation only. Note that we do not plan to support the legacy
> LegacySourceTransformation.
> 2. Choosing the partitioner after the source will be based on the
> changelog mode of the source + the existence of the primary key in source
> schema. If the source will produce update/delete message but a primary key
> does not exist, an exception will be thrown.
>
> Best,
> Zhanghao Chen
> 
> 发件人: Jane Chan 
> 发送时间: 2023年9月20日 15:13
> 收件人: dev@flink.apache.org 
> 主题: Re: [DISCUSS] FLIP-367: Support Setting Parallelism for Table/SQL
> Sources
>
> Hi Zhanghao,
>
> Thanks for the update. The FLIP now looks good to me in general, and I have
> two minor comments.
>
> 1. Compared with other subclasses like `CacheTransformation` or
> `PartitionTransformation`, the name  `WrapperTransformation` seems too
> general. What about `SourceTransformationWrapper`, which is more specific
> and descriptive, WDYT?
>
> 2.
>
> > When the source generates update and delete data (determined by checking
> > the existence of a primary key in the source schema), the source will use
> > hash partitioner to send data.
>
>
> It might not be sufficient to determine whether the source is a CDC source
> solely based on checking the existence of the primary key. It's better to
> check the changelog mode of the source. On the other hand, adding the hash
> partitioner requires the CDC source table to declare the primary key in the
> DDL. Therefore, it is preferable to explain this restriction in the FLIP
> and doc and throw a meaningful exception when users want to configure a
> different parallelism for a CDC source but forget to declare the primary
> key constraint.
>
> Best,
> Jane
>
> On Wed, Sep 20, 2023 at 9:20 AM Benchao Li  wrote:
>
> > Thank you for the update, the FLIP now looks good to me.
> >
> > Chen Zhanghao  于2023年9月19日周二 22:50写道：
> > >
> > > Thanks to everyone for the valuable inputs, we learnt a lot during the
> > discussion. We've updated the FLIP in three main aspects based on the
> > discussion here:
> > >
> > > - Add a new subsection on keeping downstream operators' parallelism
> > unchanged by wrapping the source transformation in a phantom
> transformation.
> > > - Add a new subsection on how to deal with changelog messages, simply
> > put, build a hash partitioner based on the primary key when a source
> > generates update/delete data.
> > > - Update the non-goals section to remove the possibly misleading
> > statement that setting parallelism for individual operators lacks public
> > interest and state that we leave it for future work due to its extra
> > complexity.
> > >
> > > Looking forward to your suggestions.
> > >
> > > Best,
> > > Zhanghao Chen
> > > 
> > > 发件人: Feng Jin 
> > > 发送时间: 2023年9月17日 0:56
> > > 收件人: dev@flink.apache.org 
> > > 主题: Re: [DISCUSS] FLIP-367: Support Setting Parallelism for Table/SQL
> > Sources
> > >
> > > Hi, Zhanghao
> > >
> > > Thank you for proposing this FLIP, it is a very meaningful feature.
> > >
> > > I agree that currently we may only consider the parallelism setting of
> > the
> > > source itself. If we consider the parallelism setting of other
> operators,
> > > it may make the entire design more complex.
> > >
> > > Regarding the situation where the parallelism of the source is
> different
> > > from that of downstream tasks, I did not find a more detailed
> description
> > > in FLIP.
> > >
> > > By default, if the parallelism between two operators is different, the
> > > rebalance partitioner will be used.
> > > But in the SQL scenario, I believe that we should keep the behavior of
> > > parallelism setting consistent with that of the sink.
> > >
> > > 1. When the source only generates insert-only data, if there is a
> > mismatch
> > > in parallelism between the source and downstream operators, rebalance
> is
> > > used by default.
> > >
> > > 2. When the source generates update and delete data, we should require
> > the
> > > source to configure a primary key and then build a hash partitioner
> based
> > > on that primary key.
> > >
> > > WDYT ？
> > >
> > >
> > > Best,
> > > Feng
> > >
> > >
> > > On Sat, Sep 16, 2023 at 5:58 PM Jane Chan 
> wrote:
> > >
> > > > Hi Zhanghao,
> > > >
> > > > Thanks for the explanation.
> > >

Re: [DISCUSS] FLIP-356: Support Nested Fields Filter Pushdown

[Martijn Visser]

Sure thing, I've made the necessary changes. Thnx for clarifying

On Thu, Sep 21, 2023 at 8:24 PM Venkatakrishnan Sowrirajan
 wrote:
>
> Got it, Martijn.
>
> Unfortunately, I don't have edit access to the already created JIRA -
> FLINK-20767 . If you can
> remove the task from the EPIC FLINK-16987 FLIP-95: Add new table source and
> sink interfaces , can
> you please change it?
>
> If not, I can open a new ticket, close this one and link the 2 tickets as
> duplicated by.
>
> Regards
> Venkata krishnan
>
>
> On Thu, Sep 21, 2023 at 12:40 AM Martijn Visser 
> wrote:
>
> > Hi Venkatakrishnan,
> >
> > The reason why I thought it's abandoned because the Jira ticket is
> > part of the umbrella ticket for FLIP-95. Let's move the Jira ticket to
> > its own dedicated task instead of nested to a FLIP-95 ticket.
> >
> > Thanks,
> >
> > Martijn
> >
> > On Wed, Sep 20, 2023 at 4:34 PM Becket Qin  wrote:
> > >
> > > Hi Martijn,
> > >
> > > This FLIP has passed voting[1]. It is a modification on top of the
> > FLIP-95
> > > interface.
> > >
> > > Thanks,
> > >
> > > Jiangjie (Becket) Qin
> > >
> > > [1]
> > https://urldefense.com/v3/__https://lists.apache.org/thread/hysv9y1f48gtpr5vx3x40wtjb6cp9ky6__;!!IKRxdwAv5BmarQ!f6gg5kDGU-9tqM2rGpbK81L_7sOscG4aVKSjp0RchK1pevsMz_YrhlStS1tXduiLHoxjMP8_BjpJYrQc28-X111i$
> > >
> > > On Wed, Sep 20, 2023 at 9:29 PM Martijn Visser  > >
> > > wrote:
> > >
> > > > For clarity purposes, this FLIP is being abandoned because it was part
> > > > of FLIP-95?
> > > >
> > > > On Thu, Sep 7, 2023 at 3:01 AM Venkatakrishnan Sowrirajan
> > > >  wrote:
> > > > >
> > > > > Hi everyone,
> > > > >
> > > > > Posted a PR (
> > https://urldefense.com/v3/__https://github.com/apache/flink/pull/23313__;!!IKRxdwAv5BmarQ!f6gg5kDGU-9tqM2rGpbK81L_7sOscG4aVKSjp0RchK1pevsMz_YrhlStS1tXduiLHoxjMP8_BjpJYrQc2-0vM_Ac$
> > ) to add nested
> > > > > fields filter pushdown. Please review. Thanks.
> > > > >
> > > > > Regards
> > > > > Venkata krishnan
> > > > >
> > > > >
> > > > > On Tue, Sep 5, 2023 at 10:04 PM Venkatakrishnan Sowrirajan <
> > > > vsowr...@asu.edu>
> > > > > wrote:
> > > > >
> > > > > > Based on an offline discussion with Becket Qin, I added
> > *fieldIndices *
> > > > > > back which is the field index of the nested field at every level to
> > > > the *NestedFieldReferenceExpression
> > > > > > *in FLIP-356
> > > > > > <
> > > >
> > https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/FLINK/FLIP-356*3A*Support*Nested*Fields*Filter*Pushdown__;JSsrKysr!!IKRxdwAv5BmarQ!f6gg5kDGU-9tqM2rGpbK81L_7sOscG4aVKSjp0RchK1pevsMz_YrhlStS1tXduiLHoxjMP8_BjpJYrQc27ttpcO0$
> > > > >
> > > > > > *. *2 reasons to do it:
> > > > > >
> > > > > > 1. Agree with using *fieldIndices *as the only contract to refer
> > to the
> > > > > > column from the underlying datasource.
> > > > > > 2. To keep it consistent with *FieldReferenceExpression*
> > > > > >
> > > > > > Having said that, I see that with *projection pushdown, *index of
> > the
> > > > > > fields are used whereas with *filter pushdown (*based on scanning
> > few
> > > > > > tablesources) *FieldReferenceExpression*'s name is used for eg:
> > even in
> > > > > > the Flink's *FileSystemTableSource, IcebergSource, JDBCDatsource*.
> > This
> > > > > > way, I feel the contract is not quite clear and explicit. Wanted to
> > > > > > understand other's thoughts as well.
> > > > > >
> > > > > > Regards
> > > > > > Venkata krishnan
> > > > > >
> > > > > >
> > > > > > On Tue, Sep 5, 2023 at 5:34 PM Becket Qin 
> > > > wrote:
> > > > > >
> > > > > >> Hi Venkata,
> > > > > >>
> > > > > >>
> > > > > >> > Also I made minor changes to the
> > *NestedFieldReferenceExpression,
> > > > > >> *instead
> > > > > >> > of *fieldIndexArray* we can just do away with *fieldNames *array
> > > > that
> > > > > >> > includes fieldName at every level for the nested field.
> > > > > >>
> > > > > >>
> > > > > >> I don't think keeping only the field names array would work. At
> > the
> > > > end of
> > > > > >> the day, the contract between Flink SQL and the connectors is
> > based
> > > > on the
> > > > > >> indexes, not the names. Technically speaking, the connectors only
> > > > emit a
> > > > > >> bunch of RowData which is based on positions. The field names are
> > > > added by
> > > > > >> the SQL framework via the DDL for those RowData. In this sense,
> > the
> > > > > >> connectors may not be aware of the field names in Flink DDL at
> > all.
> > > > The
> > > > > >> common language between Flink SQL and source is just positions.
> > This
> > > > is
> > > > > >> also why ProjectionPushDown would work by only relying on the
> > > > indexes, not
> > > > > >> the field names. So I think the field index array is a must have
> > here
> > > > in
> > > > > >> the NestedFieldReferenceExpression.
> > > > > >>
> > > > > >> Thanks,
> > > > > >>
> > > > > >> Jiangjie (Becket) Qin
> > > > > >>
>

Re: [VOTE] FLIP-327: Support switching from batch to stream mode to improve throughput when processing backlog data

[Jing Ge]

+1(binding) Thanks!

Best regards,
Jing

On Fri, Sep 22, 2023 at 8:08 AM Dong Lin  wrote:

> Hi all,
>
> We would like to start the vote for FLIP-327: Support switching from batch
> to stream mode to improve throughput when processing backlog data [1]. This
> FLIP was discussed in this thread [2].
>
> The vote will be open until at least Sep 27th (at least 72
> hours), following the consensus voting process.
>
> Cheers,
> Xuannan and Dong
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-327%3A+Support+switching+from+batch+to+stream+mode+to+improve+throughput+when+processing+backlog+data
> [2] https://lists.apache.org/thread/29nvjt9sgnzvs90browb8r6ng31dcs3n
>

Re: [ANNOUNCE] Release 1.18.0, release candidate #0

[Zakelly Lan]

Hi everyone,

I want to provide an update on the benchmark results that I have been
working on. After spending some time preparing the environment and
adjusting the benchmark script, I finally got a comparison between
release 1.18 (commit: 2aeb99804ba[1]) and the commit before the old
codespeed server went down (commit: 6d62f9918ea[2]) on openjdk8. The
report is attached[3]. Note that the test has only run once on jdk8,
so the impact of single-test fluctuations is not ruled out.
Additionally, I have noticed some significant fluctuations in specific
tests when reviewing previous benchmark scores, which I have also
noted in the report. Taking all of these factors into consideration, I
think there is no obvious regression in release 1.18 *for now*. More
tests including the one on openjdk11 are on the way. Hope it does not
delay the release procedure.

Please let me know if you have any concerns.


Best,
Zakelly

[1] 
https://github.com/apache/flink/commit/2aeb99804ba56c008df0a1730f3246d3fea856b9
[2] 
https://github.com/apache/flink/commit/6d62f9918ea2cbb8a10c705a25a4ff6deab60711
[3] 
https://docs.google.com/spreadsheets/d/1-3Y974jYq_WrQNzLN-y_6lOU-NGXaIDTQBYTZd04tJ0/edit?usp=sharing

The new environment for benchmark:
ECS on Aliyun
CPU: Intel(R) Xeon(R) Platinum 8163 CPU @ 2.50GHz (8 Core available)
Memory: 64GB
OS: Alibaba Cloud Linux 3.2104 LTS 64bit
Kernel: 5.10.134-15.al8.x86_64
OpenJDK8 version: 1.8.0_372

On Thu, Sep 21, 2023 at 12:04 PM Yuxin Tan  wrote:
>
> Hi, Zakelly,
>
> No benchmark tests currently are affected by this issue. We
> may add benchmarks to guard it later. Thanks.
>
> Best,
> Yuxin
>
>
> Zakelly Lan  于2023年9月21日周四 11:56写道：
>
> > Hi Jing,
> >
> > Sure, I will run the benchmark with this fix.
> >
> > Hi Yunxin,
> >
> > I'm not familiar with the hybrid shuffle. Is there any specific
> > benchmark test that may be affected by this issue? I will pay special
> > attention to it.
> > Thanks.
> >
> >
> > Best,
> > Zakelly
> >
> > On Thu, Sep 21, 2023 at 10:08 AM Yuxin Tan  wrote:
> > >
> > > Hi, Jing, Qingsheng,
> > >
> > > Thanks a lot.
> > > The fix has been backported.
> > >
> > > Best,
> > > Yuxin
> > >
> > >
> > > Jing Ge  于2023年9月21日周四 00:42写道：
> > >
> > > > Hi Lijie,
> > > >
> > > > Thanks for reaching out. Please backport it to release-1.18.
> > > >
> > > > Best regards,
> > > > Jing
> > > >
> > > > On Wed, Sep 20, 2023 at 4:35 PM Lijie Wang 
> > > > wrote:
> > > >
> > > > > Hi community and release managers:
> > > > >
> > > > > We found a critical bug[1] of the rest client a few days ago, which
> > may
> > > > > cause the inode to be used up. Now the fix-PR[2] is ready for
> > merging, I
> > > > > hope to backport it to release-1.18.
> > > > >
> > > > > Please let me know if you have any concerns. Thanks.
> > > > >
> > > > > [1] https://issues.apache.org/jira/browse/FLINK-32974
> > > > > [2] https://github.com/apache/flink/pull/23363
> > > > >
> > > > > Best,
> > > > > Lijie
> > > > >
> > > > > Zakelly Lan  于2023年9月19日周二 17:26写道：
> > > > >
> > > > > > Hi Yuan and Jing,
> > > > > >
> > > > > > Thank you for sharing your thoughts. I completely agree that it is
> > our
> > > > > > top priority to ensure that there are no regressions from the last
> > > > > > commit the previous benchmark pipeline covered to the final commit
> > of
> > > > > > this release. I will try to get this result first.
> > > > > >
> > > > > >
> > > > > > Best,
> > > > > > Zakelly
> > > > > >
> > > > > > On Tue, Sep 19, 2023 at 4:55 PM Jing Ge  > >
> > > > > > wrote:
> > > > > > >
> > > > > > > Hi
> > > > > > >
> > > > > > > Thanks Zakelly and Yuan for your effort and update. Since we
> > changed
> > > > > the
> > > > > > > hardware, IMHO, if we are able to reach a consensus in the
> > community
> > > > > that
> > > > > > > there is no regression with the benchmarks, we could consider
> > > > releasing
> > > > > > rc1
> > > > > > > without waiting for the new baseline scores which might take
> > days.
> > > > > > >
> > > > > > > Best regards,
> > > > > > > Jing
> > > > > > >
> > > > > > > On Tue, Sep 19, 2023 at 10:42 AM Yuan Mei <
> > yuanmei.w...@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > > Hey Zakelly,
> > > > > > > >
> > > > > > > > Thanks very much for the efforts to re-build the entire
> > benchmark
> > > > > > > > environment.
> > > > > > > >
> > > > > > > > As long as we have
> > > > > > > > 1) the pipeline set up and ready (no need for the entire portal
> > > > > ready),
> > > > > > > > 2) get benchmark comparison numbers (comparing with the commit
> > just
> > > > > > before
> > > > > > > > the benchmark pipeline is down) and
> > > > > > > > 3) confirmed no-regression, it should be good enough.
> > > > > > > >
> > > > > > > > Thanks again!
> > > > > > > >
> > > > > > > > Best
> > > > > > > > Yuan
> > > > > > > >
> > > > > > > > On Tue, Sep 19, 2023 at 4:26 PM Zakelly Lan <
> > zakelly@gmail.com
> > > > >
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi everyone,
> > > > > > > > >

Re: [ANNOUNCE] Release 1.18.0, release candidate #0

[Jing Ge]

Hi Zakelly,

Thanks for your effort and the update! Since Java 8 has been deprecated[1],
let's wait for the result with Java 11. It should be available after the
weekend and there should be no big surprise. WDYT?

Best regards,
Jing

[1]
https://nightlies.apache.org/flink/flink-docs-master/release-notes/flink-1.15/#jdk-upgrade

On Fri, Sep 22, 2023 at 11:26 AM Zakelly Lan  wrote:

> Hi everyone,
>
> I want to provide an update on the benchmark results that I have been
> working on. After spending some time preparing the environment and
> adjusting the benchmark script, I finally got a comparison between
> release 1.18 (commit: 2aeb99804ba[1]) and the commit before the old
> codespeed server went down (commit: 6d62f9918ea[2]) on openjdk8. The
> report is attached[3]. Note that the test has only run once on jdk8,
> so the impact of single-test fluctuations is not ruled out.
> Additionally, I have noticed some significant fluctuations in specific
> tests when reviewing previous benchmark scores, which I have also
> noted in the report. Taking all of these factors into consideration, I
> think there is no obvious regression in release 1.18 *for now*. More
> tests including the one on openjdk11 are on the way. Hope it does not
> delay the release procedure.
>
> Please let me know if you have any concerns.
>
>
> Best,
> Zakelly
>
> [1]
> https://github.com/apache/flink/commit/2aeb99804ba56c008df0a1730f3246d3fea856b9
> [2]
> https://github.com/apache/flink/commit/6d62f9918ea2cbb8a10c705a25a4ff6deab60711
> [3]
> https://docs.google.com/spreadsheets/d/1-3Y974jYq_WrQNzLN-y_6lOU-NGXaIDTQBYTZd04tJ0/edit?usp=sharing
>
> The new environment for benchmark:
> ECS on Aliyun
> CPU: Intel(R) Xeon(R) Platinum 8163 CPU @ 2.50GHz (8 Core available)
> Memory: 64GB
> OS: Alibaba Cloud Linux 3.2104 LTS 64bit
> Kernel: 5.10.134-15.al8.x86_64
> OpenJDK8 version: 1.8.0_372
>
> On Thu, Sep 21, 2023 at 12:04 PM Yuxin Tan  wrote:
> >
> > Hi, Zakelly,
> >
> > No benchmark tests currently are affected by this issue. We
> > may add benchmarks to guard it later. Thanks.
> >
> > Best,
> > Yuxin
> >
> >
> > Zakelly Lan  于2023年9月21日周四 11:56写道：
> >
> > > Hi Jing,
> > >
> > > Sure, I will run the benchmark with this fix.
> > >
> > > Hi Yunxin,
> > >
> > > I'm not familiar with the hybrid shuffle. Is there any specific
> > > benchmark test that may be affected by this issue? I will pay special
> > > attention to it.
> > > Thanks.
> > >
> > >
> > > Best,
> > > Zakelly
> > >
> > > On Thu, Sep 21, 2023 at 10:08 AM Yuxin Tan 
> wrote:
> > > >
> > > > Hi, Jing, Qingsheng,
> > > >
> > > > Thanks a lot.
> > > > The fix has been backported.
> > > >
> > > > Best,
> > > > Yuxin
> > > >
> > > >
> > > > Jing Ge  于2023年9月21日周四 00:42写道：
> > > >
> > > > > Hi Lijie,
> > > > >
> > > > > Thanks for reaching out. Please backport it to release-1.18.
> > > > >
> > > > > Best regards,
> > > > > Jing
> > > > >
> > > > > On Wed, Sep 20, 2023 at 4:35 PM Lijie Wang <
> wangdachui9...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi community and release managers:
> > > > > >
> > > > > > We found a critical bug[1] of the rest client a few days ago,
> which
> > > may
> > > > > > cause the inode to be used up. Now the fix-PR[2] is ready for
> > > merging, I
> > > > > > hope to backport it to release-1.18.
> > > > > >
> > > > > > Please let me know if you have any concerns. Thanks.
> > > > > >
> > > > > > [1] https://issues.apache.org/jira/browse/FLINK-32974
> > > > > > [2] https://github.com/apache/flink/pull/23363
> > > > > >
> > > > > > Best,
> > > > > > Lijie
> > > > > >
> > > > > > Zakelly Lan  于2023年9月19日周二 17:26写道：
> > > > > >
> > > > > > > Hi Yuan and Jing,
> > > > > > >
> > > > > > > Thank you for sharing your thoughts. I completely agree that
> it is
> > > our
> > > > > > > top priority to ensure that there are no regressions from the
> last
> > > > > > > commit the previous benchmark pipeline covered to the final
> commit
> > > of
> > > > > > > this release. I will try to get this result first.
> > > > > > >
> > > > > > >
> > > > > > > Best,
> > > > > > > Zakelly
> > > > > > >
> > > > > > > On Tue, Sep 19, 2023 at 4:55 PM Jing Ge
>  > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > Hi
> > > > > > > >
> > > > > > > > Thanks Zakelly and Yuan for your effort and update. Since we
> > > changed
> > > > > > the
> > > > > > > > hardware, IMHO, if we are able to reach a consensus in the
> > > community
> > > > > > that
> > > > > > > > there is no regression with the benchmarks, we could consider
> > > > > releasing
> > > > > > > rc1
> > > > > > > > without waiting for the new baseline scores which might take
> > > days.
> > > > > > > >
> > > > > > > > Best regards,
> > > > > > > > Jing
> > > > > > > >
> > > > > > > > On Tue, Sep 19, 2023 at 10:42 AM Yuan Mei <
> > > yuanmei.w...@gmail.com>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hey Zakelly,
> > > > > > > > >
> > > > > > > > > Thanks very much for the efforts to re-build the entire
> > > benchma

Re: [VOTE] FLIP-327: Support switching from batch to stream mode to improve throughput when processing backlog data

[Rui Fan]

+1(binding), thanks for driving this proposal.

Best,
Rui

On Fri, Sep 22, 2023 at 5:16 PM Jing Ge  wrote:

> +1(binding) Thanks!
>
> Best regards,
> Jing
>
> On Fri, Sep 22, 2023 at 8:08 AM Dong Lin  wrote:
>
> > Hi all,
> >
> > We would like to start the vote for FLIP-327: Support switching from
> batch
> > to stream mode to improve throughput when processing backlog data [1].
> This
> > FLIP was discussed in this thread [2].
> >
> > The vote will be open until at least Sep 27th (at least 72
> > hours), following the consensus voting process.
> >
> > Cheers,
> > Xuannan and Dong
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-327%3A+Support+switching+from+batch+to+stream+mode+to+improve+throughput+when+processing+backlog+data
> > [2] https://lists.apache.org/thread/29nvjt9sgnzvs90browb8r6ng31dcs3n
> >
>

Re: [ANNOUNCE] Release 1.18.0, release candidate #0

[Zakelly Lan]

Hi Jing,

I agree we could wait for the result with Java 11. And it should be
available next Monday.
Additionally, I could also build a pipeline with Java 17 later since
it is supported in 1.18[1].


Best regards,
Zakelly

[1] 
https://github.com/apache/flink/commit/9c1318ca7fa5b2e7b11827068ad1288483aaa464#diff-8310c97396d60e96766a936ca8680f1e2971ef486cfc2bc55ec9ca5a5333c47fR53

On Fri, Sep 22, 2023 at 5:57 PM Jing Ge  wrote:
>
> Hi Zakelly,
>
> Thanks for your effort and the update! Since Java 8 has been deprecated[1],
> let's wait for the result with Java 11. It should be available after the
> weekend and there should be no big surprise. WDYT?
>
> Best regards,
> Jing
>
> [1]
> https://nightlies.apache.org/flink/flink-docs-master/release-notes/flink-1.15/#jdk-upgrade
>
> On Fri, Sep 22, 2023 at 11:26 AM Zakelly Lan  wrote:
>
> > Hi everyone,
> >
> > I want to provide an update on the benchmark results that I have been
> > working on. After spending some time preparing the environment and
> > adjusting the benchmark script, I finally got a comparison between
> > release 1.18 (commit: 2aeb99804ba[1]) and the commit before the old
> > codespeed server went down (commit: 6d62f9918ea[2]) on openjdk8. The
> > report is attached[3]. Note that the test has only run once on jdk8,
> > so the impact of single-test fluctuations is not ruled out.
> > Additionally, I have noticed some significant fluctuations in specific
> > tests when reviewing previous benchmark scores, which I have also
> > noted in the report. Taking all of these factors into consideration, I
> > think there is no obvious regression in release 1.18 *for now*. More
> > tests including the one on openjdk11 are on the way. Hope it does not
> > delay the release procedure.
> >
> > Please let me know if you have any concerns.
> >
> >
> > Best,
> > Zakelly
> >
> > [1]
> > https://github.com/apache/flink/commit/2aeb99804ba56c008df0a1730f3246d3fea856b9
> > [2]
> > https://github.com/apache/flink/commit/6d62f9918ea2cbb8a10c705a25a4ff6deab60711
> > [3]
> > https://docs.google.com/spreadsheets/d/1-3Y974jYq_WrQNzLN-y_6lOU-NGXaIDTQBYTZd04tJ0/edit?usp=sharing
> >
> > The new environment for benchmark:
> > ECS on Aliyun
> > CPU: Intel(R) Xeon(R) Platinum 8163 CPU @ 2.50GHz (8 Core available)
> > Memory: 64GB
> > OS: Alibaba Cloud Linux 3.2104 LTS 64bit
> > Kernel: 5.10.134-15.al8.x86_64
> > OpenJDK8 version: 1.8.0_372
> >
> > On Thu, Sep 21, 2023 at 12:04 PM Yuxin Tan  wrote:
> > >
> > > Hi, Zakelly,
> > >
> > > No benchmark tests currently are affected by this issue. We
> > > may add benchmarks to guard it later. Thanks.
> > >
> > > Best,
> > > Yuxin
> > >
> > >
> > > Zakelly Lan  于2023年9月21日周四 11:56写道：
> > >
> > > > Hi Jing,
> > > >
> > > > Sure, I will run the benchmark with this fix.
> > > >
> > > > Hi Yunxin,
> > > >
> > > > I'm not familiar with the hybrid shuffle. Is there any specific
> > > > benchmark test that may be affected by this issue? I will pay special
> > > > attention to it.
> > > > Thanks.
> > > >
> > > >
> > > > Best,
> > > > Zakelly
> > > >
> > > > On Thu, Sep 21, 2023 at 10:08 AM Yuxin Tan 
> > wrote:
> > > > >
> > > > > Hi, Jing, Qingsheng,
> > > > >
> > > > > Thanks a lot.
> > > > > The fix has been backported.
> > > > >
> > > > > Best,
> > > > > Yuxin
> > > > >
> > > > >
> > > > > Jing Ge  于2023年9月21日周四 00:42写道：
> > > > >
> > > > > > Hi Lijie,
> > > > > >
> > > > > > Thanks for reaching out. Please backport it to release-1.18.
> > > > > >
> > > > > > Best regards,
> > > > > > Jing
> > > > > >
> > > > > > On Wed, Sep 20, 2023 at 4:35 PM Lijie Wang <
> > wangdachui9...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi community and release managers:
> > > > > > >
> > > > > > > We found a critical bug[1] of the rest client a few days ago,
> > which
> > > > may
> > > > > > > cause the inode to be used up. Now the fix-PR[2] is ready for
> > > > merging, I
> > > > > > > hope to backport it to release-1.18.
> > > > > > >
> > > > > > > Please let me know if you have any concerns. Thanks.
> > > > > > >
> > > > > > > [1] https://issues.apache.org/jira/browse/FLINK-32974
> > > > > > > [2] https://github.com/apache/flink/pull/23363
> > > > > > >
> > > > > > > Best,
> > > > > > > Lijie
> > > > > > >
> > > > > > > Zakelly Lan  于2023年9月19日周二 17:26写道：
> > > > > > >
> > > > > > > > Hi Yuan and Jing,
> > > > > > > >
> > > > > > > > Thank you for sharing your thoughts. I completely agree that
> > it is
> > > > our
> > > > > > > > top priority to ensure that there are no regressions from the
> > last
> > > > > > > > commit the previous benchmark pipeline covered to the final
> > commit
> > > > of
> > > > > > > > this release. I will try to get this result first.
> > > > > > > >
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Zakelly
> > > > > > > >
> > > > > > > > On Tue, Sep 19, 2023 at 4:55 PM Jing Ge
> >  > > > >
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Hi
> > > > > > > > >
> > > > > > > > > Thanks Zakelly and Yuan for y

Re: [DISCUSS] FLIP-365: Introduce flush interval to adjust the interval of emitting results with idempotent semantics

[Zakelly Lan]

Hi Yunfeng,

Thank you for providing the information. Here are my opinions:

1. Could you please include the implementation details in your FLIP? I
believe it would be helpful for further discussion. Additionally, I
have a concern regarding the usage of ValueState/ListState. Would it
introduce more serialization/deserialization overhead?
2.1  I think the buffered intermediate results *themselves* are not
included in the next checkpoint based on your description. They are
cleared and subsumed in the original states right before the
checkpoint. However, the deprecated data and tombstones reside in the
LSM-tree files, which brings in unnecessary checkpoint consumption.
2.2 Based on my experience, many jobs face bottlenecks on state
backends. Mini-batching helps in reducing state access overhead,
leading to performance advantages in certain scenarios. I can see the
benefits you mentioned in the FLIP, but I am concerned about whether
it is a good choice to increase the burden on the bottleneck and use
such a resource-intensive component. I have noticed some "useless" but
performance-related design aspects in keyed state for your use case,
such as time-to-live (wrapped as part of the value), KeyedGroup
(calculating hash during key iteration, serialized as a prefix in
Rocksdb), and checkpoint (CPU/IO/network consumption). What if we
design a new buffer holder, borrowing only "useful" designs from the
state backend, would it necessarily be better than the design of the
current state backends?

I'm not very familiar with mini-batching, and since this FLIP proposed
to deprecate the mini-batching, it is better to involve more experts
to discuss this topic.


Best,
Zakelly


On Fri, Sep 22, 2023 at 9:10 AM Yunfeng Zhou
 wrote:
>
> Hi Zakelly,
>
> Thanks for your comments on this FLIP. Please let me attempt to
> clarify these points.
>
> 1. Yes, this FLIP proposes to buffer the outputs in the state backend.
> As only the latest one of each type of StreamElement is about to be
> buffered, a ValueState in keyed context or a ListState in non-keyed
> context would be enough to hold each type of StreamElement. The value
> to be stored in the ValueState/ListState would be the original
> StreamRecord/Watermark/WatermarkStatus/LatencyMarker. Besides, the
> KeyedStateBackend#applyToAllKeys method makes it possible to access
> states for all keys in one keyed context.
>
> 2.1 The buffered intermediate results need to be included in the next
> checkpoint to preserve exactly-once semantics during failover. The
> buffer would be cleared in the `flush()` operation, but `flush()` need
> not be triggered before checkpoints. I agree with it that saving
> buffered results to state would increase the workload about state
> access operations, but given that the state buffer would be enabled on
> aggregation operators which already involve states, the additional
> buffer results would not increase the time complexity of state
> accesses or the memory(state) complexity. If we could exchange one
> state read/write operation and the space of a ValueState with all
> computations in downstream operators to process one intermediate
> result, I believe the optimization to throughput would be worth the
> tradeoff in states.
>
> 2.2 Not considering checkpoints, it might still be meaningful to
> discuss the alternative solutions to store buffered results during
> runtime as proposed in your suggestions. At least for keyed streams,
> I'm concerned that saving all buffered results in memory would easily
> cause OOM problems, as there is no guarantee on the number of keyed
> states to store between a flush interval. I'm also wondering whether a
> file-based map would have better performance than state backends, and
> why Flink haven't introduced FileSystemStateBackend if file-based map
> could be better. Could you please provide more illustrations on the
> pros & cons of state backend v.s. memory/filesystem?
>
> Best regards,
> Yunfeng
>
> On Thu, Sep 21, 2023 at 4:10 PM Zakelly Lan  wrote:
> >
> > Hi Yunfeng and Dong,
> >
> > Thanks for this FLIP. I have reviewed it briefly and have a few questions:
> >
> > 1. Is this FLIP proposing to buffer the output in the state backend?
> > If so, what is the data format of this buffer (what type of state does
> > it use and what is the value)? Additionally, how does the operator
> > retrieve all the buffer data from the state backend during the
> > `flush()` operation (while the keyed states can only be accessed under
> > a keyed context)?
> > 2. Are the buffered intermediate results required to be included in
> > the next checkpoint? Or are they deleted and subsumed in the original
> > states during the `flush()` operation before triggering the
> > checkpoint? I'm asking because if they are not included in the
> > checkpoint, it may be more efficient to avoid using keyed states for
> > buffering. In this scenario, a simple heap-based or even file-based
> > map could be more efficient. Frequent writes and clears

回复: [Discuss] FLIP-366: Support standard YAML for FLINK configuration

[Chen Zhanghao]

Hi Junrui,

Thanks for driving this, +1 for it

Best,
Zhanghao Chen

发件人: Junrui Lee 
发送时间: 2023年9月20日 11:06
收件人: dev@flink.apache.org 
主题: [Discuss] FLIP-366: Support standard YAML for FLINK configuration

Hi devs,

I would like to start a discussion about FLIP-366:
Support standard YAML for FLINK configuration[1]

The current flink-conf.yaml parser in FLINK is not a standard YAML parser,
which has some shortcomings.
Firstly, it does not support nested structure configuration items and only
supports key-value pairs, resulting in poor readability. Secondly, if the
value is a collection type, such as a List or Map, users are required to
write the value in a FLINK-specific pattern, which is inconvenient to use.
Additionally, the parser of FLINK has some differences in syntax compared
to the standard YAML parser, such as the syntax for parsing comments and
null values. These inconsistencies can cause confusion for users, as seen
in FLINK-15358 and FLINK-32740.

By supporting standard YAML, these issues can be resolved, and users can
create a Flink configuration file using third-party tools and leverage
some advanced YAML features. Therefore, we propose to support standard
YAML for FLINK configuration.

You can find more details in the FLIP-366[1]. Looking forward to your
feedback.

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-366%3A+Support+standard+YAML+for+FLINK+configuration

Best,
Junrui

回复: [VOTE] FLIP-363: Unify the Representation of TaskManager Location in REST API and Web UI

[Chen Zhanghao]

Thank you all! Closing the vote. The result will be sent in a separate email.

Best,
Zhanghao Chen

发件人: Matt Wang 
发送时间: 2023年9月20日 20:54
收件人: dev@flink.apache.org 
主题: Re: [VOTE] FLIP-363: Unify the Representation of TaskManager Location in 
REST API and Web UI

+1 (non-binding)


Thanks for driving this FLIP


--

Best,
Matt Wang


 Replied Message 
| From | Weihua Hu |
| Date | 09/19/2023 19:17 |
| To |  |
| Subject | Re: [VOTE] FLIP-363: Unify the Representation of TaskManager 
Location in REST API and Web UI |
+1(binding)

Best,
Weihua


On Tue, Sep 19, 2023 at 6:16 PM Jing Ge  wrote:

+1(binding) Thanks!

Best regards,
Jing

On Tue, Sep 19, 2023 at 9:01 AM Chen Zhanghao 
wrote:

Hi Devs,

Thanks for all the feedbacks on FLIP-363: Unify the Representation of
TaskManager Location in REST API and Web UI [1][2]. Given that the
consensus on the naming issue has been reached (using "endpoint" instead
of
"location"),  I'd like to restart the vote for it. The vote will be open
for at least 72 hours (until Sep 22th 12:00 PM GMT) unless there is an
objection or insufficient votes.

[1]

https://cwiki.apache.org/confluence/display/FLINK/FLIP-363%3A+Unify+the+Representation+of+TaskManager+Location+in+REST+API+and+Web+UI
[2] https://lists.apache.org/thread/sls1196mmk25w8nm2qf585254nbjr9hd

Best,
Zhanghao Chen

发件人: Chen Zhanghao 
发送时间: 2023年9月18日 19:19
收件人: dev@flink.apache.org ; Jing Ge <
j...@ververica.com>
主题: 回复: [VOTE] FLIP-363: Unify the Representation of TaskManager Location
in REST API and Web UI

Thanks for pointing that out. Let's give it a bit more time for reaching
consensus on the naming issue and postpone the voting for now. Sorry for
the inconvenience here. Will send another email once the voting restarts.

Best,
Zhanghao Chen

发件人: Rui Fan <1996fan...@gmail.com>
发送时间: 2023年9月18日 11:55
收件人: dev@flink.apache.org ; Jing Ge <
j...@ververica.com>
主题: Re: [VOTE] FLIP-363: Unify the Representation of TaskManager Location
in REST API and Web UI

A gentle reminder about the location naming.
The naming of location is a little unclear, but
I can't think of any other better naming.

So I +1(binding) first.

Ping @Jing Ge  to help double check the name again.

Sorry for mentioning naming in the VOTE thread,
I didn't know this VOTE would be so early.

Best,
Rui

On Mon, Sep 18, 2023 at 11:44 AM Yangze Guo  wrote:

+1 (binding)

Best,
Yangze Guo

On Mon, Sep 18, 2023 at 11:37 AM Chen Zhanghao
 wrote:

Hi All,

Thanks for all the feedback on FLIP-363: Unify the Representation of
TaskManager Location in REST API and Web UI [1][2]

I'd like to start a vote for FLIP-363. The vote will be open for at
least 72 hours unless there is an objection or insufficient votes.

[1]


https://cwiki.apache.org/confluence/display/FLINK/FLIP-363%3A+Unify+the+Representation+of+TaskManager+Location+in+REST+API+and+Web+UI
[2] https://lists.apache.org/thread/sls1196mmk25w8nm2qf585254nbjr9hd

Best,
Zhanghao Chen

[RESULT][VOTE] FLIP-363: Unify the Representation of TaskManager Location in REST API and Web UI

[Chen Zhanghao]

Hi everyone,

The proposal, FLIP-363: Unify the Representation of TaskManager Location in 
REST API and Web UI, has been unanimously accepted with 5 votes (4 binding):

+1 votes:

 - Yangze Guo (binding)
 - Rui Fan (binding)
 - Jing Ge (binding)
 - Weihua Hu (binding)
 - Matt Wang

Thanks again to everyone who participated in the discussion and voting.

Best,
Zhanghao Chen

[jira] [Created] (FLINK-33128) TestValuesRuntimeFunctions$TestValuesLookupFunction does not call open() on converter

[Jerome Gagnon (Jira)]

Jerome Gagnon created FLINK-33128:
-

 Summary: TestValuesRuntimeFunctions$TestValuesLookupFunction does 
not call open() on converter
 Key: FLINK-33128
 URL: https://issues.apache.org/jira/browse/FLINK-33128
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Planner
Affects Versions: 1.16.2
Reporter: Jerome Gagnon
 Fix For: 1.16.2


When using the TestValues connector with nested Row values relying on 
`BinaryArrayWriter` we face the following issue : 
```

java.lang.NullPointerException: Cannot invoke 
"org.apache.flink.table.data.writer.BinaryArrayWriter.getNumElements()" because 
"this.reuseWriter" is null
    at 
org.apache.flink.table.data.conversion.ArrayObjectArrayConverter.allocateWriter(ArrayObjectArrayConverter.java:140)
    at 
org.apache.flink.table.data.conversion.ArrayObjectArrayConverter.toBinaryArrayData(ArrayObjectArrayConverter.java:114)
    at 
org.apache.flink.table.data.conversion.ArrayObjectArrayConverter.toInternal(ArrayObjectArrayConverter.java:93)
    at 
org.apache.flink.table.data.conversion.ArrayObjectArrayConverter.toInternal(ArrayObjectArrayConverter.java:40)
    at 
org.apache.flink.table.data.conversion.DataStructureConverter.toInternalOrNull(DataStructureConverter.java:61)
    at 
org.apache.flink.table.data.conversion.RowRowConverter.toInternal(RowRowConverter.java:90)
    at 
org.apache.flink.table.data.conversion.RowRowConverter.toInternal(RowRowConverter.java:37)
    at 
org.apache.flink.table.data.conversion.DataStructureConverter.toInternalOrNull(DataStructureConverter.java:61)
    at 
org.apache.flink.table.data.conversion.RowRowConverter.toInternal(RowRowConverter.java:90)
    at 
org.apache.flink.table.data.conversion.RowRowConverter.toInternal(RowRowConverter.java:37)
    at 
org.apache.flink.table.data.conversion.DataStructureConverter.toInternalOrNull(DataStructureConverter.java:61)
    at 
org.apache.flink.table.data.conversion.RowRowConverter.toInternal(RowRowConverter.java:75)
    at 
org.apache.flink.table.data.conversion.RowRowConverter.toInternal(RowRowConverter.java:37)
    at 
org.apache.flink.table.data.conversion.DataStructureConverter.toInternalOrNull(DataStructureConverter.java:61)
    at 
org.apache.flink.table.runtime.connector.source.DataStructureConverterWrapper.toInternal(DataStructureConverterWrapper.java:51)
    at 
org.apache.flink.table.planner.factories.TestValuesRuntimeFunctions$TestValuesLookupFunction.lambda$indexDataByKey$0(TestValuesRuntimeFunctions.java:626)
    at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
    at 
org.apache.flink.table.planner.factories.TestValuesRuntimeFunctions$TestValuesLookupFunction.indexDataByKey(TestValuesRuntimeFunctions.java:624)
    at 
org.apache.flink.table.planner.factories.TestValuesRuntimeFunctions$TestValuesLookupFunction.open(TestValuesRuntimeFunctions.java:601)
    at LookupFunction$370.open(Unknown Source)
    at 
org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:34)
    at 
org.apache.flink.table.runtime.operators.join.lookup.LookupJoinRunner.open(LookupJoinRunner.java:67)
    at 
org.apache.flink.table.runtime.operators.join.lookup.LookupJoinWithCalcRunner.open(LookupJoinWithCalcRunner.java:51)
    at 
org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:34)
    at 
org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:100)
    at 
org.apache.flink.streaming.api.operators.ProcessOperator.open(ProcessOperator.java:56)
    at 
org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.initializeStateAndOpenOperators(RegularOperatorChain.java:107)
    at 
org.apache.flink.streaming.runtime.tasks.StreamTask.restoreGates(StreamTask.java:731)
    at 
org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.call(StreamTaskActionExecutor.java:100)
    at 
org.apache.flink.streaming.runtime.tasks.StreamTask.restoreInternal(StreamTask.java:706)
    at 
org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:672)
    at 
org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:935)
    at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:904)
    at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:728)
    at org.apache.flink.runtime.taskmanager.Task.run(Task.java:550

```

 

This is happening because open() is being not called from 
TestValuesLookupFunction.open() and the underlying converter writer never gets 
initialized.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] FLIP-367: Support Setting Parallelism for Table/SQL Sources

[Lincoln Lee]

Hi Zhanghao,

Thanks for the FLIP and discussion!  Hope this reply isn't too late : )
Firstly I'm fully agreed with the motivation of this FLIP and the value for
the users, but there are a few things we should consider(please correct me
if I'm misunderstanding):

*1.  *It seems that the current solution only takes care of part of the
requirement, the need to set source's parallelism may be different in
different jobs,  for example, consider the following two job topologies(one
{} simply represents a vertex):
a. {source -> calc -> sink}

b. {source -> calc} -> {aggregate} -> {sink}

For job a, if there is a bottleneck in calc operator, but source
parallelism cannot be scaled up (e.g., limited by kafka's partition
number), so the calc operator cannot be scaled up to achieve higher
throughput because the operators in source vertex are chained together,
then current solution is reasonable (break the chain, add a shuffle).

But for job b, if the bottleneck is the aggregate operator (not calc), it's
more likely be better to scale up the aggregate operator/vertex and without
breaking the {source -> calc} chain, as this will incur additional shuffle
cost.
So if we decide to add this new feature, I would recommend that both cases
be taken care of.


2. the assumption that a cdc source must have pk(primary key) may not be
reasonable, for example, mysql cdc supports the case without pk(
https://ververica.github.io/flink-cdc-connectors/master/content/connectors/mysql-cdc.html#tables-without-primary-keys),
so we can not just raise an error here.


3. for the new SourceTransformationWrapper I have some concerns about the
future evolution, if we need to add support for other operators, do we
continue to add new xxWrappers?

I've also revisited the previous discussion on FLIP-146[1], there were no
clear conclusions or good ideas about similar support issues for the source
before, and I also noticed that the new capability to change per-vertex
parallelism via rest api in 1.18 (part of FLIP-291[2][3], but there is
actually an issue about sql job's parallelism change which may require a
hash shuffle to ensure the order of update stream, this needs to be
followed up in FLIP-291, a jira will be created later).  So perhaps, we
need to think about it more (the next version is not yet launched, so we
still have time)

[1] https://lists.apache.org/thread/gtpswl42jzv0c9o3clwqskpllnw8rh87
[2]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-291%3A+Externalized+Declarative+Resource+Management
[3] https://issues.apache.org/jira/browse/FLINK-31471


Best,
Lincoln Lee


Chen Zhanghao  于2023年9月22日周五 16:00写道：

> Thanks to everyone who participated in the discussion here. If no further
> questions/concerns are raised, we'll start voting next Monday afternoon
> (GMT+8).
>
> Best,
> Zhanghao Chen
> 
> 发件人: Jane Chan 
> 发送时间: 2023年9月22日 15:35
> 收件人: dev@flink.apache.org 
> 主题: Re: [DISCUSS] FLIP-367: Support Setting Parallelism for Table/SQL
> Sources
>
> Hi Zhanghao,
>
> Thanks for the update; +1 for the proposal!
>
> Best,
> Jane
>
> On Fri, Sep 22, 2023 at 2:13 PM Chen Zhanghao 
> wrote:
>
> > Hi Jane,
> >
> > Thanks for the suggestions and totally agree with them. I've updated the
> > FLIP with the following two changes:
> >
> > 1. Rename WrapperTransformation to SourceTransformationWrapper that wraps
> > a SourceTransformation only. Note that we do not plan to support the
> legacy
> > LegacySourceTransformation.
> > 2. Choosing the partitioner after the source will be based on the
> > changelog mode of the source + the existence of the primary key in source
> > schema. If the source will produce update/delete message but a primary
> key
> > does not exist, an exception will be thrown.
> >
> > Best,
> > Zhanghao Chen
> > 
> > 发件人: Jane Chan 
> > 发送时间: 2023年9月20日 15:13
> > 收件人: dev@flink.apache.org 
> > 主题: Re: [DISCUSS] FLIP-367: Support Setting Parallelism for Table/SQL
> > Sources
> >
> > Hi Zhanghao,
> >
> > Thanks for the update. The FLIP now looks good to me in general, and I
> have
> > two minor comments.
> >
> > 1. Compared with other subclasses like `CacheTransformation` or
> > `PartitionTransformation`, the name  `WrapperTransformation` seems too
> > general. What about `SourceTransformationWrapper`, which is more specific
> > and descriptive, WDYT?
> >
> > 2.
> >
> > > When the source generates update and delete data (determined by
> checking
> > > the existence of a primary key in the source schema), the source will
> use
> > > hash partitioner to send data.
> >
> >
> > It might not be sufficient to determine whether the source is a CDC
> source
> > solely based on checking the existence of the primary key. It's better to
> > check the changelog mode of the source. On the other hand, adding the
> hash
> > partitioner requires the CDC source table to declare the primary key in
> the
> > DDL. Therefore, it is preferable to explain this restriction in the FLIP
>

[jira] [Created] (FLINK-33129) Can't create RowDataToAvroConverter for LocalZonedTimestampType logical type

[Zhaoyang Shao (Jira)]

Zhaoyang Shao created FLINK-33129:
-

 Summary: Can't create RowDataToAvroConverter for 
LocalZonedTimestampType logical type
 Key: FLINK-33129
 URL: https://issues.apache.org/jira/browse/FLINK-33129
 Project: Flink
  Issue Type: Bug
  Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
Affects Versions: 1.17.1
Reporter: Zhaoyang Shao
 Fix For: 1.17.1


While creating converter using `RowDataToAvroConverters.createConverter` with 
LocalZonedTimestampType logical type, the method will throw exception. This is 
because the switch clause is missing a clause for 
`LogicalTypeRoot.TIMESTAMP_WITH_LOCAL_TIME_ZON`.

Code: 
[https://github.com/apache/flink/blob/master/flink-formats/flink-avro/src/main/java/org/apache/flink/formats/avro/RowDataToAvroConverters.java#L75]

 

We can convert the value to `LocalDateTime` and then `TimestampData` using 
method below. Then we can apply the same converter as 
TIMESTAMP_WITHOUT_TIME_ZONE? 
 
`TimestampData fromLocalDateTime(LocalDateTime dateTime)`

Can Flink team help adding the support for this logical type and logical type 
root?

This is now a blocker for creating Flink Iceberg consumer with Avro 
GenericRecord when IcebergTable has `TimestampTZ` type field which will be 
converted to LocalZonedTimestampType.

See error below:
 Unsupported type: TIMESTAMP_LTZ(6) 
stack: [ [-] 
  
org.apache.flink.formats.avro.RowDataToAvroConverters.createConverter(RowDataToAvroConverters.java:186)
 
  
java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195) 
  
java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655) 
  java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) 
  
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) 
  java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:550) 
  
java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)
 
  
java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:517) 
  
org.apache.flink.formats.avro.RowDataToAvroConverters.createRowConverter(RowDataToAvroConverters.java:224)
 
  
org.apache.flink.formats.avro.RowDataToAvroConverters.createConverter(RowDataToAvroConverters.java:178)
 
  
org.apache.iceberg.flink.source.RowDataToAvroGenericRecordConverter.(RowDataToAvroGenericRecordConverter.java:46)
 
  
org.apache.iceberg.flink.source.RowDataToAvroGenericRecordConverter.fromIcebergSchema(RowDataToAvroGenericRecordConverter.java:60)
 
  
org.apache.iceberg.flink.source.reader.AvroGenericRecordReaderFunction.lazyConverter(AvroGenericRecordReaderFunction.java:93)
 
  
org.apache.iceberg.flink.source.reader.AvroGenericRecordReaderFunction.createDataIterator(AvroGenericRecordReaderFunction.java:85)
 
  
org.apache.iceberg.flink.source.reader.DataIteratorReaderFunction.apply(DataIteratorReaderFunction.java:39)
 
  
org.apache.iceberg.flink.source.reader.DataIteratorReaderFunction.apply(DataIteratorReaderFunction.java:27)
 
  
org.apache.iceberg.flink.source.reader.IcebergSourceSplitReader.fetch(IcebergSourceSplitReader.java:74)
 
  
org.apache.flink.connector.base.source.reader.fetcher.FetchTask.run(FetchTask.java:58)
 
  
org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.runOnce(SplitFetcher.java:162)
 
  
org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.run(SplitFetcher.java:114)
 
  
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) 
  java.util.concurrent.FutureTask.run(FutureTask.java:264) 
  
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) 
  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) 
  java.lang.Thread.run(Thread.java:829)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[RESULT][VOTE] FLIP-312: Prometheus Sink Connector

[Lorenzo Nicora]

Hello

The proposal  FLIP-312: Prometheus Sink Connector, has been approved with 8
votes (6 binding)

Thanks everybody
Lorenzo

Re: [DISCUSS] FLIP-356: Support Nested Fields Filter Pushdown

[Venkatakrishnan Sowrirajan]

Thanks Martijn.

Regards
Venkata krishnan


On Fri, Sep 22, 2023 at 1:51 AM Martijn Visser 
wrote:

> Sure thing, I've made the necessary changes. Thnx for clarifying
>
> On Thu, Sep 21, 2023 at 8:24 PM Venkatakrishnan Sowrirajan
>  wrote:
> >
> > Got it, Martijn.
> >
> > Unfortunately, I don't have edit access to the already created JIRA -
> > FLINK-20767 <
> https://urldefense.com/v3/__https://issues.apache.org/jira/browse/FLINK-20767__;!!IKRxdwAv5BmarQ!c8v3R7qHJbTKwloBaUOXL-W6HQU65q11mB4vgpETHmUEkDx0nPNXpt_ZZceCQtPpnAgrau3ua42_YMRK6jvX0h8h$
> >. If you can
> > remove the task from the EPIC FLINK-16987 FLIP-95: Add new table source
> and
> > sink interfaces <
> https://urldefense.com/v3/__https://issues.apache.org/jira/browse/FLINK-16987__;!!IKRxdwAv5BmarQ!c8v3R7qHJbTKwloBaUOXL-W6HQU65q11mB4vgpETHmUEkDx0nPNXpt_ZZceCQtPpnAgrau3ua42_YMRK6t5j3KbX$
> >, can
> > you please change it?
> >
> > If not, I can open a new ticket, close this one and link the 2 tickets as
> > duplicated by.
> >
> > Regards
> > Venkata krishnan
> >
> >
> > On Thu, Sep 21, 2023 at 12:40 AM Martijn Visser <
> martijnvis...@apache.org>
> > wrote:
> >
> > > Hi Venkatakrishnan,
> > >
> > > The reason why I thought it's abandoned because the Jira ticket is
> > > part of the umbrella ticket for FLIP-95. Let's move the Jira ticket to
> > > its own dedicated task instead of nested to a FLIP-95 ticket.
> > >
> > > Thanks,
> > >
> > > Martijn
> > >
> > > On Wed, Sep 20, 2023 at 4:34 PM Becket Qin 
> wrote:
> > > >
> > > > Hi Martijn,
> > > >
> > > > This FLIP has passed voting[1]. It is a modification on top of the
> > > FLIP-95
> > > > interface.
> > > >
> > > > Thanks,
> > > >
> > > > Jiangjie (Becket) Qin
> > > >
> > > > [1]
> > >
> https://urldefense.com/v3/__https://lists.apache.org/thread/hysv9y1f48gtpr5vx3x40wtjb6cp9ky6__;!!IKRxdwAv5BmarQ!f6gg5kDGU-9tqM2rGpbK81L_7sOscG4aVKSjp0RchK1pevsMz_YrhlStS1tXduiLHoxjMP8_BjpJYrQc28-X111i$
> > > >
> > > > On Wed, Sep 20, 2023 at 9:29 PM Martijn Visser <
> martijnvis...@apache.org
> > > >
> > > > wrote:
> > > >
> > > > > For clarity purposes, this FLIP is being abandoned because it was
> part
> > > > > of FLIP-95?
> > > > >
> > > > > On Thu, Sep 7, 2023 at 3:01 AM Venkatakrishnan Sowrirajan
> > > > >  wrote:
> > > > > >
> > > > > > Hi everyone,
> > > > > >
> > > > > > Posted a PR (
> > >
> https://urldefense.com/v3/__https://github.com/apache/flink/pull/23313__;!!IKRxdwAv5BmarQ!f6gg5kDGU-9tqM2rGpbK81L_7sOscG4aVKSjp0RchK1pevsMz_YrhlStS1tXduiLHoxjMP8_BjpJYrQc2-0vM_Ac$
> > > ) to add nested
> > > > > > fields filter pushdown. Please review. Thanks.
> > > > > >
> > > > > > Regards
> > > > > > Venkata krishnan
> > > > > >
> > > > > >
> > > > > > On Tue, Sep 5, 2023 at 10:04 PM Venkatakrishnan Sowrirajan <
> > > > > vsowr...@asu.edu>
> > > > > > wrote:
> > > > > >
> > > > > > > Based on an offline discussion with Becket Qin, I added
> > > *fieldIndices *
> > > > > > > back which is the field index of the nested field at every
> level to
> > > > > the *NestedFieldReferenceExpression
> > > > > > > *in FLIP-356
> > > > > > > <
> > > > >
> > >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/FLINK/FLIP-356*3A*Support*Nested*Fields*Filter*Pushdown__;JSsrKysr!!IKRxdwAv5BmarQ!f6gg5kDGU-9tqM2rGpbK81L_7sOscG4aVKSjp0RchK1pevsMz_YrhlStS1tXduiLHoxjMP8_BjpJYrQc27ttpcO0$
> > > > > >
> > > > > > > *. *2 reasons to do it:
> > > > > > >
> > > > > > > 1. Agree with using *fieldIndices *as the only contract to
> refer
> > > to the
> > > > > > > column from the underlying datasource.
> > > > > > > 2. To keep it consistent with *FieldReferenceExpression*
> > > > > > >
> > > > > > > Having said that, I see that with *projection pushdown, *index
> of
> > > the
> > > > > > > fields are used whereas with *filter pushdown (*based on
> scanning
> > > few
> > > > > > > tablesources) *FieldReferenceExpression*'s name is used for eg:
> > > even in
> > > > > > > the Flink's *FileSystemTableSource, IcebergSource,
> JDBCDatsource*.
> > > This
> > > > > > > way, I feel the contract is not quite clear and explicit.
> Wanted to
> > > > > > > understand other's thoughts as well.
> > > > > > >
> > > > > > > Regards
> > > > > > > Venkata krishnan
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Sep 5, 2023 at 5:34 PM Becket Qin <
> becket@gmail.com>
> > > > > wrote:
> > > > > > >
> > > > > > >> Hi Venkata,
> > > > > > >>
> > > > > > >>
> > > > > > >> > Also I made minor changes to the
> > > *NestedFieldReferenceExpression,
> > > > > > >> *instead
> > > > > > >> > of *fieldIndexArray* we can just do away with *fieldNames
> *array
> > > > > that
> > > > > > >> > includes fieldName at every level for the nested field.
> > > > > > >>
> > > > > > >>
> > > > > > >> I don't think keeping only the field names array would work.
> At
> > > the
> > > > > end of
> > > > > > >> the day, the contract between Flink SQL and the connectors is
> > > based
> > > > > on the
> > > > > > >> indexes, not the names. Technically speaking, th

Re: [Discuss] FLIP-362: Support minimum resource limitation

[xiangyu feng]

Hi all,

Thank you for the comments.

If there is no further comment, we will open the voting thread in 3 days.

Regards,
Xiangyu

Yangze Guo  于2023年9月21日周四 11:19写道：

> Thanks for the reply, Shammon.
>
> As the example described in my last response, an application could
> contain multiple jobs, both batch and streaming. I don't lean to
> disable it in Application mode in case users want to leverage it to
> accelerate the preceding batch jobs in their application.
>
> Best,
> Yangze Guo
>
> On Thu, Sep 21, 2023 at 11:15 AM Shammon FY  wrote:
> >
> > Hi,
> >
> > I agree that `minimum resource limitation` will bring values for flink
> > session clusters, but for `Application Mode`, is it useful for streaming
> > and batch jobs? Is it necessary for us to not support the application
> mode,
> > rather than relying on the default value 0?
> >
> > Best,
> > Shammon FY
> >
> > On Thu, Sep 21, 2023 at 10:18 AM Yangze Guo  wrote:
> >
> > > Thanks for the comments, Jing.
> > >
> > > > Will the minimum resource configuration also take effect for
> streaming
> > > jobs in application mode?
> > > > Since it is not recommended to configure
> slotmanager.number-of-slots.max
> > > for streaming jobs, does it make sense to disable it for common
> streaming
> > > jobs? At least disable the check for avoiding the oscillation?
> > >
> > > Yes. The minimum resource configuration will only disabled in
> > > standalone cluster atm. I agree it make sense to disable it for a pure
> > > streaming job, however:
> > > - By default, the minimum resource is configured to 0. If users do not
> > > proactively set it, either the oscillation check or the minimum
> > > restriction can be considered as disabled.
> > > - The minimum resource is a cluster-level configuration rather than a
> > > job-level configuration. If a user has an application with two batch
> > > jobs preceding the streaming job, they may also require this
> > > configuration to accelerate the execution of batch jobs.
> > >
> > > WDYT?
> > >
> > > Best,
> > > Yangze Guo
> > >
> > > On Thu, Sep 21, 2023 at 4:49 AM Jing Ge 
> > > wrote:
> > > >
> > > > Hi Xiangyu,
> > > >
> > > > Thanks for driving it! There is one thing I am not really sure if I
> > > > understand you correctly.
> > > >
> > > > According to the FLIP: "The minimum resource limitation will be
> > > implemented
> > > > in the DefaultResourceAllocationStrategy of FineGrainedSlotManager.
> > > >
> > > > Each time when SlotManager needs to reconcile the cluster resources
> or
> > > > fulfill job resource requirements, the
> DefaultResourceAllocationStrategy
> > > > will check if the minimum resource requirement has been fulfilled.
> If it
> > > is
> > > > not, DefaultResourceAllocationStrategy will request new
> > > PendingTaskManagers
> > > > and FineGrainedSlotManager will allocate new worker resources
> > > accordingly."
> > > >
> > > > "To avoid this oscillation, we need to check the worker number
> derived
> > > from
> > > > minimum and maximum resource configuration is consistent before
> starting
> > > > SlotManager."
> > > >
> > > > Will the minimum resource configuration also take effect for
> streaming
> > > jobs
> > > > in application mode? Since it is not recommended to
> > > > configure slotmanager.number-of-slots.max for streaming jobs, does it
> > > make
> > > > sense to disable it for common streaming jobs? At least disable the
> check
> > > > for avoiding the oscillation?
> > > >
> > > > Best regards,
> > > > Jing
> > > >
> > > >
> > > > On Tue, Sep 19, 2023 at 4:58 PM Chen Zhanghao <
> zhanghao.c...@outlook.com
> > > >
> > > > wrote:
> > > >
> > > > > Thanks for driving this, Xiangyu. We use Session clusters for
> quick SQL
> > > > > debugging internally, and found cold-start job submission slow due
> to
> > > lack
> > > > > of the exact minimum resource reservation feature proposed here.
> This
> > > > > should improve the experience a lot for running short lived-jobs in
> > > session
> > > > > clusters.
> > > > >
> > > > > Best,
> > > > > Zhanghao Chen
> > > > > 
> > > > > 发件人: Yangze Guo 
> > > > > 发送时间: 2023年9月19日 13:10
> > > > > 收件人: xiangyu feng 
> > > > > 抄送: dev@flink.apache.org 
> > > > > 主题: Re: [Discuss] FLIP-362: Support minimum resource limitation
> > > > >
> > > > > Thanks for driving this @Xiangyu. This is a feature that many users
> > > > > have requested for a long time. +1 for the overall proposal.
> > > > >
> > > > > Best,
> > > > > Yangze Guo
> > > > >
> > > > > On Tue, Sep 19, 2023 at 11:48 AM xiangyu feng <
> xiangyu...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > Hi Devs,
> > > > > >
> > > > > > I'm opening this thread to discuss FLIP-362: Support minimum
> resource
> > > > > limitation. The design doc can be found at:
> > > > > > FLIP-362: Support minimum resource limitation
> > > > > >
> > > > > > Currently, the Flink cluster only requests Task Managers (TMs)
> when
> > > > > there is a resource requirement, and idle TMs are release

[jira] [Created] (FLINK-33130) Source ans Sink

[xiebin (Jira)]

xiebin created FLINK-33130:
--

 Summary: Source ans Sink
 Key: FLINK-33130
 URL: https://issues.apache.org/jira/browse/FLINK-33130
 Project: Flink
  Issue Type: Improvement
Reporter: xiebin






--
This message was sent by Atlassian Jira
(v8.20.10#820010)