Re: [DISCUSS] FLIP-378: Support Avro timestamp with local timezone

[Gyula Fóra]

+1

Gyula

On Thu, Nov 2, 2023 at 6:18 AM Martijn Visser 
wrote:

> +1
>
> On Thu, Nov 2, 2023 at 12:44 PM Leonard Xu  wrote:
> >
> >
> > > Thanks @Leonard Xu . Two minor versions are
> definitely needed for flip the configs.
> >
> > Sorry, Peter. I thought the next minor versions are 1.19、2.0， but
> actually it should be 1.19、1.20、2.0 from current community version plan
> IIUC, so remove the config in 2.0 should be okay if the 1.20 version exists
> .
> >
> > Best,
> > Leonard
> >
> >
> > >
> > > On Mon, Oct 30, 2023 at 8:55 PM Leonard Xu  xbjt...@gmail.com>> wrote:
> > > Thanks @Peter for driving this FLIP
> > >
> > > +1 from my side, the timestamp semantics mapping looks good to me.
> > >
> > > >  In the end, the legacy behavior will be dropped in
> > > > Flink 2.0
> > > > I don’t think we can drop this option which introduced in 1.19 and
> drop in 2.0, the API removal requires at least two minor versions.
> > >
> > >
> > > Best,
> > > Leonard
> > >
> > > > 2023年10月31日 上午11:18，Peter Huang  huangzhenqiu0...@gmail.com>> 写道：
> > > >
> > > > Hi Devs,
> > > >
> > > > Currently, Flink Avro Format doesn't support the Avro time
> (milli/micros)
> > > > with local timezone type.
> > > > Although the Avro timestamp (millis/micros) type is supported and is
> mapped
> > > > to flink timestamp without timezone,
> > > > it is not compliant to semantics defined in Consistent timestamp
> types in
> > > > Hadoop SQL engines
> > > > <
> https://docs.google.com/document/d/1gNRww9mZJcHvUDCXklzjFEQGpefsuR_akCDfWsdE35Q/edit#heading=h.n699ftkvhjlo
> <
> https://docs.google.com/document/d/1gNRww9mZJcHvUDCXklzjFEQGpefsuR_akCDfWsdE35Q/edit#heading=h.n699ftkvhjlo
> >>
> > > > .
> > > >
> > > > I propose to support Avro timestamps with the compliance to the
> mapping
> > > > semantics [1] by using a configuration flag.
> > > > To keep back compatibility, current mapping is kept as default
> behavior.
> > > > Users can explicitly turn on the new mapping
> > > > by setting it to false. In the end, the legacy behavior will be
> dropped in
> > > > Flink 2.0
> > > >
> > > > Looking forward to your feedback.
> > > >
> > > >
> > > > [1]
> > > >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-378%3A+Support+Avro+timestamp+with+local+timezone
> <
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-378%3A+Support+Avro+timestamp+with+local+timezone
> >
> > > >
> > > >
> > > > Best Regards
> > > >
> > > > Peter Huang
> > >
> >
>

Re: [DISCUSS] FLIP-381: Deprecate configuration getters/setters that return/set complex Java objects

[Jing Ge]

Hi Junrui,

Thanks for bringing the proposal to our attention. It looks overall great.

I am just a little bit confused with the content of Flip with the following
questions:

1. How many complex java classes will be deprecated?
2. After reading the section title "Deprecate following classes, fields and
methods", I thought that e.g.
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment will
be deprecated. I would suggest restructure the content and point out
clearly which entire classes will be deprecated and which methods and
fields and not the class itself will be deprecated instead of mixing them
up. WDYT?

Best regards,
Jing


On Fri, Nov 3, 2023 at 8:59 AM Wencong Liu  wrote:

> Thanks Junrui for your effort!
>
> Making all configuration code paths lead to ConfigOption is a more
> standardized
> approach to configuring Flink applications.
>
> +1 for this proposal.
>
> Best,
> Wencong Liu
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> At 2023-11-02 10:10:14, "Junrui Lee"  wrote:
> >Hi devs,
> >
> >I would like to start a discussion on FLIP-381: Deprecate configuration
> >getters/setters that return/set complex Java objects[1].
> >
> >Currently, the job configuration in FLINK is spread out across different
> >components, which leads to inconsistencies and confusion. To address this
> >issue, it is necessary to migrate non-ConfigOption complex Java objects to
> >use ConfigOption and adopt a single Configuration object to host all the
> >configuration.
> >However, there is a significant blocker in implementing this solution.
> >These complex Java objects in StreamExecutionEnvironment,
> CheckpointConfig,
> >and ExecutionConfig have already been exposed through the public API,
> >making it challenging to modify the existing implementation.
> >
> >Therefore, I propose to deprecate these Java objects and their
> >corresponding getter/setter interfaces, ultimately removing them in
> >FLINK-2.0.
> >
> >Your feedback and thoughts on this proposal are highly appreciated.
> >
> >Best regards,
> >Junrui Lee
> >
> >[1]
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=278464992
>

Re: Request to release flink 1.6.3

[Jing Ge]

Hi folks,

Just like Matthias mentioned in his reply and Rui followed. Since 1.18.0
has been released, I'd suggest that we vote to make 1.16.3 the final bugix
release of 1.16, ask and wait for any objections. WDYT? Background info
could be found at [1].

Best regards,
Jing

[1] https://lists.apache.org/thread/szq23kr3rlkm80rw7k9n95js5vqpsnbv


On Fri, Nov 3, 2023 at 7:38 PM Rui Fan <1996fan...@gmail.com> wrote:

> Thanks Yun and Yu,
>
> That would be great! Let's move forward!
>
> Best,
> Rui
>
> On Sat, Nov 4, 2023 at 10:10 AM Yun Tang  wrote:
>
> > Hi Rui,
> >
> > I could help to join the release management of 1.17.2, and I will also
> ask
> > Yu Chen for help with some working items.
> >
> > Let's move forward and make it.
> >
> > Best
> > Yun Tang
> > 
> > From: Rui Fan <1996fan...@gmail.com>
> > Sent: Tuesday, October 31, 2023 23:06
> > To: dev@flink.apache.org 
> > Subject: Re: Request to release flink 1.6.3
> >
> > Thanks Vikas for the ask!
> >
> > Hi devs,
> >
> > Is anyone willing to pick up the release of 1.16.3 and 1.17.2 with me? If
> > so, I can volunteer to release one of the versions. If no one picks it up
> > for more than three days, I volunteer to release two versions. After it’s
> > determined, the official discussion can be started.
> >
> > Looking forward to other committer or  PMC join this release!
> >
> >
> > Hi Matthias,
> >
> > Thank you for sorting out these 2 lists. 1.16.3 may be the final version
> of
> > 1.16 series, it makes sense to sort out all left over issues.
> >
> > Also, some of release processes need PMC  permission, would you mind
> > helping it? Thanks~
> >
> > Best,
> > Rui
> >
> > On Tue, 31 Oct 2023 at 21:42, vikas patil 
> wrote:
> >
> > > Hello Rui,
> > >
> > > Do we need more votes for this or are we good to go with the release of
> > > 1.6.3 ? Please let me know. Thanks.
> > >
> > > -Vikas
> > >
> > > On Tue, Oct 24, 2023 at 9:27 AM Rui Fan <1996fan...@gmail.com> wrote:
> > >
> > > > Hi Vikas,
> > > >
> > > > Thanks for your feedback!
> > > >
> > > > Do you mean flink 1.16.3 instead of 1.6.3?
> > > >
> > > > The 1.16.2 and 1.17.1 were released on 2023-05-25,
> > > > it’s been 5 months. And the flink community has fixed
> > > > many bugs in the past 5 months. Usually, there is a
> > > > fix(minor) version every three or four months, so I propose
> > > > to release 1.16.3 and 1.17.2 now.
> > > >
> > > > If the community agrees to create this new patch release, I could
> > > > volunteer as the release manager for one of the 1.16.3 or 1.17.2.
> > > >
> > > > Looking forward to feedback from the community, thank you
> > > >
> > > > [1]
> > > >
> > > >
> > >
> >
> https://flink.apache.org/2023/05/25/apache-flink-1.16.2-release-announcement/
> > > > [2]
> > > >
> > > >
> > >
> >
> https://flink.apache.org/2023/05/25/apache-flink-1.17.1-release-announcement/
> > > >
> > > > Best,
> > > > Rui
> > > >
> > > > On Tue, Oct 24, 2023 at 9:50 PM vikas patil  >
> > > > wrote:
> > > >
> > > > > Hello All,
> > > > >
> > > > > Facing this FLINK-28185 <
> > > > https://issues.apache.org/jira/browse/FLINK-28185
> > > > > >
> > > > > issue for one of the flink jobs. We are running flink version 1.6.1
> > but
> > > > it
> > > > > looks like the backport <
> https://github.com/apache/flink/pull/22505>
> > > to
> > > > > 1.6
> > > > > was never released as 1.6.3. The latest that was released is 1.6.2
> > > > > . By any chance we
> can
> > > get
> > > > > the 1.6.3 released ?
> > > > >
> > > > > Also we use the official flink docker <
> > https://hub.docker.com/_/flink/
> > > >
> > > > > image. Not sure if that needs to be updated as well manually.
> Thanks.
> > > > >
> > > > > -Vikas
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS][FLINK-33240] Document deprecated options as well

[Jing Ge]

Hi folks,

I started the same task and discussions in Feb. [1]. There were some
objections.

Best regards,
Jing
[1] https://issues.apache.org/jira/browse/FLINK-30862

On Wed, Nov 1, 2023 at 8:30 AM Junrui Lee  wrote:

> Hi Zhanghao,
>
> Thank you for the proposal.
>
> +1 from my side. It would be more user-friendly to have the deprecated
> options in the same section as the non-deprecated ones. Therefore, adding
> them in the same section sounds good to me.
>
> Best regards,
> Junrui
>
> Zhanghao Chen  于2023年11月1日周三 21:10写道：
>
> > Hi Samrat and Ruan,
> >
> > Thanks for the suggestion. I'm actually in favor of adding the deprecated
> > options in the same section as the non-deprecated ones. This would make
> > user search for descriptions of the replacement options more easily. It
> > would be a different story for options deprecated because the related
> > API/module is entirely deprecated, e.g. DataSet API. In that case, users
> > would not search for replacement on an individual option but rather need
> to
> > migrate to a new API, and it would be better to move these options to a
> > separate section. WDYT?
> >
> > Best,
> > Zhanghao Chen
> > 
> > From: Samrat Deb 
> > Sent: Wednesday, November 1, 2023 15:31
> > To: dev@flink.apache.org 
> > Cc: u...@flink.apache.org 
> > Subject: Re: [DISCUSS][FLINK-33240] Document deprecated options as well
> >
> > Thanks for the proposal ,
> > +1 for adding deprecated identifier
> >
> > [Thought] Can we have seperate section / page for deprecated configs ?
> Wdut
> > ?
> >
> >
> > Bests,
> > Samrat
> >
> >
> > On Tue, 31 Oct 2023 at 3:44 PM, Alexander Fedulov <
> > alexander.fedu...@gmail.com> wrote:
> >
> > > Hi Zhanghao,
> > >
> > > Thanks for the proposition.
> > > In general +1, this sounds like a good idea as long it is clear that
> the
> > > usage of these settings is discouraged.
> > > Just one minor concern - the configuration page is already very long,
> do
> > > you have a rough estimate of how many more options would be added with
> > this
> > > change?
> > >
> > > Best,
> > > Alexander Fedulov
> > >
> > > On Mon, 30 Oct 2023 at 18:24, Matthias Pohl  > > .invalid>
> > > wrote:
> > >
> > > > Thanks for your proposal, Zhanghao Chen. I think it adds more
> > > transparency
> > > > to the configuration documentation.
> > > >
> > > > +1 from my side on the proposal
> > > >
> > > > On Wed, Oct 11, 2023 at 2:09 PM Zhanghao Chen <
> > zhanghao.c...@outlook.com
> > > >
> > > > wrote:
> > > >
> > > > > Hi Flink users and developers,
> > > > >
> > > > > Currently, Flink won't generate doc for the deprecated options.
> This
> > > > might
> > > > > confuse users when upgrading from an older version of Flink: they
> > have
> > > to
> > > > > either carefully read the release notes or check the source code
> for
> > > > > upgrade guidance on deprecated options.
> > > > >
> > > > > I propose to document deprecated options as well, with a
> > "(deprecated)"
> > > > > tag placed at the beginning of the option description to highlight
> > the
> > > > > deprecation status [1].
> > > > >
> > > > > Looking forward to your feedbacks on it.
> > > > >
> > > > > [1] https://issues.apache.org/jira/browse/FLINK-33240
> > > > >
> > > > > Best,
> > > > > Zhanghao Chen
> > > > >
> > > >
> > >
> >
>

[jira] [Created] (FLINK-33458) Add env.java.opts.gateway option in flink-conf.yaml

[Sharath Avadoot Gururaj (Jira)]

Sharath Avadoot Gururaj created FLINK-33458:
---

 Summary: Add env.java.opts.gateway option in flink-conf.yaml
 Key: FLINK-33458
 URL: https://issues.apache.org/jira/browse/FLINK-33458
 Project: Flink
  Issue Type: Improvement
  Components: Table SQL / Gateway
Affects Versions: 1.18.0
Reporter: Sharath Avadoot Gururaj


Currently, JobManager, TaskManager, HistoryServer, Client have their own 
configuration keys to set java options in flink-conf.yaml

However it is missing for SQL Gateway. It should be added for completeness. It 
can be useful, for example, to add Remote Debugging options.

I propose this config key 

`env.java.opts.gateway` in flink-conf.yaml which will be applied to SQL 
Gateway. I already have a working patch. Its a pretty small one. If the 
community is OK, I will raise a PR



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Request to release flink 1.6.3

[Rui Fan]

Hi Jing,

Thanks for your feedback.

Yun, Yu and me have a offline discussion about release 1.16.3 and 1.17.2,
we will start 2 discuss tomorrow. And I will consider 1.16.3 as the final
release of 1.16.

Best,
Rui

On Sun, 5 Nov 2023 at 00:49, Jing Ge  wrote:

> Hi folks,
>
> Just like Matthias mentioned in his reply and Rui followed. Since 1.18.0
> has been released, I'd suggest that we vote to make 1.16.3 the final bugix
> release of 1.16, ask and wait for any objections. WDYT? Background info
> could be found at [1].
>
> Best regards,
> Jing
>
> [1] https://lists.apache.org/thread/szq23kr3rlkm80rw7k9n95js5vqpsnbv
>
>
> On Fri, Nov 3, 2023 at 7:38 PM Rui Fan <1996fan...@gmail.com> wrote:
>
> > Thanks Yun and Yu,
> >
> > That would be great! Let's move forward!
> >
> > Best,
> > Rui
> >
> > On Sat, Nov 4, 2023 at 10:10 AM Yun Tang  wrote:
> >
> > > Hi Rui,
> > >
> > > I could help to join the release management of 1.17.2, and I will also
> > ask
> > > Yu Chen for help with some working items.
> > >
> > > Let's move forward and make it.
> > >
> > > Best
> > > Yun Tang
> > > 
> > > From: Rui Fan <1996fan...@gmail.com>
> > > Sent: Tuesday, October 31, 2023 23:06
> > > To: dev@flink.apache.org 
> > > Subject: Re: Request to release flink 1.6.3
> > >
> > > Thanks Vikas for the ask!
> > >
> > > Hi devs,
> > >
> > > Is anyone willing to pick up the release of 1.16.3 and 1.17.2 with me?
> If
> > > so, I can volunteer to release one of the versions. If no one picks it
> up
> > > for more than three days, I volunteer to release two versions. After
> it’s
> > > determined, the official discussion can be started.
> > >
> > > Looking forward to other committer or  PMC join this release!
> > >
> > >
> > > Hi Matthias,
> > >
> > > Thank you for sorting out these 2 lists. 1.16.3 may be the final
> version
> > of
> > > 1.16 series, it makes sense to sort out all left over issues.
> > >
> > > Also, some of release processes need PMC  permission, would you mind
> > > helping it? Thanks~
> > >
> > > Best,
> > > Rui
> > >
> > > On Tue, 31 Oct 2023 at 21:42, vikas patil 
> > wrote:
> > >
> > > > Hello Rui,
> > > >
> > > > Do we need more votes for this or are we good to go with the release
> of
> > > > 1.6.3 ? Please let me know. Thanks.
> > > >
> > > > -Vikas
> > > >
> > > > On Tue, Oct 24, 2023 at 9:27 AM Rui Fan <1996fan...@gmail.com>
> wrote:
> > > >
> > > > > Hi Vikas,
> > > > >
> > > > > Thanks for your feedback!
> > > > >
> > > > > Do you mean flink 1.16.3 instead of 1.6.3?
> > > > >
> > > > > The 1.16.2 and 1.17.1 were released on 2023-05-25,
> > > > > it’s been 5 months. And the flink community has fixed
> > > > > many bugs in the past 5 months. Usually, there is a
> > > > > fix(minor) version every three or four months, so I propose
> > > > > to release 1.16.3 and 1.17.2 now.
> > > > >
> > > > > If the community agrees to create this new patch release, I could
> > > > > volunteer as the release manager for one of the 1.16.3 or 1.17.2.
> > > > >
> > > > > Looking forward to feedback from the community, thank you
> > > > >
> > > > > [1]
> > > > >
> > > > >
> > > >
> > >
> >
> https://flink.apache.org/2023/05/25/apache-flink-1.16.2-release-announcement/
> > > > > [2]
> > > > >
> > > > >
> > > >
> > >
> >
> https://flink.apache.org/2023/05/25/apache-flink-1.17.1-release-announcement/
> > > > >
> > > > > Best,
> > > > > Rui
> > > > >
> > > > > On Tue, Oct 24, 2023 at 9:50 PM vikas patil <
> vikas.a.pa...@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > > > Hello All,
> > > > > >
> > > > > > Facing this FLINK-28185 <
> > > > > https://issues.apache.org/jira/browse/FLINK-28185
> > > > > > >
> > > > > > issue for one of the flink jobs. We are running flink version
> 1.6.1
> > > but
> > > > > it
> > > > > > looks like the backport <
> > https://github.com/apache/flink/pull/22505>
> > > > to
> > > > > > 1.6
> > > > > > was never released as 1.6.3. The latest that was released is
> 1.6.2
> > > > > > . By any chance we
> > can
> > > > get
> > > > > > the 1.6.3 released ?
> > > > > >
> > > > > > Also we use the official flink docker <
> > > https://hub.docker.com/_/flink/
> > > > >
> > > > > > image. Not sure if that needs to be updated as well manually.
> > Thanks.
> > > > > >
> > > > > > -Vikas
> > > > > >
> > > > >
> > > >
> > >
> >
>