Re: [Discuss] FLIP-366: Support standard YAML for FLINK configuration

[Chesnay Schepler]

It is a unfortunate that we'll need a separate config file but the FLIP 
does a good job justifying it.


Personally I'd just name it "config.yaml"; I never quite understood why 
there was a flink prefix to begin with, and the current proposal 
("flink-configuration.yaml") seems unnecessarily long.


For the deprecation process we could consider logging a warning if the 
old parser is used.


In terms of scope, is the migration of existing e2e tests and the docker 
image to the new parser part of the FLIP?


On 22/09/2023 09:32, Jane Chan wrote:

Hi, Junrui,

Sorry for the late reply. The update looks good to me and thanks for your
effort!

Best,
Jane

On Fri, Sep 22, 2023 at 2:25 PM Yuxin Tan  wrote:


Hi, Junrui

+1 for the proposal.
Thanks for your effort.

Best,
Yuxin


Samrat Deb  于2023年9月22日周五 13:23写道：


Hello Junrui,

+1 for the proposal.


Bests,
Samrat

On Fri, Sep 22, 2023 at 10:18 AM Shammon FY  wrote:


+1 for the proposal, thanks for driving.

Bet,
Shammon FY

On Fri, Sep 22, 2023 at 12:41 PM Yangze Guo 

wrote:

Thanks for driving this, +1 for the proposal.

Best,
Yangze Guo


On Fri, Sep 22, 2023 at 11:59 AM Lijie Wang <

wangdachui9...@gmail.com>

wrote:

Hi Junrui,

+1 for this proposal, thanks for driving.

Best,
Lijie

ConradJam  于2023年9月22日周五 10:07写道：


+1 Support for standard YAML format facilitates specification

Jing Ge  于2023年9月22日周五 02:23写道：


Hi Junrui,

+1 for following the standard. Thanks for your effort!

Best regards,
Jing

On Thu, Sep 21, 2023 at 5:09 AM Junrui Lee <

jrlee@gmail.com>

wrote:

Hi Jane,

Thank you for your valuable feedback and suggestions.
I agree with your point about differentiating between

"flink-config.yaml"

and "flink-conf.yaml" to determine the standard syntax at a

glance.

While I understand your suggestion of using

"flink-conf-default.yaml"

to

represent the default YAML file for Flink 1.x, I have been

considering

the option of using "flink-configuration.yaml" as the file

name

for the

new configuration file.
This name "flink-configuration.yaml" provides a clear

distinction

between

the new and old configuration files based on their names, and

it

does

not

introduce any additional semantics. Moreover, this name
"flink-configuration.yaml" can continue to be used in future

versions

FLINK-2.0.

WDYT? If we can reach a consensus on this, I will update the

FLIP

documentation
accordingly.

Best regards,
Junrui

Jane Chan  于2023年9月20日周三 23:38写道：


Hi Junrui,

Thanks for driving this FLIP. +1 for adoption of the

standard

YAML

syntax.

I just have one minor suggestion. It's a little bit

challenging

to

differentiate between `flink-config.yaml` and

`flink-conf.yaml`

to

determine which one uses the standard syntax at a glance.

How

about

using `flink-conf-default.yaml` to represent the default

yaml

file

for

Flink 1.x?

Best,
Jane

On Wed, Sep 20, 2023 at 11:06 AM Junrui Lee <

jrlee@gmail.com

wrote:

Hi devs,

I would like to start a discussion about FLIP-366:
Support standard YAML for FLINK configuration[1]

The current flink-conf.yaml parser in FLINK is not a

standard

YAML

parser,

which has some shortcomings.
Firstly, it does not support nested structure

configuration

items

and

only

supports key-value pairs, resulting in poor readability.

Secondly,

if

the

value is a collection type, such as a List or Map, users

are

required

to

write the value in a FLINK-specific pattern, which is

inconvenient

to

use.

Additionally, the parser of FLINK has some differences in

syntax

compared

to the standard YAML parser, such as the syntax for

parsing

comments

and

null values. These inconsistencies can cause confusion

for

users,

as

seen

in FLINK-15358 and FLINK-32740.

By supporting standard YAML, these issues can be

resolved,

and

users

can

create a Flink configuration file using third-party tools

and

leverage

some advanced YAML features. Therefore, we propose to

support

standard

YAML for FLINK configuration.

You can find more details in the FLIP-366[1]. Looking

forward

to

your

feedback.

[1]



https://cwiki.apache.org/confluence/display/FLINK/FLIP-366%3A+Support+standard+YAML+for+FLINK+configuration

Best,
Junrui



--
Best

ConradJam

[jira] [Created] (FLINK-33178) Highly parallel apps suffer from bottleneck in NativePRNG

[Emre Kartoglu (Jira)]

Emre Kartoglu created FLINK-33178:
-

 Summary: Highly parallel apps suffer from bottleneck in NativePRNG 
 Key: FLINK-33178
 URL: https://issues.apache.org/jira/browse/FLINK-33178
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Network
Affects Versions: 1.13.2
Reporter: Emre Kartoglu


I observed the below thread dumps that highlighted a potential bottleneck in 
Flink/Netty/JDK. The application (Flink 1.13) from which I took the thread 
dumps had very high parallelism and was distributed on nodes with >150GB random 
access memory.

It appears that there is a call to "Arrays.copyOfRange" in a syncrhonized block 
in "sun.security.provider.NativePRNG", which blocks other threads waiting for 
the lock to the same synchronized block. This appears to be a problem only with 
highly parallel applications. I don't know exactly at what parallelism it 
starts becoming a problem, and how much of a bottleneck it actually is.

I was also slightly hesitant about creating a Flink ticket as the improvement 
could well be made in Netty or even JDK. But I believe we should have a record 
of the issue in Flink Jira.

Related: [https://bugs.openjdk.org/browse/JDK-8278371]

 

 
{code:java}
"Flink Netty Server (6121) Thread 43" #930 daemon prio=5 os_prio=0 
cpu=2298176.43ms elapsed=44352.31s allocated=155G defined_classes=0 
tid=0x7f0a3397f800 nid=0x519 waiting for monitor entry  [0x7efc5d549000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at 
sun.security.provider.NativePRNG$RandomIO.implNextBytes(java.base@11.0.18/NativePRNG.java:544)
        - waiting to lock <0x7f0b62c2eee8> (a java.lang.Object)
        at 
sun.security.provider.NativePRNG.engineNextBytes(java.base@11.0.18/NativePRNG.java:220)
        at 
java.security.SecureRandom.nextBytes(java.base@11.0.18/SecureRandom.java:751)
        at 
sun.security.ssl.SSLCipher$T11BlockWriteCipherGenerator$BlockWriteCipher.encrypt(java.base@11.0.18/SSLCipher.java:1498)
        at 
sun.security.ssl.OutputRecord.t10Encrypt(java.base@11.0.18/OutputRecord.java:441)
        at 
sun.security.ssl.OutputRecord.encrypt(java.base@11.0.18/OutputRecord.java:345)
        at 
sun.security.ssl.SSLEngineOutputRecord.encode(java.base@11.0.18/SSLEngineOutputRecord.java:287)
        at 
sun.security.ssl.SSLEngineOutputRecord.encode(java.base@11.0.18/SSLEngineOutputRecord.java:189)
        at 
sun.security.ssl.SSLEngineImpl.encode(java.base@11.0.18/SSLEngineImpl.java:285)
        at 
sun.security.ssl.SSLEngineImpl.writeRecord(java.base@11.0.18/SSLEngineImpl.java:231)
        at 
sun.security.ssl.SSLEngineImpl.wrap(java.base@11.0.18/SSLEngineImpl.java:136)
        - eliminated <0x7f0b6aab70c8> (a sun.security.ssl.SSLEngineImpl)
        at 
sun.security.ssl.SSLEngineImpl.wrap(java.base@11.0.18/SSLEngineImpl.java:116)
        - locked <0x7f0b6aab70c8> (a sun.security.ssl.SSLEngineImpl)
        at javax.net.ssl.SSLEngine.wrap(java.base@11.0.18/SSLEngine.java:522)
        at 
org.apache.flink.shaded.netty4.io.netty.handler.ssl.SslHandler.wrap(SslHandler.java:1071)
        at 
org.apache.flink.shaded.netty4.io.netty.handler.ssl.SslHandler.wrap(SslHandler.java:843)
        at 
org.apache.flink.shaded.netty4.io.netty.handler.ssl.SslHandler.wrapAndFlush(SslHandler.java:811)
        at 
org.apache.flink.shaded.netty4.io.netty.handler.ssl.SslHandler.flush(SslHandler.java:792)
        at 
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:750)
        at 
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:742)
        at 
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:728)
        at 
org.apache.flink.shaded.netty4.io.netty.channel.ChannelOutboundHandlerAdapter.flush(ChannelOutboundHandlerAdapter.java:125)
        at 
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:750)
        at 
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:765)
        at 
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:790)
        at 
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:758)
        at 
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:808)
        at 
org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1025)
        at 
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel.write

Rest ports

[David Radley]

Hello,
I notice:

  *the web UI listens on port 8081,
  *   The default port for that Flink listens on for rest calls is 8081.
  *   Specifying rest.port  in the configuration changes the port for both 
listeners.


We would like the 2 ports to be different so they can be secured differently, 
with the UI prompting an authentication for an invalid token, whereas the rest 
API would not.

We would need to do something equivalent for HA - 
https://nightlies.apache.org/flink/flink-docs-release-1.11/ops/jobmanager_high_availability.html#configuration
 where it says:

“To enable JobManager High Availability you have to set the high-availability 
mode to zookeeper, configure a ZooKeeper quorum and set up a masters file with 
all JobManagers hosts and their web UI ports.”

Have I missed anything here? What are your thoughts?
Kind regards, David.













Unless otherwise stated above:

IBM United Kingdom Limited
Registered in England and Wales with number 741598
Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU

RE: [Discuss] FLIP-366: Support standard YAML for FLINK configuration

[David Radley]

Hi,
I agree this is a standardising, simplifying change for read,  simplifying 
programmatically authoring the config file as well.  As you know the mapping 
for the old config form to the new form, could we write a migration tool, that 
would convert the old config files to the new format, with errors and warnings 
if the content is such that the user would need to manually fix up the file.
If there are minimal user fixups, we should consider automatically migrating 
the config file to the new format,
Kind regards, David.




From: Chesnay Schepler 
Date: Tuesday, 3 October 2023 at 11:17
To: dev@flink.apache.org 
Subject: [EXTERNAL] Re: [Discuss] FLIP-366: Support standard YAML for FLINK 
configuration
It is a unfortunate that we'll need a separate config file but the FLIP
does a good job justifying it.

Personally I'd just name it "config.yaml"; I never quite understood why
there was a flink prefix to begin with, and the current proposal
("flink-configuration.yaml") seems unnecessarily long.

For the deprecation process we could consider logging a warning if the
old parser is used.

In terms of scope, is the migration of existing e2e tests and the docker
image to the new parser part of the FLIP?

On 22/09/2023 09:32, Jane Chan wrote:
> Hi, Junrui,
>
> Sorry for the late reply. The update looks good to me and thanks for your
> effort!
>
> Best,
> Jane
>
> On Fri, Sep 22, 2023 at 2:25 PM Yuxin Tan  wrote:
>
>> Hi, Junrui
>>
>> +1 for the proposal.
>> Thanks for your effort.
>>
>> Best,
>> Yuxin
>>
>>
>> Samrat Deb  于2023年9月22日周五 13:23写道：
>>
>>> Hello Junrui,
>>>
>>> +1 for the proposal.
>>>
>>>
>>> Bests,
>>> Samrat
>>>
>>> On Fri, Sep 22, 2023 at 10:18 AM Shammon FY  wrote:
>>>
 +1 for the proposal, thanks for driving.

 Bet,
 Shammon FY

 On Fri, Sep 22, 2023 at 12:41 PM Yangze Guo 
>> wrote:
> Thanks for driving this, +1 for the proposal.
>
> Best,
> Yangze Guo
>
>
> On Fri, Sep 22, 2023 at 11:59 AM Lijie Wang <
>> wangdachui9...@gmail.com>
> wrote:
>> Hi Junrui,
>>
>> +1 for this proposal, thanks for driving.
>>
>> Best,
>> Lijie
>>
>> ConradJam  于2023年9月22日周五 10:07写道：
>>
>>> +1 Support for standard YAML format facilitates specification
>>>
>>> Jing Ge  于2023年9月22日周五 02:23写道：
>>>
 Hi Junrui,

 +1 for following the standard. Thanks for your effort!

 Best regards,
 Jing

 On Thu, Sep 21, 2023 at 5:09 AM Junrui Lee <
>> jrlee@gmail.com>
> wrote:
> Hi Jane,
>
> Thank you for your valuable feedback and suggestions.
> I agree with your point about differentiating between
>>> "flink-config.yaml"
> and "flink-conf.yaml" to determine the standard syntax at a
 glance.
> While I understand your suggestion of using
> "flink-conf-default.yaml"
>>> to
> represent the default YAML file for Flink 1.x, I have been
> considering
> the option of using "flink-configuration.yaml" as the file
>> name
> for the
> new configuration file.
> This name "flink-configuration.yaml" provides a clear
>>> distinction
>>> between
> the new and old configuration files based on their names, and
>>> it
> does
>>> not
> introduce any additional semantics. Moreover, this name
> "flink-configuration.yaml" can continue to be used in future
> versions
> FLINK-2.0.
>
> WDYT? If we can reach a consensus on this, I will update the
>>> FLIP
> documentation
> accordingly.
>
> Best regards,
> Junrui
>
> Jane Chan  于2023年9月20日周三 23:38写道：
>
>> Hi Junrui,
>>
>> Thanks for driving this FLIP. +1 for adoption of the
>> standard
> YAML
> syntax.
>> I just have one minor suggestion. It's a little bit
>>> challenging
> to
>> differentiate between `flink-config.yaml` and
>>> `flink-conf.yaml`
> to
>> determine which one uses the standard syntax at a glance.
>> How
> about
>> using `flink-conf-default.yaml` to represent the default
>> yaml
> file
>>> for
>> Flink 1.x?
>>
>> Best,
>> Jane
>>
>> On Wed, Sep 20, 2023 at 11:06 AM Junrui Lee <
 jrlee@gmail.com
 wrote:
>>> Hi devs,
>>>
>>> I would like to start a discussion about FLIP-366:
>>> Support standard YAML for FLINK configuration[1]
>>>
>>> The current flink-conf.yaml parser in FLINK is not a
>>> standard
> YAML
>> parser,
>>> which has some shortcomings.
>>> Firstly, it does not support nested structure
>> configuration
> items
>>> and
>> only
>>> supports key-value pairs, resulting in poor readability.
> Secondly,
>>> if
> the
>>

Re: [Discuss] FLIP-362: Support minimum resource limitation

[xiangyu feng]

Hi David,

Thx for your feedback.

First of all, for keeping some spare resources around, do you mean
'Redundant TaskManagers'[1]? If not, what is the difference between the
spare resources and redundant taskmanagers?

Secondly, IMHO the difference between min-reserved resource and spare
resources is that we could configure a rather large min-reserved resource
for user cases submitting lots of short-lived jobs concurrently, but we
don't want to configure a large spare resource since this might double the
total resource usage and lead to resource waste.

Looking forward to hearing from you.

Regards,
Xiangyu

[1] https://issues.apache.org/jira/browse/FLINK-18625

David Morávek  于2023年10月3日周二 05:00写道：

> H Xiangyui,
>
> The sentiment of the FLIP makes sense, but I keep wondering whether this
> is the best way to think about the problem. I assume that "interactive
> session cluster" users always want to keep some spare resources around (up
> to a configured threshold) to reduce cold start instead of statically
> configuring the minimum.
>
> It's just a tiny change from the original proposal, but it could make all
> the difference (eliminate overprovisioning, maintain latencies with a
> growing # of jobs, ..)
>
> WDYT?
>
> Best,
> D.
>
> On Mon, Sep 25, 2023 at 5:11 PM Jing Ge 
> wrote:
>
>> Hi Yangze,
>>
>> Thanks for the clarification. The example of two batch jobs team up with
>> one streaming job is interesting.
>>
>> Best regards,
>> Jing
>>
>> On Wed, Sep 20, 2023 at 7:19 PM Yangze Guo  wrote:
>>
>> > Thanks for the comments, Jing.
>> >
>> > > Will the minimum resource configuration also take effect for streaming
>> > jobs in application mode?
>> > > Since it is not recommended to configure
>> slotmanager.number-of-slots.max
>> > for streaming jobs, does it make sense to disable it for common
>> streaming
>> > jobs? At least disable the check for avoiding the oscillation?
>> >
>> > Yes. The minimum resource configuration will only disabled in
>> > standalone cluster atm. I agree it make sense to disable it for a pure
>> > streaming job, however:
>> > - By default, the minimum resource is configured to 0. If users do not
>> > proactively set it, either the oscillation check or the minimum
>> > restriction can be considered as disabled.
>> > - The minimum resource is a cluster-level configuration rather than a
>> > job-level configuration. If a user has an application with two batch
>> > jobs preceding the streaming job, they may also require this
>> > configuration to accelerate the execution of batch jobs.
>> >
>> > WDYT?
>> >
>> > Best,
>> > Yangze Guo
>> >
>> > On Thu, Sep 21, 2023 at 4:49 AM Jing Ge 
>> > wrote:
>> > >
>> > > Hi Xiangyu,
>> > >
>> > > Thanks for driving it! There is one thing I am not really sure if I
>> > > understand you correctly.
>> > >
>> > > According to the FLIP: "The minimum resource limitation will be
>> > implemented
>> > > in the DefaultResourceAllocationStrategy of FineGrainedSlotManager.
>> > >
>> > > Each time when SlotManager needs to reconcile the cluster resources or
>> > > fulfill job resource requirements, the
>> DefaultResourceAllocationStrategy
>> > > will check if the minimum resource requirement has been fulfilled. If
>> it
>> > is
>> > > not, DefaultResourceAllocationStrategy will request new
>> > PendingTaskManagers
>> > > and FineGrainedSlotManager will allocate new worker resources
>> > accordingly."
>> > >
>> > > "To avoid this oscillation, we need to check the worker number derived
>> > from
>> > > minimum and maximum resource configuration is consistent before
>> starting
>> > > SlotManager."
>> > >
>> > > Will the minimum resource configuration also take effect for streaming
>> > jobs
>> > > in application mode? Since it is not recommended to
>> > > configure slotmanager.number-of-slots.max for streaming jobs, does it
>> > make
>> > > sense to disable it for common streaming jobs? At least disable the
>> check
>> > > for avoiding the oscillation?
>> > >
>> > > Best regards,
>> > > Jing
>> > >
>> > >
>> > > On Tue, Sep 19, 2023 at 4:58 PM Chen Zhanghao <
>> zhanghao.c...@outlook.com
>> > >
>> > > wrote:
>> > >
>> > > > Thanks for driving this, Xiangyu. We use Session clusters for quick
>> SQL
>> > > > debugging internally, and found cold-start job submission slow due
>> to
>> > lack
>> > > > of the exact minimum resource reservation feature proposed here.
>> This
>> > > > should improve the experience a lot for running short lived-jobs in
>> > session
>> > > > clusters.
>> > > >
>> > > > Best,
>> > > > Zhanghao Chen
>> > > > 
>> > > > 发件人: Yangze Guo 
>> > > > 发送时间: 2023年9月19日 13:10
>> > > > 收件人: xiangyu feng 
>> > > > 抄送: dev@flink.apache.org 
>> > > > 主题: Re: [Discuss] FLIP-362: Support minimum resource limitation
>> > > >
>> > > > Thanks for driving this @Xiangyu. This is a feature that many users
>> > > > have requested for a long time. +1 for the overall proposal.
>> > > >
>> > > > Best,
>> > > > Yangze Guo
>> > > >
>>

[jira] [Created] (FLINK-33179) Improve reporting serialisation issues

[Dawid Wysakowicz (Jira)]

Dawid Wysakowicz created FLINK-33179:


 Summary: Improve reporting serialisation issues
 Key: FLINK-33179
 URL: https://issues.apache.org/jira/browse/FLINK-33179
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / Planner
Reporter: Dawid Wysakowicz
Assignee: Dawid Wysakowicz


FLINK-33158 shows that serialisation exceptions are not reported in a helpful 
manner. We should improve error reporting so that it gives more context what 
went wrong.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [ANNOUNCE] Release 1.18.0, release candidate #1

[Ferenc Csaky]

Thanks everyone for the efforts!

Checked the following:

- Downloaded artifacts
- Built Flink from source
- Verified checksums/signatures
- Verified NOTICE, LICENSE files
- Deployed dummy SELECT job via SQL gateway on standalone cluster, things 
seemed fine according to the log files

+1 (non-binding)

Best,
Ferenc


--- Original Message ---
On Friday, September 29th, 2023 at 22:12, Gabor Somogyi 
 wrote:


> 
> 
> Thanks for the efforts!
> 
> +1 (non-binding)
> 
> * Verified versions in the poms
> * Built from source
> * Verified checksums and signatures
> * Started basic workloads with kubernetes operator
> * Verified NOTICE and LICENSE files
> 
> G
> 
> On Fri, Sep 29, 2023, 18:16 Matthias Pohl matthias.p...@aiven.io.invalid
> 
> wrote:
> 
> > Thanks for creating RC1. I did the following checks:
> > 
> > * Downloaded artifacts
> > * Built Flink from sources
> > * Verified SHA512 checksums GPG signatures
> > * Compared checkout with provided sources
> > * Verified pom file versions
> > * Went over NOTICE file/pom files changes without finding anything
> > suspicious
> > * Deployed standalone session cluster and ran WordCount example in batch
> > and streaming: Nothing suspicious in log files found
> > 
> > +1 (binding)
> > 
> > On Fri, Sep 29, 2023 at 10:34 AM Etienne Chauchot echauc...@apache.org
> > wrote:
> > 
> > > Hi all,
> > > 
> > > Thanks to the team for this RC.
> > > 
> > > I did a quick check of this RC against user pipelines (1) coded with
> > > DataSet (even if deprecated and soon removed), DataStream and SQL APIs
> > > 
> > > based on the small scope of this test, LGTM
> > > 
> > > +1 (non-binding)
> > > 
> > > [1] https://github.com/echauchot/tpcds-benchmark-flink
> > > 
> > > Best
> > > Etienne
> > > 
> > > Le 28/09/2023 à 19:35, Jing Ge a écrit :
> > > 
> > > > Hi everyone,
> > > > 
> > > > The RC1 for Apache Flink 1.18.0 has been created. The related voting
> > > > process will be triggered once the announcement is ready. The RC1 has
> > > > all
> > > > the artifacts that we would typically have for a release, except for
> > > > the
> > > > release note and the website pull request for the release announcement.
> > > > 
> > > > The following contents are available for your review:
> > > > 
> > > > - Confirmation of no benchmarks regression at the thread[1].
> > > > - The preview source release and binary convenience releases [2], which
> > > > are signed with the key with fingerprint 96AE0E32CBE6E0753CE6 [3].
> > > > - all artifacts that would normally be deployed to the Maven
> > > > Central Repository [4].
> > > > - source code tag "release-1.18.0-rc1" [5]
> > > > 
> > > > Your help testing the release will be greatly appreciated! And we'll
> > > > create the rc1 release and the voting thread as soon as all the efforts
> > > > are
> > > > finished.
> > > > 
> > > > [1]https://lists.apache.org/thread/yxyphglwwvq57wcqlfrnk3qo9t3sr2ro
> > > > [2]https://dist.apache.org/repos/dist/dev/flink/flink-1.18.0-rc1/
> > > > [3]https://dist.apache.org/repos/dist/release/flink/KEYS
> > > > [4]
> > > > https://repository.apache.org/content/repositories/orgapacheflink-1657
> > > > [5]https://github.com/apache/flink/releases/tag/release-1.18.0-rc1
> > > > 
> > > > Best regards,
> > > > Qingsheng, Sergei, Konstantin and Jing

Re: [DISCUSS] [FLINK-32873] Add a config to allow disabling Query hints

[Dawid Wysakowicz]

Hey all,
My understanding was that from the first message we were discussing 
throwing an exception. Oracle was only shown as an example of a system 
that have a flag for hints behaviour.


Let's get back to the discussion and agree on the behavior. My 
suggestion is to introduce an enum instead of a boolean flag since 
apparently there are different requirements. My take is that it is worth 
to have an option to throw an exception if hints are disabled and are 
provided in the SQL query. This is the same behavior as disabling 
OPTIONS hint we already have[1]


Since you both @Jing and @Sergey would rather like to have an option to 
ignore them we can introduce


table.optimizer.query-options = ENABLED/DISABLED/IGNORED

ENABLED: hints just work

DISABLED: throw an exception

IGNORED: ignore hints

Are you two fine with that option @Jing @Sergey?

Since this thread had a few misunderstandings already, I'd suggest to 
convert it to a FLIP and follow with a VOTE shortly. @Bonnie would you 
like to help with that?


Best,

Dawid

[1] 
https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/config/#table-dynamic-table-options-enabled


On 02/10/2023 18:18, Jing Ge wrote:

Hi,

I have the same concern as Sergey and would like to make sure the purpose
of this discussion is to globally ignore hints without changing any other
behaviours, if I am not mistaken. Thanks!

Best regards,
Jing

On Mon, Oct 2, 2023 at 3:40 PM Sergey Nuyanzin  wrote:


Hi Bonnie,

I think changing it to something like .enabled
could lead to misunderstanding
for instance when we set this flag to false what should it mean?
1. Just ignore hints and execute a query like the same query however with
removed hints
2. Fail on validation because hints are disabled
3. something else

I tend to think that we are talking about just ignoring hints, so option 1
(and Oracle follows option 1 as well as mentioned at [1]).
So I would suggest to keep ignore in property name to make it more clear

Please let me know if I miss anything

[1]

https://docs.oracle.com/en/database/oracle/oracle-database/23/refrn/OPTIMIZER_IGNORE_HINTS.html#GUID-D62CA6D8-D0D8-4A20-93EA-EEB4B3144347

On Fri, Sep 29, 2023 at 7:20 PM Bonnie Arogyam Varghese
 wrote:


Hi Jark,
  A minor suggestion. Would it be ok if we changed the config name to `
table.optimizer.query-options.enabled`?
This is inline with other existing configurations such as:

table.dynamic-table-options.enabled
table.optimizer.distinct-agg.split.enabled
table.optimizer.dynamic-filtering.enabled


On Wed, Sep 27, 2023 at 9:57 AM Bonnie Arogyam Varghese <
bvargh...@confluent.io> wrote:


Hi Martjin,
Yes, the suggestion for the configuration name made by Jark sounds

good.

Also, thanks to everyone who participated in this discussion.

On Tue, Sep 19, 2023 at 2:40 PM Martijn Visser <

martijnvis...@apache.org

wrote:


Hey Jark,

Sounds fine to me.
@Bonnie does that also work for you?

Best regards,

Martijn

On Fri, Sep 15, 2023 at 1:28 PM Jark Wu  wrote:

Hi Martijn,

Thanks for the investigation. I found the blog[1] shows a use case
that they use "OPTIMIZER_IGNORE_HINTS" to check if embedded
hints can be removed in legacy code. I think this is a useful tool

to

improve queries without complex hints strewn throughout the code.
Therefore, I'm fine to support this now.

Maybe we can follow Oracle to name the config
"table.optimizer.ignore-query-hints=false"?

Best,
Jark

[1]: https://dbaora.com/optimizer_ignore_hints-oracle-database-18c/

On Fri, 15 Sept 2023 at 17:57, Martijn Visser <

martijnvis...@apache.org

wrote:


Hi Jark,

Oracle has/had a generic "OPTIMIZER_IGNORE_HINTS" [1]. It looks

like

MSSQL

has configuration options to disable specific hints [2] instead

of a

generic solution.

[1]



https://docs.oracle.com/en/database/oracle/oracle-database/23/refrn/OPTIMIZER_IGNORE_HINTS.html#GUID-D62CA6D8-D0D8-4A20-93EA-EEB4B3144347

[2]



https://www.mssqltips.com/sqlservertip/4175/disabling-sql-server-optimizer-rules-with-queryruleoff/

Best regards,

Martijn

On Fri, Sep 15, 2023 at 10:53 AM Jark Wu 

wrote:

Hi Martijn,

I can understand that.
Is there any database/system that supports disabling/enabling

query

hints

  with a configuration? This might help us to understand the use

case,

and follow the approach.

Best,
Jark

On Fri, 15 Sept 2023 at 15:34, Martijn Visser <

martijnvis...@apache.org>

wrote:


Hi all,

I think Jark has a valid point with:


Does this mean that in the future we might add an option to

disable

each

feature?

I don't think that's a reasonable outcome indeed, but we are

currently

in a

situation where we don't have clear guidelines on when to add

a

configuration option, and when not to add one. From a platform

perspective,

there might not be an imminent or obvious security

implication,

but you

want to minimize a potential attack surface.


We should try to remove the unnecessary configuration from

the

list

in

Flink 2.0.

I agree with that too.

With t

Re: Support AWS SDK V2 for Flink's S3 FileSystem

[Martijn Visser]

+1 for David's suggestion. We should get away from the current
approach with two abstractions and get to one rock solid one.

On Mon, Oct 2, 2023 at 11:13 PM David Morávek  wrote:
>
> Hi Maomao,
>
> I wonder whether it would make sense to take a stab at consolidating the S3
> filesystems instead and introduce a native one. The whole Hadoop wrapper
> around the S3 client exists for legacy reasons, and it adds complexity and
> probably an unnecessary performance penalty.
>
> If you take a look at the underlying presto implementation, it's actually
> not too complex to adapt to Flink interfaces (since you're proposing to
> maintain a copy of it anyway).
>
> Overall, the S3 FS is probably the most used one that we have so this could
> be rather high impact. It would also eliminate user confusion when choosing
> the implementation to use.
>
> WDYT?
>
> Best,
> D.
>
> On Fri, Sep 29, 2023 at 2:41 PM Min, Maomao 
> wrote:
>
> > Hi Flink Dev,
> >
> > I’m Maomao, a developer from AWS EMR.
> >
> > Recently, our team is working on adding AWS SDK V2 support for Flink’s S3
> > Filesystem. During development, we found out that our work was blocked by
> > Presto. This is because that Presto still uses AWS SDK V1 and won’t add
> > support for AWS SDK V2 in short term. To unblock, our team proposed several
> > options and I’ve created a JIRA issue as here<
> > https://issues.apache.org/jira/browse/FLINK-33157>.
> >
> > Since our team plans to contribute this work back to the community later,
> > we’d like to collect feedback from the community about the options we
> > proposed in the long term so that the community won’t need to duplicate
> > this work in the future.
> >
> > Best,
> > Maomao
> >
> >

[jira] [Created] (FLINK-33180) Implement support for processing subset of shards/traffic

[Chandra Prakash (Jira)]

Chandra Prakash created FLINK-33180:
---

 Summary: Implement support for processing subset of  shards/traffic
 Key: FLINK-33180
 URL: https://issues.apache.org/jira/browse/FLINK-33180
 Project: Flink
  Issue Type: Sub-task
Reporter: Chandra Prakash


Support processing of subset of shards/traffic. This feature is require to 
implement canary / onebox.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Close orphaned/stale PRs

[Venkatakrishnan Sowrirajan]

Gentle ping to surface this up for more discussions.

Regards
Venkata krishnan


On Tue, Sep 26, 2023 at 4:59 PM Venkatakrishnan Sowrirajan 
wrote:

> Hi Martijn,
>
> Agree with your point that closing a PR without any review feedback even
> after 'X' days is discouraging to a new contributor. I understand that this
> is a capacity problem. Capacity problem cannot be solved by this proposal
> and it is beyond the scope of this proposal.
>
> Regarding your earlier question,
> > What's the added value of
> closing these PRs
>
>- Having lots of inactive PRs lingering around shows the project is
>less active. I am not saying this is the only way to determine how active a
>project is, but this is one of the key factors.
>- A large number of PRs open can be discouraging for (new)
>contributors but on the other hand I agree closing an inactive PR without
>any reviews can also drive contributors away.
>
> Having said all of that, I agree closing PRs that don't have any reviews
> to start with should be avoided from the final proposal.
>
> > I'm +1 for (automatically) closing up PRs after X days which:
> a) Don't have a CI that has passed
> b) Don't follow the code contribution guide (like commit naming
> conventions)
> c) Have changes requested but aren't being followed-up by the contributor
>
> In general, I'm largely +1 on your above proposal except for the
> implementation feasibility.
>
> Also, I have picked a few other popular projects that have implemented the
> Github's actions stale rule to see if we can borrow some ideas. Below
> projects are listed in the order of the most invasive (for lack of a better
> word) to the least invasive actions taken wrt PR without any updates for a
> long period of time.
>
> 1. Trino
>
> TL;DR - No updates in the PR for the last 21 days, tag other maintainers
> for review. If there are no updates for 21 days after that, close the PR
> with this message - "*Closing this pull request, as it has been stale for
> six weeks. Feel free to re-open at any time.*"
> Trino's stale PR Github action rule (stale.yaml)
> 
>
>
> 2. Apache Spark
>
> TL;DR - No updates in the PR in the last 100 days, closing the PR with
> this message - "*We're closing this PR because it hasn't been updated in
> a while. This isn't a judgement on the merit of the PR in any way. It's
> just a way of keeping the PR queue manageable. If you'd like to revive this
> PR, please reopen it and ask a committer to remove the Stale tag!*"
> Spark's discussion in their mailing list
>  on
> closing stale PRs. Spark's stale PR github action rule (stale.yaml
> 
> ).
>
> 3. Python
>
> TL;DR - No updates in the PR for the last 30 days, then tag the PR as
> stale. Note: Python project *doesn't* close the stale PRs.
>
> Python discussion
> 
> in the mailing list to close stale PRs. Python's stale PR github action
> rule (stale.yaml
> )
>
> Few others Apache Beam
>  
> (closes
> inactive PRs after 60+ days), Apache Airflow
>  
> (closes
> inactive PRs after 50 days)
>
> Let me know what you think. Looking forward to hearing from others in the
> community and their experiences.
>
> [1] Github Action - Close Stale Issues -
> https://github.com/marketplace/actions/close-stale-issues
>
> Regards
> Venkata krishnan
>
>
> On Thu, Sep 21, 2023 at 6:03 AM Martijn Visser 
> wrote:
>
>> Hi all,
>>
>> I really believe that the problem of the number of open PRs is just
>> that there aren't enough reviewers/resources available to review them.
>>
>> > Stale PRs can clutter the repository, and closing them helps keep it
>> organized and ensures that only relevant and up-to-date PRs are present.
>>
>> Sure, but what's the indicator that the PR is stale? The fact that
>> there has been no reviewer yet to review it, doesn't mean that the PR
>> is stale. For me, a stale PR is a PR that has been reviewed, changes
>> have been requested and the contributor isn't participating in the
>> discussion anymore. But that's a different story compared to closing
>> PRs where there has been no review done at all.
>>
>> > It mainly helps the project maintainers/reviewers to focus on only the
>> actively updated trimmed list of PRs that are ready for review.
>>
>> I disagree that closing PRs helps with this. If you want to help
>> maintainers/reviewers, we should have a situation where it's obvious
>> that a PR is really ready (meaning, CI has passed, PR contents/commit
>> message etc are