[jira] [Created] (FLINK-33513) Metastore delegation-token can be cached?

[katty he (Jira)]

katty he created FLINK-33513:


 Summary: Metastore delegation-token can be cached?
 Key: FLINK-33513
 URL: https://issues.apache.org/jira/browse/FLINK-33513
 Project: Flink
  Issue Type: Improvement
  Components: Connectors / Hive
Reporter: katty he


Now, every time, getDelegationToken wil be called when asking for metastore, 
how about build a cache, we cache the token for the first time, then we can 
just get token from cache?



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33514) FlinkScalaKryoInstantiator class not found in KryoSerializer

[Jake.zhang (Jira)]

Jake.zhang created FLINK-33514:
--

 Summary: FlinkScalaKryoInstantiator class not found in 
KryoSerializer
 Key: FLINK-33514
 URL: https://issues.apache.org/jira/browse/FLINK-33514
 Project: Flink
  Issue Type: Bug
  Components: API / Core
Affects Versions: 1.18.0
Reporter: Jake.zhang


{code:java}
16:03:13,402 INFO  
org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer [] - Kryo 
serializer scala extensions are not available.
java.lang.ClassNotFoundException: 
org.apache.flink.runtime.types.FlinkScalaKryoInstantiator
    at java.net.URLClassLoader.findClass(URLClassLoader.java:387) ~[?:1.8.0_341]
    at java.lang.ClassLoader.loadClass(ClassLoader.java:418) ~[?:1.8.0_341]
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355) 
~[?:1.8.0_341]
    at java.lang.ClassLoader.loadClass(ClassLoader.java:351) ~[?:1.8.0_341]
    at java.lang.Class.forName0(Native Method) ~[?:1.8.0_341]
    at java.lang.Class.forName(Class.java:264) ~[?:1.8.0_341]
    at 
org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.getKryoInstance(KryoSerializer.java:487)
 ~[flink-core-1.18.0.jar:1.18.0]
    at 
org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.checkKryoInitialized(KryoSerializer.java:522)
 ~[flink-core-1.18.0.jar:1.18.0]
    at 
org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.deserialize(KryoSerializer.java:394)
 ~[flink-core-1.18.0.jar:1.18.0]
    at 
org.apache.flink.api.java.typeutils.runtime.PojoSerializer.deserialize(PojoSerializer.java:412)
 ~[flink-core-1.18.0.jar:1.18.0]
    at 
org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer.deserialize(StreamElementSerializer.java:190)
 ~[flink-streaming-java-1.18.0.jar:1.18.0]
    at 
org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer.deserialize(StreamElementSerializer.java:43)
 ~[flink-streaming-java-1.18.0.jar:1.18.0]
    at 
org.apache.flink.runtime.plugable.NonReusingDeserializationDelegate.read(NonReusingDeserializationDelegate.java:53)
 ~[flink-runtime-1.18.0.jar:1.18.0]
    at 
org.apache.flink.runtime.io.network.api.serialization.NonSpanningWrapper.readInto(NonSpanningWrapper.java:337)
 ~[flink-runtime-1.18.0.jar:1.18.0]
    at 
org.apache.flink.runtime.io.network.api.serialization.SpillingAdaptiveSpanningRecordDeserializer.readNonSpanningRecord(SpillingAdaptiveSpanningRecordDeserializer.java:128)
 ~[flink-runtime-1.18.0.jar:1.18.0]
    at 
org.apache.flink.runtime.io.network.api.serialization.SpillingAdaptiveSpanningRecordDeserializer.readNextRecord(SpillingAdaptiveSpanningRecordDeserializer.java:103)
 ~[flink-runtime-1.18.0.jar:1.18.0]
    at 
org.apache.flink.runtime.io.network.api.serialization.SpillingAdaptiveSpanningRecordDeserializer.getNextRecord(SpillingAdaptiveSpanningRecordDeserializer.java:93)
 ~[flink-runtime-1.18.0.jar:1.18.0]
    at 
org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:100)
 ~[flink-streaming-java-1.18.0.jar:1.18.0]
    at 
org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
 ~[flink-streaming-java-1.18.0.jar:1.18.0]
    at 
org.apache.flink.streaming.runtime.io.StreamMultipleInputProcessor.processInput(StreamMultipleInputProcessor.java:85)
 ~[flink-streaming-java-1.18.0.jar:1.18.0]
    at 
org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:562)
 ~[flink-streaming-java-1.18.0.jar:1.18.0]
    at 
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:231)
 ~[flink-streaming-java-1.18.0.jar:1.18.0]
    at 
org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:858)
 ~[flink-streaming-java-1.18.0.jar:1.18.0]
    at 
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:807) 
~[flink-streaming-java-1.18.0.jar:1.18.0]
    at 
org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:953)
 [flink-runtime-1.18.0.jar:1.18.0]
    at 
org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:932) 
[flink-runtime-1.18.0.jar:1.18.0]
    at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:746) 
[flink-runtime-1.18.0.jar:1.18.0]
    at org.apache.flink.runtime.taskmanager.Task.run(Task.java:562) 
[flink-runtime-1.18.0.jar:1.18.0]
    at java.lang.Thread.run(Thread.java:750) [?:1.8.0_341] {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33515) PythonDriver need to stream python process output to log instead of collecting it in memory

[Gabor Somogyi (Jira)]

Gabor Somogyi created FLINK-33515:
-

 Summary: PythonDriver need to stream python process output to log 
instead of collecting it in memory
 Key: FLINK-33515
 URL: https://issues.apache.org/jira/browse/FLINK-33515
 Project: Flink
  Issue Type: Bug
  Components: API / Python
Affects Versions: 1.19.0
Reporter: Gabor Somogyi


PythonDriver now collects the python process output in a Stringbuilder instead 
of streaming it. It can cause OOM when the python process is generating huge 
amount of output.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33516) Create dedicated PyFlink channel

[Martijn Visser (Jira)]

Martijn Visser created FLINK-33516:
--

 Summary: Create dedicated PyFlink channel
 Key: FLINK-33516
 URL: https://issues.apache.org/jira/browse/FLINK-33516
 Project: Flink
  Issue Type: Improvement
  Components: Documentation
Reporter: Martijn Visser
Assignee: Martijn Visser


See https://lists.apache.org/thread/ynb5drhqqbd84w4o4337qv47100cp67h

1. Create new Slack channel
2. Update website 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

RE: Request a release of flink-connector-kafka version 3.1.0 (to consume kafka 3.4.0 with Flink 1.18)

[Jean-Marc Paulin]

Hi,

I am not exactly thrilled by the False positive statement. This always leads to 
a difficult discussion with customers.

Is there a chance of releasing a version of the connector to just add support 
for Kafka 3.4.0, in conjunction with Flink 1.18 ?

Kind regards

Jean-Marc

From: Martijn Visser 
Sent: Thursday, November 9, 2023 13:51
To: dev@flink.apache.org ; Mason Chen 

Subject: [EXTERNAL] Re: Request a release of flink-connector-kafka version 
3.1.0 (to consume kafka 3.4.0 with Flink 1.18)

Hi,

The CVE is related to the Kafka Connect API and I think of that as a
false-positive for the Flink Kafka connector. I would be inclined to
preferably get https://issues.apache.org/jira/browse/FLINK-32197  in,
and then do a release afterwards. But I would like to understand from
Mason if he thinks that's feasible.

Best regards,

Martijn

On Tue, Nov 7, 2023 at 9:45 AM Jean-Marc Paulin  wrote:
>
> Hi,
>
> I had a chat on [FLINK-31599] Update kafka version to 3.4.0 by Ge · Pull 
> Request #11 · apache/flink-connector-kafka 
> (github.com) .
>
> We are consuming Flink 1.18, and the flink-connector-kafka 3.0.1.
> Flink 3.2.3 currently in use has the  
> CVE-2023-25194  >  vulnerability addressed in Kafka 3.4.0. We will need to move to Kafka 
> 3.4.0 for our customers. I have tried to consume Kafka client 3.4.0 but that 
> fails after a while. I tracked that down to a change required in the 
> flink-connector-kafka source code. The PR11 above has the required changes, 
> and is merge in main, but is not currently released.
>
> I would really appreciate if you could release a newer version of the 
> flink-connector-kafka that would enable us to use Kafka 3.4.0.
>
> Many thanks
>
> JM
>
> [https://opengraph.githubassets.com/54669eeddff74373a431b6540c3602aefd5fb25232da040f59d9dbb1254615c6/apache/flink-connector-kafka/pull/11
>  ]
> [FLINK-31599] Update kafka version to 3.4.0 by Ge · Pull Request #11 · 
> apache/flink-connector-kafka  >
> Apache flink. Contribute to apache/flink-connector-kafka development by 
> creating an account on GitHub.
> github.com
>
> Unless otherwise stated above:
>
> IBM United Kingdom Limited
> Registered in England and Wales with number 741598
> Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU

Unless otherwise stated above:

IBM United Kingdom Limited
Registered in England and Wales with number 741598
Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU

[jira] [Created] (FLINK-33517) Implement restore tests for Value node

[Jacky Lau (Jira)]

Jacky Lau created FLINK-33517:
-

 Summary: Implement restore tests for Value node
 Key: FLINK-33517
 URL: https://issues.apache.org/jira/browse/FLINK-33517
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / Planner
Affects Versions: 1.19.0
Reporter: Jacky Lau
 Fix For: 1.19.0






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Adding a new channel to Apache Flink slack workspace

[Martijn Visser]

Hi all,

I've created https://issues.apache.org/jira/browse/FLINK-33516 and
I'll work on this. Thanks!

Cheers, Martijn

On Fri, Nov 10, 2023 at 7:50 AM Yun Tang  wrote:
>
> +1 and glad to see more guys are using PyFlink.
>
> Best
> Yun Tang
> 
> From: Jing Ge 
> Sent: Wednesday, November 8, 2023 3:18
> To: ro...@decodable.co.invalid 
> Cc: dev@flink.apache.org 
> Subject: Re: Adding a new channel to Apache Flink slack workspace
>
> +1 since there are so many questions wrt PyFlink.
>
> Best regards,
> Jing
>
> On Tue, Nov 7, 2023 at 2:23 AM Robin Moffatt 
> wrote:
>
> > Since there have been no objections, can we go ahead and get this channel
> > created please?
> >
> > thanks :)
> >
> > On Thu, 26 Oct 2023 at 16:00, Alexander Fedulov <
> > alexander.fedu...@gmail.com>
> > wrote:
> >
> > > +1
> > > I agree that the topic is distinct enough to justify a dedicated channel
> > > and this could lead to more active participation from people who work on
> > > it.
> > >
> > > Best,
> > > Alexander
> > >
> > > On Wed, 25 Oct 2023 at 20:03, Robin Moffatt  wrote:
> > >
> > > > Hi,
> > > >
> > > > I'd like to propose adding a PyFlink channel to the Apache Flink slack
> > > > workspace.
> > > >
> > > > By creating a channel focussed on this it will help people find
> > previous
> > > > discussions as well as target new discussions and questions to the
> > > correct
> > > > place. PyFlink is a sufficiently distinct component to make a dedicated
> > > > channel viable and useful IMHO.
> > > >
> > > > There was a brief discussion of this on Slack already, the archive for
> > > > which can be found here:
> > > >
> > > >
> > >
> > https://www.linen.dev/s/apache-flink/t/16006099/re-surfacing-for-the-admins-https-apache-flink-slack-com-arc#1c7e-177a-4c37-8a34-a917883152ac
> > > >
> > > > thanks,
> > > >
> > > > Robin.
> > > >
> > >
> >

[jira] [Created] (FLINK-33518) Implement restore tests for WatermarkAssigner node

[Jacky Lau (Jira)]

Jacky Lau created FLINK-33518:
-

 Summary: Implement restore tests for WatermarkAssigner node
 Key: FLINK-33518
 URL: https://issues.apache.org/jira/browse/FLINK-33518
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / Planner
Affects Versions: 1.19.0
Reporter: Jacky Lau
 Fix For: 1.19.0






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Request a release of flink-connector-kafka version 3.1.0 (to consume kafka 3.4.0 with Flink 1.18)

[Martijn Visser]

Hi Jean-Marc,

To be fair, the Flink project has a lot of dependencies that have
false-positives from a Flink pov. We just can't fix all of them.

Let's see what others say on this topic.

Best regards,

Martijn

On Fri, Nov 10, 2023 at 10:56 AM Jean-Marc Paulin  wrote:
>
> Hi,
>
> I am not exactly thrilled by the False positive statement. This always leads 
> to a difficult discussion with customers.
>
> Is there a chance of releasing a version of the connector to just add support 
> for Kafka 3.4.0, in conjunction with Flink 1.18 ?
>
> Kind regards
>
> Jean-Marc
> 
> From: Martijn Visser 
> Sent: Thursday, November 9, 2023 13:51
> To: dev@flink.apache.org ; Mason Chen 
> 
> Subject: [EXTERNAL] Re: Request a release of flink-connector-kafka version 
> 3.1.0 (to consume kafka 3.4.0 with Flink 1.18)
>
> Hi,
>
> The CVE is related to the Kafka Connect API and I think of that as a
> false-positive for the Flink Kafka connector. I would be inclined to
> preferably get https://issues.apache.org/jira/browse/FLINK-32197  in,
> and then do a release afterwards. But I would like to understand from
> Mason if he thinks that's feasible.
>
> Best regards,
>
> Martijn
>
> On Tue, Nov 7, 2023 at 9:45 AM Jean-Marc Paulin  wrote:
> >
> > Hi,
> >
> > I had a chat on [FLINK-31599] Update kafka version to 3.4.0 by Ge · 
> > Pull Request #11 · apache/flink-connector-kafka 
> > (github.com) .
> >
> > We are consuming Flink 1.18, and the flink-connector-kafka 3.0.1.
> > Flink 3.2.3 currently in use has the  
> > CVE-2023-25194 >  >  vulnerability addressed in Kafka 3.4.0. We will need to move to Kafka 
> > 3.4.0 for our customers. I have tried to consume Kafka client 3.4.0 but 
> > that fails after a while. I tracked that down to a change required in the 
> > flink-connector-kafka source code. The PR11 above has the required changes, 
> > and is merge in main, but is not currently released.
> >
> > I would really appreciate if you could release a newer version of the 
> > flink-connector-kafka that would enable us to use Kafka 3.4.0.
> >
> > Many thanks
> >
> > JM
> >
> > [https://opengraph.githubassets.com/54669eeddff74373a431b6540c3602aefd5fb25232da040f59d9dbb1254615c6/apache/flink-connector-kafka/pull/11
> >  ]
> > [FLINK-31599] Update kafka version to 3.4.0 by Ge · Pull Request #11 · 
> > apache/flink-connector-kafka >  >
> > Apache flink. Contribute to apache/flink-connector-kafka development by 
> > creating an account on GitHub.
> > github.com
> >
> > Unless otherwise stated above:
> >
> > IBM United Kingdom Limited
> > Registered in England and Wales with number 741598
> > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
>
> Unless otherwise stated above:
>
> IBM United Kingdom Limited
> Registered in England and Wales with number 741598
> Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU

Re: [VOTE] FLIP-381: Deprecate configuration getters/setters that return/set complex Java objects

[Rui Fan]

+1(binding)

Best,
Rui

On Fri, Nov 10, 2023 at 11:58 AM Junrui Lee  wrote:

> Hi everyone,
>
> Thank you to everyone for the feedback on FLIP-381: Deprecate configuration
> getters/setters that return/set complex Java objects[1] which has been
> discussed in this thread [2].
>
> I would like to start a vote for it. The vote will be open for at least 72
> hours (excluding weekends) unless there is an objection or not enough
> votes.
>
> [1]
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=278464992
> [2]https://lists.apache.org/thread/y5owjkfxq3xs9lmpdbl6d6jmqdgbjqxo
>

Re: [DISCUSS] FLIP-364: Improve the restart-strategy

[Rui Fan]

I'll start voting next Monday if there isn't any other comment.

Best,
Rui

On Thu, Oct 19, 2023 at 6:59 PM Rui Fan <1996fan...@gmail.com> wrote:

> Hi Konstantin and Max,
>
> Thanks for your feedback!
>
> Sorry, I forgot to mention the default value of
> `restart-strategy.exponential-delay.max-attempts-before-reset-backoff`.
>
> Retrying forever sounds good to me, I have added it to the FLIP:
>
> The default value of
> `restart-strategy.exponential-delay.max-attempts-before-reset-backoff` is
> Integer.MAX_VALUE.
>
> Best,
> Rui
>
> On Thu, Oct 19, 2023 at 6:29 PM Maximilian Michels  wrote:
>
>> Hey Rui,
>>
>> +1 for making exponential backoff the default. I agree with Konstantin
>> that retrying forever is a good default for exponential backoff
>> because oftentimes the issue will resolve eventually. The purpose of
>> exponential backoff is precisely to continue to retry without causing
>> too much load. However, I'm not against adding an optional max number
>> of retries.
>>
>> -Max
>>
>> On Thu, Oct 19, 2023 at 11:35 AM Konstantin Knauf 
>> wrote:
>> >
>> > Hi Rui,
>> >
>> > Thank you for this proposal and working on this. I also agree that
>> > exponential back off makes sense as a new default in general. I think
>> > restarting indefinitely (no max attempts) makes sense by default,
>> though,
>> > but of course allowing users to change is valuable.
>> >
>> > So, overall +1.
>> >
>> > Cheers,
>> >
>> > Konstantin
>> >
>> > Am Di., 17. Okt. 2023 um 07:11 Uhr schrieb Rui Fan <
>> 1996fan...@gmail.com>:
>> >
>> > > Hi all,
>> > >
>> > > I would like to start a discussion on FLIP-364: Improve the
>> > > restart-strategy[1]
>> > >
>> > > As we know, the restart-strategy is critical for flink jobs, it mainly
>> > > has two functions:
>> > > 1. When an exception occurs in the flink job, quickly restart the job
>> > > so that the job can return to the running state.
>> > > 2. When a job cannot be recovered after frequent restarts within
>> > > a certain period of time, Flink will not retry but will fail the job.
>> > >
>> > > The current restart-strategy support for function 2 has some issues:
>> > > 1. The exponential-delay doesn't have the max attempts mechanism,
>> > > it means that flink will restart indefinitely even if it fails
>> frequently.
>> > > 2. For multi-region streaming jobs and all batch jobs, the failure of
>> > > each region will increase the total number of job failures by +1,
>> > > even if these failures occur at the same time. If the number of
>> > > failures increases too quickly, it will be difficult to set a
>> reasonable
>> > > number of retries.
>> > > If the maximum number of failures is set too low, the job can easily
>> > > reach the retry limit, causing the job to fail. If set too high, some
>> jobs
>> > > will never fail.
>> > >
>> > > In addition, when the above two problems are solved, we can also
>> > > discuss whether exponential-delay can replace fixed-delay as the
>> > > default restart-strategy. In theory, exponential-delay is smarter and
>> > > friendlier than fixed-delay.
>> > >
>> > > I also thank Zhu Zhu for his suggestions on the option name in
>> > > FLINK-32895[2] in advance.
>> > >
>> > > Looking forward to and welcome everyone's feedback and suggestions,
>> thank
>> > > you.
>> > >
>> > > [1] https://cwiki.apache.org/confluence/x/uJqzDw
>> > > [2] https://issues.apache.org/jira/browse/FLINK-32895
>> > >
>> > > Best,
>> > > Rui
>> > >
>> >
>> >
>> > --
>> > https://twitter.com/snntrable
>> > https://github.com/knaufk
>>
>

Re: [VOTE] FLIP-381: Deprecate configuration getters/setters that return/set complex Java objects

[Yuepeng Pan]

+1(non-binding)

Best,
Roc

On 2023/11/10 03:58:10 Junrui Lee wrote:
> Hi everyone,
> 
> Thank you to everyone for the feedback on FLIP-381: Deprecate configuration
> getters/setters that return/set complex Java objects[1] which has been
> discussed in this thread [2].
> 
> I would like to start a vote for it. The vote will be open for at least 72
> hours (excluding weekends) unless there is an objection or not enough votes.
> 
> [1]
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=278464992
> [2]https://lists.apache.org/thread/y5owjkfxq3xs9lmpdbl6d6jmqdgbjqxo
> 

Re: [DISCUSS] Release flink-connector-pulsar 4.1.0

[tison]

> does it include support for Flink 1.18?

Not yet. Tests for 1.16 and 1.17 can pass, but the latest 1.18-SNAPSHOT is
not (and ditto 1.18.0). I'm afraid it's not a trivial fix so let's mark it
as an issue but not a blocker.

Best,
tison.


Martijn Visser  于2023年11月9日周四 16:21写道：

> Hi Tison,
>
> I would be +1 for releasing it, but does it include support for Flink
> 1.18? I think that the tests still failed for it, and I think that
> support should be in place before releasing a new version of the
> connector. What do you think?
>
> Best regards,
>
> Martijn
>
> On Thu, Nov 9, 2023 at 7:09 AM Leonard Xu  wrote:
> >
> > Hey, Tison.
> >
> > +1 to release  flink-connector-pulsar 4.1.0.
> >
> > I’m glad to offer help for the release.
> >
> >
> > Best,
> > Leonard
> >
> >
> >
> > > 2023年11月9日 下午1:30，tison  写道：
> > >
> > > Hi,
> > >
> > > I'd propose to cut a new release for flink-connector-pulsar 4.1.0[1].
> > >
> > > From the last release (4.0.0), we mainly achieved:
> > >
> > > 1. Implement table connector (integrated with Flink SQL)
> > > 2. Drop the requirement for using adminURL
> > > 3. Support JDK 11
> > >
> > > I can help in driving the release but perhaps we need some more PMC
> > > members' attention and help.
> > >
> > > What do you think?
> > >
> > > Best,
> > > tison.
> > >
> > > [1] https://github.com/apache/flink-connector-pulsar
> >
>

[jira] [Created] (FLINK-33519) standalone mode could not create keytab secret

[chaoran.su (Jira)]

chaoran.su created FLINK-33519:
--

 Summary: standalone mode could not create keytab secret
 Key: FLINK-33519
 URL: https://issues.apache.org/jira/browse/FLINK-33519
 Project: Flink
  Issue Type: Bug
  Components: Kubernetes Operator
Affects Versions: kubernetes-operator-1.6.1, kubernetes-operator-1.6.0, 
kubernetes-operator-1.5.0, kubernetes-operator-1.3.1, 
kubernetes-operator-1.4.0, kubernetes-operator-1.3.0
Reporter: chaoran.su


when standalone build cluster, and configuration with security.kerberos.login.* 
configurations.

flink-kubernetes module will modify the path of security.kerberos.login.keytab 
configuration to /opt/kerberos/kerberos-keytab, and then create secret for job 
manager. the secret data from operator pod's keytab file.

after job manager created, creating task manager process will find the keytab 
file from the location from security.kerberos.login.keytab configuration, then 
it throws a exception says keytabs file not find. 

the bug is because of the configuration modified once, and reused it when 
create tm. Native mode didn't exist this issue.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33520) FileNotFoundException when running GPUDriverTest

[Ryan Skraba (Jira)]

Ryan Skraba created FLINK-33520:
---

 Summary: FileNotFoundException when running GPUDriverTest
 Key: FLINK-33520
 URL: https://issues.apache.org/jira/browse/FLINK-33520
 Project: Flink
  Issue Type: Technical Debt
  Components: Tests
Affects Versions: 1.18.0
Reporter: Ryan Skraba


I'd been running into a mysterious error running the 
{{flink-external-resources}} module tests:

{code}
java.io.FileNotFoundException: The gpu discovery script does not exist in path 
/opt/asf/flink/src/test/resources/testing-gpu-discovery.sh.
at 
org.apache.flink.externalresource.gpu.GPUDriver.(GPUDriver.java:98)
at 
org.apache.flink.externalresource.gpu.GPUDriverTest.testGPUDriverWithInvalidAmount(GPUDriverTest.java:64)
at
{code}

>From the command line and IntelliJ, when it seems to works, it _always_ works, 
>and when it fails it _always_ fails. I finally took a moment to figure it out: 
>if the {{FLINK_HOME}} environment variable is set (to a valid Flink 
>distribution of any version), this test fails.

This is a very minor irritation, but it's pretty easy to fix.

The workaround is to launch the unit test in an environment where this 
environment variable is not set.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33521) Implement restore tests for PythonCalc node

[Jim Hughes (Jira)]

Jim Hughes created FLINK-33521:
--

 Summary: Implement restore tests for PythonCalc node
 Key: FLINK-33521
 URL: https://issues.apache.org/jira/browse/FLINK-33521
 Project: Flink
  Issue Type: Sub-task
Reporter: Jim Hughes






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] FLIP-390: Support System out and err to be redirected to LOG or discarded

[Piotr Nowojski]

Thanks! :)

Best, Piotrek

czw., 9 lis 2023 o 16:15 Rui Fan <1996fan...@gmail.com> napisał(a):

> Hi Piotr,
>
> Thanks for your feedback!
>
> > Or implement your own loop? It shouldn't be more than a couple of lines.
>
> Implementing it directly is fine, I have updated the FLIP.
> And this logic can be found in the  `isLineEnded` method.
>
> Best,
> Rui
>
> On Thu, Nov 9, 2023 at 11:00 PM Piotr Nowojski 
> wrote:
>
> > Hi Rui,
> >
> > > I see java8 doesn't have `Arrays.equals(int[] a, int aFromIndex, int
> > > aToIndex, int[] b, int bFromIndex, int bToIndex)`,
> > > and java11 has it. Do you have any other suggestions for java8?
> >
> > Maybe use `ByteBuffer.wrap`?
> >
> > ByteBuffer.wrap(array, ..., ...).equals(ByteBuffer.wrap(array2, ...,
> ...))
> >
> > This shouldn't have overheads as far as I remember.
> >
> > Or implement your own loop? It shouldn't be more than a couple of lines.
> >
> > Best,
> > Piotrek
> >
> > czw., 9 lis 2023 o 06:43 Rui Fan <1996fan...@gmail.com> napisał(a):
> >
> > > Hi Piotr, Archit, Feng and Hangxiang:
> > >
> > > Thanks a lot for your feedback!
> > >
> > > Following is my comment, please correct me if I misunderstood anything!
> > >
> > > To Piotr:
> > >
> > > > Is there a reason why you are suggesting to copy out bytes from `buf`
> > to
> > > `bytes`,
> > > > instead of using `Arrays.equals(int[] a, int aFromIndex, int
> aToIndex,
> > > int[] b, int bFromIndex, int bToIndex)`?
> > >
> > > I see java8 doesn't have `Arrays.equals(int[] a, int aFromIndex, int
> > > aToIndex, int[] b, int bFromIndex, int bToIndex)`,
> > > and java11 has it. Do you have any other suggestions for java8?
> > >
> > > Also, this code doesn't run in production. As the comment of
> > > System.lineSeparator():
> > >
> > > > On UNIX systems, it returns {@code "\n"}; on Microsoft
> > > > Windows systems it returns {@code "\r\n"}.
> > >
> > > So Mac and Linux just return one character, we will compare
> > > one byte directly.
> > >
> > >
> > >
> > > To Feng:
> > >
> > > > Will they be written to the taskManager.log file by default
> > > > or the taskManager.out file?
> > >
> > > I prefer LOG as the default value for taskmanager.system-out.mode.
> > > It's useful for job stability and doesn't introduce significant impact
> to
> > > users. Also, our production has already used this feature for
> > > more than 1 years, it works well.
> > >
> > > However, I write the DEFAULT as the default value for
> > > taskmanager.system-out.mode, because when the community introduces
> > > new options, the default value often selects the original behavior.
> > >
> > > Looking forward to hear more thoughts from community about this
> > > default value.
> > >
> > > > If we can make taskmanager.out splittable and rolling, would it be
> > > > easier for users to use this feature?
> > >
> > > Making taskmanager.out splittable and rolling is a good choice!
> > > I have some concerns about it:
> > >
> > > 1. Users may also want to use LOG.info in their code and just
> > >   accidentally use System.out.println. It is possible that they will
> > >   also find the logs directly in taskmanager.log.
> > > 2. I'm not sure whether the rolling strategy is easy to implement.
> > >   If we do it, it's necessary to define a series of flink options
> similar
> > >   to log options, such as: fileMax(how many files should be retained),
> > >   fileSize(The max size each file), fileNamePatten (The suffix of file
> > > name),
> > > 3. Check the file size periodically: all logs are written by log
> plugin,
> > >   they can check the log file size after writing. However, System.out
> > >   are written directly. And flink must start a thread to check the
> latest
> > >   taskmanager.out size periodically. If it's too quick, most of job
> > aren't
> > >   necessary. If it's too slow, the file size cannot be controlled
> > properly.
> > >
> > > Redirect it to LOG.info may be a reasonable and easy choice.
> > > The user didn't really want to log into taskmanager.out, it just
> > > happened by accident.
> > >
> > >
> > >
> > > To Hangxiang:
> > >
> > > > 1. I have a similar concern as Feng. Will we redirect to another log
> > file
> > > > not taskManager.log ?
> > >
> > > Please see my last comment, thanks!
> > >
> > > > taskManager.log contains lots of important information like init log.
> > It
> > > > will be rolled quickly if we redirect out and error here.
> > >
> > > IIUC, this issue isn't caused by System.out, and it can happen if user
> > > call a lot of LOG.info. As I mentioned before: the user didn't really
> > want
> > > to log into taskmanager.out, it just happened by accident.
> > > So, if users change the System.out to LOG.info, it still happen.
> > >
> > > > 2. Since we have redirected to LOG mode, Could we also log the
> subtask
> > > info
> > > > ? It may help us to debug granularly.
> > >
> > > I'm not sure what `log the subtask info` means. Let me confirm with you
> > > first.
> > > Do you mean like this: LOG.info("taskName {} : {}

Re: Adding a new channel to Apache Flink slack workspace

[Robin Moffatt]

wonderful, thanks Martin!

On Fri, 10 Nov 2023 at 09:56, Martijn Visser 
wrote:

> Hi all,
>
> I've created https://issues.apache.org/jira/browse/FLINK-33516 and
> I'll work on this. Thanks!
>
> Cheers, Martijn
>
> On Fri, Nov 10, 2023 at 7:50 AM Yun Tang  wrote:
> >
> > +1 and glad to see more guys are using PyFlink.
> >
> > Best
> > Yun Tang
> > 
> > From: Jing Ge 
> > Sent: Wednesday, November 8, 2023 3:18
> > To: ro...@decodable.co.invalid 
> > Cc: dev@flink.apache.org 
> > Subject: Re: Adding a new channel to Apache Flink slack workspace
> >
> > +1 since there are so many questions wrt PyFlink.
> >
> > Best regards,
> > Jing
> >
> > On Tue, Nov 7, 2023 at 2:23 AM Robin Moffatt  >
> > wrote:
> >
> > > Since there have been no objections, can we go ahead and get this
> channel
> > > created please?
> > >
> > > thanks :)
> > >
> > > On Thu, 26 Oct 2023 at 16:00, Alexander Fedulov <
> > > alexander.fedu...@gmail.com>
> > > wrote:
> > >
> > > > +1
> > > > I agree that the topic is distinct enough to justify a dedicated
> channel
> > > > and this could lead to more active participation from people who
> work on
> > > > it.
> > > >
> > > > Best,
> > > > Alexander
> > > >
> > > > On Wed, 25 Oct 2023 at 20:03, Robin Moffatt 
> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I'd like to propose adding a PyFlink channel to the Apache Flink
> slack
> > > > > workspace.
> > > > >
> > > > > By creating a channel focussed on this it will help people find
> > > previous
> > > > > discussions as well as target new discussions and questions to the
> > > > correct
> > > > > place. PyFlink is a sufficiently distinct component to make a
> dedicated
> > > > > channel viable and useful IMHO.
> > > > >
> > > > > There was a brief discussion of this on Slack already, the archive
> for
> > > > > which can be found here:
> > > > >
> > > > >
> > > >
> > >
> https://www.linen.dev/s/apache-flink/t/16006099/re-surfacing-for-the-admins-https-apache-flink-slack-com-arc#1c7e-177a-4c37-8a34-a917883152ac
> > > > >
> > > > > thanks,
> > > > >
> > > > > Robin.
> > > > >
> > > >
> > >
>

Re: Adding a new channel to Apache Flink slack workspace

[Robin Moffatt]

*Martijn, sorry!

On Fri, 10 Nov 2023 at 16:54, Robin Moffatt  wrote:

> wonderful, thanks Martin!
>
> On Fri, 10 Nov 2023 at 09:56, Martijn Visser 
> wrote:
>
>> Hi all,
>>
>> I've created https://issues.apache.org/jira/browse/FLINK-33516 and
>> I'll work on this. Thanks!
>>
>> Cheers, Martijn
>>
>> On Fri, Nov 10, 2023 at 7:50 AM Yun Tang  wrote:
>> >
>> > +1 and glad to see more guys are using PyFlink.
>> >
>> > Best
>> > Yun Tang
>> > 
>> > From: Jing Ge 
>> > Sent: Wednesday, November 8, 2023 3:18
>> > To: ro...@decodable.co.invalid 
>> > Cc: dev@flink.apache.org 
>> > Subject: Re: Adding a new channel to Apache Flink slack workspace
>> >
>> > +1 since there are so many questions wrt PyFlink.
>> >
>> > Best regards,
>> > Jing
>> >
>> > On Tue, Nov 7, 2023 at 2:23 AM Robin Moffatt > >
>> > wrote:
>> >
>> > > Since there have been no objections, can we go ahead and get this
>> channel
>> > > created please?
>> > >
>> > > thanks :)
>> > >
>> > > On Thu, 26 Oct 2023 at 16:00, Alexander Fedulov <
>> > > alexander.fedu...@gmail.com>
>> > > wrote:
>> > >
>> > > > +1
>> > > > I agree that the topic is distinct enough to justify a dedicated
>> channel
>> > > > and this could lead to more active participation from people who
>> work on
>> > > > it.
>> > > >
>> > > > Best,
>> > > > Alexander
>> > > >
>> > > > On Wed, 25 Oct 2023 at 20:03, Robin Moffatt 
>> wrote:
>> > > >
>> > > > > Hi,
>> > > > >
>> > > > > I'd like to propose adding a PyFlink channel to the Apache Flink
>> slack
>> > > > > workspace.
>> > > > >
>> > > > > By creating a channel focussed on this it will help people find
>> > > previous
>> > > > > discussions as well as target new discussions and questions to the
>> > > > correct
>> > > > > place. PyFlink is a sufficiently distinct component to make a
>> dedicated
>> > > > > channel viable and useful IMHO.
>> > > > >
>> > > > > There was a brief discussion of this on Slack already, the
>> archive for
>> > > > > which can be found here:
>> > > > >
>> > > > >
>> > > >
>> > >
>> https://www.linen.dev/s/apache-flink/t/16006099/re-surfacing-for-the-admins-https-apache-flink-slack-com-arc#1c7e-177a-4c37-8a34-a917883152ac
>> > > > >
>> > > > > thanks,
>> > > > >
>> > > > > Robin.
>> > > > >
>> > > >
>> > >
>>
>

[jira] [Created] (FLINK-33522) Savepoint upgrade mode fails despite the savepoint succeeding

[Maximilian Michels (Jira)]

Maximilian Michels created FLINK-33522:
--

 Summary: Savepoint upgrade mode fails despite the savepoint 
succeeding
 Key: FLINK-33522
 URL: https://issues.apache.org/jira/browse/FLINK-33522
 Project: Flink
  Issue Type: Bug
  Components: Kubernetes Operator
Affects Versions: kubernetes-operator-1.6.1, kubernetes-operator-1.6.0
Reporter: Maximilian Michels
Assignee: Maximilian Michels
 Fix For: kubernetes-operator-1.7.0


Under certain circumstances, savepoint creation can succeed but the job fails 
afterwards. One example is when there are messages being distributed by the 
source coordinator to finished tasks. This is possibly a Flink bug although 
it's not clear how to solve this issue.

After the savepoint succeeded Flink fails the job like this:
{noformat}
Source (1/2) 
(cd4d56ddb71c0e763cc400bcfe2fd8ac_4081cf0163fcce7fe6af0cf07ad2d43c_0_0) 
switched from RUNNING to FAILED on host-taskmanager-1-1 @ ip(dataPort=36519). 
{noformat}
{noformat}
An OperatorEvent from an OperatorCoordinator to a task was lost. Triggering 
task failover to ensure consistency. Event: 'AddSplitEvents[[[B@722a23fa]]', 
targetTask: Source (1/2) - execution #0
Caused by:
org.apache.flink.runtime.operators.coordination.TaskNotRunningException: Task 
is not running, but in state FINISHED
   at 
org.apache.flink.runtime.taskmanager.Task.deliverOperatorEvent(Task.java:1502)
   at org.apache.flink.runtime.taskexecutor.TaskExecutor.sendOperatorEventToTask
{noformat}

Inside the operator this is processed as:

{noformat}
java.util.concurrent.CompletionException: 
org.apache.flink.runtime.scheduler.stopwithsavepoint.StopWithSavepointStoppingException:
 A savepoint has been created at: s3://..., but the corresponding job 
1b1a3061194c62ded6e2fe823b61b2ea failed during stopping. The savepoint is 
consistent, but might have uncommitted transactions. If you want to commit the 
transaction please restart a job from this savepoint. 

  
java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395) 
  
java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2022) 
  
org.apache.flink.kubernetes.operator.service.AbstractFlinkService.cancelJob(AbstractFlinkService.java:319)
 
  
org.apache.flink.kubernetes.operator.service.NativeFlinkService.cancelJob(NativeFlinkService.java:121)
 
  
org.apache.flink.kubernetes.operator.reconciler.deployment.ApplicationReconciler.cancelJob(ApplicationReconciler.java:223)
 
  
org.apache.flink.kubernetes.operator.reconciler.deployment.AbstractJobReconciler.reconcileSpecChange(AbstractJobReconciler.java:122)
 
 
org.apache.flink.kubernetes.operator.reconciler.deployment.AbstractFlinkResourceReconciler.reconcile(AbstractFlinkResourceReconciler.java:163)
  
org.apache.flink.kubernetes.operator.controller.FlinkDeploymentController.reconcile(FlinkDeploymentController.java:136)
 
  
org.apache.flink.kubernetes.operator.controller.FlinkDeploymentController.reconcile(FlinkDeploymentController.java:56)
 
  
io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:138)
 
  
io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:96) 
  
org.apache.flink.kubernetes.operator.metrics.OperatorJosdkMetrics.timeControllerExecution(OperatorJosdkMetrics.java:80)
 
  
io.javaoperatorsdk.operator.processing.Controller.reconcile(Controller.java:95) 
  
io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.reconcileExecution(ReconciliationDispatcher.java:139)
 
  
io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleReconcile(ReconciliationDispatcher.java:119)
 
  
io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleDispatch(ReconciliationDispatcher.java:89)
 
  
io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleExecution(ReconciliationDispatcher.java:62)
 
  
io.javaoperatorsdk.operator.processing.event.EventProcessor$ReconcilerExecutor.run(EventProcessor.java:414)
 
  
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) 
  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) 
  java.lang.Thread.run(Thread.java:829) 
{noformat}

Subsequently we get the following because HA metadata is not available anymore. 
It has been cleared up after the terminal job failure:

{noformat}
org.apache.flink.kubernetes.operator.exception.RecoveryFailureException","message":"HA
 metadata not available to restore from last state. It is possible that the job 
has finished or terminally failed, or the configmaps have been deleted. 
{noformat}

The deployment needs to be manually restored from a savepoint.



--
This message was sent by 

Re: [DISCUSS] FLIP-389: Annotate SingleThreadFetcherManager and FutureCompletingBlockingQueue as PublicEvolving

[Becket Qin]

Hi Hongshun and Martijn,

Sorry for the late reply as I was on travel and still catching up with the
emails. Please allow me to provide more context.

1. The original design of SplitFetcherManager and its subclasses was to
make them public to the Source developers. The goal is to let us take care
of the threading model, while the Source developers can just focus on the
SplitReader implementation. Therefore, I think making SplitFetcherManater /
SingleThreadFetcherManager public aligns with the original design. That is
also why these classes are exposed in the constructor of SourceReaderBase.

2. For FutureCompletingBlockingQueue, as a hindsight, it might be better to
not expose it to the Source developers. They are unlikely to use it
anywhere other than just constructing it. The reason that
FutureCompletingBlockingQueue is currently exposed in the SourceReaderBase
constructor is because both the SplitFetcherManager and SourceReaderBase
need it. One way to hide the FutureCompletingBlockingQueue from the public
API is to make SplitFetcherManager the only owner class of the queue, and
expose some of its methods via SplitFetcherManager. This way, the
SourceReaderBase can invoke the methods via SplitFetcherManager. I believe
this also makes the code slightly cleaner.

Thanks,

Jiangjie (Becket) Qin

On Fri, Nov 10, 2023 at 12:28 PM Hongshun Wang 
wrote:

> @Martijn, I agree with you.
>
>
> I also have two questions at the beginning:
>
>- Why is an Internal class
>exposed as a constructor param of a Public class?
>- Should these classes be exposed as public?
>
> For the first question,  I noticed that before the original Jira[1] ,
> all these classes missed the annotate , so it was not abnormal that
> FutureCompletingBlockingQueue and SingleThreadFetcherManager were
> constructor params of SingleThreadMultiplexSourceReaderBase.
>  However,
> this jira marked FutureCompletingBlockingQueue and
> SingleThreadFetcherManager as Internal, while marked
> SingleThreadMultiplexSourceReaderBase as Public. It's a good choice,
> but also forget that FutureCompletingBlockingQueue and
> SingleThreadFetcherManager have already been  exposed by
> SingleThreadMultiplexSourceReaderBase.
>  Thus, this problem occurs because we didn't
> clearly define the boundaries at the origin design. We should pay more
> attention to it when creating a new class.
>
>
> For the second question, I think at least SplitFetcherManager
> should be Public. There are few reasons:
>
>-  Connector developers want to decide their own
>thread mode. For example, Whether to recycle fetchers by overriding
>SplitFetcherManager#maybeShutdownFinishedFetchers
>when idle. Sometimes, developers want SplitFetcherManager react as a
>FixedThreadPool, because
>each time a thread is recycled then recreated, the context
> resources need to be rebuilt. I met a related issue in flink cdc[2].
>-
>KafkaSourceFetcherManager[3] also  extends
> SingleThreadFetcherManager to commitOffsets. But now kafka souce is
> not in Flink repository, so it's not allowed any more.
>
> [1] https://issues.apache.org/jira/browse/FLINK-22358
>
> [2]
>
> https://github.com/ververica/flink-cdc-connectors/pull/2571#issuecomment-1797585418
>
> [3]
>
> https://github.com/apache/flink-connector-kafka/blob/979791c4c71e944c16c51419cf9a84aa1f8fea4c/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/source/reader/fetcher/KafkaSourceFetcherManager.java#L52
>
> Looking forward to hearing from you.
>
> Best regards,
> Hongshun
>
> On Thu, Nov 9, 2023 at 11:46 PM Martijn Visser 
> wrote:
>
> > Hi all,
> >
> > I'm looking at the original Jira that introduced these stability
> > designations [1] and I'm just curious if it was intended that these
> > Internal classes would be used directly, or if we just haven't created
> > the right abstractions? The reason for asking is because moving
> > something from Internal to a public designation is an easy fix, but I
> > want to make sure that it's also the right fix. If we are missing good
> > abstractions, then I would rather invest in those.
> >
> > Best regards,
> >
> > Martijn
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-22358
> >
> > On Wed, Nov 8, 2023 at 12:40 PM Leonard Xu  wrote:
> > >
> > > Thanks Hongshun for starting this discussion.
> > >
> > > +1 from my side.
> > >
> > > IIRC, @Jiangjie(Becket) also mentioned this in FLINK-31324 comment[1].
> > >
> > > Best,
> > > Leonard
> > >
> > > [1]
> >
> https://issues.apache.org/jira/browse/FLINK-31324?focusedCommentId=17696756&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17696756
> > >
> > >
> > >
> > > > 2023年11月8日 下午5:42，Hongshun Wang  写道：
> > > >
> > > > Hi devs,
> > > >
> > > > I would like to start a discussion on FLIP-389: Annotate
> > > > SingleThreadFetcherManager and FutureCompletingBlockingQueue as
> > > > PublicEvolving.[
> > > > <
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pa

[jira] [Created] (FLINK-33523) DataType ARRAY fails to cast into Object[]

[Prabhu Joseph (Jira)]

Prabhu Joseph created FLINK-33523:
-

 Summary: DataType ARRAY fails to cast into Object[]
 Key: FLINK-33523
 URL: https://issues.apache.org/jira/browse/FLINK-33523
 Project: Flink
  Issue Type: Bug
Affects Versions: 1.18.0
Reporter: Prabhu Joseph


When upgrading Iceberg's Flink version to 1.18, we found the Flink-related unit 
test case broken due to this issue. The below code used to work fine in Flink 
1.17 but failed after upgrading to 1.18. DataType ARRAY fails to 
cast into Object[].

**Error:**

{code}
Exception in thread "main" java.lang.ClassCastException: [I cannot be cast to 
[Ljava.lang.Object;
at FlinkArrayIntNotNullTest.main(FlinkArrayIntNotNullTest.java:18)
{code}

**Repro:**

{code}

  import org.apache.flink.table.data.ArrayData;
  import org.apache.flink.table.data.GenericArrayData;
  import org.apache.flink.table.api.EnvironmentSettings;
  import org.apache.flink.table.api.TableEnvironment;
  import org.apache.flink.table.api.TableResult;

  public class FlinkArrayIntNotNullTest {

public static void main(String[] args) throws Exception {

  EnvironmentSettings settings = 
EnvironmentSettings.newInstance().inBatchMode().build();
  TableEnvironment env = TableEnvironment.create(settings);

  env.executeSql("CREATE TABLE filesystemtable2 (id INT, data ARRAY) WITH ('connector' = 'filesystem', 'path' = 
'/tmp/FLINK/filesystemtable2', 'format'='json')");
  env.executeSql("INSERT INTO filesystemtable2 VALUES (4,ARRAY [1,2,3])");
  TableResult tableResult = env.executeSql("SELECT * from 
filesystemtable2");

  ArrayData actualArrayData = new GenericArrayData((Object[]) 
tableResult.collect().next().getField(1));
}
  }

{code}






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33524) IntervalJoinOperator 's judgment on late data has bug

[ZhangTao (Jira)]

ZhangTao created FLINK-33524:


 Summary: IntervalJoinOperator 's judgment on late data has bug
 Key: FLINK-33524
 URL: https://issues.apache.org/jira/browse/FLINK-33524
 Project: Flink
  Issue Type: Bug
  Components: API / DataStream
Affects Versions: 1.18.0
 Environment: Due to the Watermark calculation method :
{code:java}
public void onPeriodicEmit(WatermarkOutput output) {
            output.emitWatermark(new Watermark(maxTs - delayTime - 1L));
        }{code}
 data that was delayed by 1 millisecond in this method was incorrectly 
determined
{code:java}
private boolean isLate(long timestamp) {
long currentWatermark = internalTimerService.currentWatermark();
return timestamp < currentWatermark;
} {code}
 
Reporter: ZhangTao


package:
org.apache.flink.streaming.api.operators.co;
 
class: IntervalJoinOperator
 
method:
isLate
 
When data with a 1-millisecond delay enters the judgment, an incorrect value 
will be returned
{code:java}
private boolean isLate(long timestamp) {
long currentWatermark = internalTimerService.currentWatermark();
return timestamp < currentWatermark;
} {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)