dev@flink.apache.org

2022-09-30 Thread Martijn Visser
Hi Peter,

I think this also depends on the support SLA that the technology that you
connect to provides. For example, with Flink and Elasticsearch, we choose
to follow Elasticsearch supported versions. So that means that when support
for Elasticsearch 8 is introduced, support for Elasticsearch 6 should be
dropped (since Elastic only support the last major version and the latest
minor version prior to that)

I don't see value in having different connectors for Iceberg 0.14 and 0.15
in separate repositories. I think that will confuse the user. I would
expect that with modules you should be able to have support for multiple
versions in one repository.

Best regards,

Martijn

On Fri, Sep 30, 2022 at 7:44 AM Péter Váry 
wrote:

> Thanks for the quick response!
>
> Would this mean, that we have different connectors for Iceberg 0.14, and
> Iceberg 0.15. Would these different versions kept in different repository?
>
> My feeling is that this model is fine for the stable/slow moving systems
> like Hive/HBase. For other systems, which are evolving faster, this is less
> than ideal.
>
> For those, who have more knowledge about the Flink ecosystem: How do you
> feel? What is the distribution of the connectors between the slow moving
> and the fast moving systems?
>
> Thanks, Peter
>
>
> On Thu, Sep 29, 2022, 16:46 Danny Cranmer  wrote:
>
> > If you look at ElasticSearch [1] as an example there are different
> variants
> > of the connector depending on the "connected" system:
> > - flink-connector-elasticsearch6
> > - flink-connector-elasticsearch7
> >
> > Looks like Hive and HBase follow a similar pattern in the main Flink
> repo/
> >
> > [1] https://github.com/apache/flink-connector-elasticsearch
> >
> > On Thu, Sep 29, 2022 at 3:17 PM Péter Váry 
> > wrote:
> >
> > > Hi Team,
> > >
> > > Just joining the conversation for the first time, so pardon me if I
> > repeat
> > > already answered questions.
> > >
> > > It might be already discussed, but I think the version for the
> > "connected"
> > > system could be important as well.
> > >
> > > There might be some API changes between Iceberg 0.14.2, and 1.0.0,
> which
> > > would require as to rewrite part of the code for the Flink-Iceberg
> > > connector.
> > > It would be important for the users:
> > > - Which Flink version(s) are this connector working with?
> > > - Which Iceberg version(s) are this connector working with?
> > > - Which code version we have for this connector?
> > >
> > > Does this make sense? What is the community's experience with the
> > connected
> > > systems? Are they stable enough for omitting their version number from
> > the
> > > naming of the connectors? Would this worth the proliferation of the
> > > versions?
> > >
> > > Thanks,
> > > Peter
> > >
> > > Chesnay Schepler  ezt írta (időpont: 2022. szept.
> > 29.,
> > > Cs, 14:11):
> > >
> > > > 2) No; the branch names would not have a Flink version in them;
> v1.0.0,
> > > > v1.0.1 etc.
> > > >
> > > > On 29/09/2022 14:03, Martijn Visser wrote:
> > > > > If I summarize it correctly, that means that:
> > > > >
> > > > > 1. The versioning scheme would be  > > > > version>-, where there will
> > never
> > > > be a
> > > > > patch release for a minor version if a newer minor version already
> > > > exists.
> > > > > E.g., 1.0.0-1.15; 1.0.1-1.15; 1.1.0-1.15; 1.2.0-1.15;
> > > > >
> > > > > 2. The branch naming scheme would be
> > > vmajor.minor-flink-major.flink-minor
> > > > > E.g., v1.0.0-1.15; v1.0.1-1.15; v1.1.0-1.15; v1.2.0-1.15;
> > > > >
> > > > > I would +1 that.
> > > > >
> > > > > Best regards,
> > > > >
> > > > > Martijn
> > > > >
> > > > > On Tue, Sep 20, 2022 at 2:21 PM Chesnay Schepler <
> ches...@apache.org
> > >
> > > > wrote:
> > > > >
> > > > >>   > After 1.16, only patches are accepted for 1.2.0-1.15.
> > > > >>
> > > > >> I feel like this is a misunderstanding that both you and Danny ran
> > > into.
> > > > >>
> > > > >> What I meant in the original proposal is that the last 2 _major_
> > > > >> /connector /versions are supported, with the latest receiving
> > > additional
> > > > >> features.
> > > > >> (Provided that the previous major version still works against a
> > > > >> currently supported Flink version!)
> > > > >> There will never be patch releases for a minor version if a newer
> > > minor
> > > > >> version exists.
> > > > >>
> > > > >> IOW, the minor/patch releases within a major version do not form a
> > > tree
> > > > >> (like in Flink), but a line.
> > > > >>
> > > > >> 1.0.0 -> 1.0.1 -> 1.1.0 -> 1.2.0 -> ...
> > > > >> NOT
> > > > >> 1.0.0 -> 1.0.1 -> 1.0.2
> > > > >>  |-> 1.1.0 -> 1.1.1
> > > > >>
> > > > >> If we actually follow semantic versioning then it's just not
> > necessary
> > > > >> to publish a patch for a previous version.
> > > > >>
> > > > >> So if 2.x exists, then (the latest) 2.x gets features and patches,
> > and
> > > > >> the latest 1.x gets patches.
> > > > >>
> > > > >> I hope that clears things up.
> > > > >>
> > > > >> On 20/0

[VOTE] Apache Flink Kubernetes Operator Release 1.2.0, release candidate #2

2022-09-30 Thread Gyula Fóra
Hi everyone,

Please review and vote on the release candidate #2 for the version 1.2.0 of
Apache Flink Kubernetes Operator,
as follows:
[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)

**Release Overview**

As an overview, the release consists of the following:
a) Kubernetes Operator canonical source distribution (including the
Dockerfile), to be deployed to the release repository at dist.apache.org
b) Kubernetes Operator Helm Chart to be deployed to the release repository
at dist.apache.org
c) Maven artifacts to be deployed to the Maven Central Repository
d) Docker image to be pushed to dockerhub

**Staging Areas to Review**

The staging areas containing the above mentioned artifacts are as follows,
for your review:
* All artifacts for a,b) can be found in the corresponding dev repository
at dist.apache.org [1]
* All artifacts for c) can be found at the Apache Nexus Repository [2]
* The docker image for d) is staged on github [3]

All artifacts are signed with the key 21F06303B87DAFF1 [4]

Other links for your review:
* JIRA release notes [5]
* source code tag "release-1.2.0-rc2" [6]
* PR to update the website Downloads page to include Kubernetes Operator
links [7]

**Vote Duration**

The voting time will run for at least 72 hours.
It is adopted by majority approval, with at least 3 PMC affirmative votes.

**Note on Verification**

You can follow the basic verification guide here[8].
Note that you don't need to verify everything yourself, but please make
note of what you have tested together with your +- vote.

Thanks,
Gyula Fora

[1]
https://dist.apache.org/repos/dist/dev/flink/flink-kubernetes-operator-1.2.0-rc2/
[2] https://repository.apache.org/content/repositories/orgapacheflink-1538/
[3] ghcr.io/apache/flink-kubernetes-operator:95128bf
[4] https://dist.apache.org/repos/dist/release/flink/KEYS
[5]
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12352091
[6]
https://github.com/apache/flink-kubernetes-operator/tree/release-1.2.0-rc2
[7] https://github.com/apache/flink-web/pull/573
[8]
https://cwiki.apache.org/confluence/display/FLINK/Verifying+a+Flink+Kubernetes+Operator+Release


Re: [DISCUSS] FLIP-262 MongoDB Connector

2022-09-30 Thread Jiabao Sun
Hi Qingsheng,
Thanks For the feedback.
Other metrics mentioned in FLIP-33, we will also implement it.
Best,
Jiabao
--
From:Qingsheng Ren 
Send Time:2022年9月28日(星期三) 18:36
To:孙家宝 
Cc:dev 
Subject:Re: [DISCUSS] FLIP-262 MongoDB Connector
Thanks for the update Jiabao. 
c. It’ll be nice to have all metrics defined in FLIP-33 in the connector. Also 
it’s better to have some MongoDB specific metrics, which will be very helpful 
for users to monitor their jobs.
Best,
Qingsheng
On Sep 27, 2022, at 11:59, Jiabao Sun mailto:jiabao@xtransfer.cn >> wrote:
Hi Qingsheng,
Thanks for the great suggestion.
a. Currently we have implemented the Source of FLIP-177 in 
https://github.com/apache/flink/pull/20848 
.
 Sorry for some mistakes in the Public Interface of FLIP-262, the interface of 
Sink has been adjusted from FLIP-143 to FLIP-177.
b. The AsyncWriter mentioned by FLIP-171 looks helpful, I'll try to implement 
it.
c. We have provided some common metrics mentioned by FLIP-33. 
 Source: numRecordsSend
 Sink: numRecordsSend/currentSendTime
Best,
Jiabao
--
From:Qingsheng Ren mailto:re...@apache.org >>
Send Time:2022年9月27日(星期二) 10:43
To:dev mailto:dev@flink.apache.org >>
Cc:孙家宝 mailto:jiabao@xtransfer.cn >>
Subject:Re: [DISCUSS] FLIP-262 MongoDB Connector
Hi Jiabao,
Thanks for the FLIP! Very glad to see having a new connector in Flink 
eco-system, so overall +1 from my side.
In addition to those FLIPs you mention in the proposal, here’s some other FLIPs 
that you may be interested in:
a. FLIP-177 [1] extended Sink API by Sink API v2 with some additional features, 
so it’ll be great to implement the sink directly on the latest interface. 
b. FLIP-171 [2] introduced async sink that you can to reuse if at-least-once 
semantic could fulfill the requirement of MongoDB sink.
c. FLIP-33 [3] defined general source and sink metrics that connectors should 
report. 
Looking forward to the new connector!
[1] 
https://cwiki.apache.org/confluence/display/FLINK/FLIP-177%3A+Extend+Sink+API 

[2] https://cwiki.apache.org/confluence/display/FLINK/FLIP-171%3A+Async+Sink 

[3] 
https://cwiki.apache.org/confluence/display/FLINK/FLIP-33%3A+Standardize+Connector+Metrics
 

Best,
Qingsheng
On Sep 23, 2022, at 00:01, 孙家宝 mailto:jiabao@xtransfer.cn.INVALID >> wrote:
Hi everyone,
I would like to start a discussion thread on FLIP-262 for a MongoDB Connector 
that will support both 
DataStream, Table API and SQL implementations with Lookup ability. 
This would be a new connector in its own external repository being
https://www.github.com/apache/flink-connector-mongodb 
 
 >.
This repository doesn't exist yet, so we created a FLIP to discuss whether we 
need to create it.
https://cwiki.apache.org/confluence/display/FLINK/FLIP-262%3A+Introduce+MongoDB+connector
 
 
 >
Looking forward to any comments or feedback.
Best Regards,
Jiabao


[jira] [Created] (FLINK-29482) Ingress always forces ClusterIP rest service type

2022-09-30 Thread Gyula Fora (Jira)
Gyula Fora created FLINK-29482:
--

 Summary: Ingress always forces ClusterIP rest service type
 Key: FLINK-29482
 URL: https://issues.apache.org/jira/browse/FLINK-29482
 Project: Flink
  Issue Type: Improvement
  Components: Kubernetes Operator
Affects Versions: kubernetes-operator-1.1.0, kubernetes-operator-1.2.0
Reporter: Gyula Fora
Assignee: Matyas Orhidi


Currently the ingress logic always overrides ClusterIP service type even if the 
user configured it otherwise.

This might simply be a bug but there could be a good reason for it, we cannot 
tell at first.

We need to investigate



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Created] (FLINK-29483) flink python udf arrow in thread model bug

2022-09-30 Thread jackylau (Jira)
jackylau created FLINK-29483:


 Summary: flink python udf arrow in thread model bug
 Key: FLINK-29483
 URL: https://issues.apache.org/jira/browse/FLINK-29483
 Project: Flink
  Issue Type: Bug
  Components: API / Python
Affects Versions: 1.15.2, 1.16.0
Reporter: jackylau
 Fix For: 1.16.0, 1.17.0, 1.15.3
 Attachments: image-2022-09-30-17-03-05-005.png

!image-2022-09-30-17-03-05-005.png!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Created] (FLINK-29484) Support orderless check of elements in SourceTestSuiteBase and SinkTestSuiteBase

2022-09-30 Thread Etienne Chauchot (Jira)
Etienne Chauchot created FLINK-29484:


 Summary: Support orderless check of elements in 
SourceTestSuiteBase and SinkTestSuiteBase
 Key: FLINK-29484
 URL: https://issues.apache.org/jira/browse/FLINK-29484
 Project: Flink
  Issue Type: Improvement
  Components: Tests
Reporter: Etienne Chauchot


Some backend source and sinks are distributed and to not guarantee order. So 
the test suites must consider that the tests pass even if the data is out of 
order. We should add a configuration to the ExternalContext to set orderless 
per source/sink.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


Re: [VOTE] FLIP-262: MongoDB Connector

2022-09-30 Thread Martijn Visser
Thanks Jiabao!
+1 (binding)

Cheers, Martijn

On Fri, Sep 30, 2022 at 11:04 AM jiabao.sun 
wrote:

> Hi everyone,
>
>
> Thanks for all your feedback for FLIP-262[1]: MongoDB Connector in the
> discussion thread[2],
> I'd like to start a vote for it.
>
>
> The vote will be open for at least 72 hours.
>
>
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-262%3A+Introduce+MongoDB+connector
> [2] https://lists.apache.org/thread/bhzj70t9g6ofdk8hqtfjjlxqnl0l4xwn
>
>
> Best,
> Jiabao Sun


[jira] [Created] (FLINK-29485) TypeSerializerUpgradeTestBase doesn't use FlinkVersions.current()

2022-09-30 Thread Matthias Pohl (Jira)
Matthias Pohl created FLINK-29485:
-

 Summary: TypeSerializerUpgradeTestBase doesn't use 
FlinkVersions.current()
 Key: FLINK-29485
 URL: https://issues.apache.org/jira/browse/FLINK-29485
 Project: Flink
  Issue Type: Bug
  Components: Tests
Affects Versions: 1.16.0, 1.17.0
Reporter: Matthias Pohl


{{TypeSerializerUpgradeTestBase}} still refers to 1.15 as the current version. 
We could use {{FlinkVersions.current()}} instead to avoid running into this 
issue again for future major updates.

I didn't check other occurrences of {{FlinkVersions}}. It should be verified as 
part of this Jira issue that we don't have the same issue in other locations as 
well.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[VOTE] Release 1.16.0, release candidate #1

2022-09-30 Thread Xingbo Huang
Hi everyone,

Please review and vote on the release candidate #1 for the version 1.16.0,
as follows:
[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)

The complete staging area is available for your review, which includes:
* JIRA release notes [1],
* the official Apache source release and binary convenience releases to be
deployed to dist.apache.org [2], which are signed with the key with
fingerprint 3C2C9FFB59DF9F3E [3],
* all artifacts to be deployed to the Maven Central Repository [4],
* source code tag "release-1.16.0-rc1" [5],
* website pull request listing the new release and adding announcement blog
post [6].

The vote will be open for at least 72 hours. It is adopted by majority
approval, with at least 3 PMC affirmative votes.

Thanks,
Chesnay, Martijn, Godfrey & Xingbo

[1]
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12351275
[2] https://dist.apache.org/repos/dist/dev/flink/flink-1.16.0-rc1
[3] https://dist.apache.org/repos/dist/release/flink/KEYS
[4] https://repository.apache.org/content/repositories/orgapacheflink-1537
[5] https://github.com/apache/flink/tree/release-1.16.0-rc1
[6] https://github.com/apache/flink-web/pull/574


[jira] [Created] (FLINK-29486) Enable SQL Client to Connect SQL Gateway in Remote Mode

2022-09-30 Thread yuzelin (Jira)
yuzelin created FLINK-29486:
---

 Summary: Enable SQL Client to Connect SQL Gateway in Remote Mode
 Key: FLINK-29486
 URL: https://issues.apache.org/jira/browse/FLINK-29486
 Project: Flink
  Issue Type: New Feature
  Components: Table SQL / Client
Affects Versions: 1.17.0
Reporter: yuzelin






--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Created] (FLINK-29487) RpcService should implement AutoCloseableAsync

2022-09-30 Thread Chesnay Schepler (Jira)
Chesnay Schepler created FLINK-29487:


 Summary: RpcService should implement AutoCloseableAsync
 Key: FLINK-29487
 URL: https://issues.apache.org/jira/browse/FLINK-29487
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Coordination
Reporter: Chesnay Schepler
Assignee: Chesnay Schepler
 Fix For: 1.17.0






--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Created] (FLINK-29488) MetricRegistryImpl should implement AutoCloseableAsync

2022-09-30 Thread Chesnay Schepler (Jira)
Chesnay Schepler created FLINK-29488:


 Summary: MetricRegistryImpl should implement AutoCloseableAsync
 Key: FLINK-29488
 URL: https://issues.apache.org/jira/browse/FLINK-29488
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Metrics
Reporter: Chesnay Schepler
Assignee: Chesnay Schepler
 Fix For: 1.17.0






--
This message was sent by Atlassian Jira
(v8.20.10#820010)


Re: [VOTE] Release 1.16.0, release candidate #1

2022-09-30 Thread Matthias Pohl
-1 (non-binding)

Hi Xingbo,
I just noticed that we haven't updated the current Flink version in
TypeSerializerUpgradeTest. It is missing in the release-1.16 branch and on
master. That means that the serialization tests are not executed for Flink
1.16. See FLINK-29485 [1].

[1] https://issues.apache.org/jira/browse/FLINK-29485

On Fri, Sep 30, 2022 at 11:28 AM Xingbo Huang  wrote:

> Hi everyone,
>
> Please review and vote on the release candidate #1 for the version 1.16.0,
> as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
> The complete staging area is available for your review, which includes:
> * JIRA release notes [1],
> * the official Apache source release and binary convenience releases to be
> deployed to dist.apache.org [2], which are signed with the key with
> fingerprint 3C2C9FFB59DF9F3E [3],
> * all artifacts to be deployed to the Maven Central Repository [4],
> * source code tag "release-1.16.0-rc1" [5],
> * website pull request listing the new release and adding announcement blog
> post [6].
>
> The vote will be open for at least 72 hours. It is adopted by majority
> approval, with at least 3 PMC affirmative votes.
>
> Thanks,
> Chesnay, Martijn, Godfrey & Xingbo
>
> [1]
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12351275
> [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.16.0-rc1
> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> [4] https://repository.apache.org/content/repositories/orgapacheflink-1537
> [5] https://github.com/apache/flink/tree/release-1.16.0-rc1
> [6] https://github.com/apache/flink-web/pull/574
>


Re: [VOTE] Release 1.16.0, release candidate #1

2022-09-30 Thread Matthias Pohl
Looking into the git history, there are numerous locations that need to be
updated in the release-1.16 branch. Yun Gao did a few commits around that
topic (da9e6be..6f69f4e). But these changes were committed close to the
actual release date rather than the release branch creation date. Is this
part of the release steps [1]. It feels like it should be part of the
section where we create the release branch. But cannot find the pointer.

[1]
https://cwiki.apache.org/confluence/display/FLINK/Creating+a+Flink+Release

On Fri, Sep 30, 2022 at 2:47 PM Matthias Pohl 
wrote:

> -1 (non-binding)
>
> Hi Xingbo,
> I just noticed that we haven't updated the current Flink version in
> TypeSerializerUpgradeTest. It is missing in the release-1.16 branch and on
> master. That means that the serialization tests are not executed for Flink
> 1.16. See FLINK-29485 [1].
>
> [1] https://issues.apache.org/jira/browse/FLINK-29485
>
> On Fri, Sep 30, 2022 at 11:28 AM Xingbo Huang  wrote:
>
>> Hi everyone,
>>
>> Please review and vote on the release candidate #1 for the version 1.16.0,
>> as follows:
>> [ ] +1, Approve the release
>> [ ] -1, Do not approve the release (please provide specific comments)
>>
>> The complete staging area is available for your review, which includes:
>> * JIRA release notes [1],
>> * the official Apache source release and binary convenience releases to be
>> deployed to dist.apache.org [2], which are signed with the key with
>> fingerprint 3C2C9FFB59DF9F3E [3],
>> * all artifacts to be deployed to the Maven Central Repository [4],
>> * source code tag "release-1.16.0-rc1" [5],
>> * website pull request listing the new release and adding announcement
>> blog
>> post [6].
>>
>> The vote will be open for at least 72 hours. It is adopted by majority
>> approval, with at least 3 PMC affirmative votes.
>>
>> Thanks,
>> Chesnay, Martijn, Godfrey & Xingbo
>>
>> [1]
>>
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12351275
>> [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.16.0-rc1
>> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
>> [4]
>> https://repository.apache.org/content/repositories/orgapacheflink-1537
>> [5] https://github.com/apache/flink/tree/release-1.16.0-rc1
>> [6] https://github.com/apache/flink-web/pull/574
>>
>


Re: [VOTE] Release 1.16.0, release candidate #1

2022-09-30 Thread Chesnay Schepler

Could you be more specific as to what you believe should be updated?

IIRC the release-1.16 branch only gets updated ta all once the release 
is out (e.g., mark docs as stable, update japicmp reference).


On 30/09/2022 15:32, Matthias Pohl wrote:

Looking into the git history, there are numerous locations that need to be
updated in the release-1.16 branch. Yun Gao did a few commits around that
topic (da9e6be..6f69f4e). But these changes were committed close to the
actual release date rather than the release branch creation date. Is this
part of the release steps [1]. It feels like it should be part of the
section where we create the release branch. But cannot find the pointer.

[1]
https://cwiki.apache.org/confluence/display/FLINK/Creating+a+Flink+Release

On Fri, Sep 30, 2022 at 2:47 PM Matthias Pohl 
wrote:


-1 (non-binding)

Hi Xingbo,
I just noticed that we haven't updated the current Flink version in
TypeSerializerUpgradeTest. It is missing in the release-1.16 branch and on
master. That means that the serialization tests are not executed for Flink
1.16. See FLINK-29485 [1].

[1] https://issues.apache.org/jira/browse/FLINK-29485

On Fri, Sep 30, 2022 at 11:28 AM Xingbo Huang  wrote:


Hi everyone,

Please review and vote on the release candidate #1 for the version 1.16.0,
as follows:
[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)

The complete staging area is available for your review, which includes:
* JIRA release notes [1],
* the official Apache source release and binary convenience releases to be
deployed to dist.apache.org [2], which are signed with the key with
fingerprint 3C2C9FFB59DF9F3E [3],
* all artifacts to be deployed to the Maven Central Repository [4],
* source code tag "release-1.16.0-rc1" [5],
* website pull request listing the new release and adding announcement
blog
post [6].

The vote will be open for at least 72 hours. It is adopted by majority
approval, with at least 3 PMC affirmative votes.

Thanks,
Chesnay, Martijn, Godfrey & Xingbo

[1]

https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12351275
[2] https://dist.apache.org/repos/dist/dev/flink/flink-1.16.0-rc1
[3] https://dist.apache.org/repos/dist/release/flink/KEYS
[4]
https://repository.apache.org/content/repositories/orgapacheflink-1537
[5] https://github.com/apache/flink/tree/release-1.16.0-rc1
[6] https://github.com/apache/flink-web/pull/574





Re: [VOTE] Release 1.16.0, release candidate #1

2022-09-30 Thread Matthias Pohl
Probably, my expectations were wrong here: I expected that we want to test
the compatibility in these tests between different major versions and that
we would want to verify the current version of the release branch as well
before releasing the artifacts. What's the rationale behind doing it after
the artifacts are released?

On Fri, Sep 30, 2022 at 3:37 PM Chesnay Schepler  wrote:

> Could you be more specific as to what you believe should be updated?
>
> IIRC the release-1.16 branch only gets updated ta all once the release
> is out (e.g., mark docs as stable, update japicmp reference).
>
> On 30/09/2022 15:32, Matthias Pohl wrote:
> > Looking into the git history, there are numerous locations that need to
> be
> > updated in the release-1.16 branch. Yun Gao did a few commits around that
> > topic (da9e6be..6f69f4e). But these changes were committed close to the
> > actual release date rather than the release branch creation date. Is this
> > part of the release steps [1]. It feels like it should be part of the
> > section where we create the release branch. But cannot find the pointer.
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/FLINK/Creating+a+Flink+Release
> >
> > On Fri, Sep 30, 2022 at 2:47 PM Matthias Pohl 
> > wrote:
> >
> >> -1 (non-binding)
> >>
> >> Hi Xingbo,
> >> I just noticed that we haven't updated the current Flink version in
> >> TypeSerializerUpgradeTest. It is missing in the release-1.16 branch and
> on
> >> master. That means that the serialization tests are not executed for
> Flink
> >> 1.16. See FLINK-29485 [1].
> >>
> >> [1] https://issues.apache.org/jira/browse/FLINK-29485
> >>
> >> On Fri, Sep 30, 2022 at 11:28 AM Xingbo Huang  wrote:
> >>
> >>> Hi everyone,
> >>>
> >>> Please review and vote on the release candidate #1 for the version
> 1.16.0,
> >>> as follows:
> >>> [ ] +1, Approve the release
> >>> [ ] -1, Do not approve the release (please provide specific comments)
> >>>
> >>> The complete staging area is available for your review, which includes:
> >>> * JIRA release notes [1],
> >>> * the official Apache source release and binary convenience releases
> to be
> >>> deployed to dist.apache.org [2], which are signed with the key with
> >>> fingerprint 3C2C9FFB59DF9F3E [3],
> >>> * all artifacts to be deployed to the Maven Central Repository [4],
> >>> * source code tag "release-1.16.0-rc1" [5],
> >>> * website pull request listing the new release and adding announcement
> >>> blog
> >>> post [6].
> >>>
> >>> The vote will be open for at least 72 hours. It is adopted by majority
> >>> approval, with at least 3 PMC affirmative votes.
> >>>
> >>> Thanks,
> >>> Chesnay, Martijn, Godfrey & Xingbo
> >>>
> >>> [1]
> >>>
> >>>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12351275
> >>> [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.16.0-rc1
> >>> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> >>> [4]
> >>> https://repository.apache.org/content/repositories/orgapacheflink-1537
> >>> [5] https://github.com/apache/flink/tree/release-1.16.0-rc1
> >>> [6] https://github.com/apache/flink-web/pull/574
> >>>
>
>


Re: [VOTE] Release 1.16.0, release candidate #1

2022-09-30 Thread Matthias Pohl
Ok, I was wrong. The step is actually documented at the end of the Flink
release documentation [1] in item 15) in the "Checklist to declare the
process completed" subsection. I missed that one. Sorry for the confusion.

I revoke my veto and close FLINK-29485 [2].

[1]
https://cwiki.apache.org/confluence/display/FLINK/Creating+a+Flink+Release
[2] https://issues.apache.org/jira/browse/FLINK-29485

On Fri, Sep 30, 2022 at 3:50 PM Matthias Pohl 
wrote:

> Probably, my expectations were wrong here: I expected that we want to test
> the compatibility in these tests between different major versions and that
> we would want to verify the current version of the release branch as well
> before releasing the artifacts. What's the rationale behind doing it after
> the artifacts are released?
>
> On Fri, Sep 30, 2022 at 3:37 PM Chesnay Schepler 
> wrote:
>
>> Could you be more specific as to what you believe should be updated?
>>
>> IIRC the release-1.16 branch only gets updated ta all once the release
>> is out (e.g., mark docs as stable, update japicmp reference).
>>
>> On 30/09/2022 15:32, Matthias Pohl wrote:
>> > Looking into the git history, there are numerous locations that need to
>> be
>> > updated in the release-1.16 branch. Yun Gao did a few commits around
>> that
>> > topic (da9e6be..6f69f4e). But these changes were committed close to the
>> > actual release date rather than the release branch creation date. Is
>> this
>> > part of the release steps [1]. It feels like it should be part of the
>> > section where we create the release branch. But cannot find the pointer.
>> >
>> > [1]
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/Creating+a+Flink+Release
>> >
>> > On Fri, Sep 30, 2022 at 2:47 PM Matthias Pohl 
>> > wrote:
>> >
>> >> -1 (non-binding)
>> >>
>> >> Hi Xingbo,
>> >> I just noticed that we haven't updated the current Flink version in
>> >> TypeSerializerUpgradeTest. It is missing in the release-1.16 branch
>> and on
>> >> master. That means that the serialization tests are not executed for
>> Flink
>> >> 1.16. See FLINK-29485 [1].
>> >>
>> >> [1] https://issues.apache.org/jira/browse/FLINK-29485
>> >>
>> >> On Fri, Sep 30, 2022 at 11:28 AM Xingbo Huang  wrote:
>> >>
>> >>> Hi everyone,
>> >>>
>> >>> Please review and vote on the release candidate #1 for the version
>> 1.16.0,
>> >>> as follows:
>> >>> [ ] +1, Approve the release
>> >>> [ ] -1, Do not approve the release (please provide specific comments)
>> >>>
>> >>> The complete staging area is available for your review, which
>> includes:
>> >>> * JIRA release notes [1],
>> >>> * the official Apache source release and binary convenience releases
>> to be
>> >>> deployed to dist.apache.org [2], which are signed with the key with
>> >>> fingerprint 3C2C9FFB59DF9F3E [3],
>> >>> * all artifacts to be deployed to the Maven Central Repository [4],
>> >>> * source code tag "release-1.16.0-rc1" [5],
>> >>> * website pull request listing the new release and adding announcement
>> >>> blog
>> >>> post [6].
>> >>>
>> >>> The vote will be open for at least 72 hours. It is adopted by majority
>> >>> approval, with at least 3 PMC affirmative votes.
>> >>>
>> >>> Thanks,
>> >>> Chesnay, Martijn, Godfrey & Xingbo
>> >>>
>> >>> [1]
>> >>>
>> >>>
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12351275
>> >>> [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.16.0-rc1
>> >>> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
>> >>> [4]
>> >>>
>> https://repository.apache.org/content/repositories/orgapacheflink-1537
>> >>> [5] https://github.com/apache/flink/tree/release-1.16.0-rc1
>> >>> [6] https://github.com/apache/flink-web/pull/574
>> >>>
>>
>>


[jira] [Created] (FLINK-29489) Display issue when querying complex, deeply nested fields

2022-09-30 Thread Justin (Jira)
Justin created FLINK-29489:
--

 Summary: Display issue when querying complex, deeply nested fields
 Key: FLINK-29489
 URL: https://issues.apache.org/jira/browse/FLINK-29489
 Project: Flink
  Issue Type: Bug
Reporter: Justin
 Attachments: FlinkQueries.txt, flink-screenshot1.jpg, 
flink-screenshot2.jpg

Using Flink 1.15, I’m observing some strange behavior when querying fields that 
have complex, deeply nested fields.

The attached file, FlinkQueries.txt, has two create table statements.

In the first table, I create a Flink table on top of the `tfmsStatusOutput` 
field. I have no issues querying this field using the SQL client or Java API.

The second table, I create a Flink table on top of the `fltdOutput` field. This 
field is more complex and has deeply nested fields. Using the SQL client, when 
I run a simple bounded query such as `SELECT * FROM TBL2 LIMIT 1;` I get a 
stack trace dump in my display window and when I press any key on my keyboard 
it returns to the result window as if it's waiting for the results. Those are 
screenshots  (flink-screenshot1 and flink-screenshot2).

 

Using the Java API, I experience something similar. 

It looks like it is going to return something good:

Starting the SELECT...
Job has been submitted with JobID 0b38a2c51e8357e5fef471b38ac839d0
+++++
| op |                fltdOutput |

but then it starts spitting out some form of diagnostics that doesn’t make much 
sense:

/* 1 */public final class GeneratedCastExecutor$45 implements 
org.apache.flink.table.data.utils.CastExecutor {
/* 2 */java.lang.StringBuilder builder$46 = new java.lang.StringBuilder();
/* 3 */java.lang.StringBuilder builder$50 = new java.lang.StringBuilder();
/* 4 */java.lang.StringBuilder builder$55 = new java.lang.StringBuilder();
…
/* 845 */java.lang.StringBuilder builder$7642 = new java.lang.StringBuilder();
/* 846 */java.lang.StringBuilder builder$7654 = new java.lang.StringBuilder();
/* 847 */java.lang.StringBuilder builder$7664 = new java.lang.StringBuilder();
/* 848 */java.lang.StringBuilder builder$7672 = new java.lang.StringBuilder();
/* 849 */public GeneratedCastExecutor$45() {
/* 850 */}
/* 851 */@Override public Object cast(Object _myInputObj) throws 
org.apache.flink.table.api.TableException {
/* 852 */org.apache.flink.table.data.RowData _myInput = 
((org.apache.flink.table.data.RowData)(_myInputObj));
/* 853 */boolean _myInputIsNull = _myInputObj == null;
/* 854 */boolean isNull$0;
/* 855 */org.apache.flink.table.data.binary.BinaryStringData result$1;
/* 856 */boolean isNull$2;
/* 857 */org.apache.flink.table.data.binary.BinaryStringData result$3;
/* 858 */boolean isNull$4;

etc…

 

I couldn't find anything in the task manager logs.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


dev@flink.apache.org

2022-09-30 Thread Péter Váry
+1 having an option storing every version of a connector in one repo

Also, it would be good to have the major(.minor) version of the connected
system in the name of the connector jar, depending of the compatibility. I
think this compatibility is mostly system dependent.

Thanks, Peter


On Fri, Sep 30, 2022, 09:32 Martijn Visser  wrote:

> Hi Peter,
>
> I think this also depends on the support SLA that the technology that you
> connect to provides. For example, with Flink and Elasticsearch, we choose
> to follow Elasticsearch supported versions. So that means that when support
> for Elasticsearch 8 is introduced, support for Elasticsearch 6 should be
> dropped (since Elastic only support the last major version and the latest
> minor version prior to that)
>
> I don't see value in having different connectors for Iceberg 0.14 and 0.15
> in separate repositories. I think that will confuse the user. I would
> expect that with modules you should be able to have support for multiple
> versions in one repository.
>
> Best regards,
>
> Martijn
>
> On Fri, Sep 30, 2022 at 7:44 AM Péter Váry 
> wrote:
>
> > Thanks for the quick response!
> >
> > Would this mean, that we have different connectors for Iceberg 0.14, and
> > Iceberg 0.15. Would these different versions kept in different
> repository?
> >
> > My feeling is that this model is fine for the stable/slow moving systems
> > like Hive/HBase. For other systems, which are evolving faster, this is
> less
> > than ideal.
> >
> > For those, who have more knowledge about the Flink ecosystem: How do you
> > feel? What is the distribution of the connectors between the slow moving
> > and the fast moving systems?
> >
> > Thanks, Peter
> >
> >
> > On Thu, Sep 29, 2022, 16:46 Danny Cranmer 
> wrote:
> >
> > > If you look at ElasticSearch [1] as an example there are different
> > variants
> > > of the connector depending on the "connected" system:
> > > - flink-connector-elasticsearch6
> > > - flink-connector-elasticsearch7
> > >
> > > Looks like Hive and HBase follow a similar pattern in the main Flink
> > repo/
> > >
> > > [1] https://github.com/apache/flink-connector-elasticsearch
> > >
> > > On Thu, Sep 29, 2022 at 3:17 PM Péter Váry <
> peter.vary.apa...@gmail.com>
> > > wrote:
> > >
> > > > Hi Team,
> > > >
> > > > Just joining the conversation for the first time, so pardon me if I
> > > repeat
> > > > already answered questions.
> > > >
> > > > It might be already discussed, but I think the version for the
> > > "connected"
> > > > system could be important as well.
> > > >
> > > > There might be some API changes between Iceberg 0.14.2, and 1.0.0,
> > which
> > > > would require as to rewrite part of the code for the Flink-Iceberg
> > > > connector.
> > > > It would be important for the users:
> > > > - Which Flink version(s) are this connector working with?
> > > > - Which Iceberg version(s) are this connector working with?
> > > > - Which code version we have for this connector?
> > > >
> > > > Does this make sense? What is the community's experience with the
> > > connected
> > > > systems? Are they stable enough for omitting their version number
> from
> > > the
> > > > naming of the connectors? Would this worth the proliferation of the
> > > > versions?
> > > >
> > > > Thanks,
> > > > Peter
> > > >
> > > > Chesnay Schepler  ezt írta (időpont: 2022.
> szept.
> > > 29.,
> > > > Cs, 14:11):
> > > >
> > > > > 2) No; the branch names would not have a Flink version in them;
> > v1.0.0,
> > > > > v1.0.1 etc.
> > > > >
> > > > > On 29/09/2022 14:03, Martijn Visser wrote:
> > > > > > If I summarize it correctly, that means that:
> > > > > >
> > > > > > 1. The versioning scheme would be  > > > > > version>-, where there will
> > > never
> > > > > be a
> > > > > > patch release for a minor version if a newer minor version
> already
> > > > > exists.
> > > > > > E.g., 1.0.0-1.15; 1.0.1-1.15; 1.1.0-1.15; 1.2.0-1.15;
> > > > > >
> > > > > > 2. The branch naming scheme would be
> > > > vmajor.minor-flink-major.flink-minor
> > > > > > E.g., v1.0.0-1.15; v1.0.1-1.15; v1.1.0-1.15; v1.2.0-1.15;
> > > > > >
> > > > > > I would +1 that.
> > > > > >
> > > > > > Best regards,
> > > > > >
> > > > > > Martijn
> > > > > >
> > > > > > On Tue, Sep 20, 2022 at 2:21 PM Chesnay Schepler <
> > ches...@apache.org
> > > >
> > > > > wrote:
> > > > > >
> > > > > >>   > After 1.16, only patches are accepted for 1.2.0-1.15.
> > > > > >>
> > > > > >> I feel like this is a misunderstanding that both you and Danny
> ran
> > > > into.
> > > > > >>
> > > > > >> What I meant in the original proposal is that the last 2 _major_
> > > > > >> /connector /versions are supported, with the latest receiving
> > > > additional
> > > > > >> features.
> > > > > >> (Provided that the previous major version still works against a
> > > > > >> currently supported Flink version!)
> > > > > >> There will never be patch releases for a minor version if a
> newer
> > > > minor
> > > > > >> version exists.
> > > > > >>
> > 

Re: [VOTE] Apache Flink Kubernetes Operator Release 1.2.0, release candidate #2

2022-09-30 Thread Jim Busche
+1 (not-binding)

Thank you Gyula,


Helm install from flink-kubernetes-operator-1.2.0-helm.tgz looks good, logs 
look normal

podman Dockerfile build from source looks good.

twistlock security scans of the proposed image look good:  
ghcr.io/apache/flink-kubernetes-operator:95128bf

UI and basic sample look good.

Checksums looked good.

Tested on OpenShift 4.10.25.  Will try additional versions (4.8 and 4.11) if I 
get an opportunity, but I don't expect issues.



Thank you,

James Busche


[jira] [Created] (FLINK-29490) Timestamp LTZ is unsupported in table store

2022-09-30 Thread Jingsong Lee (Jira)
Jingsong Lee created FLINK-29490:


 Summary: Timestamp LTZ is unsupported in table store 
 Key: FLINK-29490
 URL: https://issues.apache.org/jira/browse/FLINK-29490
 Project: Flink
  Issue Type: Bug
  Components: Table Store
Reporter: Jingsong Lee
 Fix For: table-store-0.3.0


Due to orc format limitation, timestamp ltz is unsupported now. We should fix 
this, and validate this type cross multiple engines (hive spark trino).
We need to careful about time zone.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Created] (FLINK-29491) Primary key without partition field can be supported from full changelog

2022-09-30 Thread Jingsong Lee (Jira)
Jingsong Lee created FLINK-29491:


 Summary: Primary key without partition field can be supported from 
full changelog
 Key: FLINK-29491
 URL: https://issues.apache.org/jira/browse/FLINK-29491
 Project: Flink
  Issue Type: Improvement
  Components: Table Store
Reporter: Jingsong Lee
 Fix For: table-store-0.3.0


When pk does not contain partition fields, an exception will be thrown under 
any circumstances. We can relax this restriction. When the input is a complete 
changelog.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Created] (FLINK-29492) Kafka exactly-once sink causes OutOfMemoryError

2022-09-30 Thread Robert Metzger (Jira)
Robert Metzger created FLINK-29492:
--

 Summary: Kafka exactly-once sink causes OutOfMemoryError
 Key: FLINK-29492
 URL: https://issues.apache.org/jira/browse/FLINK-29492
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Kafka
Affects Versions: 1.15.2
Reporter: Robert Metzger


My Kafka exactly-once sinks are periodically failing with a `OutOfMemoryError: 
Java heap space`.

This looks very similar to FLINK-28250. But I am running 1.15.2, which contains 
a fix for FLINK-28250.

Exception:
{code:java}
java.io.IOException: Could not perform checkpoint 2281 for operator 
http_events[3]: Writer (1/1)#1.
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpointOnBarrier(StreamTask.java:1210)
at 
org.apache.flink.streaming.runtime.io.checkpointing.CheckpointBarrierHandler.notifyCheckpoint(CheckpointBarrierHandler.java:147)
at 
org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler.triggerCheckpoint(SingleCheckpointBarrierHandler.java:287)
at 
org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler.access$100(SingleCheckpointBarrierHandler.java:64)
at 
org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler$ControllerImpl.triggerGlobalCheckpoint(SingleCheckpointBarrierHandler.java:493)
at 
org.apache.flink.streaming.runtime.io.checkpointing.AbstractAlignedBarrierHandlerState.triggerGlobalCheckpoint(AbstractAlignedBarrierHandlerState.java:74)
at 
org.apache.flink.streaming.runtime.io.checkpointing.AbstractAlignedBarrierHandlerState.barrierReceived(AbstractAlignedBarrierHandlerState.java:66)
at 
org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler.lambda$processBarrier$2(SingleCheckpointBarrierHandler.java:234)
at 
org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler.markCheckpointAlignedAndTransformState(SingleCheckpointBarrierHandler.java:262)
at 
org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler.processBarrier(SingleCheckpointBarrierHandler.java:231)
at 
org.apache.flink.streaming.runtime.io.checkpointing.CheckpointedInputGate.handleEvent(CheckpointedInputGate.java:181)
at 
org.apache.flink.streaming.runtime.io.checkpointing.CheckpointedInputGate.pollNext(CheckpointedInputGate.java:159)
at 
org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:110)
at 
org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:519)
at 
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:203)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:804)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:753)
at 
org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:948)
at 
org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:927)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:741)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:563)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: org.apache.flink.runtime.checkpoint.CheckpointException: Could not 
complete snapshot 2281 for operator http_events[3]: Writer (1/1)#1. Failure 
reason: Checkpoint was declined.
at 
org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.snapshotState(StreamOperatorStateHandler.java:269)
at 
org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.snapshotState(StreamOperatorStateHandler.java:173)
at 
org.apache.flink.streaming.api.operators.AbstractStreamOperator.snapshotState(AbstractStreamOperator.java:348)
at 
org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.checkpointStreamOperator(RegularOperatorChain.java:227)
at 
org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.buildOperatorSnapshotFutures(RegularOperatorChain.java:212)
at 
org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.snapshotState(RegularOperatorChain.java:192)
at 
org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.takeSnapshotSync(SubtaskCheckpointCoordinatorImpl.java:647)
at 
org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.checkpointState(SubtaskCheckpointCoordinatorImpl.java:320)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$performCheckpoint$12(StreamTask.java:1253)
at 
org.apache.flink.streaming.runtime.t