[jira] [Created] (FLINK-29617) Cost too much time to start SourceCoordinator of hdfsFileSource when start JobMaster

LI Mingkun created FLINK-29617: -- Summary: Cost too much time to start SourceCoordinator of hdfsFileSource when start JobMaster Key: FLINK-29617 URL: https://issues.apache.org/jira/browse/FLINK-29617 Proj

[jira] [Created] (FLINK-29618) YARNSessionFIFOSecuredITCase.testDetachedMode timed out in Azure CI

Matthias Pohl created FLINK-29618: - Summary: YARNSessionFIFOSecuredITCase.testDetachedMode timed out in Azure CI Key: FLINK-29618 URL: https://issues.apache.org/jira/browse/FLINK-29618 Project: Flink

[jira] [Created] (FLINK-29619) Remove redundant MeterView updater thread from KubernetesClientMetrics

Matyas Orhidi created FLINK-29619: - Summary: Remove redundant MeterView updater thread from KubernetesClientMetrics Key: FLINK-29619 URL: https://issues.apache.org/jira/browse/FLINK-29619 Project: Fli

[jira] [Created] (FLINK-29620) Flink deployment stuck in UPGRADING state when changing configuration

liad shachoach created FLINK-29620: -- Summary: Flink deployment stuck in UPGRADING state when changing configuration Key: FLINK-29620 URL: https://issues.apache.org/jira/browse/FLINK-29620 Project: Fl

Re: [VOTE] Drop Gelly

Hi Yu, The vote is for option 1, remove all code and documentation of Gelly. For the current users, they can stay on a previous version of Flink which still contains Gelly. Since Gelly (and the DataSet API) haven't been touched in multiple releases, there's most likely no need for them to upgrade

[jira] [Created] (FLINK-29621) Append-only with eventual log.consistency can not work

Jingsong Lee created FLINK-29621: Summary: Append-only with eventual log.consistency can not work Key: FLINK-29621 URL: https://issues.apache.org/jira/browse/FLINK-29621 Project: Flink Issue

Re: [VOTE] Remove HCatalog connector

+1 (non-binding) Thanks Martijn for driving this On Thu, Oct 13, 2022 at 8:47 AM Qingsheng Ren wrote: > +1 > > Thanks for driving this Martijn! > > Best, > Qingsheng > > > On Oct 13, 2022, at 03:45, Martijn Visser > wrote: > > > > Hi everyone, > > > > Since no comments were made, I'm opening a

Re: [ANNOUNCE] Apache Flink Table Store 0.2.1 released

Congratulations and thanks to all those involved! On Thu, Oct 13, 2022 at 4:47 AM Jingsong Lee wrote: > The Apache Flink community is very happy to announce the release of > Apache Flink Table Store 0.2.1. > > Apache Flink Table Store is a unified storage to build dynamic tables > for both strea

Re: [VOTE] Drop Gelly

Hi all, For the successor of the Gelly library, currently in flink-ml we have already implemented a new iteration framework based on the DataStream API, which is necessary for a graph processing framework. It could also provide abilities like allowing for arbitrary states across rounds and async

[jira] [Created] (FLINK-29622) KerberosDelegationTokenManager fails to load DelegationTokenProvider due to NoClassDefFoundError in various tests

Matthias Pohl created FLINK-29622: - Summary: KerberosDelegationTokenManager fails to load DelegationTokenProvider due to NoClassDefFoundError in various tests Key: FLINK-29622 URL: https://issues.apache.org/jira/b

Re: [VOTE] Drop Gelly

+1 On Wed, Oct 12, 2022 at 11:00 PM Martijn Visser wrote: > Hi everyone, > > I would like to open a vote for dropping Gelly, which was discussed a long > time ago but never put to a vote [1]. > > Voting will be open for at least 72 hours. > > Best regards, > > Martijn > https://twitter.com/Marti

[jira] [Created] (FLINK-29623) Bump Prometheus Java Client to 0.16.0

Martijn Visser created FLINK-29623: -- Summary: Bump Prometheus Java Client to 0.16.0 Key: FLINK-29623 URL: https://issues.apache.org/jira/browse/FLINK-29623 Project: Flink Issue Type: Technic

[jira] [Created] (FLINK-29624) Upgrade org.apache.commons:commons-lang3 from 3.3.2 to 3.12.0

Martijn Visser created FLINK-29624: -- Summary: Upgrade org.apache.commons:commons-lang3 from 3.3.2 to 3.12.0 Key: FLINK-29624 URL: https://issues.apache.org/jira/browse/FLINK-29624 Project: Flink

[jira] [Created] (FLINK-29625) Optimize changelog normalize

Jiabao Sun created FLINK-29625: -- Summary: Optimize changelog normalize Key: FLINK-29625 URL: https://issues.apache.org/jira/browse/FLINK-29625 Project: Flink Issue Type: Improvement Co

[jira] [Created] (FLINK-29626) Update Akka to 2.6.20

Martijn Visser created FLINK-29626: -- Summary: Update Akka to 2.6.20 Key: FLINK-29626 URL: https://issues.apache.org/jira/browse/FLINK-29626 Project: Flink Issue Type: Technical Debt

Re: [Discuss]- Donate Iceberg Flink Connector

+1. Thanks for driving it. Hope I can find some chances to take part in the future development of Iceberg Flink Connector. Best regards, Yuxia - 原始邮件 - 发件人: "Zheng Yu Chen" 收件人: "dev" 发送时间: 星期四, 2022年 10 月 13日 上午 11:26:29 主题: Re: [Discuss]- Donate Iceberg Flink Connector +1, thanks to

dev@flink.apache.org

I mean minor. I always get confused on the Flink side because we use "major" instead of "minor" releases in many places. On 12/10/2022 20:18, Danny Cranmer wrote: Thanks for the concise summary Chesnay. +1 from me (binding) Just one clarification, for "3.1) The Flink versions supported by the

dev@flink.apache.org

@Thomas: Version-specific modules that either contain a connector or shims to support that Flink version. Alternatively, since the addition of such code (usually) goes beyond a patch release you'd create a new minor version and could have that only support the later version. On 13/10/2022 02:

Re: [VOTE] Drop Gelly

+1 Am Do., 13. Okt. 2022 um 10:56 Uhr schrieb Niels Basjes : > +1 > > On Wed, Oct 12, 2022 at 11:00 PM Martijn Visser > wrote: > > > Hi everyone, > > > > I would like to open a vote for dropping Gelly, which was discussed a > long > > time ago but never put to a vote [1]. > > > > Voting will be

Re: [DISCUSS] FLIP-265 Deprecate and remove Scala API support

Support for records has not been investigated yet. We're still at the stage of getting things to run at all on Java 17. It _may_ be possible, it _may_ not be. On 13/10/2022 07:39, Salva Alcántara wrote: Hi Martijn, Maybe a bit of an off-topic, but regarding Java 17 support, will it be possib

Re: [VOTE] Remove HCatalog connector

+1 On 12/10/2022 21:45, Martijn Visser wrote: Hi everyone, Since no comments were made, I'm opening a vote to remove the HCatalog connector [1] The voting period will be open at least 72hrs. Best regards, Martijn [1] https://lists.apache.org/thread/j8jc5zrhnqlv8y3lkmc3wdo9ysgmsr84

[jira] [Created] (FLINK-29627) Sink - Duplicate key exception during recover more than 1 committable.

Krzysztof Chmielewski created FLINK-29627: - Summary: Sink - Duplicate key exception during recover more than 1 committable. Key: FLINK-29627 URL: https://issues.apache.org/jira/browse/FLINK-29627

dev@flink.apache.org

+1 (binding) for the vote and thanks for the explanation On Thu, Oct 13, 2022 at 5:58 AM Chesnay Schepler wrote: > @Thomas: > Version-specific modules that either contain a connector or shims to > support that Flink version. > Alternatively, since the addition of such code (usually) goes beyond

[jira] [Created] (FLINK-29628) Bump aws-java-sdk-s3 to 1.12.319

Martijn Visser created FLINK-29628: -- Summary: Bump aws-java-sdk-s3 to 1.12.319 Key: FLINK-29628 URL: https://issues.apache.org/jira/browse/FLINK-29628 Project: Flink Issue Type: Technical De

dev@flink.apache.org

+1 non-binding! I've been following (and generally agreeing) with the thread -- it's a perfectly reasonable way to start, and I'm sure we can adjust the process if it turns out to be unsuitable or unexpected as the connectors evolve in their external repositories. On Thu, Oct 13, 2022 at 12:37 PM

[jira] [Created] (FLINK-29629) FlameGraph is empty for Legacy Source Threads

Peter Vary created FLINK-29629: -- Summary: FlameGraph is empty for Legacy Source Threads Key: FLINK-29629 URL: https://issues.apache.org/jira/browse/FLINK-29629 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-29630) Junit 5.8.1 run unit test with temporary directory will occur Failed to delete temp directory.

Aiden Gong created FLINK-29630: -- Summary: Junit 5.8.1 run unit test with temporary directory will occur Failed to delete temp directory. Key: FLINK-29630 URL: https://issues.apache.org/jira/browse/FLINK-29630

Re: [DISCUSS] FLIP-265 Deprecate and remove Scala API support

Hi Martjin, After some more careful consideration I am in favor of dropping the Scala API support in with Flink 2.0 given that we add Java 17 support earlier or latest at the same time. Best, Marton On Thu, Oct 13, 2022 at 12:01 PM Chesnay Schepler wrote: > Support for records has not been inv

Re: [VOTE] Remove HCatalog connector

+1 (non-binding) Bests, Samrat On Thu, 13 Oct 2022 at 8:23 AM, yuxia wrote: > +1 (non-binding) > Thanks for driving. > > Best regards, > Yuxia > > - 原始邮件 - > 发件人: "Hang Ruan" > 收件人: "dev" > 发送时间: 星期四, 2022年 10 月 13日 上午 10:16:46 > 主题: Re: [VOTE] Remove HCatalog connector > > +1, > > B

Re: [ANNOUNCE] New Apache Flink PMC Member - Danny Cranmer

Congratulations Danny! Best, Yang Hang Ruan 于2022年10月13日周四 10:58写道： > Congratulations Danny! > > Best, > Hang > > Yun Gao 于2022年10月13日周四 10:56写道： > > > Congratulations Danny! > > Best, > > Yun Gao > > -- > > From:yuxia > > Send T

Re: [DISCUSS] Reference operator from Flink Kubernetes deployment docs

+1 for increasing the visibility of flink-kubernetes-operator. Best, Yang Thomas Weise 于2022年10月13日周四 07:49写道： > +1 > > > On Wed, Oct 12, 2022 at 5:03 PM Martijn Visser > wrote: > > > +1 from my end to include the operator in the related Kubernetes sections > > of the Flink docs > > > > On Wed

Re: [DISCUSS] Reference operator from Flink Kubernetes deployment docs

+1 good idea! On Thu, Oct 13, 2022 at 2:42 PM Yang Wang wrote: > +1 for increasing the visibility of flink-kubernetes-operator. > > Best, > Yang > > Thomas Weise 于2022年10月13日周四 07:49写道： > > > +1 > > > > > > On Wed, Oct 12, 2022 at 5:03 PM Martijn Visser > > > wrote: > > > > > +1 from my end to

Re: [DISCUSS] Reverting sink metric name changes made in 1.15

Hi Jing, Thanks for the reply! Let me rephrase my proposal: we’d like to use numXXXOut registered on SinkWriterOperator to reflect the traffic to the external system for compatibility with old versions before 1.15, and make numXXXSend have the same value as numXXXOut for compatibility within 1.15

Re: [ANNOUNCE] New Apache Flink PMC Member - Danny Cranmer

Congratulations Danny! Well deserved :) -Max On Thu, Oct 13, 2022 at 2:40 PM Yang Wang wrote: > Congratulations Danny! > > Best, > Yang > > Hang Ruan 于2022年10月13日周四 10:58写道： > > > Congratulations Danny! > > > > Best, > > Hang > > > > Yun Gao 于2022年10月13日周四 10:56写道： > > > > > Congratulations D

Re: [VOTE] Drop Gelly

+1 On Thu, Oct 13, 2022 at 12:00 PM Konstantin Knauf wrote: > +1 > > Am Do., 13. Okt. 2022 um 10:56 Uhr schrieb Niels Basjes : > > > +1 > > > > On Wed, Oct 12, 2022 at 11:00 PM Martijn Visser < > martijnvis...@apache.org> > > wrote: > > > > > Hi everyone, > > > > > > I would like to open a vote

Re: [DISCUSS] Reference operator from Flink Kubernetes deployment docs

+1 Agree, good idea! On Thu, Oct 13, 2022 at 2:52 PM Jing Ge wrote: > +1 good idea! > > On Thu, Oct 13, 2022 at 2:42 PM Yang Wang wrote: > > > +1 for increasing the visibility of flink-kubernetes-operator. > > > > Best, > > Yang > > > > Thomas Weise 于2022年10月13日周四 07:49写道： > > > > > +1 > > > >

Re: State of the Rescale API

Thank you everyone for the clarification. It looks like the Rescale API will require some more work before it can be fully taken advantage of. I'm wondering whether in addition to the Rescale API we want to provide a means to do a rescale without an explicit API call but as part of the job submissi

dev@flink.apache.org

+1 and I would suggest giving a concrete example to explain 4) support. It is still not quite easy to understand the text. Not many (future) connector developers could join this discussion. It is better to make it as clear as possible at the beginning than spend more time explaining multiple times.

Re: [DISCUSS] Reverting sink metric name changes made in 1.15

Hi Qingsheng, Thanks for the clarification. +1, I like the idea. Pointing both numXXXOut and numXXXSend to the same external data transfer metric does not really break the new SinkV2 design, since there was no requirement to monitor the internal traffic. So, I think both developer and user can liv

[jira] [Created] (FLINK-29631) [CVE-2022-42003] flink-shaded-jackson

Sergio Sainz created FLINK-29631: Summary: [CVE-2022-42003] flink-shaded-jackson Key: FLINK-29631 URL: https://issues.apache.org/jira/browse/FLINK-29631 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-29632) Support nodeSelector in helm template for flink operator deployment

Jeesmon Jacob created FLINK-29632: - Summary: Support nodeSelector in helm template for flink operator deployment Key: FLINK-29632 URL: https://issues.apache.org/jira/browse/FLINK-29632 Project: Flink

[VOTE] Reverting sink metric name changes made in 1.15

Hi devs, I'd like to start a vote about reverting sink metric name changes made in 1.15 considering compatibility issues. These metrics include: - numRecordsSend -> numRecordsOut - numRecordsSendPerSecond -> numRecordsOutPerSecond - numBytesSend -> numBytesOut - numBytesSendPerSecond -> numBytesO

Re: [DISCUSS] Reverting sink metric name changes made in 1.15

Hi devs and users, It looks like we are getting an initial consensus in the discussion so I started a voting thread [1] just now. Looking forward to your feedback! [1] https://lists.apache.org/thread/ozlf82mkm6ndx2n1vdgq532h156p4lt6 Best, Qingsheng On Thu, Oct 13, 2022 at 10:41 PM Jing Ge wro

[jira] [Created] (FLINK-29633) Operator doesn't pass initialSavepointPath as fromSavepoint argument

Yaroslav Tkachenko created FLINK-29633: -- Summary: Operator doesn't pass initialSavepointPath as fromSavepoint argument Key: FLINK-29633 URL: https://issues.apache.org/jira/browse/FLINK-29633 Proj

Re: [VOTE] Reverting sink metric name changes made in 1.15

Do we have a PoC that achieves this without re-introducing the bug where the numRecordsOut was simply wrong because it counted both records written to the external system and the downstream committer? It's gonna be quite the dirty hack I assume. On 13/10/2022 19:24, Qingsheng Ren wrote: Hi dev

dev@flink.apache.org

I will write this all down in the wiki once the vote is over, but here are some example. Let's start out with a happy-case scenario with one connector supporting the last 2 Flink versions. This will commonly be the scenario for connectors when they have been externalized: v1: 1.14-1.15 No

[jira] [Created] (FLINK-29634) Support periodic checkpoint triggering

Thomas Weise created FLINK-29634: Summary: Support periodic checkpoint triggering Key: FLINK-29634 URL: https://issues.apache.org/jira/browse/FLINK-29634 Project: Flink Issue Type: New Featur

Re:[VOTE] Remove HCatalog connector

 +1 (non-binding) Thanks for driving. Best regards, Roc Marshal. At 2022-10-13 03:45:57, "Martijn Visser" wrote: >Hi everyone, > >Since no comments were made, I'm opening a vote to remove the HCatalog >connector [1] > >The voting period will be open at least 72hrs. > >Best regards, > >Martijn >

[jira] [Created] (FLINK-29635) Hive sink should supports merge small files in batch mode

luoyuxia created FLINK-29635: Summary: Hive sink should supports merge small files in batch mode Key: FLINK-29635 URL: https://issues.apache.org/jira/browse/FLINK-29635 Project: Flink Issue Type:

Re: Utilizing Kafka headers in Flink Kafka connector

hi. You can use SQL API to parse or write the header in the Kafka record[1] if you are using Flink SQL. Best, Shengkai [1] https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/kafka/#available-metadata Yaroslav Tkachenko 于2022年10月13日周四 02:21写道： > Hi, > > You can implemen

Re: [DISCUSS] Reverting sink metric name changes made in 1.15

@Qingsheng, I'm overall +1 to your proposal, with only one question: How complicated is it to come up with a metric for the internal traffic? I'm asking because, as the new feature is already out for 1.15 & 1.16, it would be nice if the corresponding new metrics can also be available in these ver

Re: [VOTE] Reverting sink metric name changes made in 1.15

+1 from me (non-binding). The documents about these Kafka sink metrics( Flink metrics and Kafka sink monitoring

[jira] [Created] (FLINK-29636) Add micro benchmark module in flink table store

Shammon created FLINK-29636: --- Summary: Add micro benchmark module in flink table store Key: FLINK-29636 URL: https://issues.apache.org/jira/browse/FLINK-29636 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-29637) Improve to reuse threads in TaskManager for different tasks between jobs

Shammon created FLINK-29637: --- Summary: Improve to reuse threads in TaskManager for different tasks between jobs Key: FLINK-29637 URL: https://issues.apache.org/jira/browse/FLINK-29637 Project: Flink

Re: [DISCUSS] FLIP-263: Improve resolving schema compatibility

Hi, Zakelly. Thanks for your suggestion and I think it makes sense. I think both two options should be adopted. The option 2 is just what I mentioned in step 2. At the same time, we still need option 1 because we should make sure the existing user code could run normally without modifying any code

Re: [VOTE] Reverting sink metric name changes made in 1.15

Thanks a lot for driving this, Qingsheng! +1 (binding) Best, Jark On Fri, 14 Oct 2022 at 10:16, Hang Ruan wrote: > +1 from me (non-binding). > > The documents about these Kafka sink metrics( Flink metrics > < > https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/ops/metrics//#scop

Re: [DISCUSS] Reverting sink metric name changes made in 1.15

Hi Xintong, In terms of code, I think it's not complicated. It's all about we need a public discussion for the new metric name. And we don't want to block the release for the rarely used metric. Best, Jark On Fri, 14 Oct 2022 at 10:07, Xintong Song wrote: > @Qingsheng, > > I'm overall +1 to yo

Re: [Discuss]- Donate Iceberg Flink Connector

Thank Abid for the discussion, I'm also fine with maintaining it under the Flink project. But I'm also interested in the response to Martijn's question. Besides, once the code is moved to the Flink project, are there any initial maintainers for the connector we can find? In addition, do we still

RE: [DISCUSS] FLIP-263: Improve resolving schema compatibility

Hi Hangxiang, Thanks for the proposal. It seems more reasonable to let the new serializer claim the compatibility in the cases you mentioned. I have but one question here. What happens in the case of “compatibleAfterMigration” after we completely reverse the direction (in step 3)? To be speci

Re: [VOTE] Reverting sink metric name changes made in 1.15

+1 (binding) Best, Xintong On Fri, Oct 14, 2022 at 11:19 AM Jark Wu wrote: > Thanks a lot for driving this, Qingsheng! > > +1 (binding) > > > Best, > Jark > > On Fri, 14 Oct 2022 at 10:16, Hang Ruan wrote: > > > +1 from me (non-binding). > > > > The documents about these Kafka sink metrics(

Re: [VOTE] Reverting sink metric name changes made in 1.15

+1 Best, Xingbo Xintong Song 于2022年10月14日周五 11:42写道： > +1 (binding) > > Best, > > Xintong > > > > On Fri, Oct 14, 2022 at 11:19 AM Jark Wu wrote: > > > Thanks a lot for driving this, Qingsheng! > > > > +1 (binding) > > > > > > Best, > > Jark > > > > On Fri, 14 Oct 2022 at 10:16, Hang Ruan wro