Re: [ANNOUNCE] Performance Daily Monitoring Moved from Ververica to Apache Flink Slack Channel

Hi Yanfei Lei, Thanks for setting this up! It would be interesting to also know which aspects of Flink are monitored for "performance". I'm assuming there are specific pieces of functionality that are performance tested, but it would be great if this would be written down somewhere (next to a proc

[VOTE] Dedicated AWS externalized connector repo

Hello all, As discussed in the discussion thread [1], I propose to create a dedicated repository for AWS connectors called flink-connector-aws. This will house 3x connectors: Amazon Kinesis Data Streams, Amazon Kinesis Data Firehose and Amazon DynamoDB and any future AWS connectors. We will also e

Re: [DISCUSS] Release Flink 1.15.3

+1, thanks for driving this Fabian. Danny, On Wed, Oct 26, 2022 at 2:22 AM yuxia wrote: > Thanks for driving this. > +1 for release 1.15.3 > > Best regards, > Yuxia > > - 原始邮件 - > 发件人: "Leonard Xu" > 收件人: "dev" > 发送时间: 星期二, 2022年 10 月 25日 下午 10:00:47 > 主题: Re: [DISCUSS] Release Flink

Re: [ANNOUNCE] Performance Daily Monitoring Moved from Ververica to Apache Flink Slack Channel

Hi yanfei, Thanks for driving this! It's a great help. I would like to join as a maintainer. Best, Zakelly On Wed, Oct 26, 2022 at 11:32 AM yanfei lei wrote: > > Hi everyone, > > As discussed earlier, we plan to create a benchmark channel in Apache Flink > slack[1], but the plan was shelved fo

[jira] [Created] (FLINK-29764) Automatic judgment of parallelism of source

waywtdcc created FLINK-29764: Summary: Automatic judgment of parallelism of source Key: FLINK-29764 URL: https://issues.apache.org/jira/browse/FLINK-29764 Project: Flink Issue Type: New Feature

Re: [DISCUSS] FLIP-263: Improve resolving schema compatibility

Hey Huangxiang, The section of `Rejected Alternatives` may also need an update. Current plan sounds like a reasonable one. I am fine with it. Thanks for driving this. Best Yuan On Tue, Oct 25, 2022 at 5:11 PM Hangxiang Yu wrote: > （Resend the mail to fix the format issue） > Hi, everyone. > >

[jira] [Created] (FLINK-29763) TaskManager heatbeat timeout exception in Github CI for python tests

Yunfeng Zhou created FLINK-29763: Summary: TaskManager heatbeat timeout exception in Github CI for python tests Key: FLINK-29763 URL: https://issues.apache.org/jira/browse/FLINK-29763 Project: Flink

[ANNOUNCE] Performance Daily Monitoring Moved from Ververica to Apache Flink Slack Channel

Hi everyone, As discussed earlier, we plan to create a benchmark channel in Apache Flink slack[1], but the plan was shelved for a while[2]. So I went on with this work, and created the #flink-dev-benchmarks channel for performance regression notifications. We have a regression report script[3] th

Re: [DISCUSS] Release Flink 1.15.3

Thanks for driving this. +1 for release 1.15.3 Best regards, Yuxia - 原始邮件 - 发件人: "Leonard Xu" 收件人: "dev" 发送时间: 星期二, 2022年 10 月 25日 下午 10:00:47 主题: Re: [DISCUSS] Release Flink 1.15.3 Thanks Fabian for driving this. +1 to release 1.15.3. The bug tickets FLINK-26394 and FLINK-27148 shou

Re: flink-s3-fs-hadoop dependencies

Hi all, I have been thinking that we should consider creating one new, rock solid S3 connector for Flink. I think it's confusing for users that there is an S3 Presto and an S3 Hadoop implementation, which both are not perfect. I'm not sure that creating another bandaid is a good idea. I'm not su

[jira] [Created] (FLINK-29762) Can not create a standalone cluster with reactive mode using the operator

yuvipanda created FLINK-29762: - Summary: Can not create a standalone cluster with reactive mode using the operator Key: FLINK-29762 URL: https://issues.apache.org/jira/browse/FLINK-29762 Project: Flink

[jira] [Created] (FLINK-29761) Simplify HadoopModule

Gabor Somogyi created FLINK-29761: - Summary: Simplify HadoopModule Key: FLINK-29761 URL: https://issues.apache.org/jira/browse/FLINK-29761 Project: Flink Issue Type: Sub-task Repo

Re: flink-s3-fs-hadoop dependencies

Thanks for the answer Gabor! Just for the sake of clarity: - The issue is that the `flink-s3-fs-hadoop` does not even read the `core-site.xml` if it is not on the classpath Do I understand correctly that the proposal is: - Write a new `getHadoopConfiguration` method somewhere without using the de

Re: [Cassandra] source connector

Ok thanks everyone. Etienne Le 24/10/2022 à 20:03, Martijn Visser a écrit : Hi all, @Etienne many thanks for the PR for the Cassandra Source. Hopefully we can make this available after the 1.16 release. With regards to the connector externalization comments, while the wiki for the release pla

[jira] [Created] (FLINK-29760) Introduce snapshots metadata table

Jingsong Lee created FLINK-29760: Summary: Introduce snapshots metadata table Key: FLINK-29760 URL: https://issues.apache.org/jira/browse/FLINK-29760 Project: Flink Issue Type: Sub-task

Re: [DISCUSS] Release Flink 1.15.3

Thanks Fabian for driving this. +1 to release 1.15.3. The bug tickets FLINK-26394 and FLINK-27148 should be fixed as well, I’ll help to address them soon. Best, Leonard Xu > 2022年10月25日 下午8:28，Jing Ge 写道： > > +1 The timing is good to have 1.15.3 release. Thanks Fabian for bringing > this t

Re: [DISCUSS] Issue tracking workflow

Thank you for starting this discussion Xintong! I would also prefer option 1. The ASF Jira is probably one of the largest, public Jira instances on the internet. Most other Jiras are internal within companies, so Atlassian is probably not putting a lot of effort into automatically detecting and p

[jira] [Created] (FLINK-29759) Cast type in LEFT JOIN

Alexandre Decuq created FLINK-29759: --- Summary: Cast type in LEFT JOIN Key: FLINK-29759 URL: https://issues.apache.org/jira/browse/FLINK-29759 Project: Flink Issue Type: Bug Compon

[jira] [Created] (FLINK-29758) Enum

Alexandre Decuq created FLINK-29758: --- Summary: Enum Key: FLINK-29758 URL: https://issues.apache.org/jira/browse/FLINK-29758 Project: Flink Issue Type: Bug Components: API / Type S

[jira] [Created] (FLINK-29757) ContinuousFileSplitEnumerator skip unprocessed splits when the file is splittable

Hanley Yang created FLINK-29757: --- Summary: ContinuousFileSplitEnumerator skip unprocessed splits when the file is splittable Key: FLINK-29757 URL: https://issues.apache.org/jira/browse/FLINK-29757 Proje

Re: [DISCUSS] Release Flink 1.15.3

+1 The timing is good to have 1.15.3 release. Thanks Fabian for bringing this to our attention. I just checked PRs and didn't find the 1.15 backport of FLINK-29567 . Please be aware of it. Thanks! Best regards, Jing On Tue, Oct 25, 2022 at 11:44

Re: [DISCUSS] Issue tracking workflow

Hi everyone, while I see some benefits in moving to Github Issues completely, we need to be aware that Github Issues lacks many features that Jira has. From the top of my head: * there are no issue types * no priorities * issues can only be assigned to one milestone So, you need to work a lot with

[jira] [Created] (FLINK-29756) Support materialized column to improve query performance for complex types

Nicholas Jiang created FLINK-29756: -- Summary: Support materialized column to improve query performance for complex types Key: FLINK-29756 URL: https://issues.apache.org/jira/browse/FLINK-29756 Projec

Re: flink-s3-fs-hadoop dependencies

Hi Peter, > would this cause issues for the users? I think yes, it is going to make trouble for users who want to use S3 without HDFS client. Adding HDFS client may happen but enforcing it is not a good direction. As mentioned I've realized that we have 6 different ways how Hadoop conf is loaded

Re: [VOTE] Release 1.16.0, release candidate #2

+1(binding) * Built from source * Verified signature and checksum * Build docker image with flink binary * Submit/stop a streaming and batch job with Flink Kubernetes Operator and everything works well * Check the metrics and logs via ingress webUI Best, Yang Mason Chen 于2022年10月25日周二 14:43写道：

[jira] [Created] (FLINK-29755) PulsarSourceUnorderedE2ECase.testSavepoint failed because of missing TaskManagers

Matthias Pohl created FLINK-29755: - Summary: PulsarSourceUnorderedE2ECase.testSavepoint failed because of missing TaskManagers Key: FLINK-29755 URL: https://issues.apache.org/jira/browse/FLINK-29755 P

Re: [DISCUSS] Release Flink 1.15.3

Thanks for bringing this up, Fabian. +1 for creating a 1.15.3 release. I've also seen users requiring this version [1]. I can help with actions that require a PMC role, if needed. Best, Xintong [1] https://lists.apache.org/thread/501q4l1c6gs8hwh433bw3v1y8fs9cw2n On Tue, Oct 25, 2022 at 5:1

flink-s3-fs-hadoop dependencies

Hi Team, I have recently faced the issue that the S3 FileSystem read my core-site.xml until it was on the classpath, but later when I tried to add it using the HADOOP_CONF_DIR then the configuration file was not loaded. Filed a jira [1] and created a PR [2] for fixing it. HadoopUtils.getHadoopCon

[DISCUSS] Release Flink 1.15.3

Hi all, I want to start the discussion of creating a new 1.15 patch release (1.15.3). The last 1.15 release is almost two months old, and since then, ~60 tickets have been closed, targeting 1.15.3. It includes critical changes to the sink architecture, including: - Reverting the sink metric namin

Re: [DISCUSS] FLIP-263: Improve resolving schema compatibility

（Resend the mail to fix the format issue） Hi, everyone. Thanks for your suggestions! Let me summarize the remaining questions in the thread and share my ideas based on your suggestions: 1. Should we put the new opposite interface in TypeSerializer or TypeSerializerSnapshot ? Just as I repl

Re: [DISCUSS] Create a dedicated aws-base connector repository

Thanks all for the feedback. +1 on the single repo and version for AWS connectors. The reduced maintenance cost and complexity is a clear winner here. I will open a vote thread for this matter. Thanks all! On Tue, 25 Oct 2022, 03:04 Jark Wu, wrote: > TBH, I suspect the way of “a single reposit

[jira] [Created] (FLINK-29754) HadoopConfigLoader should consider Hadoop configuration files

Peter Vary created FLINK-29754: -- Summary: HadoopConfigLoader should consider Hadoop configuration files Key: FLINK-29754 URL: https://issues.apache.org/jira/browse/FLINK-29754 Project: Flink Is

Re: [DISCUSS] Changing the minimal supported version of Hadoop to 2.10.2

> Additionally, having code that hasn't been touch for a while increases the risk of it Sorry about this incomplete confusing sentence. I was about to remove it when accidentally pushing the shortcut for sending the message out to the mailing list. On Tue, Oct 25, 2022 at 10:18 AM Matthias Pohl w

Re: Limiting backpressure during checkpoints

Hi Robin. Could you share how you got the metric of CPU usage ? By summing all used CPU cores of TMs or evaluating it by the busy metric in Flink UI ? I think it's the first thing we need to align. > network (async) part of the checkpoint should in theory not cause backpressure since resources wou

[jira] [Created] (FLINK-29753) FileSource throws exception reading file with name that ends with xz

Xuannan Su created FLINK-29753: -- Summary: FileSource throws exception reading file with name that ends with xz Key: FLINK-29753 URL: https://issues.apache.org/jira/browse/FLINK-29753 Project: Flink

Re: [DISCUSS] Changing the minimal supported version of Hadoop to 2.10.2

I guess upgrading the minimal version should also mean cleaning up the codebase, i.e. removing code segments that have been around to allow support for older versions. The overall goal should be to improve the Flink codebase in my opinion. Considering what David said in the old thread about Hadoop

[jira] [Created] (FLINK-29752) Modify Flink Table Store connector to trigger full compaction constantly when full changelog is needed

Caizhi Weng created FLINK-29752: --- Summary: Modify Flink Table Store connector to trigger full compaction constantly when full changelog is needed Key: FLINK-29752 URL: https://issues.apache.org/jira/browse/FLINK-297