Re: [DISCUSS] FLIP-379: Dynamic source parallelism inference for batch jobs

Thanks Xia and Zhu Zhu for kickoff this discussion. The dynamic source parallelism inference is a useful feature for batch story. I’ve some comments about current design. 1.How user disable the parallelism inference if they want to use fixed source parallelism? They can configure fixed paralle

Re: [DISCUSS] Release Flink 1.16.3

+1 Sergey Nuyanzin 于2023年11月8日周三 13:08写道： > +1 for the final release > and thanks for the efforts > > On Wed, Nov 8, 2023 at 4:09 AM Leonard Xu wrote: > > > Thanks Rui for driving this. > > > > +1 to release 1.16.3 and make it as the final bugix release of 1.16 > > series. > > > > Best, > > Le

Re: [DISCUSS] FLIP-384: Introduce TraceReporter and use it to create checkpointing and recovery traces

Hi Zakelly, Thanks for the comments. Quick answer for both of your questions would be that it probably should be left as a future work. For more detailed answers please take a look below :) > Does it mean the inclusion and subdivision relationships of spans defined > by "parent_id" are not suppor

Re: [VOTE] FLIP-376: Add DISTRIBUTED BY clause

+1(binding) Best, Leonard > 2023年11月8日 下午1:05，Sergey Nuyanzin 写道： > > +1 (binding) > > On Wed, Nov 8, 2023 at 6:02 AM Zhanghao Chen > wrote: > >> +1 (non-binding) >> >> Best, >> Zhanghao Chen >> >> From: Timo Walther >> Sent: Monday, November 6, 2023 19:38

[DISCUSS] FLIP-389: Annotate SingleThreadFetcherManager and FutureCompletingBlockingQueue as PublicEvolving

Hi devs, I would like to start a discussion on FLIP-389: Annotate SingleThreadFetcherManager and FutureCompletingBlockingQueue as PublicEvolving.[ 1]. Though the SingleThreadFetcherManager is annotated as Internal, it ac

[jira] [Created] (FLINK-33482) Flink benchmark regression check in the new machines

Zakelly Lan created FLINK-33482: --- Summary: Flink benchmark regression check in the new machines Key: FLINK-33482 URL: https://issues.apache.org/jira/browse/FLINK-33482 Project: Flink Issue Type

[DISCUSS] FLIP-390: Support System out and err to be redirected to LOG or discarded

Hi all! I would like to start a discussion of FLIP-390: Support System out and err to be redirected to LOG or discarded[1]. In various production environments, either cloud native or physical machines, the disk space that Flink TaskManager can use is limited. In general, the flink users shouldn'

Re: [DISCUSS] FLIP-389: Annotate SingleThreadFetcherManager and FutureCompletingBlockingQueue as PublicEvolving

Thanks Hongshun for starting this discussion. +1 from my side. IIRC, @Jiangjie(Becket) also mentioned this in FLINK-31324 comment[1]. Best, Leonard [1] https://issues.apache.org/jira/browse/FLINK-31324?focusedCommentId=17696756&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-ta

Re: [DISCUSS] Connector releases for Flink 1.18

Hi, Thanks for starting the discussion, I already started testing Cassandra connector on Flink 1.18. The error in the nightly build linked here is a known issue tracked by FLINK-32353. To fix it, it needs a change in connector-parent and in the ci utils that are tracked by FLINK-32563. Best

[jira] [Created] (FLINK-33483) Why is “UNDEFINED” defined in the Flink task status?

Xin Chen created FLINK-33483: Summary: Why is “UNDEFINED” defined in the Flink task status? Key: FLINK-33483 URL: https://issues.apache.org/jira/browse/FLINK-33483 Project: Flink Issue Type: Impr

[jira] [Created] (FLINK-33484) Flink Kafka Connector Offset Lag Issue with Transactional Data and Read Committed Isolation Level

Darcy Lin created FLINK-33484: - Summary: Flink Kafka Connector Offset Lag Issue with Transactional Data and Read Committed Isolation Level Key: FLINK-33484 URL: https://issues.apache.org/jira/browse/FLINK-33484

[jira] [Created] (FLINK-33485) Optimize the EXISTS sub-query by Metadata RowCount

Sergey Nuyanzin created FLINK-33485: --- Summary: Optimize the EXISTS sub-query by Metadata RowCount Key: FLINK-33485 URL: https://issues.apache.org/jira/browse/FLINK-33485 Project: Flink Issu

Re: [DISCUSS] FLIP-384: Introduce TraceReporter and use it to create checkpointing and recovery traces

Hi Piotr, Thanks for driving this proposal! I strongly agree that the existing metric APIs are not suitable for monitoring restore/checkpoint behavior! I think the TM-level recovery/checkpointing traces are necessary in the future. In our production environment, we sometimes encounter that job

Re: [VOTE] Release flink-connector-gcp-pubsub v3.0.2, release candidate #1

Hey Leonard, Thanks for helping to verify the release. The 1.16.0 Flink version in the pom is a miss, ideally it should be updated to 1.18.0, additionally this should have been updated to 1.17.x previously. I would not consider it a hard blocker since the Maven build overrides this variable based

Re: [DISCUSS] FLIP-390: Support System out and err to be redirected to LOG or discarded

Hi Rui, Thanks for the proposal. +1 I don't have any major comments :) One nit. In `SystemOutRedirectToLog` in this code: System.arraycopy(buf, count - LINE_SEPARATOR_LENGTH, bytes, 0, LINE_SEPARATOR_LENGTH); return Arrays.equals(LINE_SEPARATOR_BYTES, bytes) Is there a r

Fink 1.17 vs 1.18

Hello Flink dev team, This question might have been answered already but I want to ask again after the official release 1.18. What are the major differences between Fink 1.17 and 1.18? Is 1.18 faster for streaming processing? Thank you very much. Best, Amir

[jira] [Created] (FLINK-33486) Pulsar Client Send Timeout Terminates TaskManager

Jason Kania created FLINK-33486: --- Summary: Pulsar Client Send Timeout Terminates TaskManager Key: FLINK-33486 URL: https://issues.apache.org/jira/browse/FLINK-33486 Project: Flink Issue Type: B

[jira] [Created] (FLINK-33487) Add the new Snowflake connector to supported list

Mohsen Rezaei created FLINK-33487: - Summary: Add the new Snowflake connector to supported list Key: FLINK-33487 URL: https://issues.apache.org/jira/browse/FLINK-33487 Project: Flink Issue Typ

Re: [DISCUSS] FLIP-390: Support System out and err to be redirected to LOG or discarded

Hi Rui, Thanks for the proposal. The proposed solution of supporting System out and err to be redirected to LOG or discarded and introducing an enum and two options to manage this, seems reasonable. +1 Thanks, Archit Goyal From: Piotr Nowojski Date: Wednesday, November 8, 2023 at 7:38 AM T

[jira] [Created] (FLINK-33488) Implement restore tests for Deduplicate node

Jim Hughes created FLINK-33488: -- Summary: Implement restore tests for Deduplicate node Key: FLINK-33488 URL: https://issues.apache.org/jira/browse/FLINK-33488 Project: Flink Issue Type: Sub-task

Re: Fink 1.17 vs 1.18

Hi Amir, You can find the major differences in Flink 1.18's release note[1], which has also described the performance improvements in 1.18. [1] https://flink.apache.org/2023/10/24/announcing-the-release-of-apache-flink-1.18/ Best, Yunfeng Zhou On Wed, Nov 8, 2023 at 11:50 PM Amir Hossein Shari

Re: [DISCUSS] FLIP-390: Support System out and err to be redirected to LOG or discarded

Hi, Rui. Thank you for initiating this proposal. I have a question regarding redirecting stdout and stderr to LOG: Will they be written to the taskManager.log file by default or the taskManager.out file? If we can make taskmanager.out splittable and rolling, would it be easier for users to use t

Re: [VOTE] Release flink-connector-gcp-pubsub v3.0.2, release candidate #1

Thanks Danny for the reply. -1 (binding) Let’s fix the outdated version in the source code and spin a new rc2. I’d like to open a PR to fix it, and hope everything OK in your Flink Forward Trip. Best, Leonard > Thanks for helping to verify the release. The 1.16.0 Flink version in the > pom

Re: [DISCUSS] FLIP-384: Introduce TraceReporter and use it to create checkpointing and recovery traces

Hi Piotr, Thanks for your detailed explanation! I could see the challenge of implementing traces with multiple spans and agree to put it in the future work. I personally prefer the idea of generating multi span traces for checkpoints on the JM only. > I'm not sure if I understand the proposal - I

Re: [DISCUSS] FLIP-390: Support System out and err to be redirected to LOG or discarded

Hi, Rui. Thanks for the proposal. It sounds reasonable. I have some questions, PTAL: 1. I have a similar concern as Feng. Will we redirect to another log file not taskManager.log ? taskManager.log contains lots of important information like init log. It will be rolled quickly if we redirect out and

Re: [DISCUSS] Release 1.17.2

Hi All, Thank you for your feedback! As there are no other concerns or objections, and currently I am not aware of any unresolved blockers. I will kick off the release process and start preparing for the RC1 version from today. Best Yun Tang From: Leonard Xu

Re: [DISCUSS] Release Flink 1.16.3

Hi All, Thank you for your feedback! As there are no other concerns or objections, and currently I am not aware of any unresolved blockers. I will kick off the release process and start preparing for the RC1 version from today. Best, Rui On Wed, Nov 8, 2023 at 4:23 PM ConradJam wrote: > +1 >

[jira] [Created] (FLINK-33489) LISTAGG with generating partial-final agg will case wrong result

xuyang created FLINK-33489: -- Summary: LISTAGG with generating partial-final agg will case wrong result Key: FLINK-33489 URL: https://issues.apache.org/jira/browse/FLINK-33489 Project: Flink Issue T

Re: [DISCUSS] FLIP-384: Introduce TraceReporter and use it to create checkpointing and recovery traces

Hi, Piotr. Thanks for the proposal. Just as we discussed in FLINK-23411, +1 for supporting trace/span to monitor metrics like checkpoint and recovery. We could also do many things based on this mechanism: 1. more fine-grained metrics about checkpoint and recovery. For example, some stage info abou

Re: [DISCUSS] FLIP-384: Introduce TraceReporter and use it to create checkpointing and recovery traces

Hi Piotr: Thanks for your reply! > About structured logging (basically events?) I vaguely remember some > discussions about that. It might be a much larger topic, so I would prefer > to leave it out of the scope of this FLIP. Sounds make sense to me! > I think those could be indeed useful. If y

[DISCUSS] Release flink-connector-pulsar 4.1.0

Hi, I'd propose to cut a new release for flink-connector-pulsar 4.1.0[1]. >From the last release (4.0.0), we mainly achieved: 1. Implement table connector (integrated with Flink SQL) 2. Drop the requirement for using adminURL 3. Support JDK 11 I can help in driving the release but perhaps we ne

Re: [DISCUSS] FLIP-390: Support System out and err to be redirected to LOG or discarded

Hi Piotr, Archit, Feng and Hangxiang: Thanks a lot for your feedback! Following is my comment, please correct me if I misunderstood anything! To Piotr: > Is there a reason why you are suggesting to copy out bytes from `buf` to `bytes`, > instead of using `Arrays.equals(int[] a, int aFromIndex,

Re: [DISCUSS] Release flink-connector-pulsar 4.1.0

Hey, Tison. +1 to release flink-connector-pulsar 4.1.0. I’m glad to offer help for the release. Best, Leonard > 2023年11月9日 下午1:30，tison 写道： > > Hi, > > I'd propose to cut a new release for flink-connector-pulsar 4.1.0[1]. > > From the last release (4.0.0), we mainly achieved: > > 1. Im

[jira] [Created] (FLINK-33490) Validate the name conflicts when creating view

Shengkai Fang created FLINK-33490: - Summary: Validate the name conflicts when creating view Key: FLINK-33490 URL: https://issues.apache.org/jira/browse/FLINK-33490 Project: Flink Issue Type: