Re: [DISCUSS] FLIP-250: Support Customized Kubernetes Schedulers Proposal

Hi all, Thanks for the FLIP. I have a couple of remarks/questions: * Regarding the motivation, it mentions that the development trend is that Flink supports both batch and stream processing. I think the vision and trend is that we have unified batch- and stream processing. What I'm missing is the

Re: [DISCUSS] Add new JobStatus fields to Flink Kubernetes Operator CRD

Hi Daren, Could you list the benefits for the users of Flink? I do think that an internal AWS requirement is not a good argument for getting something done in Flink. Best regards, Martijn Op di 12 jul. 2022 om 21:17 schreef WONG, DAREN : > Hi Yang, > > The requirement to add *plan* currently o

Re: [DISCUSS] FLIP-217 Support watermark alignment of source splits

Hi Sebastian, Thanks for updating the FLIP wiki. Just to double confirm, I was thinking of a configuration like "allow.coarse.grained.watermark.alignment". This will allow the coarse grained watermark alignment as a fallback instead of bubbling up an exception if split pausing is not supported in

Re: Not able to run a simple table API file streaming sink

Hi all, For completeness, for this issue a ticket has been created at https://issues.apache.org/jira/browse/FLINK-28513 Best regards, Martijn Op di 12 jul. 2022 om 03:49 schreef Jaya Ananthram : > Hello There, > > I am trying to write a simple table API S3 streaming sink using flink > 1.15.1 a

Re: [DISCUSS] FLIP-251: Support collecting arbitrary number of streams

+1 Op zo 10 jul. 2022 om 20:30 schreef Konstantin Knauf : > Makes sense to me. Thank you, Chesnay. +1 > > David Anderson schrieb am Fr., 8. Juli 2022, 13:56: > > > I've found that with our current tooling it's frustrating to try to write > > good end-to-end tests for real-world jobs with multipl

Re: [DISCUSS] FLIP-243: Dedicated Opensearch connectors

Hi Andriy, Thanks for that. If there's no more comments I think you can open up a vote thread. Best regards, Martijn Op ma 4 jul. 2022 om 18:32 schreef Andriy Redko : > Hi Martijn, > > Thanks a lot supporting the FLIP! Regarding your question, yes, > Elasticsearch > has a new client in v8 (dep

[jira] [Created] (FLINK-28533) SchemaChange supports updateColumnNullability and updateColumnComment

Jane Chan created FLINK-28533: - Summary: SchemaChange supports updateColumnNullability and updateColumnComment Key: FLINK-28533 URL: https://issues.apache.org/jira/browse/FLINK-28533 Project: Flink

[jira] [Created] (FLINK-28534) Spec change event is triggered twice per upgrade

Gyula Fora created FLINK-28534: -- Summary: Spec change event is triggered twice per upgrade Key: FLINK-28534 URL: https://issues.apache.org/jira/browse/FLINK-28534 Project: Flink Issue Type: New

Re: [DISCUSS] FLIP-217 Support watermark alignment of source splits

Hi Becket, Hi Thomas, Hi Piotrek, Thanks for the feedback. I would like to highlight some concerns: 1. Major: A configuration parameter like "allow coarse grained alignment" defines a semantic that mixes two contexts conditionally as follows: "ignore incapability to pause splits in Sourc

[jira] [Created] (FLINK-28535) Support create database/table for SparkCatalog

Jane Chan created FLINK-28535: - Summary: Support create database/table for SparkCatalog Key: FLINK-28535 URL: https://issues.apache.org/jira/browse/FLINK-28535 Project: Flink Issue Type: New Feat

[jira] [Created] (FLINK-28536) Adds an internal abstraction `PhysicalDagProcessor` for physical plan processing

lincoln lee created FLINK-28536: --- Summary: Adds an internal abstraction `PhysicalDagProcessor` for physical plan processing Key: FLINK-28536 URL: https://issues.apache.org/jira/browse/FLINK-28536 Projec

[jira] [Created] (FLINK-28537) OffsetsInitializer.timestamp() fails if no element exceeds timestamps

Chesnay Schepler created FLINK-28537: Summary: OffsetsInitializer.timestamp() fails if no element exceeds timestamps Key: FLINK-28537 URL: https://issues.apache.org/jira/browse/FLINK-28537 Project

Re: [DISCUSS] FLIP-249: Flink Web UI Enhancement for Speculative Execution

Hi Gen, > The way the speculative executions are presented should be almost the same as the job was running. Users can still find the executions folded in the subtask list page. It's a more complicated operation to check all vertex and all subtasks list page. It's better to have an easier way to

Re: [VOTE] FLIP-249: Flink Web UI Enhancement for Speculative Execution

+1 (binding) Best, Jing Zhang Gen Luo 于2022年7月13日周三 14:49写道： > Hi Jing, > > I have replied in the discussion thread about the questions. Hope that > would be helpful. > > Best, > Gen > > On Tue, Jul 12, 2022 at 8:43 PM Jing Zhang wrote: > > > Hi, Gen Luo, > > > > I left two minor questions in

Re: [DISCUSS] FLIP-243: Dedicated Opensearch connectors

Hi Martijn, Awesome, will do that shortly, thank you! Best Regards, Andriy Redko > Hi Andriy, > Thanks for that. If there's no more comments I think you can open up a vote > thread. > Best regards, > Martijn > Op ma 4 jul. 2022 om 18:32 schreef Andriy Redko : >> Hi Martijn, >> Thanks

[REQUEST] Edit Permissions for FLIP

Hi folks! My name is Steve Su, and I am currently a third-year master’s student at Tsinghua University. I am now participating in a Flink-related student project in OSPP (Open Source Promotion Plan[1]). The project aims to design and implement the generic rate limiter for the FLIP-27 source and a

Re: [DISCUSS] FLIP-248: Introduce dynamic partition pruning

Hi Godfrey, Thanks for driving this discussion. This is an important improvement for batch sql jobs. I agree with Jingsong to expand the capability to more than just partitions. Besides, I have two points: 1. Based on FLIP-248[1], > Dynamic partition pruning mechanism can improve performance by av

[jira] [Created] (FLINK-28538) Eagerly emit a watermark the first time we exceed Long.MIN_VALUE

Chesnay Schepler created FLINK-28538: Summary: Eagerly emit a watermark the first time we exceed Long.MIN_VALUE Key: FLINK-28538 URL: https://issues.apache.org/jira/browse/FLINK-28538 Project: Fli

[jira] [Created] (FLINK-28539) Enable CompactionDynamicLevelBytes in FLASH_SSD_OPTIMIZED

Usamah Jassat created FLINK-28539: - Summary: Enable CompactionDynamicLevelBytes in FLASH_SSD_OPTIMIZED Key: FLINK-28539 URL: https://issues.apache.org/jira/browse/FLINK-28539 Project: Flink I

Re: [REQUEST] Edit Permissions for FLIP

Hi Steve, Welcome to the Flink community. I have granted the edit permission to you. Looking forward to your FLIP. Best, Jark On Wed, 13 Jul 2022 at 19:56, Steve Yurong Su wrote: > Hi folks! > > My name is Steve Su, and I am currently a third-year master’s student > at Tsinghua University. > >

Re: [DISCUSS] FLIP-217 Support watermark alignment of source splits

Thanks for the explanation, Sebastian. I understand your concern now. 1. About the major concern. Personally I'd consider the coarse grained watermark alignment as a special case for backward compatibility. In the future, if for whatever reason we want to pause a split and that is not supported, i

[jira] [Created] (FLINK-28540) Unaligned checkpoint waiting in 'start delay' with AsyncDataStream

Nathan P Sharp created FLINK-28540: -- Summary: Unaligned checkpoint waiting in 'start delay' with AsyncDataStream Key: FLINK-28540 URL: https://issues.apache.org/jira/browse/FLINK-28540 Project: Flink

[jira] [Created] (FLINK-28541) Add OwnerReferences to FlinkDeployment CR in jobmanager Deployment

ChangZhuo Chen (陳昌倬) created FLINK-28541: Summary: Add OwnerReferences to FlinkDeployment CR in jobmanager Deployment Key: FLINK-28541 URL: https://issues.apache.org/jira/browse/FLINK-28541 Pr

Re: [REQUEST] Edit Permissions for FLIP

Hi Jark, Thanks a lot :) Steve Jark Wu 于2022年7月13日周三 22:48写道： > > Hi Steve, > > Welcome to the Flink community. I have granted the edit permission to you. > Looking forward to your FLIP. > > Best, > Jark > > On Wed, 13 Jul 2022 at 19:56, Steve Yurong Su wrote: > > > Hi folks! > > > > My name i

Re: [REQUEST] Edit Permissions for FLIP

Hi Steve, in the FLIP-238 [1] I propose to also introduce a rate limiter with a simple API at the level of SourceReader: RateLimiter.java Guav

[VOTE] FLIP-243: Dedicated Opensearch connectors

Hey Folks, Thanks a lot for all the feedback and comments so far. Based on the discussion [1], it seems like there is a genuine interest in supporting OpenSearch [2] natively. With that being said, I would like to start a vote on FLIP-243 [3]. The vote will last for at least 72 hours unless t

[jira] [Created] (FLINK-28542) [JUnit5 Migration] FileSystemBehaviorTestSuite

Ryan Skraba created FLINK-28542: --- Summary: [JUnit5 Migration] FileSystemBehaviorTestSuite Key: FLINK-28542 URL: https://issues.apache.org/jira/browse/FLINK-28542 Project: Flink Issue Type: Sub-

[jira] [Created] (FLINK-28543) After compiling Flink, you should provide the artifact file in tar format instead of a folder

MengYao created FLINK-28543: --- Summary: After compiling Flink, you should provide the artifact file in tar format instead of a folder Key: FLINK-28543 URL: https://issues.apache.org/jira/browse/FLINK-28543 P

[jira] [Created] (FLINK-28544) Elasticsearch6SinkE2ECase failed with no space left on device

Huang Xingbo created FLINK-28544: Summary: Elasticsearch6SinkE2ECase failed with no space left on device Key: FLINK-28544 URL: https://issues.apache.org/jira/browse/FLINK-28544 Project: Flink

[jira] [Created] (FLINK-28545) FlinkKafkaProducerITCase.testRestoreToCheckpointAfterExceedingProducersPool failed with TimeoutException: Topic flink-kafka-producer-fail-before-notify not present in m

Huang Xingbo created FLINK-28545: Summary: FlinkKafkaProducerITCase.testRestoreToCheckpointAfterExceedingProducersPool failed with TimeoutException: Topic flink-kafka-producer-fail-before-notify not present in metadata after 6 ms

[jira] [Created] (FLINK-28546) Add the release logic for py39 and m1 wheel package

Huang Xingbo created FLINK-28546: Summary: Add the release logic for py39 and m1 wheel package Key: FLINK-28546 URL: https://issues.apache.org/jira/browse/FLINK-28546 Project: Flink Issue Typ

[jira] [Created] (FLINK-28547) Add IT cases for SpeculativeScheduler

Lijie Wang created FLINK-28547: -- Summary: Add IT cases for SpeculativeScheduler Key: FLINK-28547 URL: https://issues.apache.org/jira/browse/FLINK-28547 Project: Flink Issue Type: Sub-task

Re: [DISCUSS] FLIP-250: Support Customized Kubernetes Schedulers Proposal

Hi Martijn, Thank you for your comments. I will answer the questions one by one. "" * Regarding the motivation, it mentions that the development trend is that Flink supports both batch and stream processing. I think the vision and trend is that we have unified batch- and stream processing. What I

Re: [DISCUSS] FLIP-249: Flink Web UI Enhancement for Speculative Execution

Hi Jing, I agree that in this version it's not really convenient to check out if there are speculative attempts. I think what we need to improve this is not only the enhancement of the current Web UI, but a new page summarizing the information of speculative execution on the history server, or bot

[jira] [Created] (FLINK-28548) The commit partition base path is not created when no data is sent which may cause FileNotFoundException

Liu created FLINK-28548: --- Summary: The commit partition base path is not created when no data is sent which may cause FileNotFoundException Key: FLINK-28548 URL: https://issues.apache.org/jira/browse/FLINK-28548

Re: [VOTE] FLIP-249: Flink Web UI Enhancement for Speculative Execution

+1 (binding) Thanks for driving this! Best, Yun Gao -- From:Jing Zhang Send Time:2022 Jul. 13 (Wed.) 19:07 To:undefined Subject:Re: [VOTE] FLIP-249: Flink Web UI Enhancement for Speculative Execution +1 (binding) Best, Jing Zha

Re: [REQUEST] Edit Permissions for FLIP

Hi Alexander, The reference [3] mentioned in my previous email is your discussion thread on FLIP-238. We have noticed your idea of providing a new rate-limiter, which is great and matches our starting point. I was just about to chat with you on slack about this feature :) I tend to propose a sepa