[jira] [Created] (FLINK-28319) test_ci tests times out after/during running org.apache.flink.test.streaming.experimental

2022-06-30 Thread Martijn Visser (Jira)
Martijn Visser created FLINK-28319: -- Summary: test_ci tests times out after/during running org.apache.flink.test.streaming.experimental Key: FLINK-28319 URL: https://issues.apache.org/jira/browse/FLINK-28319

[jira] [Created] (FLINK-28320) ShuffleMasterTest hangs and doesn't produce any output for 900 seconds

2022-06-30 Thread Martijn Visser (Jira)
Martijn Visser created FLINK-28320: -- Summary: ShuffleMasterTest hangs and doesn't produce any output for 900 seconds Key: FLINK-28320 URL: https://issues.apache.org/jira/browse/FLINK-28320 Project: F

[jira] [Created] (FLINK-28321) HiveDialectQueryITCase fails with error code 137

2022-06-30 Thread Martijn Visser (Jira)
Martijn Visser created FLINK-28321: -- Summary: HiveDialectQueryITCase fails with error code 137 Key: FLINK-28321 URL: https://issues.apache.org/jira/browse/FLINK-28321 Project: Flink Issue Ty

Re: [VOTE] FLIP-242: Introduce configurable RateLimitingStrategy for Async Sink

2022-06-30 Thread Danny Cranmer
Thanks, Hong. +1 (binding) from me. On Wed, Jun 29, 2022 at 8:32 PM Martijn Visser wrote: > This is looking good! Thanks for the efforts. +1 (binding) > > Op wo 29 jun. 2022 om 20:55 schreef Piotr Nowojski > > > Thanks for starting the voting thread. +1 (binding) from my side. > > > > Best, >

[jira] [Created] (FLINK-28322) DataStreamScanProvider's new method is not compatible

2022-06-30 Thread Jingsong Lee (Jira)
Jingsong Lee created FLINK-28322: Summary: DataStreamScanProvider's new method is not compatible Key: FLINK-28322 URL: https://issues.apache.org/jira/browse/FLINK-28322 Project: Flink Issue T

Re: [VOTE] Release 1.15.1, release candidate #1

2022-06-30 Thread Jingsong Li
Hi David, Thanks for creating this RC. -1 We found an incompatible modification in 1.15.0 [1] I think we should fix it. [1] https://issues.apache.org/jira/browse/FLINK-28322 Best, Jingsong On Tue, Jun 28, 2022 at 8:45 PM Robert Metzger wrote: > > +1 (binding) > > - staging repo contents look

[jira] [Created] (FLINK-28323) Support using new KafkaSource in PyFlink

2022-06-30 Thread Juntao Hu (Jira)
Juntao Hu created FLINK-28323: - Summary: Support using new KafkaSource in PyFlink Key: FLINK-28323 URL: https://issues.apache.org/jira/browse/FLINK-28323 Project: Flink Issue Type: New Feature

Re: Re:Re: [DISCUSS] FLIP-218: Support SELECT clause in CREATE TABLE(CTAS)

2022-06-30 Thread Martijn Visser
Hi Mang, I have two questions/remarks: 1. The FLIP mentions that if the user doesn't specify the WITH option part in the query of the sink table, it will be assumed that the user wants to create a managed table. What will happen if the user doesn't have Table Store configured/installed? Will we t

[DISCUSS] Introduce multi delete API to Flink's FileSystem class

2022-06-30 Thread Yun Tang
Hi guys, As more and more teams move to cloud-based environments. Cloud object storage has become the factual technical standard for big data ecosystems. >From our experience, the performance of writing/deleting objects in object >storage could vary in each call, the FLIP of changelog state-back

Re: [DISCUSS] FLIP-239: Port JDBC Connector Source to FLIP-27

2022-06-30 Thread João Boto
Hi, On source we could improve the JdbcParameterValuesProvider.. to be defined as a query(s) or something more dynamic. The most time if your job is dynamic or have some condition to be met (based on data on table) you have to create a connection an get that info from database If we are going t

Re:[DISCUSS] Introduce multi delete API to Flink's FileSystem class

2022-06-30 Thread Feifan Wang
Thanks a lot for the proposal @Yun Tang ! It sounds great and I can't find any reason not to make this improvement. —— Name: Feifan Wang Email: zoltar9...@163.com Replied Message | From | Yun Tang | | Date | 06/30/2022 16:56 | | To | dev@flink.apache.org | | Subject | [D

Re: [DISCUSS] Introduce multi delete API to Flink's FileSystem class

2022-06-30 Thread Martijn Visser
Hi Yun Tang, +1 for addressing this problem and your approach. Best regards, Martijn Op do 30 jun. 2022 om 11:12 schreef Feifan Wang : > Thanks a lot for the proposal @Yun Tang ! It sounds great and I can't > find any reason not to make this improvement. > > > —— > Name: Feifan Wa

[jira] [Created] (FLINK-28324) JUnit5 Migration] Module: flink-sql-client

2022-06-30 Thread zl (Jira)
zl created FLINK-28324: -- Summary: JUnit5 Migration] Module: flink-sql-client Key: FLINK-28324 URL: https://issues.apache.org/jira/browse/FLINK-28324 Project: Flink Issue Type: Improvement Comp

Re: [DISCUSS] Introduce multi delete API to Flink's FileSystem class

2022-06-30 Thread Zakelly Lan
Hi Yun, Thanks for bringing this into discussion. I'm +1 to this idea. And IIUC, Flink implements the OSS and S3 filesystem based on the hadoop filesystem interface, which does not provide the multi-delete API, it may take some effort to implement this. Best, Zakelly On Thu, Jun 30, 2022 at 5:36

[jira] [Created] (FLINK-28325) DataOutputSerializer#writeBytes increase position twice

2022-06-30 Thread huweihua (Jira)
huweihua created FLINK-28325: Summary: DataOutputSerializer#writeBytes increase position twice Key: FLINK-28325 URL: https://issues.apache.org/jira/browse/FLINK-28325 Project: Flink Issue Type: B

Re: [DISCUSS] Introduce multi delete API to Flink's FileSystem class

2022-06-30 Thread Piotr Nowojski
Hi, I presume this would mostly supersede the recursive deletes [1]? I remember an argument that the recursive deletes were not obviously better, even if the underlying FS was supporting it. I'm not saying that this would have been a counter argument against this effort, since every FileSystem cou

[jira] [Created] (FLINK-28326) ResultPartitionTest.testIdleAndBackPressuredTime failed with AssertError

2022-06-30 Thread Huang Xingbo (Jira)
Huang Xingbo created FLINK-28326: Summary: ResultPartitionTest.testIdleAndBackPressuredTime failed with AssertError Key: FLINK-28326 URL: https://issues.apache.org/jira/browse/FLINK-28326 Project: Fli

[jira] [Created] (FLINK-28327) Make table store src codes compiled with Flink 1.14

2022-06-30 Thread Jingsong Lee (Jira)
Jingsong Lee created FLINK-28327: Summary: Make table store src codes compiled with Flink 1.14 Key: FLINK-28327 URL: https://issues.apache.org/jira/browse/FLINK-28327 Project: Flink Issue Ty

[jira] [Created] (FLINK-28328) RescaleCheckpointManuallyITCase.testCheckpointRescalingInKeyedState failed with IllegalStateException

2022-06-30 Thread Huang Xingbo (Jira)
Huang Xingbo created FLINK-28328: Summary: RescaleCheckpointManuallyITCase.testCheckpointRescalingInKeyedState failed with IllegalStateException Key: FLINK-28328 URL: https://issues.apache.org/jira/browse/FLINK-2

[jira] [Created] (FLINK-28329) List top 15 biggest directories in terms of used disk space

2022-06-30 Thread Martijn Visser (Jira)
Martijn Visser created FLINK-28329: -- Summary: List top 15 biggest directories in terms of used disk space Key: FLINK-28329 URL: https://issues.apache.org/jira/browse/FLINK-28329 Project: Flink

[jira] [Created] (FLINK-28330) Remove old delegation token framework code when new is working fine

2022-06-30 Thread Gabor Somogyi (Jira)
Gabor Somogyi created FLINK-28330: - Summary: Remove old delegation token framework code when new is working fine Key: FLINK-28330 URL: https://issues.apache.org/jira/browse/FLINK-28330 Project: Flink

Re: [DISCUSS] Introduce multi delete API to Flink's FileSystem class

2022-06-30 Thread Yun Tang
Hi Piotr, As I said in the original email, you cannot delete folders recursively for incremental checkpoints. And If you take a close look at the original email, I have shared the experimental results, which proved 29x improvement: "A simple experiment shows that deleting 1000 objects with each

Re: [DISCUSS] Contribution of Multi Cluster Kafka Source

2022-06-30 Thread Martijn Visser
Hi Mason, I'm sure there's a PMC (*hint*) out there who can grant you access to create a FLIP. Looking forward to it, this sounds like an improvement that users are looking forward to. Best regards, Martijn Op di 28 jun. 2022 om 09:21 schreef Mason Chen : > Hi all, > > Thanks for the feedback!

RE: Re: [DISCUSS] FLIP-239: Port JDBC Connector Source to FLIP-27

2022-06-30 Thread Roc Marshal
Hi, Boto. Thanks for your reply. +1 to me on watermark strategy definition in ‘streaming’ & table source. I'm not sure if FLIP-202[1] is suitable for a separate discussion, but I think your proposal is very helpful to the new source. It would be great if the new source could be compa

Re: [DISCUSS] Contribution of Multi Cluster Kafka Source

2022-06-30 Thread Thomas Weise
Hi Mason, I added mason6345 to the Flink confluence space, you should be able to add a FLIP now. Looking forward to the contribution! Thomas On Thu, Jun 30, 2022 at 9:25 AM Martijn Visser wrote: > > Hi Mason, > > I'm sure there's a PMC (*hint*) out there who can grant you access to > create a

Re:Re: Re:Re: [DISCUSS] FLIP-218: Support SELECT clause in CREATE TABLE(CTAS)

2022-06-30 Thread Mang Zhang
Hi Martijn, Thank you for your reply, these are two good questions. >1. The FLIP mentions that if the user doesn't specify the WITH option part >in the query of the sink table, it will be assumed that the user wants to >create a managed table. What will happen if the user doesn't have Table >Store

[jira] [Created] (FLINK-28331) Persist status after every observe loop

2022-06-30 Thread Matyas Orhidi (Jira)
Matyas Orhidi created FLINK-28331: - Summary: Persist status after every observe loop Key: FLINK-28331 URL: https://issues.apache.org/jira/browse/FLINK-28331 Project: Flink Issue Type: Improve

Re: [DISCUSS] Introduce multi delete API to Flink's FileSystem class

2022-06-30 Thread Piotr Nowojski
Hi, Yes, I know that you can not use recursive deletes for incremental checkpoints and I didn't suggest it anywhere. I just pointed out that I would expect multi/bulk deletes to supersede the recursive deletes feature assuming good underlying implementation. Also I'm not surprised that multi delet

[jira] [Created] (FLINK-28332) GlueSchemaRegistryJsonKinesisITCase is being Ignored due to `Access key not configured`

2022-06-30 Thread Ahmed Hamdy (Jira)
Ahmed Hamdy created FLINK-28332: --- Summary: GlueSchemaRegistryJsonKinesisITCase is being Ignored due to `Access key not configured` Key: FLINK-28332 URL: https://issues.apache.org/jira/browse/FLINK-28332

[jira] [Created] (FLINK-28333) GlueSchemaRegistryAvroKinesisITCase is being Ignored due to `Access key not configured`

2022-06-30 Thread Ahmed Hamdy (Jira)
Ahmed Hamdy created FLINK-28333: --- Summary: GlueSchemaRegistryAvroKinesisITCase is being Ignored due to `Access key not configured` Key: FLINK-28333 URL: https://issues.apache.org/jira/browse/FLINK-28333

[jira] [Created] (FLINK-28334) PushProjectIntoTableSourceScanRule should cover the case when table source SupportsReadingMetadata and not SupportsProjectionPushDown

2022-06-30 Thread lincoln lee (Jira)
lincoln lee created FLINK-28334: --- Summary: PushProjectIntoTableSourceScanRule should cover the case when table source SupportsReadingMetadata and not SupportsProjectionPushDown Key: FLINK-28334 URL: https://issues.a

Re: [DISCUSS] Introduce multi delete API to Flink's FileSystem class

2022-06-30 Thread Roman Khachatryan
Hi, Thanks for the proposal Yun, I think that's a good idea and it could solve the issue you mentioned (FLINK-26590) in many cases (though not all, depending on deletion speed; but in practice it may be enough). Having a separate interface (BulkDeletingFileSystem) would probably help in increment

Re: [VOTE] Release 1.15.1, release candidate #1

2022-06-30 Thread Danny Cranmer
Hello all, -1 The Kinesis Data Streams consumer does not work with Stop With Savepoint [1]. We are planning to have a fix ready to merge tomorrow and would appreciate getting this in 1.15.1. [1] https://issues.apache.org/jira/browse/FLINK-23528 Thanks, Danny On Thu, Jun 30, 2022 at 9:31 AM Jin

Re: [DISCUSS] FLIP-238: Introduce FLIP-27-based Data Generator Source

2022-06-30 Thread Alexander Fedulov
Hi all, getting back to the idea of reusing FlinkConnectorRateLimiter: it is designed for the SourceFunction API and has an open() method that takes a RuntimeContext. Therefore, we need to add a different interface for the new Source API. This is where I see a certain limitation for the rate-limi

[jira] [Created] (FLINK-28335) Delete topic after tests

2022-06-30 Thread Jingsong Lee (Jira)
Jingsong Lee created FLINK-28335: Summary: Delete topic after tests Key: FLINK-28335 URL: https://issues.apache.org/jira/browse/FLINK-28335 Project: Flink Issue Type: Bug Components

Enquiry about contributing to Flink

2022-06-30 Thread Qing Lim
Hi Flink Dev, My company uses flink, and we have some code changes that we'd like to contribute to the main fork. Can I get your advice on what to do here? The features we wish to contribute are as follow: 1. JDBC connector filter pushdown, already tracked on JIRA: https://issues.apache.o

Re: Re: [DISCUSS] FLIP-239: Port JDBC Connector Source to FLIP-27

2022-06-30 Thread Lijie Wang
Hi Roc, Thanks for driving the discussion. Could you describe in detail what the JdbcSourceSplit represents? It looks like something wrong with the comments of JdbcSourceSplit in FLIP(it describe as "A {@link SourceSplit} that represents a file, or a region of a file"). Best, Lijie Roc Mar

[jira] [Created] (FLINK-28336) Support parquet-avro format in PyFlink DataStream

2022-06-30 Thread Juntao Hu (Jira)
Juntao Hu created FLINK-28336: - Summary: Support parquet-avro format in PyFlink DataStream Key: FLINK-28336 URL: https://issues.apache.org/jira/browse/FLINK-28336 Project: Flink Issue Type: New F

[VOTE] Apache Flink ML Release 2.1.0, release candidate #2

2022-06-30 Thread Zhipeng Zhang
Hi everyone, Please review and vote on the release candidate #2 for the version 2.1.0 of Apache Flink ML as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) **Testing Guideline** You can find here [1] a page in the project wiki on in

[jira] [Created] (FLINK-28337) java.lang.IllegalArgumentException: Table identifier not set

2022-06-30 Thread wei (Jira)
wei created FLINK-28337: --- Summary: java.lang.IllegalArgumentException: Table identifier not set Key: FLINK-28337 URL: https://issues.apache.org/jira/browse/FLINK-28337 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-28338) org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonParseException: Invalid numeric value: Leading zeroes not allowed

2022-06-30 Thread wangkang (Jira)
wangkang created FLINK-28338: Summary: org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonParseException: Invalid numeric value: Leading zeroes not allowed Key: FLINK-28338 URL: https://issues.apache.org

Re: [VOTE] Apache Flink ML Release 2.1.0, release candidate #2

2022-06-30 Thread Dong Lin
Thanks for the update! +1 (non-binding) Here is what I checked. All required checks are included. - Verified that the checksums and GPG files match the corresponding release files. - Verified that the source distributions do not contain any binaries. - Built the source distribution and ensured t

[jira] [Created] (FLINK-28339) Introduce SparkCatalog in table store

2022-06-30 Thread Jingsong Lee (Jira)
Jingsong Lee created FLINK-28339: Summary: Introduce SparkCatalog in table store Key: FLINK-28339 URL: https://issues.apache.org/jira/browse/FLINK-28339 Project: Flink Issue Type: New Feature

RE: Re: Re: [DISCUSS] FLIP-239: Port JDBC Connector Source to FLIP-27

2022-06-30 Thread Roc Marshal
Hi, Lijie. Thanks for the catching and sorry for the typo in sorting out the draft. I updated it. Best regards, Roc Marshal On 2022/07/01 02:24:56 Lijie Wang wrote: > Hi Roc, > > Thanks for driving the discussion. > > Could you describe in detail what the JdbcSourceSplit represents? It loo

[jira] [Created] (FLINK-28340) Support using system conda env for PyFlink tests

2022-06-30 Thread Juntao Hu (Jira)
Juntao Hu created FLINK-28340: - Summary: Support using system conda env for PyFlink tests Key: FLINK-28340 URL: https://issues.apache.org/jira/browse/FLINK-28340 Project: Flink Issue Type: Improv

Re: Enquiry about contributing to Flink

2022-06-30 Thread Martijn Visser
Hi Qing, Thanks a lot for wanting to contribute your improvements back to Flink. The code contribution process is documented and highly recommend to read through [1] For the 1st item, I can assign it to you so you can open a PR. The 2nd item also has a Jira [2]. For the 3rd item, I would recommen