[DISCUSS] Roadmap update for 1.15

Dear Flink Community, I created a PR to update the roadmap. Please have a look at the new feature radar as well as on the overview of the most important efforts that are coming up. https://github.com/apache/flink-web/pull/527 Best, Joe

[jira] [Created] (FLINK-27216) CheckpointCoordinatorTest.testMinCheckpointPause failed on azure

Yun Gao created FLINK-27216: --- Summary: CheckpointCoordinatorTest.testMinCheckpointPause failed on azure Key: FLINK-27216 URL: https://issues.apache.org/jira/browse/FLINK-27216 Project: Flink Issue

[jira] [Created] (FLINK-27217) Support partition filter push down when there exists default_parition

luoyuxia created FLINK-27217: Summary: Support partition filter push down when there exists default_parition Key: FLINK-27217 URL: https://issues.apache.org/jira/browse/FLINK-27217 Project: Flink

[jira] [Created] (FLINK-27218) Serializer in OperatorState has not been updated when new Serializers are NOT incompatible

Yue Ma created FLINK-27218: -- Summary: Serializer in OperatorState has not been updated when new Serializers are NOT incompatible Key: FLINK-27218 URL: https://issues.apache.org/jira/browse/FLINK-27218 Projec

Re: [DISCUSS] Roadmap update for 1.15

Hi Joe, Thanks for driving. Looks very good~ As a part of Apache Flink，Flink Table Store [1][2] is a new feature on Flink Table. It is now in Beta status. Now Flink Table Store is waiting for the release of Flink 1.15 as it depends on Flink 1.15, so it is ready to release the first version after

Re: [DISCUSS] Roadmap update for 1.15

Hi Jingsong, I already included it in the roadmap [1], will also mention it in the release notes. Best, Joe [1] https://github.com/apache/flink-web/pull/527/files#diff-51abe96bc493857fdfba599c6b70e3a6c09200cf29e130d15b599fef538e32dfR87-R89 > On 13.04.2022, at 09:19, Jingsong Li wrote: > >

Re: [ANNOUNCE] New scalafmt formatter has been merged

Thank you for this Francesco! It will really improve the lives of everyone touching scala code! Best, Marios On Wed, Apr 13, 2022 at 9:55 AM Timo Walther wrote: > Thanks for the great work Francesco! > > This will improve the contributor productivity a lot and ease reviews. > This change was l

Re: [DISCUSS] Roadmap update for 1.15

Thanks Joe! Best, Jingsong On Wed, Apr 13, 2022 at 3:22 PM Johannes Moser wrote: > > Hi Jingsong, > > I already included it in the roadmap [1], will also mention it in the release > notes. > > Best, > Joe > > > [1] > https://github.com/apache/flink-web/pull/527/files#diff-51abe96bc493857fdfba5

Re: [DISCUSS] Roadmap update for 1.15

Hi Joe, Thanks for the update! I've left a couple of comments, which I'll repeat here: - Kafka, File [via Unified Sink API] should go to Production Ready & Evolving. We should also add Pulsar there. - We're missing Kinesis and Firehose as a connector, which should probably be in Beta. - NiFi Sour

Re: Update regarding externalizing existing connectors and new connectors

Thanks a lot for the update Martijn and all your efforts, and those of everyone involved in this! Best, Marios On Tue, Apr 12, 2022 at 2:41 PM Martijn Visser wrote: > Hi everyone, > > After our discussion thread on "Creating an external connector repository" > [1], "Moving connectors from Flin

Re: FLINK-11746 work

Hi Chen, Thanks for restarting the discussion. I would +1 on Yu, let's agree on the scope and the design. Could you elaborate a bit on how you're using Thrift? When reading more about this, it feels like Thrift is somewhat in the middle between a connector and a format from a Flink perspective. It

Re: [DISCUSS] The abstraction of cache lookupFunction and cache metric

Hi Yuan, Sorry for pending such long time on this thread. I think adding unified abstraction and metrics for cache is quite important for users and developers to optimize and improve their jobs with lookup join. We also have our inner cache abstraction and implementation, so I took a deeper obs

OLM Bundle planned?

Hello Is integrating the helm-based flink-kubernetes-operator into Operator Community Marketplace and the operator lifecycle manager planned in the future? https://sdk.operatorframework.io/docs/olm-integration/tutorial-bundle/ Sven Thoms Senior IT Architect Data Science Banking T +41 81 28

[jira] [Created] (FLINK-27219) CliClientITCase.testSqlStatements failed on azure with jdk11

Yun Gao created FLINK-27219: --- Summary: CliClientITCase.testSqlStatements failed on azure with jdk11 Key: FLINK-27219 URL: https://issues.apache.org/jira/browse/FLINK-27219 Project: Flink Issue Typ

[jira] [Created] (FLINK-27220) Remove redundant null-check for int parameter.

da created FLINK-27220: -- Summary: Remove redundant null-check for int parameter. Key: FLINK-27220 URL: https://issues.apache.org/jira/browse/FLINK-27220 Project: Flink Issue Type: Improvement

Re: Discuss making KafkaSubscriber Public

Thanks for the proposal Mason! I think exposing `KafkaSubscriber` as public API is helpful for users to implement more complex subscription logics. +1 (non-binding) Cheers, Qingsheng > On Apr 12, 2022, at 11:46, Mason Chen wrote: > > Hi Flink Devs, > > I was looking to contribute to > ht

[DISCUSS] FLIP-217 Support watermark alignment of source splits

Dear Flink developers, I would like to open a discussion on FLIP 217 [1] for an extension of Watermark Alignment to perform alignment also in SplitReaders. To do so, SplitReaders must be able to suspend and resume reading from split sources where the SourceOperator coordinates and controlls suspen

Re: [ANNOUNCE] New scalafmt formatter has been merged

Thanks a lot for this great work Francesco! Regards, Dian On Wed, Apr 13, 2022 at 3:23 PM Marios Trivyzas wrote: > Thank you for this Francesco! > > It will really improve the lives of everyone touching scala code! > > Best, > Marios > > On Wed, Apr 13, 2022 at 9:55 AM Timo Walther wrote: > >

Re: [DISCUSS] FLIP-220: Temporal State

David, thanks for the feedback, much appreciated! I'm hoping you can explain a bit more about how Iterable> readRange(long minTimestamp, long limitTimestamp); would be used (and perhaps, implemented) in practice. I worry that this might either prevent certain optimizations and/or be rather more

[jira] [Created] (FLINK-27221) Improvements to YAML parsing

yao.zhou created FLINK-27221: Summary: Improvements to YAML parsing Key: FLINK-27221 URL: https://issues.apache.org/jira/browse/FLINK-27221 Project: Flink Issue Type: Improvement Compon

Re: [DISCUSS] FLIP-220: Temporal State

Aitozi, Our POC can be seen at [1]. My personal opinion is the namespace is an implementation detail that is better not exposed directly. Flink state is already quite complex to understand, and I fear that if we expose the namespaces the additional flexibility this offers will be more confusing t

Re: [DISCUSS] FLIP-220: Temporal State

Yun Tang and Jingsong, Some flavor of OrderedMapState is certainly feasible, and I do see some appeal in supporting Binary**State. However, I haven't seen a motivating use case for this generalization, and would rather keep this as simple as possible. By handling Longs we can already optimize a w

Re: [ANNOUNCE] New scalafmt formatter has been merged

Hi, Francesco Thanks for your work! Best, Guowei On Wed, Apr 13, 2022 at 5:35 PM Dian Fu wrote: > Thanks a lot for this great work Francesco! > > Regards, > Dian > > On Wed, Apr 13, 2022 at 3:23 PM Marios Trivyzas wrote: > > > Thank you for this Francesco! > > > > It will really improve the li

[jira] [Created] (FLINK-27222) Execution history limit can lead to eviction of critical local-recovery information

Chesnay Schepler created FLINK-27222: Summary: Execution history limit can lead to eviction of critical local-recovery information Key: FLINK-27222 URL: https://issues.apache.org/jira/browse/FLINK-27222

[jira] [Created] (FLINK-27223) State access doesn't work as expected when cache size is set to 0

Dian Fu created FLINK-27223: --- Summary: State access doesn't work as expected when cache size is set to 0 Key: FLINK-27223 URL: https://issues.apache.org/jira/browse/FLINK-27223 Project: Flink Issu

Re: [ANNOUNCE] New scalafmt formatter has been merged

Thanks for your work, Francesco Best, Jingsong On Wed, Apr 13, 2022 at 6:37 PM Guowei Ma wrote: > > Hi, Francesco > Thanks for your work! > Best, > Guowei > > > On Wed, Apr 13, 2022 at 5:35 PM Dian Fu wrote: > > > Thanks a lot for this great work Francesco! > > > > Regards, > > Dian > > > > On

Re: [DISCUSS] FLIP-220: Temporal State

Hi David, It seems to me that at least with the heap-based state backend, readRange > is going to have to do a lot of unnecessary work to implement this > isEmpty() operation, since it have will to consider the entire range from > MIN_VALUE to MAX_VALUE. (Maybe we should add an explicit isEmpty me

[jira] [Created] (FLINK-27224) Drop redundant flink.forkCountTestPackage property

Chesnay Schepler created FLINK-27224: Summary: Drop redundant flink.forkCountTestPackage property Key: FLINK-27224 URL: https://issues.apache.org/jira/browse/FLINK-27224 Project: Flink Is

Re: [DISCUSS] FLIP-220: Temporal State

Here is a very naive implementation [1] from a prototype I did few months back that uses list and insertion sort. Since the list is sorted we can use binary search to create sub-list, that could leverage the same thing I've described above. I think back then I didn't go for the SortedMap as it wou

[jira] [Created] (FLINK-27225) Remove redundant reuseForks settings

Chesnay Schepler created FLINK-27225: Summary: Remove redundant reuseForks settings Key: FLINK-27225 URL: https://issues.apache.org/jira/browse/FLINK-27225 Project: Flink Issue Type: Tech

[jira] [Created] (FLINK-27226) How to implement two partition methods for an operator at the same time?

Underwood created FLINK-27226: - Summary: How to implement two partition methods for an operator at the same time? Key: FLINK-27226 URL: https://issues.apache.org/jira/browse/FLINK-27226 Project: Flink

[jira] [Created] (FLINK-27227) ScalarFunctionsTest#testExtract fails with Extract operator does not support unit

Sergey Nuyanzin created FLINK-27227: --- Summary: ScalarFunctionsTest#testExtract fails with Extract operator does not support unit Key: FLINK-27227 URL: https://issues.apache.org/jira/browse/FLINK-27227

[jira] [Created] (FLINK-27228) Redistributed modules across CI profiles

Chesnay Schepler created FLINK-27228: Summary: Redistributed modules across CI profiles Key: FLINK-27228 URL: https://issues.apache.org/jira/browse/FLINK-27228 Project: Flink Issue Type:

Re: Discuss making KafkaSubscriber Public

Could you expand a bit on possible alternative implementations that require this interface to become public, opposed to providing more built-in ways to subscribe? On 13/04/2022 11:26, Qingsheng Ren wrote: Thanks for the proposal Mason! I think exposing `KafkaSubscriber` as public API is helpf

[jira] [Created] (FLINK-27229) Cassandra overrides netty version in tests

Dawid Wysakowicz created FLINK-27229: Summary: Cassandra overrides netty version in tests Key: FLINK-27229 URL: https://issues.apache.org/jira/browse/FLINK-27229 Project: Flink Issue Type

回复：Re: [DISCUSS] FLIP-216 Decouple Hive connector with Flink planner

Hi all, Sorry for the late reply for this thread. About decoupling Hive Connector, it is actually mainly for decoupling Hive dialect. So, I think it's a good timing to introduce pluagble dialect mechanism for Flink and make Hive dialect as the first. Based on this point, I have updated the FLIP-2

[jira] [Created] (FLINK-27230) Unnecessary entries in connector-kinesis NOTICE file

Dawid Wysakowicz created FLINK-27230: Summary: Unnecessary entries in connector-kinesis NOTICE file Key: FLINK-27230 URL: https://issues.apache.org/jira/browse/FLINK-27230 Project: Flink

[jira] [Created] (FLINK-27231) SQL pulsar connector lists dependencies under wrong license

Dawid Wysakowicz created FLINK-27231: Summary: SQL pulsar connector lists dependencies under wrong license Key: FLINK-27231 URL: https://issues.apache.org/jira/browse/FLINK-27231 Project: Flink

[jira] [Created] (FLINK-27232) .scalafmt.conf can

Chesnay Schepler created FLINK-27232: Summary: .scalafmt.conf can Key: FLINK-27232 URL: https://issues.apache.org/jira/browse/FLINK-27232 Project: Flink Issue Type: Technical Debt

[jira] [Created] (FLINK-27233) Unnecessary entries in connector-elasticsearch7 in NOTICE file

Dawid Wysakowicz created FLINK-27233: Summary: Unnecessary entries in connector-elasticsearch7 in NOTICE file Key: FLINK-27233 URL: https://issues.apache.org/jira/browse/FLINK-27233 Project: Flink

Re: [VOTE] Release 1.15.0, release candidate #2

-1, I checked new modules and modules with updated dependencies for proper listing LICENSE files. I found a number of issues, of which one justifies cancelling the rc, imo. Blocker: * https://issues.apache.org/jira/browse/FLINK-27231 Minor: * https://issues.apache.org/jira/browse/FLINK-2

Re: [VOTE] Release 1.15.0, release candidate #2

Re why CI didn't pick up on it: Currently, excessive NOTICE entries don't fail CI, only missing ones do. They should be logged though! The reason being that only missing entries are problems (for us, in a legal sense). That would be easy to change though. On 13/04/2022 17:40, Dawid Wysakowicz

Re: Discuss making KafkaSubscriber Public

Hi Chesnay, Typically, users want to plug in a KafkaSubscriber that depends on an external system [1][2]. We could also provide a higher level interface that doesn’t depend on the Kafka Admin Client, but I think it would be more flexible to be able to re-use the one created by the enumerator if ne

[jira] [Created] (FLINK-27234) Enable fork-reuse for connector-jdbc

Chesnay Schepler created FLINK-27234: Summary: Enable fork-reuse for connector-jdbc Key: FLINK-27234 URL: https://issues.apache.org/jira/browse/FLINK-27234 Project: Flink Issue Type: Tech

[jira] [Created] (FLINK-27235) Publish Flink k8s Operator Helm Charts via Github Actions

Gezim Sejdiu created FLINK-27235: Summary: Publish Flink k8s Operator Helm Charts via Github Actions Key: FLINK-27235 URL: https://issues.apache.org/jira/browse/FLINK-27235 Project: Flink Iss

[jira] [Created] (FLINK-27236) No task slot allocated for job in larege-scale job

yanpengshi created FLINK-27236: -- Summary: No task slot allocated for job in larege-scale job Key: FLINK-27236 URL: https://issues.apache.org/jira/browse/FLINK-27236 Project: Flink Issue Type: Bu

[jira] [Created] (FLINK-27237) Partition table statement enhancement

dalongliu created FLINK-27237: - Summary: Partition table statement enhancement Key: FLINK-27237 URL: https://issues.apache.org/jira/browse/FLINK-27237 Project: Flink Issue Type: New Feature

[jira] [Created] (FLINK-27238) The HiveGenericUDTF should support primitive array，for example Array Array ...

hehuiyuan created FLINK-27238: - Summary: The HiveGenericUDTF should support primitive array，for example Array Array ... Key: FLINK-27238 URL: https://issues.apache.org/jira/browse/FLINK-27238 Project: Fli

[jira] [Created] (FLINK-27239) rewrite PreValidateReWriter from scala to java

xuyang created FLINK-27239: -- Summary: rewrite PreValidateReWriter from scala to java Key: FLINK-27239 URL: https://issues.apache.org/jira/browse/FLINK-27239 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-27240) Support ADD PARTITION statement

dalongliu created FLINK-27240: - Summary: Support ADD PARTITION statement Key: FLINK-27240 URL: https://issues.apache.org/jira/browse/FLINK-27240 Project: Flink Issue Type: Sub-task Re

[jira] [Created] (FLINK-27241) Support DROP PARTITION statement

dalongliu created FLINK-27241: - Summary: Support DROP PARTITION statement Key: FLINK-27241 URL: https://issues.apache.org/jira/browse/FLINK-27241 Project: Flink Issue Type: Sub-task R

[jira] [Created] (FLINK-27243) Support SHOW PARTITIONS statement for partitioned table

dalongliu created FLINK-27243: - Summary: Support SHOW PARTITIONS statement for partitioned table Key: FLINK-27243 URL: https://issues.apache.org/jira/browse/FLINK-27243 Project: Flink Issue Type:

[jira] [Created] (FLINK-27242) Support RENAME PARTITION statement for partitioned table

dalongliu created FLINK-27242: - Summary: Support RENAME PARTITION statement for partitioned table Key: FLINK-27242 URL: https://issues.apache.org/jira/browse/FLINK-27242 Project: Flink Issue Type

Re: Re: [VOTE] Release 1.15.0, release candidate #2

Very thanks Dawid for the check! I think the issue of wrong licences should indeed be a blocker. I'll also have a look at these issues. Due to the issues, the RC2 would be officially canceled and I'll create the RC3 after these issues are fixed. --Original Mail --

[jira] [Created] (FLINK-27244) Support subdirectories with Hive tables

luoyuxia created FLINK-27244: Summary: Support subdirectories with Hive tables Key: FLINK-27244 URL: https://issues.apache.org/jira/browse/FLINK-27244 Project: Flink Issue Type: Sub-task

Re: Re: Re: Re: Re: 回复：Re:[DISCUSS] FLIP-214 Support Advanced Function DDL

Hi, everyone First of all, thanks for the valuable suggestions received about this FLIP. After some discussion, it looks like all concerns have been addressed for now, so I will start a vote about this FLIP in two or three days later. Also, further feedback is very welcome. Best, Ron >