Re: [DISCUSS] Make Kubernetes Operator config "dynamic" and consider merging with flinkConfiguration

2022-04-01 Thread Őrhidi Mátyás
Thanks Gyula for bringing this topic up! Although the suggestion would indeed simplify the configuration handling I have some concerns about opening the operator configuration for end users in certain cases. In a multitenant scenario for example, how could we protect against one user messing up the

[jira] [Created] (FLINK-27002) Optimize batch multiple partitions inserting

2022-04-01 Thread Jingsong Lee (Jira)
Jingsong Lee created FLINK-27002: Summary: Optimize batch multiple partitions inserting Key: FLINK-27002 URL: https://issues.apache.org/jira/browse/FLINK-27002 Project: Flink Issue Type: Sub-

回复: [GitHub] [flink] flinkbot edited a comment on pull request #15519: [FLINK-22120] Remove duplicate code in generated code for map get

2022-04-01 Thread 范 佳兴
退订 发件人: GitBox 发送时间: 2021年4月9日 16:28 收件人: iss...@flink.apache.org 主题: [GitHub] [flink] flinkbot edited a comment on pull request #15519: [FLINK-22120] Remove duplicate code in generated code for map get flinkbot edited a comment on pull request #15519: URL: htt

[jira] [Created] (FLINK-27001) Support to specify the resource of the operator

2022-04-01 Thread Aitozi (Jira)
Aitozi created FLINK-27001: -- Summary: Support to specify the resource of the operator Key: FLINK-27001 URL: https://issues.apache.org/jira/browse/FLINK-27001 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-27000) Support to set JVM args for operator

2022-04-01 Thread Aitozi (Jira)
Aitozi created FLINK-27000: -- Summary: Support to set JVM args for operator Key: FLINK-27000 URL: https://issues.apache.org/jira/browse/FLINK-27000 Project: Flink Issue Type: Improvement Co

Re: [VOTE] Apache Flink Kubernetes Operator Release 0.1.0, release candidate #3

2022-04-01 Thread Aitozi
+1 (non-binding) - verified building and installing operator - verified session cluster and application job submit and upgrade Cheers, Aitozi. Xintong Song 于2022年4月1日周五 19:12写道: > Thanks Gyula, the doc pr LGTM. > > +1 (binding) > > - verified checksum & signature > - installed via helm and tri

[jira] [Created] (FLINK-26999) Introduce ClickHouse Connector

2022-04-01 Thread ZhuoYu Chen (Jira)
ZhuoYu Chen created FLINK-26999: --- Summary: Introduce ClickHouse Connector Key: FLINK-26999 URL: https://issues.apache.org/jira/browse/FLINK-26999 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-26998) Remove log-related configuration from predefined options in RocksDB state backend

2022-04-01 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-26998: --- Summary: Remove log-related configuration from predefined options in RocksDB state backend Key: FLINK-26998 URL: https://issues.apache.org/jira/browse/FLINK-26998 Proj

Re: [DISCUSS] Make Kubernetes Operator config "dynamic" and consider merging with flinkConfiguration

2022-04-01 Thread Yang Wang
I also like the proposal 2. Maybe it could be named with *KubernetesOperatorConfigOptions*, which just looks like all other ConfigOption(e.g. *KubernetesConfigOptions, YarnConfigOptions*) in Flink. The proposal 2 is more natural and easy to use for Flink users. Best, Yang Gyula Fóra 于2022年4月2日周

Re: Wrong format when passing arguments with space

2022-04-01 Thread Kevin Lee
It's a typo I run this demo on yarn cluster The mode is *"run-application"* instead of "run" ./bin/flink run-application -c com.lmk.QuotaTest --rate 10 --time "2022-03-28 11:53:21" The JM log shows program-args as follow: $internal.application.program-args, --rate;10;--time;'"2022-03-30 1

[jira] [Created] (FLINK-26997) Add table store links to Community page

2022-04-01 Thread Jingsong Lee (Jira)
Jingsong Lee created FLINK-26997: Summary: Add table store links to Community page Key: FLINK-26997 URL: https://issues.apache.org/jira/browse/FLINK-26997 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-26996) Break the reconcile after first create session cluster

2022-04-01 Thread Aitozi (Jira)
Aitozi created FLINK-26996: -- Summary: Break the reconcile after first create session cluster Key: FLINK-26996 URL: https://issues.apache.org/jira/browse/FLINK-26996 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-26995) Enable fork-reuse for all unit tests

2022-04-01 Thread Chesnay Schepler (Jira)
Chesnay Schepler created FLINK-26995: Summary: Enable fork-reuse for all unit tests Key: FLINK-26995 URL: https://issues.apache.org/jira/browse/FLINK-26995 Project: Flink Issue Type: Tech

[jira] [Created] (FLINK-26994) Merge libraries CI profile into core

2022-04-01 Thread Chesnay Schepler (Jira)
Chesnay Schepler created FLINK-26994: Summary: Merge libraries CI profile into core Key: FLINK-26994 URL: https://issues.apache.org/jira/browse/FLINK-26994 Project: Flink Issue Type: Tech

[jira] [Created] (FLINK-26993) CheckpointCoordinatorTest#testMinCheckpointPause

2022-04-01 Thread Chesnay Schepler (Jira)
Chesnay Schepler created FLINK-26993: Summary: CheckpointCoordinatorTest#testMinCheckpointPause Key: FLINK-26993 URL: https://issues.apache.org/jira/browse/FLINK-26993 Project: Flink Issu

[DISCUSS] Make Kubernetes Operator config "dynamic" and consider merging with flinkConfiguration

2022-04-01 Thread Gyula Fóra
Hi Devs! *Background*: With more and more features and options added to the flink kubernetes operator it would make sense to not expose everything as first class options in the deployment/jobspec (same as we do for flink configuration currently). Furthermore it would be beneficial if users could

[jira] [Created] (FLINK-26992) PojoSerializer may cause concurrent exception passing directly between threads

2022-04-01 Thread Yuan Mei (Jira)
Yuan Mei created FLINK-26992: Summary: PojoSerializer may cause concurrent exception passing directly between threads Key: FLINK-26992 URL: https://issues.apache.org/jira/browse/FLINK-26992 Project: Flink

[jira] [Created] (FLINK-26991) Limit operator docs build to main repository

2022-04-01 Thread Jira
Márton Balassi created FLINK-26991: -- Summary: Limit operator docs build to main repository Key: FLINK-26991 URL: https://issues.apache.org/jira/browse/FLINK-26991 Project: Flink Issue Type:

[jira] [Created] (FLINK-26990) BatchArrowPythonGroupWindowAggregateFunctionOperatorTest.testFinishBundleTriggeredByTime failed

2022-04-01 Thread Matthias Pohl (Jira)
Matthias Pohl created FLINK-26990: - Summary: BatchArrowPythonGroupWindowAggregateFunctionOperatorTest.testFinishBundleTriggeredByTime failed Key: FLINK-26990 URL: https://issues.apache.org/jira/browse/FLINK-26990

[jira] [Created] (FLINK-26989) Fix the potential lost of secondary resource when watching a namespace

2022-04-01 Thread Aitozi (Jira)
Aitozi created FLINK-26989: -- Summary: Fix the potential lost of secondary resource when watching a namespace Key: FLINK-26989 URL: https://issues.apache.org/jira/browse/FLINK-26989 Project: Flink I

[jira] [Created] (FLINK-26988) Better error reporting when format fails

2022-04-01 Thread Francesco Guardiani (Jira)
Francesco Guardiani created FLINK-26988: --- Summary: Better error reporting when format fails Key: FLINK-26988 URL: https://issues.apache.org/jira/browse/FLINK-26988 Project: Flink Issue

[jira] [Created] (FLINK-26987) ZooKeeperStateHandleStore.getAllAndLock ends up in a infinite loop if there's an entry marked for deletion that's not cleaned up, yet

2022-04-01 Thread Matthias Pohl (Jira)
Matthias Pohl created FLINK-26987: - Summary: ZooKeeperStateHandleStore.getAllAndLock ends up in a infinite loop if there's an entry marked for deletion that's not cleaned up, yet Key: FLINK-26987 URL: https://issu

[jira] [Created] (FLINK-26986) Remove deprecated string expressions in Python Table API

2022-04-01 Thread Dian Fu (Jira)
Dian Fu created FLINK-26986: --- Summary: Remove deprecated string expressions in Python Table API Key: FLINK-26986 URL: https://issues.apache.org/jira/browse/FLINK-26986 Project: Flink Issue Type: Bu

Re: [VOTE] Apache Flink Kubernetes Operator Release 0.1.0, release candidate #3

2022-04-01 Thread Xintong Song
Thanks Gyula, the doc pr LGTM. +1 (binding) - verified checksum & signature - installed via helm and tried example CRs - viewed online documentations Thank you~ Xintong Song On Fri, Apr 1, 2022 at 4:56 PM Gyula Fóra wrote: > Xintong, > > You are right, this is indeed missing from the docs

Re: [DISCUSS] Flink SQL Syntax for Query/Savepoint Management

2022-04-01 Thread Paul Lam
Hi Martjin, > > For any extension on the SQL syntax, there should be a FLIP. I would like > to understand how this works for both bounded and unbounded jobs, how this > works with the SQL upgrade story. Could you create one? Sure. I’m preparing one. Please give me the permission if possible. My

Re: [DISCUSS] assign SQL Table properties from environment variables

2022-04-01 Thread Qingsheng Ren
Hi Fred, Thanks for raising the discussion! I think the definition of “environment variable” varies under different context. Under Flink on K8s it means the environment variable for a container, and if you are a SQL client user it could refer to environment variable of SQL client, or even the

[jira] [Created] (FLINK-26985) With legacy restore mode, incremantal checkpoints would be deleted by mistake

2022-04-01 Thread Yun Tang (Jira)
Yun Tang created FLINK-26985: Summary: With legacy restore mode, incremantal checkpoints would be deleted by mistake Key: FLINK-26985 URL: https://issues.apache.org/jira/browse/FLINK-26985 Project: Flink

Re: [DISCUSS] Flink SQL Syntax for Query/Savepoint Management

2022-04-01 Thread Martijn Visser
Hi Paul, For any extension on the SQL syntax, there should be a FLIP. I would like to understand how this works for both bounded and unbounded jobs, how this works with the SQL upgrade story. Could you create one? I'm also copying in @Timo Walther and @Jark Wu for their opinion on this. Best r

Re: Question about community collaboration options

2022-04-01 Thread Martijn Visser
Hi Ted, This is a great question. There are usually bi-weekly sync meetings to discuss plans and progress for the next Flink release. For example, there was a regular one for the Flink 1.15 release [1] I do see some things that we could improve on as a Flink community. For example, there are quit

Re: [DISCUSS] Flink SQL Syntax for Query/Savepoint Management

2022-04-01 Thread Paul Lam
Hi Martijn, Thanks a lot for your input. > Have you already thought on how you would implement this in Flink? Yes, I roughly thought about the implementation: 1. Extending Executor to support job list via ClusterClient. 2. Extending Executor to support savepoint trigger/cancel/remove via JobCli

Re: [DISCUSS] assign SQL Table properties from environment variables

2022-04-01 Thread Martijn Visser
Hi Fred, Thanks for opening the discussion. There are probably multiple ways to solve it, you could also think about addressing this problem in a catalogue and hide any password/keywords in there. But then again, I do think your solution could also work and is straightforward. If there are no peo

Re: [DISCUSS] Flink SQL Syntax for Query/Savepoint Management

2022-04-01 Thread Martijn Visser
Hi Paul, Thanks for opening the discussion. I agree that there are opportunities in this area to increase user value. I would say that the syntax should be part of a proposal in a FLIP, because the implementation would actually be the complex part, not so much the syntax :) Especially since this

[DISCUSS] Flink SQL Syntax for Query/Savepoint Management

2022-04-01 Thread Paul Lam
Hi team, Greetings from Apache Kyuubi(incubating) community. We’re integrating Flink as a SQL engine and aiming to make it production-ready. However, query/savepoint management is a crucial but missing part in Flink SQL, thus we reach out to discuss the SQL syntax with Flink community. We pr

Re: [VOTE] Apache Flink Kubernetes Operator Release 0.1.0, release candidate #3

2022-04-01 Thread Gyula Fóra
Xintong, You are right, this is indeed missing from the docs :) I will be pushing some quickstart improvements once the release is finalized and will include it in there. Please check this doc change and let me know if it clarifies it to a good degree: https://github.com/apache/flink-kubernetes-op

[jira] [Created] (FLINK-26984) HistoryServer archiving is not retried

2022-04-01 Thread Matthias Pohl (Jira)
Matthias Pohl created FLINK-26984: - Summary: HistoryServer archiving is not retried Key: FLINK-26984 URL: https://issues.apache.org/jira/browse/FLINK-26984 Project: Flink Issue Type: Improvem

[jira] [Created] (FLINK-26983) When the startup mode in the command line submission task is filled in incorrectly, the code judgment is wrong, and the error information will be misleading

2022-04-01 Thread Jira
丛鹏 created FLINK-26983: -- Summary: When the startup mode in the command line submission task is filled in incorrectly, the code judgment is wrong, and the error information will be misleading Key: FLINK-26983 URL: https://is

Re: [VOTE] Apache Flink Kubernetes Operator Release 0.1.0, release candidate #3

2022-04-01 Thread Xintong Song
Thanks for the quick response, Gyula. I'm fine with having Helm chart pointing to ghcr.io. It should be clear from the readme how to use the dockerhub image instead. > Could you point me where to find these instructions? I don't find them in neither the Quick Start nor the Helm page in the docum

Re: [VOTE] Apache Flink Kubernetes Operator Release 0.1.0, release candidate #3

2022-04-01 Thread Gyula Fóra
Hi Xintong, I have been contemplating the same for a while. The final docker image that will be uploaded to dockerhub should be the same image as the one on ghcr.io (re-tagged and reuploaded). We made the decision to try to keep the integrity of the RC and the final release and not make any change

[jira] [Created] (FLINK-26982) strike a balance between reuse the same RelNode and project/filter/limit/partition push down

2022-04-01 Thread zoucao (Jira)
zoucao created FLINK-26982: -- Summary: strike a balance between reuse the same RelNode and project/filter/limit/partition push down Key: FLINK-26982 URL: https://issues.apache.org/jira/browse/FLINK-26982 Pro

Re: [VOTE] Apache Flink Kubernetes Operator Release 0.1.0, release candidate #3

2022-04-01 Thread Xintong Song
I have a concern, which I'm not sure whether it should be a release blocker. While trying to install the operator via helm, I noticed that the image is pulled from `ghcr.io/apache/flink-kubernetes-operator`. IIUC, in one of the previous voting threads it is mentioned that we are not using `ghcr.io

[jira] [Created] (FLINK-26981) [Umbrella] Make Flink work on Apple M1 ARM chipset

2022-04-01 Thread Martijn Visser (Jira)
Martijn Visser created FLINK-26981: -- Summary: [Umbrella] Make Flink work on Apple M1 ARM chipset Key: FLINK-26981 URL: https://issues.apache.org/jira/browse/FLINK-26981 Project: Flink Issue

[jira] [Created] (FLINK-26980) PulsarUnorderedSourceReaderTest hang on azure

2022-04-01 Thread Yun Gao (Jira)
Yun Gao created FLINK-26980: --- Summary: PulsarUnorderedSourceReaderTest hang on azure Key: FLINK-26980 URL: https://issues.apache.org/jira/browse/FLINK-26980 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-26979) JsonFunctionsITCase.test failed on azure

2022-04-01 Thread Yun Gao (Jira)
Yun Gao created FLINK-26979: --- Summary: JsonFunctionsITCase.test failed on azure Key: FLINK-26979 URL: https://issues.apache.org/jira/browse/FLINK-26979 Project: Flink Issue Type: Bug Comp

[jira] [Created] (FLINK-26978) Flink-SQL cannot display Timestamp time correctly.

2022-04-01 Thread Siqi Ouyang (Jira)
Siqi Ouyang created FLINK-26978: --- Summary: Flink-SQL cannot display Timestamp time correctly. Key: FLINK-26978 URL: https://issues.apache.org/jira/browse/FLINK-26978 Project: Flink Issue Type:

[jira] [Created] (FLINK-26977) JobMasterStopWithSavepointITCase.throwingExceptionOnCallbackWithRestartsShouldSimplyRestartInTerminate failed on azure

2022-04-01 Thread Yun Gao (Jira)
Yun Gao created FLINK-26977: --- Summary: JobMasterStopWithSavepointITCase.throwingExceptionOnCallbackWithRestartsShouldSimplyRestartInTerminate failed on azure Key: FLINK-26977 URL: https://issues.apache.org/jira/browse/

[jira] [Created] (FLINK-26976) HistoryServer archiving is not idempotent

2022-04-01 Thread Matthias Pohl (Jira)
Matthias Pohl created FLINK-26976: - Summary: HistoryServer archiving is not idempotent Key: FLINK-26976 URL: https://issues.apache.org/jira/browse/FLINK-26976 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-26975) TableTestBase will not generate exec node tree after generating rel node tree

2022-04-01 Thread xuyang (Jira)
xuyang created FLINK-26975: -- Summary: TableTestBase will not generate exec node tree after generating rel node tree Key: FLINK-26975 URL: https://issues.apache.org/jira/browse/FLINK-26975 Project: Flink