Re: [DISCUSS] FLIP-144: Native Kubernetes HA for Flink

2020-09-30 Thread tison
Hi, Generally +1 for a native k8s HA service. For leader election & publish leader information, there was a discussion[1] pointed out that since these two actions is NOT atomic, there will be always edge case where a previous leader overwrite leader information, even with versioned write. Version

[jira] [Created] (FLINK-19476) Introduce CacheManager in CatalogManager to keep track of the ClusterPartitionDescriptor

2020-09-30 Thread Xuannan Su (Jira)
Xuannan Su created FLINK-19476: -- Summary: Introduce CacheManager in CatalogManager to keep track of the ClusterPartitionDescriptor Key: FLINK-19476 URL: https://issues.apache.org/jira/browse/FLINK-19476

[jira] [Created] (FLINK-19477) Translate page 'python_table_api_connectors' into Chinese

2020-09-30 Thread hailong wang (Jira)
hailong wang created FLINK-19477: Summary: Translate page 'python_table_api_connectors' into Chinese Key: FLINK-19477 URL: https://issues.apache.org/jira/browse/FLINK-19477 Project: Flink Iss

[jira] [Created] (FLINK-19478) Translate page 'intro_to_table_api' into Chinese

2020-09-30 Thread hailong wang (Jira)
hailong wang created FLINK-19478: Summary: Translate page 'intro_to_table_api' into Chinese Key: FLINK-19478 URL: https://issues.apache.org/jira/browse/FLINK-19478 Project: Flink Issue Type:

Re: [DISCUSS] FLIP-144: Native Kubernetes HA for Flink

2020-09-30 Thread Yang Wang
Thanks till and tison for your comments. @Till Rohrmann 1. I am afraid we could not do this if we are going to use fabric8 Kubernetes client SDK for the leader election. The official Kubernetes Java client[1] also could not support it. Unless we implement a new LeaderElector in Flink based on the

Re: [DISCUSS] FLIP-144: Native Kubernetes HA for Flink

2020-09-30 Thread tison
Thanks for your explanation. It would be fine if only checking leadership & actually write information is atomic. Best, tison. Yang Wang 于2020年9月30日周三 下午3:57写道: > Thanks till and tison for your comments. > > @Till Rohrmann > 1. I am afraid we could not do this if we are going to use fabric8 >

[jira] [Created] (FLINK-19479) Allow explicitly configuring time behaviour on KeyedStream.intervalJoin()

2020-09-30 Thread Aljoscha Krettek (Jira)
Aljoscha Krettek created FLINK-19479: Summary: Allow explicitly configuring time behaviour on KeyedStream.intervalJoin() Key: FLINK-19479 URL: https://issues.apache.org/jira/browse/FLINK-19479 Pro

[jira] [Created] (FLINK-19480) Support RichFunction in Python DataStream API

2020-09-30 Thread Dian Fu (Jira)
Dian Fu created FLINK-19480: --- Summary: Support RichFunction in Python DataStream API Key: FLINK-19480 URL: https://issues.apache.org/jira/browse/FLINK-19480 Project: Flink Issue Type: Improvement

[VOTE] Release flink-shaded 12.0

2020-09-30 Thread Robert Metzger
Hi everyone, Please review and vote on the release candidate #1 for the version 12.0, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) The complete staging area is available for your review, which includes: * JIRA release notes [1], *

Re: [DISCUSS] FLIP-144: Native Kubernetes HA for Flink

2020-09-30 Thread Till Rohrmann
Thanks for the clarifications Yang Wang. 2. Keeping the HA information relevant for a component (Dispatcher, JobManager, ResourceManager) in a single ConfigMap sounds good. We should check that we don't exceed the 1 MB size limit with this approach though. The Dispatcher's ConfigMap would then con

[jira] [Created] (FLINK-19481) Add support for a flink native GCS FileSystem

2020-09-30 Thread Ben Augarten (Jira)
Ben Augarten created FLINK-19481: Summary: Add support for a flink native GCS FileSystem Key: FLINK-19481 URL: https://issues.apache.org/jira/browse/FLINK-19481 Project: Flink Issue Type: Imp

[jira] [Created] (FLINK-19482) OrcRowInputFormat does not define serialVersionUID

2020-09-30 Thread hailong wang (Jira)
hailong wang created FLINK-19482: Summary: OrcRowInputFormat does not define serialVersionUID Key: FLINK-19482 URL: https://issues.apache.org/jira/browse/FLINK-19482 Project: Flink Issue Type

[DISCUSS] Introduce MongoDB connector for Flink

2020-09-30 Thread Dominik Wosiński
Hey all, I've seen several tasks with propositions for this both in Flink and Bahir[1]. I haven't seen any discussion neither any work was actually done to implement this (that I know of). So I would like to start a discussion about whether this is something You think would be beneficial to have a

[jira] [Created] (FLINK-19483) PyFlink Table end-to-end test failed with "FileExistsError: [Errno 17] File exists: '/home/vsts/work/1/s/flink-python/dev/.conda/pkgs/cache'"

2020-09-30 Thread Dian Fu (Jira)
Dian Fu created FLINK-19483: --- Summary: PyFlink Table end-to-end test failed with "FileExistsError: [Errno 17] File exists: '/home/vsts/work/1/s/flink-python/dev/.conda/pkgs/cache'" Key: FLINK-19483 URL: https://issues.

[jira] [Created] (FLINK-19484) Kubernetes pyflink application test failed with "error executing jsonpath "{range .items[*]}{.metadata.name}{\"\\n\"}{end}": Error executing template: not in rang"

2020-09-30 Thread Dian Fu (Jira)
Dian Fu created FLINK-19484: --- Summary: Kubernetes pyflink application test failed with "error executing jsonpath "{range .items[*]}{.metadata.name}{\"\\n\"}{end}": Error executing template: not in rang" Key: FLINK-19484

Re: [DISCUSS] FLIP-144: Native Kubernetes HA for Flink

2020-09-30 Thread Yang Wang
2. Yes. This is exactly what I mean. Storing the HA information relevant to a specific component in a single ConfigMap and ensuring that “Get(check the leader)-and-Update(write back to the ConfigMap)” is a transactional operation. Since we only store the job graph stateHandler(not the real data) in

Re: [DISCUSS] FLIP-145: Support SQL windowing table-valued function

2020-09-30 Thread Jark Wu
Hi Pengcheng, Yes, the window TVF is part of the FLIP. Welcome to contribute and join the discussion. Regarding the SESSION window aggregation, users can use the existing grouped session window function. Best, Jark On Sun, 27 Sep 2020 at 21:24, liupengcheng wrote: > Hi Jark, > Thanks f

Re: [DISCUSS] FLIP-145: Support SQL windowing table-valued function

2020-09-30 Thread Jark Wu
Hi everyone, I have added a section for Performance Optimization to describe how to improve the performance in the short-term and long-term and sketch the future performance potential under the new window API. Introducing the window API is just the first step, we will continuously improve the perf