Re: [ANNOUNCE] New PMC member: Zhu Zhu

2020-10-09 Thread Zhijiang
Congratulations and welcome, Zhu Zhu! Best, Zhijiang -- From:Yun Tang Send Time:2020年10月9日(星期五) 14:20 To:dev@flink.apache.org Subject:Re: [ANNOUNCE] New PMC member: Zhu Zhu Congratulations, Zhu! Best Yun Tang _

[jira] [Created] (FLINK-19552) JobManager dies with IllegalStateException SharedSlot (physical request SlotRequestId{%}) has been released

2020-10-09 Thread Robert Metzger (Jira)
Robert Metzger created FLINK-19552: -- Summary: JobManager dies with IllegalStateException SharedSlot (physical request SlotRequestId{%}) has been released Key: FLINK-19552 URL: https://issues.apache.org/jira/brows

[jira] [Created] (FLINK-19553) The format of checkpoint Completion Time and Failure Time is changed from HH:mm:ss to yyyy-MM-dd HH:mm:ss

2020-10-09 Thread liufangliang (Jira)
liufangliang created FLINK-19553: Summary: The format of checkpoint Completion Time and Failure Time is changed from HH:mm:ss to -MM-dd HH:mm:ss Key: FLINK-19553 URL: https://issues.apache.org/jira/browse/FLIN

Re: [ANNOUNCE] New PMC member: Zhu Zhu

2020-10-09 Thread Jingsong Li
Congratulations, Zhu Zhu! On Fri, Oct 9, 2020 at 3:08 PM Zhijiang wrote: > Congratulations and welcome, Zhu Zhu! > > Best, > Zhijiang > -- > From:Yun Tang > Send Time:2020年10月9日(星期五) 14:20 > To:dev@flink.apache.org > Subject:Re: [

Re: [ANNOUNCE] New PMC member: Zhu Zhu

2020-10-09 Thread Xintong Song
Congratulations, Zhu~! Thank you~ Xintong Song On Fri, Oct 9, 2020 at 3:17 PM Jingsong Li wrote: > Congratulations, Zhu Zhu! > > On Fri, Oct 9, 2020 at 3:08 PM Zhijiang .invalid> > wrote: > > > Congratulations and welcome, Zhu Zhu! > > > > Best, > > Zhijiang > >

[DISCUSS] Add Translation Specification for Stateful Functions

2020-10-09 Thread Congxian Qiu
Hi all, As Stateful Functions continues to evolve, and there have been many attempts by Chinese users. I propose to add a translation specification for Stateful Functions like the Translation Specification for Apache Flink[1](the proposed translation specification[2] is written in Chinese, we can

Re: [DISCUSS] FLIP-145: Support SQL windowing table-valued function

2020-10-09 Thread Jark Wu
Hi all, I know we have a lot of discussion and development on going right now but it would be great if we can get FLIP-145 into a votable state. If there are no objections, I would like to start voting in the next days. Best, Jark On Thu, 1 Oct 2020 at 14:29, Jark Wu wrote: > Hi everyone, > >

Re: [DISCUSS] Add Translation Specification for Stateful Functions

2020-10-09 Thread Yu Li
+1 for adding the translation specification for stateful functions. Thanks for driving the discussion Congxian! Best Regards, Yu On Fri, 9 Oct 2020 at 15:55, Congxian Qiu wrote: > Hi all, > > > As Stateful Functions continues to evolve, and there have been many > attempts by Chinese users. I

Re: [DISCUSS] Add Translation Specification for Stateful Functions

2020-10-09 Thread Jark Wu
Thanks for driving the discussion Congxian! I'm +1 for adding translation specifications for stateful functions. What about merging the proposed translation specifications into the existing document [1]? It seems the new specifications only add some terminology translation for the glossary. Most

Re: [DISCUSS] Enforce common opinionated coding style using Spotless

2020-10-09 Thread tison
+1 on locking on one codestyle and I think that is what current checkstyle rules serving. For automatic applying part, we suggest developing by IDEA and with Checkstyle Plugin on IDEA applying checkstyle suggestion is also automatic. One short coming for spotless is that we can hardly adjust rule

Re: [DISCUSS] FLIP-145: Support SQL windowing table-valued function

2020-10-09 Thread Benchao Li
Thanks Jark for bringing this discussion, I like this FLIP very much. Especially the cumulate window, it's much like the current TUMBLE window + Fast Emit (which is an undocumented experimental feature), however, it's more powerful. And This will make the SQL semantic more standard, especially fo

Re: [DISCUSS] FLIP-145: Support SQL windowing table-valued function

2020-10-09 Thread 刘 芃成
Hi,Benchao, Welcome to join the discussion, yes, this new syntax can make SQL more clear and simpler. For your first question, the `window_start` and `window_end` columns will be added automatically, so we don't need to use auxiliary group functions to infer or access the

Re: [DISCUSS] FLIP-146: Improve new TableSource and TableSink interfaces

2020-10-09 Thread Jingsong Li
Hi Aljoscha, I want to separate `Customized parallelism` and `Parallelism inference`. ### Customized parallelism First, I want to explain the current DataStream parallelism setting: `env.fromSource(...).setParallelism(...)`. This is how users explicitly specify parallelism, and it is the only wa

Re: autoCommit for postgres jdbc streaming in Table/SQL API

2020-10-09 Thread Dylan Forciea
Jark, Thank you! I had actually mistyped the JIRA issue; autoCommit needs to be set to false for streaming to work. The default on the driver is true when the option isn’t specified. I’ve updated the issue accordingly. Setting this to false automatically on the read path would fix my issue. Ho

[jira] [Created] (FLINK-19554) Unified testing framework for connectors

2020-10-09 Thread Qingsheng Ren (Jira)
Qingsheng Ren created FLINK-19554: - Summary: Unified testing framework for connectors Key: FLINK-19554 URL: https://issues.apache.org/jira/browse/FLINK-19554 Project: Flink Issue Type: New Fe

[jira] [Created] (FLINK-19555) Remove runAsyncWithoutFencing from MesosResourceManager

2020-10-09 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-19555: - Summary: Remove runAsyncWithoutFencing from MesosResourceManager Key: FLINK-19555 URL: https://issues.apache.org/jira/browse/FLINK-19555 Project: Flink Iss

[jira] [Created] (FLINK-19556) YarnResourceManagerDriver.getPendingRequestsAndCheckConsistency only returns container requests for first matching equivalent resource

2020-10-09 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-19556: - Summary: YarnResourceManagerDriver.getPendingRequestsAndCheckConsistency only returns container requests for first matching equivalent resource Key: FLINK-19556 URL: https://is

Re: autoCommit for postgres jdbc streaming in Table/SQL API

2020-10-09 Thread Jark Wu
Thanks for the explanation. I will review the pull request. Let's move the discussion to the PR. Best, Jark On Fri, 9 Oct 2020 at 21:06, Dylan Forciea wrote: > Jark, > > > > Thank you! I had actually mistyped the JIRA issue; autoCommit needs to be > set to false for streaming to work. The defau

Re: Need help in creating Flink Streaming s3 Job for multiple path reader one by one

2020-10-09 Thread Satyaa Dixit
Hi Shesnay/Team, Thank you so much for the reply.In the continuation of the previous email, below is the block diagram where I am reading the file from s3 and pushing it to kafka.Now with the current setup, I have total 4 directory based on the readfile method from flink environment ,we are cr

[jira] [Created] (FLINK-19557) Issue retrieving leader after zookeeper session reconnect

2020-10-09 Thread Max Mizikar (Jira)
Max Mizikar created FLINK-19557: --- Summary: Issue retrieving leader after zookeeper session reconnect Key: FLINK-19557 URL: https://issues.apache.org/jira/browse/FLINK-19557 Project: Flink Issue

Re: [DISCUSS] FLIP-145: Support SQL windowing table-valued function

2020-10-09 Thread Jark Wu
Hi Benchao, 1) time attribute Yes. We don't need time attribute auxiliary function. Because the new window operations are all based on the window_start and window_end columns instead of on the time attributes. So we don't need to propagate time attributes. Cascaded window aggregate can be express

[jira] [Created] (FLINK-19558) Umbrella Ticket for upgrading the REST API

2020-10-09 Thread Matthias (Jira)
Matthias created FLINK-19558: Summary: Umbrella Ticket for upgrading the REST API Key: FLINK-19558 URL: https://issues.apache.org/jira/browse/FLINK-19558 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-19559) Remove metrics from TaskManagerDetailsHandler

2020-10-09 Thread Matthias (Jira)
Matthias created FLINK-19559: Summary: Remove metrics from TaskManagerDetailsHandler Key: FLINK-19559 URL: https://issues.apache.org/jira/browse/FLINK-19559 Project: Flink Issue Type: Sub-task

[Announce] Flink Forward Global Is Just Around the Corner

2020-10-09 Thread Seth Wiesman
Hi Everyone, Flink Forward Global is just around the corner - October 19th to the 22nd - and our amazing line up of keynotes has been announced. - Peter Chalif from Citi will explore how Apache Flink has been fundamentally changing Big Data in the financial services industry - Ricky Sa

[jira] [Created] (FLINK-19560) Translate "YARN Setup" of "Clusters & Depolyment" page into Chinese

2020-10-09 Thread onewayforlife (Jira)
onewayforlife created FLINK-19560: - Summary: Translate "YARN Setup" of "Clusters & Depolyment" page into Chinese Key: FLINK-19560 URL: https://issues.apache.org/jira/browse/FLINK-19560 Project: Flink

Re: [DISCUSS] FLIP-145: Support SQL windowing table-valued function

2020-10-09 Thread Benchao Li
Hi Jark, 2 & 3 sounds good to me. Regarding time attribute, I still have some questions, I knew it's easy to support cascaded window aggregate using new TVFs. However there are some other places where need time attribute: - CEP - interval join - order by - over window If there is no time attribut

[jira] [Created] (FLINK-19561) Corrected spelling errors in "Monitoring REST API" page of "Debugging & Monitoring"

2020-10-09 Thread weizheng (Jira)
weizheng created FLINK-19561: Summary: Corrected spelling errors in "Monitoring REST API" page of "Debugging & Monitoring" Key: FLINK-19561 URL: https://issues.apache.org/jira/browse/FLINK-19561 Project:

Re: [DISCUSS] FLIP-145: Support SQL windowing table-valued function

2020-10-09 Thread pengcheng Liu
Hi,Benchao, In TVFs, the time attributes is just passed through from parent rels, and the TVFs just add two additional window attributes(i.e. window_start & window_end). Also, I think the time columns can be not only a time attribute with type of `TimeIndicatorType` but also a regular c

Re: [DISCUSS] FLIP-145: Support SQL windowing table-valued function

2020-10-09 Thread Benchao Li
Hi pengcheng, Thanks for your response. I knew that the original time attribute column will be retained after the TVF, what I'm questioning is how do we get the time attribute column after Aggregation. Your answer did not remove my doubts about this. It's ok if we did not plan to integrate new TV

Re: [ANNOUNCE] New PMC member: Zhu Zhu

2020-10-09 Thread Yang Wang
Congratulations! Zhu Zhu. Best, Yang Xintong Song 于2020年10月9日周五 下午3:35写道: > Congratulations, Zhu~! > > Thank you~ > > Xintong Song > > > > On Fri, Oct 9, 2020 at 3:17 PM Jingsong Li wrote: > > > Congratulations, Zhu Zhu! > > > > On Fri, Oct 9, 2020 at 3:08 PM Zhijiang > .invalid> > > wrote: >

Re: [ANNOUNCE] New PMC member: Zhu Zhu

2020-10-09 Thread Kurt Young
Congratulations, Zhu Zhu! Best, Kurt On Sat, Oct 10, 2020 at 11:03 AM Yang Wang wrote: > Congratulations! Zhu Zhu. > > Best, > Yang > > Xintong Song 于2020年10月9日周五 下午3:35写道: > > > Congratulations, Zhu~! > > > > Thank you~ > > > > Xintong Song > > > > > > > > On Fri, Oct 9, 2020 at 3:17 PM Jing

Re: [DISCUSS] FLIP-145: Support SQL windowing table-valued function

2020-10-09 Thread 刘 芃成
Hi, Benchao, I think I got your point, actually, in current implementation for group window aggregation, the value of time attributes(e.g. TUMBLE_ROWTIME/TUMBLE_PROCTIME) is calculated as (window_end – 1), so I think we can just use it directly if you need this. But I think this time attr

Re: [ANNOUNCE] New PMC member: Zhu Zhu

2020-10-09 Thread 刘 芃成
Congratulations, Zhu! Best, Pengcheng 在 2020/10/10 上午11:14,“Kurt Young” 写入: Congratulations, Zhu Zhu! Best, Kurt On Sat, Oct 10, 2020 at 11:03 AM Yang Wang wrote: > Congratulations! Zhu Zhu. > > Best, > Yang > > Xintong Song 于2020年10月9日周五 下午3:35写道:

Re: [DISCUSS] FLIP-145: Support SQL windowing table-valued function

2020-10-09 Thread Jark Wu
Hi Benchao, That's a good question. IMO, the new windowed operators and the current time operators are two different sets of functions, just like time operators and non-time operators are two different sets of functions. I think it's fine if we don't support integrating them, just like time opera

Re: [DISCUSS] FLIP-145: Support SQL windowing table-valued function

2020-10-09 Thread Benchao Li
Hi Jark, Thanks for your reply, this makes sense to me. The scenario I used above is just a case to explain what I'm concerned about, not necessarily a production use case. We can leave it to the future to see whether other users have these use cases. Then I have no other concerns, +1 to start t

Re: [DISCUSS] FLIP-145: Support SQL windowing table-valued function

2020-10-09 Thread Danny Chan
Thanks for driving this Jark ~ The FLIP overall looks good, i think the window TVFs would be the main "entry point" syntax for our NTR use cases. The syntax originated from the "One SQL To Rule Them All" paper and i think we have reached an agreement. I want to make some additions to the window