[jira] [Created] (FLINK-10807) KafkaConsumer still consume removed topic after changing topics list

Jiayi Liao created FLINK-10807: -- Summary: KafkaConsumer still consume removed topic after changing topics list Key: FLINK-10807 URL: https://issues.apache.org/jira/browse/FLINK-10807 Project: Flink

Re: [DISCUSS] Table API Enhancement Outline

Hi xiaowei, Yes, I agree with you that the semantics of TableAggregateFunction emit is much more complex than AggregateFunction. The fundamental difference is that TableAggregateFunction emits a "table" while AggregateFunction outputs (a column of) a "row". In the case of AggregateFunction it only

Re: [DISCUSS] Table API Enhancement Outline

Hi Xiaowei, Thank you for mentioned such key points. Yes, I think those points are very important for the clear definition of the semantics of Table AggregateFunction！I'd like share my thoughts about the those questions: 1. Do we allow multi-staged TableAggregate in this case? >From the points of

Re: [DISCUSS] Task speculative execution for Flink batch

+1, Thanks Yangyu for proposing this very useful feature. Looking forward to the design doc. On Wed, Nov 7, 2018 at 10:15 AM SHI Xiaogang wrote: > Hi, > > +1 for the speculative execution. > > It will be more great if it can work well with exisitng checkpointing and > pipelined execution. That w

Re: [DISCUSS] FLIP-27: Refactor Source Interface

Hi Biao, Thanks for the explanation. The current API makes more sense to me now. It basically means: 1. Readers should all be non-blocking 2. The offset advancing and the record fetching are two steps. 3. After each advance() call, the currentRecord, currentTimestamp and watermark will all be upda

Re: [DISCUSS] Enhancing the functionality and productivity of Table API

Hi all, Thanks for the feedback. I enjoyed the discussions, especially the ones between Fabian and Xiaowei. I think it well revealed the motivations and design pros/cons behind this proposal. Enhancing tableAPI will not affect and limit the improvements on Flink SQL (as well as DataStream). Actual

[jira] [Created] (FLINK-10806) Support consume from earliest offset with a new topic

Jiayi Liao created FLINK-10806: -- Summary: Support consume from earliest offset with a new topic Key: FLINK-10806 URL: https://issues.apache.org/jira/browse/FLINK-10806 Project: Flink Issue Type:

Re: [DISCUSS] Task speculative execution for Flink batch

Hi, +1 for the speculative execution. It will be more great if it can work well with exisitng checkpointing and pipelined execution. That way, we can move a further step towards the unification of batch and stream processing. Regards, Xiaogang Jeff Zhang 于2018年11月7日周三 上午9:40写道： > +1 for the s

Re: [DISCUSS] Enhancing the functionality and productivity of Table API

Hi all, Thank you for your replies and comments. I have similar consideration like Piotrek. My opinion is that two APIs are enough for Flink, a declarative one (SQL) and one imperative one (DataStream). From my perspective, most of users prefer SQL at most time and turn to Data Stream when the l

Re: [DISCUSS] Task speculative execution for Flink batch

+1 for the speculative execution for Flink batch, Speculative execution is used in lots of batch execution engine like mr, tez and spark. This would be a great improvement for Flink in batch scenario. Jin Sun 于2018年11月7日周三 上午8:38写道： > I think this is target for batch at the very beginning, the id

Re: [DISCUSS] Task speculative execution for Flink batch

I think this is target for batch at the very beginning, the idea should be also work for both case, with different algorithm/strategy. Ryan, since you are working on this, I will assign FLINK-10644 to you. Jin > On Nov 6, 2018, at 4:45 AM, T

JIRA assignment

I submitted a PR[1] for FLINK-10213[2] awhile ago, but I couldn't assign the issue to myself or update it's status in JIRA. Is there anything I need to do to get JIRA updated? Thanks! -Joey [1] https://github.com/apache/flink/pull/6862 [2] https://issues.apache.org/jira/browse/FLINK-10213

[ANNOUNCE] Weekly community update #45

Dear community, this is the weekly community update thread #45. Please post any news and updates you want to share with the community to this thread. # First release candidate for Flink 1.7.0 The community has published the first release candidate for Flink 1.7.0 [0]. Please help the community b

Re: [DISCUSS] FLIP-27: Refactor Source Interface

Regarding the naming style. The advantage of `poll()` style is that basically the name of `poll` means it should be a non-blocking operator, same with `Queue` in Java API. It's easy to understand. We don't need to write too much in docs to imply the implementation should not do something heavy. Ho

[jira] [Created] (FLINK-10805) test_confluent_schema_registry.sh and test_sql_client.sh end-to-end test broken

Till Rohrmann created FLINK-10805: - Summary: test_confluent_schema_registry.sh and test_sql_client.sh end-to-end test broken Key: FLINK-10805 URL: https://issues.apache.org/jira/browse/FLINK-10805 Pro

[jira] [Created] (FLINK-10804) Transfer suppressed exceptions with SerializedThrowable

Nico Kruber created FLINK-10804: --- Summary: Transfer suppressed exceptions with SerializedThrowable Key: FLINK-10804 URL: https://issues.apache.org/jira/browse/FLINK-10804 Project: Flink Issue T

Re: [DISCUSS] Enhancing the functionality and productivity of Table API

Hi, What is our intended division/border between Table API and DataSet or DataStream? If we want Table API to drift away from SQL that would be a valid question. > Another distinguishing feature of DataStream API is that users get direct > access to state/statebackend which we intensionally avo

Re: [DISCUSS] Enhancing the functionality and productivity of Table API

Hi, An analysis of orthogonal functions would be great! There is certainly some overlap in the functions provided by the DataSet API. In the past, I found that having low-level functions helped a lot to efficiently implement complex logic. Without partitionByHash, sortPartition, sort, mapPartitio

Re: [DISCUSS] Table API Enhancement Outline

Hi, Thanks for the great design document! It answers my question regarding handling of retraction messages. Overall, I like the proposal. It is well scoped and the proposed changes are well described. I left a question regarding the handling of time attributes for multi-column output functions.

Re: [DISCUSS] Task speculative execution for Flink batch

Thanks for starting this discussion Ryan. I'm looking forward to your design document about this feature. Quick question: Will it be a batch only feature? If no, then it needs to take checkpointing into account as well. Cheers, Till On Tue, Nov 6, 2018 at 4:29 AM zhijiang wrote: > Thanks yangyu

Re: [DISCUSS] Flink Cluster Overview Dashboard Improvement Proposal

Ok Shaoxuan Wang 于2018年11月6日 周二19:26写道： > Lining, > Thanks for the proposal. > There is another ongoing ML ( > > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Change-underlying-Frontend-Architecture-for-Flink-Web-Dashboard-td24902.html > ), > where YaDong has shared some

Re: [DISCUSS] Enhancing the functionality and productivity of Table API

Hi Fabian, Thank you for your deep thoughts in this regard, I think most of questions you had mentioned are very worthy of in-depth discussion! I want share thoughts about following questions: 1. Do we need move all DataSet API functionality into the Table API? I think most of dataset functionalit

Re: [DISCUSS] Change underlying Frontend Architecture for Flink Web Dashboard

Fabian, Thanks for pointing out the Jira. Sure, we will reuse it to start the contribution. Regards, Shaoxuan On Tue, Nov 6, 2018 at 7:28 PM Fabian Wollert wrote: > i updated this JIRA already, feel free to reuse this: > https://issues.apache.org/jira/browse/FLINK-10706 > > -- > > > *Fabian Wol

Re: [DISCUSS] Enhancing the functionality and productivity of Table API

Hi Fabian, I totally agree with you that we should incrementally improve TableAPI. We don't suggest that we do anything drastic such as replacing DataSet API yet. We should see how much we can achieve by extending TableAPI cleanly. By then, we should see if there are any natural boundaries on how

Re: [DISCUSS] Change underlying Frontend Architecture for Flink Web Dashboard

i updated this JIRA already, feel free to reuse this: https://issues.apache.org/jira/browse/FLINK-10706 -- *Fabian WollertZalando SE* E-Mail: fab...@zalando.de Am Di., 6. Nov. 2018 um 12:10 Uhr schrieb Shaoxuan Wang : > Till, > Yes, it is a good idea to have a feature flag to switch the web

Re: [DISCUSS] Flink Cluster Overview Dashboard Improvement Proposal

Lining, Thanks for the proposal. There is another ongoing ML ( http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Change-underlying-Frontend-Architecture-for-Flink-Web-Dashboard-td24902.html), where YaDong has shared some sample code ( https://github.com/vthinkxie/flink-runtime-

Re: [DISCUSS] Table API Enhancement Outline

Hi Jincheng, Thanks for adding the public interfaces! I think that it's a very good start. There are a few points that we need to have more discussions. - TableAggregateFunction - this is a very complex beast, definitely the most complex user defined objects we introduced so far. I think th

Re: [DISCUSS] Change underlying Frontend Architecture for Flink Web Dashboard

Till, Yes, it is a good idea to have a feature flag to switch the web UI before we completely deprecate the old one. Yadong, It seems that everyone likes the new web UI. Can you please open a master Jira and start to merge the code to Flink master. What do you think? Regards, Shaoxuan On Mon, No

Re: [DISCUSS] Flink Cluster Overview Dashboard Improvement Proposal

We also need refactor single job show. Now, just can see vertex metrics. And if you want see other informations, have to go to other page .First, we need update vertex and operator show. - vertex show in https://issues.apache.org/jira/browse/FLINK-10802. [image: image.png] - operator show like [i

Re: [DISCUSS] Table API Enhancement Outline

Hi, Xiaowei, Thanks for bring up the discuss of Table API Enhancement Outline ! I quickly looked at the overall content, these are good expressions of our offline discussions. But from the points of my view, we should add the usage of public interfaces that we will introduce in this propose. So,

Re: [DISCUSS] Enhancing the functionality and productivity of Table API

Thanks for the replies Xiaowei and others! You are right, I did not consider the batch optimization that would be missing if the DataSet API would be ported to extend the DataStream API. By extending the scope of the Table API, we can gain a holistic logical & physical optimization which would be

[jira] [Created] (FLINK-10803) Add documentation about S3 support by the StreamingFileSink

Kostas Kloudas created FLINK-10803: -- Summary: Add documentation about S3 support by the StreamingFileSink Key: FLINK-10803 URL: https://issues.apache.org/jira/browse/FLINK-10803 Project: Flink

[jira] [Created] (FLINK-10802) Add metrics on vertex in job graph picture show

lining created FLINK-10802: -- Summary: Add metrics on vertex in job graph picture show Key: FLINK-10802 URL: https://issues.apache.org/jira/browse/FLINK-10802 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-10801) Fix sql client integrate elasticsearch connector test failure

vinoyang created FLINK-10801: Summary: Fix sql client integrate elasticsearch connector test failure Key: FLINK-10801 URL: https://issues.apache.org/jira/browse/FLINK-10801 Project: Flink Issue

Re: [DISCUSS] Enhancing the functionality and productivity of Table API

Hi Jark, Glad to see your feedback! That's Correct, The proposal is aiming to extend the functionality for Table API! I like add "drop" to fit the use case you mentioned. Not only that, if a 100-columns Table. and our UDF needs these 100 columns, we don't want to define the eval as eval(column0...c

Re: Confluence permission for FLIP creation

Hi Xuefu, I gave you permissions for Confluence. Looking forward to the FLIP. Best, Fabian Am Di., 6. Nov. 2018 um 04:09 Uhr schrieb Zhang, Xuefu < xuef...@alibaba-inc.com>: > Hi there, > > As communicated in an email thread, I'm proposing Flink-Hive metastore > integration. I have a draft desi