[jira] [Created] (FLINK-10745) Improve record writer and serializer stack

zhijiang created FLINK-10745: Summary: Improve record writer and serializer stack Key: FLINK-10745 URL: https://issues.apache.org/jira/browse/FLINK-10745 Project: Flink Issue Type: Improvement

[DISCUSS] Enhancing the functionality and productivity of Table API

Hi all, With the continuous efforts from the community, the Flink system has been continuously improved, which has attracted more and more users. Flink SQL is a canonical, widely used relational query language. However, there are still some scenarios where Flink SQL failed to meet user needs in te

回复：[DISCUSS] Proposal of external shuffle service

Thanks for the efficient response till! Thanks sunjin for the good feedbacks, we will further confirm with the comments then! :) -- 发件人：Jin Sun 发送时间：2018年11月1日(星期四) 06:42 收件人：dev 抄　送：Zhijiang(wangzhijiang999) ; Nico Kruber ; Piotr

Re: [DISCUSS] Change underlying Frontend Architecture for Flink Web Dashboard

Thanks to Fabian for starting this discussion! I liked the discussion. Here are some opinions I have: [1] Till mentioned that "I think our web UI technology stack is quite dusty by now and it would be beneficial to think about its technological future." I totally agree with that we should upgrade t

Flip23

For those who want to see an actual code, it is here: https://github.com/FlinkML/flink-modelServer for simple model serving And https://github.com/FlinkML/flink-modelServer for speculative one Boris Lub

Re: [DISCUSS] Change underlying Frontend Architecture for Flink Web Dashboard

For your interest, here are some snapshots of Flink web UI (with angular 7.0) we recently reconstructed. [image: snapshot.001.jpeg] https://drive.google.com/file/d/1p7rcmgHi3ZJpjt7h1H97VIrIhNkiBs1j/view?usp=sharing We choose Angular 7.0 due to the following reasons: 1. It is easy to migrate th

Re: [DISCUSS] FLIP-27: Refactor Source Interface

Hi, Thanks Aljoscha for starting this, it’s blocking quite a lot of other possible improvements. I have one proposal. Instead of having a method: boolean advance() throws IOException; I would replace it with /* * Return a future, which when completed means that source has more data and getN

Re: Flip23

Thanks for sharing the code with the community Boris! Cheers, Till On Thu, Nov 1, 2018 at 10:29 AM Boris Lublinsky < boris.lublin...@lightbend.com> wrote: > For those who want to see an actual code, it is here: > https://github.com/FlinkML/flink-modelServer < > https://github.com/FlinkML/flink-m

Re: Sharing state between subtasks

Hi Thomas, using Akka directly would further manifest our dependency on Scala in flink-runtime. This is something we are currently trying to get rid of. For that purpose we have added the RpcService abstraction which encapsulates all Akka specific logic. We hope that we can soon get rid of the Sca

[jira] [Created] (FLINK-10746) Need to replace transfer.sh for Travis log upload because it shuts down

Till Rohrmann created FLINK-10746: - Summary: Need to replace transfer.sh for Travis log upload because it shuts down Key: FLINK-10746 URL: https://issues.apache.org/jira/browse/FLINK-10746 Project: Fl

[jira] [Created] (FLINK-10747) CoGroupConnectedComponentsITCase deadlocked on Travis

Till Rohrmann created FLINK-10747: - Summary: CoGroupConnectedComponentsITCase deadlocked on Travis Key: FLINK-10747 URL: https://issues.apache.org/jira/browse/FLINK-10747 Project: Flink Issue

[jira] [Created] (FLINK-10748) Jobmanager in HA setup redirects to ip address instead of hostname

Jeroen Steggink created FLINK-10748: --- Summary: Jobmanager in HA setup redirects to ip address instead of hostname Key: FLINK-10748 URL: https://issues.apache.org/jira/browse/FLINK-10748 Project: Fli

Re: [DISCUSS] Enhancing the functionality and productivity of Table API

*I am sorry for the formatting of the email content. I reformat the **content** as follows---* *Hi ALL,* With the continuous efforts from the community, the Flink system has been continuously improved, which has attracted more and more users. Flink SQL is a canonical, widely used

[jira] [Created] (FLINK-10749) GraphOperationsITCase deadlocks on travis

Chesnay Schepler created FLINK-10749: Summary: GraphOperationsITCase deadlocks on travis Key: FLINK-10749 URL: https://issues.apache.org/jira/browse/FLINK-10749 Project: Flink Issue Type:

[ANNOUNCE] Apache Flink-shaded 5.0 released

The Apache Flink community is pleased to announce the release of Apache Flink-shaded 5.0. The flink-shaded project contains a number of shaded dependencies for Apache Flink. This release adds jackson-dataformat-csv to the flink-shaded-jackson module and adds a new flink-shaded-asm6 module r

Re: [DISCUSS] Enhancing the functionality and productivity of Table API

Hi Jincheng, these points sound very good! Are there any concrete proposals for changes? For example a FLIP/design document? See here for FLIPs: https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals Best, Aljoscha > On 1. Nov 2018, at 12:51, jincheng sun wrote: > >

[jira] [Created] (FLINK-10750) SocketClientSinkTest.testRetry fails on Travis

Till Rohrmann created FLINK-10750: - Summary: SocketClientSinkTest.testRetry fails on Travis Key: FLINK-10750 URL: https://issues.apache.org/jira/browse/FLINK-10750 Project: Flink Issue Type:

Re: [DISCUSS] Enhancing the functionality and productivity of Table API

Hi Jincheng, I was also thinking about introducing a process function for the Table API several times. This would allow to define more complex logic (custom windows, timers, etc.) embedded into a relational API with schema awareness and optimization around the black box. Of course this would

[jira] [Created] (FLINK-10751) Checkpoints should be retained when job reaches suspended state

Ufuk Celebi created FLINK-10751: --- Summary: Checkpoints should be retained when job reaches suspended state Key: FLINK-10751 URL: https://issues.apache.org/jira/browse/FLINK-10751 Project: Flink

[jira] [Created] (FLINK-10752) Result of AbstractYarnClusterDescriptor#validateClusterResources is ignored

Dawid Wysakowicz created FLINK-10752: Summary: Result of AbstractYarnClusterDescriptor#validateClusterResources is ignored Key: FLINK-10752 URL: https://issues.apache.org/jira/browse/FLINK-10752 P

Re: [DISCUSS] Enhancing the functionality and productivity of Table API

Hi, Aljoscha, Thanks for your feedback and suggestions. I think your are right, the detailed design/FLIP is very necessary. Before the detailed design or open a FLIP, I would like to hear the community's views on Enhancing the functionality and productivity of Table API, to ensure that it worth t

Re: [DISCUSS] Enhancing the functionality and productivity of Table API

Yes, that makes sense! > On 1. Nov 2018, at 15:51, jincheng sun wrote: > > Hi, Aljoscha, > > Thanks for your feedback and suggestions. I think your are right, the > detailed design/FLIP is very necessary. Before the detailed design or open > a FLIP, I would like to hear the community's views on

[jira] [Created] (FLINK-10753) Propagate and log snapshotting exceptions

Alexander Fedulov created FLINK-10753: - Summary: Propagate and log snapshotting exceptions Key: FLINK-10753 URL: https://issues.apache.org/jira/browse/FLINK-10753 Project: Flink Issue Typ

[jira] [Created] (FLINK-10754) Update Jepsen tests to run with activated local recovery

Till Rohrmann created FLINK-10754: - Summary: Update Jepsen tests to run with activated local recovery Key: FLINK-10754 URL: https://issues.apache.org/jira/browse/FLINK-10754 Project: Flink Is

Re: [DISCUSS] Enhancing the functionality and productivity of Table API

Hi, Timo, I am very grateful for your feedback, and I am very excited when I hear that you also consider adding a process function to the TableAPI. I agree that add support for the Process Function on the Table API, which is actually part of my proposal Enhancing the functionality of Table API. In

Re: [DISCUSS] Integrate Flink SQL well with Hive ecosystem

Hi all, As Xuefu has published the design doc on google, I agree with Shuyi's suggestion that we probably should start a new email thread like "[DISCUSS] ... Hive integration design ..." on only dev mailing list for community devs to review. The current thread sends to both dev and user list. Thi

Re: [DISCUSS] Enhancing the functionality and productivity of Table API

Thanks for the proposal, Jincheng. This makes a lot of sense. As a programming interface, Table API is especially attractive because it supports both batch and stream. However, the relational-only API often forces users to shoehorn their logic into a bunch of user defined functions. Introducing so

Re: [DISCUSS] FLIP-27: Refactor Source Interface

Thanks for the FLIP, Aljoscha. The proposal makes sense to me. Separating the split discovery and consumption is very useful as it enables Flink to better manage the sources. Looking at the interface, I have a few questions: 1. *SplitEnumerator*.*discoverNewSplits()* seems assuming that the numbe

[jira] [Created] (FLINK-10755) Port external catalogs in Table API extension points to flink-table-common

Bowen Li created FLINK-10755: Summary: Port external catalogs in Table API extension points to flink-table-common Key: FLINK-10755 URL: https://issues.apache.org/jira/browse/FLINK-10755 Project: Flink

Re: [DISCUSS] Integrate Flink SQL well with Hive ecosystem

After taking a look at how other discussion threads work, I think it's actually fine just keep our discussion here. It's up to you, Xuefu. The google doc LGTM. I left some minor comments. On Thu, Nov 1, 2018 at 10:17 AM Bowen Li wrote: > Hi all, > > As Xuefu has published the design doc on goog

Re: Flink operators for Kubernetes

Hello, I have documented our work of running Flink applications on Kubernetes using FlinkK8sOperator - here We have finished the implementation of the FlinkK8sOperator, and are currently

Re: Demo of the K8S Flink Operator at the K8s Big Data SIG

Thank you all for providing us the opportunity to present our work. Here is the presentation that we used. Also, you can find more information about FlinkK8sOperator here

[jira] [Created] (FLINK-10756) TaskManagerProcessFailureBatchRecoveryITCase did not finish on time

Bowen Li created FLINK-10756: Summary: TaskManagerProcessFailureBatchRecoveryITCase did not finish on time Key: FLINK-10756 URL: https://issues.apache.org/jira/browse/FLINK-10756 Project: Flink

[jira] [Created] (FLINK-10757) TaskManagerProcessFailureStreamingRecoveryITCase failed to initialize the cluster entrypoint StandaloneSessionClusterEntrypoint.

Bowen Li created FLINK-10757: Summary: TaskManagerProcessFailureStreamingRecoveryITCase failed to initialize the cluster entrypoint StandaloneSessionClusterEntrypoint. Key: FLINK-10757 URL: https://issues.apache.org/j

Re: [DISCUSS] Enhancing the functionality and productivity of Table API

Hi Aljoscha， Glad that you like the proposal. We have completed the prototype of most new proposed functionalities. Once collect the feedback from community, we will come up with a concrete FLIP/design doc. Regards, Shaoxuan On Thu, Nov 1, 2018 at 8:12 PM Aljoscha Krettek wrote: > Hi Jincheng,

Re: [DISCUSS] Enhancing the functionality and productivity of Table API

Hi Jincheng, Thanks a lot for your proposal. It is very encouraging! As we all know, SQL is a widely used language. It follows standards, is a descriptive language, and is easy to use. A powerful feature of SQL is that it supports optimization. Users only need to care about the logic of the progr

Re: [DISCUSS] Change underlying Frontend Architecture for Flink Web Dashboard

+1 to keep the web UI stack to Angular. This will make the stack upgrading much easier and won't break the on-going efforts already made towards the improvements of Flink Web UI. Till, Robert, and Fabian, Please take a look at the upgrading plan from Angular 1.x to 7.0 proposed by Yadong. Any feed

Re: [DISCUSS] Enhancing the functionality and productivity of Table API

Hi, Jiangjie, Thanks a lot for your feedback. And also thanks for our offline discussion! Yes, your right! The Row-based APIs which you mentioned are very friendly to flink user! In order to follow the concept of the traditional database, perhaps we named the corresponding function RowValued/TabeVa

Re: [DISCUSS] Enhancing the functionality and productivity of Table API

Hi Hequn, Thanks for your feedback! And also thanks for our offline discussion! You are right, unification of batch and streaming is very important for flink API. We will provide more detailed design later, Please let me know if you have further thoughts or feedback. Thanks, Jincheng Hequn Cheng

[jira] [Created] (FLINK-10758) Refactor TableEnvironment so that all registration calls delegate to CatalogManager

Xuefu Zhang created FLINK-10758: --- Summary: Refactor TableEnvironment so that all registration calls delegate to CatalogManager Key: FLINK-10758 URL: https://issues.apache.org/jira/browse/FLINK-10758 Pr

[jira] [Created] (FLINK-10759) Adapt SQL-client configuration file to specify external catalogs and default catalog

Xuefu Zhang created FLINK-10759: --- Summary: Adapt SQL-client configuration file to specify external catalogs and default catalog Key: FLINK-10759 URL: https://issues.apache.org/jira/browse/FLINK-10759 Pr

[jira] [Created] (FLINK-10760) Create a command line tool to migrate meta objects specified in SQL client configuration

Xuefu Zhang created FLINK-10760: --- Summary: Create a command line tool to migrate meta objects specified in SQL client configuration Key: FLINK-10760 URL: https://issues.apache.org/jira/browse/FLINK-10760

Why dont't have a csv formatter for kafka table source

Hi all， I have encountered a error When i want to register a table from kafka using csv formatter. The error is "Could not find a suitable table factory for 'org.apache.flink.table.factories.DeserializationSchemaFactory" Jocean

Re: [DISCUSS] Enhancing the functionality and productivity of Table API

Hi jingchengm Thanks a lot for your proposal! I find it is a good start point for internal optimization works and help Flink to be more user-friendly. AFAIK, DataStream is the most popular API currently that Flink users should describe their logic with detailed logic. >From a more internal view t

[DISCUSS] Flink SQL DDL Design

Hi everyone, SQL DDL support has been a long-time ask from the community. Current Flink SQL support only DML (e.g. SELECT and INSERT statements). In its current form, Flink SQL users still need to define/create table sources and sinks programmatically in Java/Scala. Also, in SQL Client, without DD

Re: [DISCUSS] Change underlying Frontend Architecture for Flink Web Dashboard

+1 for the proposal of Yadong. I'd like to add one more comment: We should provide more development guidance when developing the next dashboard via angular 7. As Till mentioned before, one big concern is lack of frontend guys in flink community, I believe more development guidance will attract mo