Re: [DISCUSS] Semantic and implementation of per-job mode

Hi all, Thanks for your participation! First of all I have to clarify two confusion in this thread. 1. The proposed "pre-program" mode is definitely a new mode opt-in. It is described in "Compatibility" section of the original email. 2. The documentation linked in the original email "Flink drive

Re: [Discussion] FLIP-79 Flink Function DDL Support

Hi Terry, Thanks for the quick response. We are on the same page. For the properties of function DDL, let's see whether there is such a need from other people. I will start voting on the design in 24 hours. Best Regards Peter Huang On Thu, Oct 31, 2019 at 3:18 AM Terry Wang wrote: > Hi

Re: [DISCUSS] Semantic and implementation of per-job mode

Hi Tison and Community, Thanks for bringing it up. Actually, we meet a similar bottleneck of using per cluster mode. Our team built a service for deploying and operating Flink jobs. The service sits in front of yarn clusters. To submit different job jars, we need to download client jar into the se

[jira] [Created] (FLINK-14591) Execute PlannerBase#mergeParameters every time of calling PlannerBase#translate method

Wei Zhong created FLINK-14591: - Summary: Execute PlannerBase#mergeParameters every time of calling PlannerBase#translate method Key: FLINK-14591 URL: https://issues.apache.org/jira/browse/FLINK-14591 Pro

Re: FW: Unit testing for ProcessAllWindowFunction

Hi Diogo, In order to test ProcessAllWindowFunction, you need to do a little more. You can refer to Flink's own test code.[1] In short: 1) Write a Flink Streaming job that uses your UDF (here is ProcessAllWindowFunction) and return a DataStream. 2) Get OneInputTransformation through DataStream,

Re: [DISCUSS] Semantic and implementation of per-job mode

Hi all, Firstly thanks @tison for bring this up and strongly +1 for the overall design. I’d like to add one more example of "multiple jobs in one program" with what I’m currently working on. I’m trying to run a TPC-DS benchmark testing (including tens of sql query job) on Flink and sufferrin

[jira] [Created] (FLINK-14590) Unify the working directory of Java process and Python process when submitting python jobs via "flink run -py"

Wei Zhong created FLINK-14590: - Summary: Unify the working directory of Java process and Python process when submitting python jobs via "flink run -py" Key: FLINK-14590 URL: https://issues.apache.org/jira/browse/FLINK

Re: Checkstyle-IDEA plugin For Java

If you can import checkstyle rules file, the version of checkstyle plugin is not very important. We don't use nightly feature IIRC. Best, tison. yanjun qiu 于2019年10月31日周四 下午7:22写道： > Hi Community, > I want to contribute code to Flink and I have followed the IDE set up > guide as below: > > ht

Re: [ANNOUNCE] Becket Qin joins the Flink PMC

Congratulations, Becket! Best, Guowei Steven Wu 于2019年11月1日周五 上午6:20写道： > Congratulations, Becket! > > On Wed, Oct 30, 2019 at 9:51 PM Shaoxuan Wang wrote: > > > Congratulations, Becket! > > > > On Mon, Oct 28, 2019 at 6:08 PM Fabian Hueske wrote: > > > > > Hi everyone, > > > > > > I'm happy

Re: [ANNOUNCE] Becket Qin joins the Flink PMC

Congratulations, Becket! On Wed, Oct 30, 2019 at 9:51 PM Shaoxuan Wang wrote: > Congratulations, Becket! > > On Mon, Oct 28, 2019 at 6:08 PM Fabian Hueske wrote: > > > Hi everyone, > > > > I'm happy to announce that Becket Qin has joined the Flink PMC. > > Let's congratulate and welcome Becket

Re: [VOTE] Accept Stateful Functions into Apache Flink

+1 (binding) Thank you to Stephan and all current and future contributors to this tool! On Thu, Oct 31, 2019 at 4:24 AM Vijay Bhaskar wrote: > +1 from me > > Regards > Bhaskar > > On Thu, Oct 31, 2019 at 11:42 AM Gyula Fóra wrote: > > > +1 from me, this is a great addition to Flink! > > > > Gy

[jira] [Created] (FLINK-14589) Redundant slot requests with the same AllocationID leads to inconsistent slot table

Hwanju Kim created FLINK-14589: -- Summary: Redundant slot requests with the same AllocationID leads to inconsistent slot table Key: FLINK-14589 URL: https://issues.apache.org/jira/browse/FLINK-14589 Proje

FW: Unit testing for ProcessAllWindowFunction

Good afternoon, After Reading the official flink testing documentation (https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/stream/testing.html) I was able to develop tests for a ProcessFunction, using a Test Harness, something like this: pendingPartitionBuilder = new PendingParti

[jira] [Created] (FLINK-14588) Support Hive version 1.0.0 and 1.0.1

Rui Li created FLINK-14588: -- Summary: Support Hive version 1.0.0 and 1.0.1 Key: FLINK-14588 URL: https://issues.apache.org/jira/browse/FLINK-14588 Project: Flink Issue Type: Task Component

[jira] [Created] (FLINK-14587) HiveTableSourceTest timeouts on Travis

Gary Yao created FLINK-14587: Summary: HiveTableSourceTest timeouts on Travis Key: FLINK-14587 URL: https://issues.apache.org/jira/browse/FLINK-14587 Project: Flink Issue Type: Bug Comp

Checkstyle-IDEA plugin For Java

Hi Community, I want to contribute code to Flink and I have followed the IDE set up guide as below: https://ci.apache.org/projects/flink/flink-docs-stable/flinkDev/ide_setup.html#checkstyle-for-java

[jira] [Created] (FLINK-14586) JobMaster issues promote calls if job is successful

Chesnay Schepler created FLINK-14586: Summary: JobMaster issues promote calls if job is successful Key: FLINK-14586 URL: https://issues.apache.org/jira/browse/FLINK-14586 Project: Flink I

Re: RemoteEnvironment cannot execute job from local.

Hi Till Thanks for your reply. Actually I modify the code like this: I commented the filter part, and re-run the code, then it works well !! The jar passed to createRemoteEnvironment is a udf jar, which does not contain my code My flink version is 1.9.0, So I’m confused about the actual behavi

Re: Builds Notification Update

> > Thank you very much. I'll prepare and start to send the test result from > next week. > >

Re: [DISCUSS] FLIP-83: Flink End-to-end Performance Testing Framework

Hi Piotr, Thanks for the comments! bq. How are you planning to execute the end-to-end benchmarks and integrate them with our build process? Great question! We plan to execute the end-to-end benchmark in a small cluster (like 3 vm nodes) to better reflect network cost, triggering it through our Je

Re: [Discussion] FLIP-79 Flink Function DDL Support

Hi Peter, I’d like to share some thoughts from mysids: 1. what's the syntax to distinguish function language ? +1 for using `[LANGUAGE JVM|PYTHON] USING JAR` 2. How to persist function language in backend catalog ? + 1 for a separate field in CatalogFunction. But as to specific ba

[jira] [Created] (FLINK-14585) flink connector for sink to apache/incubator-druid

xiaodao created FLINK-14585: --- Summary: flink connector for sink to apache/incubator-druid Key: FLINK-14585 URL: https://issues.apache.org/jira/browse/FLINK-14585 Project: Flink Issue Type: New Feat

Re: [DISCUSS] Semantic and implementation of per-job mode

Hi all, we're using a lot the multiple jobs in one program and this is why: when you fetch data from a huge number of sources and, for each source, you do some transformation and then you want to write into a single directory the union of all outputs (this assumes you're doing batch). When the numb

Reminder: please update FLIP document to latest status

Dear all, While checking 1.10 release progress, we find some FLIPs have already been accepted through voting, but still marked as "Under Discussion" in wiki document (eg. FLIP-68), and some marked as implementing but lack of JIRA link (eg. FLIP-64). So just a reminder to our FLIP authors/owners, t

[jira] [Created] (FLINK-14584) Support complex data types in Python user-defined functions

Huang Xingbo created FLINK-14584: Summary: Support complex data types in Python user-defined functions Key: FLINK-14584 URL: https://issues.apache.org/jira/browse/FLINK-14584 Project: Flink

Re: RemoteEnvironment cannot execute job from local.

In order to run the program on a remote cluster from the IDE you need to first build the jar containing your user code. This jar needs to passed to createRemoteEnvironment() so that the Flink client knows which jar to upload. Hence, please make sure that /tmp/myudf.jar contains your user code. Che

[RESULT][VOTE] FLIP-70: Flink SQL Computed Column Design

Hi all, The voting time for FLIP-70 has passed. I'm closing the vote now. There were seven +1 votes, 3 of which are binding: - David (binding) - Jark Wu (binding) - Kurt Young (binding) - Benchao Li (non-binding) - Terry Wang (non-binding) - Jingsong Li (non-binding) - Zhenghua Gao (non-binding)

Re: [DISCUSS] Semantic and implementation of per-job mode

Thanks for tison starting this exciting discussion. We also suffer a lot from the per job mode. I think the per-job cluster is a dedicated cluster for only one job and will not accept more other jobs. It has the advantage of one-step submission, do not need to start dispatcher first and then submit

Re: [DISCUSS] Semantic and implementation of per-job mode

Hi all, Regarding the compilation part: I think there are up and downsides to building the Flink job (running the main method) on the client side, however since this is the current way of doing it we should have a very powerful reason to change the default behaviour. While there is a possible wor

Re: possible backwards compatibility issue between 1.8->1.9?

Hi, (This question is more appropriate for the user mailing list, not dev - when responding to my e-mail please remove dev mailing list from the recipients, I’ve kept it just FYI that discussion has moved to user mailing list). Could it be, that the problem is caused by changes in chaining stra

Re: [VOTE] Accept Stateful Functions into Apache Flink

+1 from me Regards Bhaskar On Thu, Oct 31, 2019 at 11:42 AM Gyula Fóra wrote: > +1 from me, this is a great addition to Flink! > > Gyula > > On Thu, Oct 31, 2019, 03:52 Yun Gao wrote: > > > +1 (non-binding) > > Very thanks for bringing this to the community! > > > > > > ---

[jira] [Created] (FLINK-14583) Remove progressLock from ExecutionGraph

vinoyang created FLINK-14583: Summary: Remove progressLock from ExecutionGraph Key: FLINK-14583 URL: https://issues.apache.org/jira/browse/FLINK-14583 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-14582) Do not upload {uuid}-taskmanager-conf.yaml for each task manager container

Yang Wang created FLINK-14582: - Summary: Do not upload {uuid}-taskmanager-conf.yaml for each task manager container Key: FLINK-14582 URL: https://issues.apache.org/jira/browse/FLINK-14582 Project: Flink

Re: [DISCUSS] Introduce a location-oriented two-stage query mechanism toimprove the queryable state.

Hi Yu, Thanks for your suggestion. Yes, the community is missing some attention and exploration of the application scenario (even for the state processing api that was recently released). I do have some frustration because I have implemented and verified this feature. But it doesn't matter, let's

RemoteEnvironment cannot execute job from local.

Hi all I want to test to submit a job from my local IDE and I deployed a Flink cluster in my vm. Here is my code from Flink 1.9 document and add some of my parameters. public static void main(String[] args) throws Exception { ExecutionEnvironment env = ExecutionEnvironment .createRemoteE

[jira] [Created] (FLINK-14581) Support to run Python UDF jobs in a YARN cluster

Dian Fu created FLINK-14581: --- Summary: Support to run Python UDF jobs in a YARN cluster Key: FLINK-14581 URL: https://issues.apache.org/jira/browse/FLINK-14581 Project: Flink Issue Type: Sub-task