[jira] [Created] (FLINK-5324) JVM Opitons will be work both for YarnApplicationMasterRunner and YarnTaskManager with yarn mode

2016-12-12 Thread yuemeng (JIRA)
yuemeng created FLINK-5324: -- Summary: JVM Opitons will be work both for YarnApplicationMasterRunner and YarnTaskManager with yarn mode Key: FLINK-5324 URL: https://issues.apache.org/jira/browse/FLINK-5324 Pr

Minor Cleanup of StreamGraphGeneator

2016-12-12 Thread Chen Qin
Hi There, I am thinking of doing a minor refactor of StreamGraphGenerator. There are some transforms share same logic that can potentially reusable. Also, long list of if else seems hard to read, might better off abstract interface & offer default implementation. Bonus point would be allow confi

[jira] [Created] (FLINK-5323) CheckpointNotifier should be removed from docs

2016-12-12 Thread Abhishek Singh (JIRA)
Abhishek Singh created FLINK-5323: - Summary: CheckpointNotifier should be removed from docs Key: FLINK-5323 URL: https://issues.apache.org/jira/browse/FLINK-5323 Project: Flink Issue Type: Bu

Re: [DISCUSS] Schedule and Scope for Flink 1.2

2016-12-12 Thread Robert Metzger
Thank you all for figuring out a solution for the security pull request. Lets try to get 1.2 feature freezed as fast as possible so that we can "unblock" waiting features like FLIP-6 and the remaining security changes. *What do you think about Friday evening (6pm Berlin, 9am US west coast) for f

[jira] [Created] (FLINK-5322) yarn.taskmanager.env value does not appear in System.getenv

2016-12-12 Thread Shannon Carey (JIRA)
Shannon Carey created FLINK-5322: Summary: yarn.taskmanager.env value does not appear in System.getenv Key: FLINK-5322 URL: https://issues.apache.org/jira/browse/FLINK-5322 Project: Flink Is

Re: trouble using maven to deploy master branch to a private repo

2016-12-12 Thread Robert Metzger
I checked the flink-clients/pom.xml file and I could not find any recent changes that could cause such an error. However, the only other project that is using the create-test-dependency goal is the flink-batch connectors project, which has been removed in master recently. It could be that this cha

Re: [DISCUSS] Schedule and Scope for Flink 1.2

2016-12-12 Thread Vijay
Hi Stephan, The new handler changes for Netty layer is straightforward and it is not really an workaround. Moreover if we think there is a better way to handle it in future we can easily unwind it as it is just a pluggable handler. Pushing these changes before FLIP-6 merge certainly avoids lot

[jira] [Created] (FLINK-5321) FlinkMiniCluster does not start Jobmanager MetricQueryService

2016-12-12 Thread Chesnay Schepler (JIRA)
Chesnay Schepler created FLINK-5321: --- Summary: FlinkMiniCluster does not start Jobmanager MetricQueryService Key: FLINK-5321 URL: https://issues.apache.org/jira/browse/FLINK-5321 Project: Flink

[jira] [Created] (FLINK-5320) Fix result TypeInformation in WindowedStream.fold(ACC, FoldFunction, WindowFunction)

2016-12-12 Thread Yassine Marzougui (JIRA)
Yassine Marzougui created FLINK-5320: Summary: Fix result TypeInformation in WindowedStream.fold(ACC, FoldFunction, WindowFunction) Key: FLINK-5320 URL: https://issues.apache.org/jira/browse/FLINK-5320

Re: [DISCUSS] Schedule and Scope for Flink 1.2

2016-12-12 Thread Stephan Ewen
Hi Vijay! The workaround you suggest may be doable, but I am wondering how much that helps, because the authorization feature would be incomplete like that and thus of limited use. I would also assume that merging it properly and in full use after the 1.2 release would be a bit better - in genera

[jira] [Created] (FLINK-5319) ClassCastException when reusing an inherited method as KeySelector for different classes

2016-12-12 Thread Alexander Chermenin (JIRA)
Alexander Chermenin created FLINK-5319: -- Summary: ClassCastException when reusing an inherited method as KeySelector for different classes Key: FLINK-5319 URL: https://issues.apache.org/jira/browse/FLINK-5319

Re: Reg. custom sinks in Flink

2016-12-12 Thread Meghashyam Sandeep V
Hi Till, Thanks for the information. 1. What do you mean by 'subtask', is it every partition or every message in the stream? 2. I tried using CassandraSink with a Pojo. Is there a way to specify TTL as I can't use a query when I have a datastream with Pojo? CassandraSink.addSink(messageStream)

Re: trouble using maven to deploy master branch to a private repo

2016-12-12 Thread Aljoscha Krettek
Hi Steve, I think this part of create_release_files.sh (our release script) is helpful: mvn clean deploy -Prelease,docs-and-source --settings deploysettings.xml -DskipTests -Dgpg.executable=$GPG -Dgpg.keyname=$GPG_KEY -Dgpg.passphrase=$GPG_PASSPHRASE -DretryFailedDeploymentCount=10 I think -Prele

[jira] [Created] (FLINK-5318) Make the Rolling/Bucketing sink backwards compatible.

2016-12-12 Thread Kostas Kloudas (JIRA)
Kostas Kloudas created FLINK-5318: - Summary: Make the Rolling/Bucketing sink backwards compatible. Key: FLINK-5318 URL: https://issues.apache.org/jira/browse/FLINK-5318 Project: Flink Issue T

[jira] [Created] (FLINK-5317) Make the continuous file processing code backwards compatible,

2016-12-12 Thread Kostas Kloudas (JIRA)
Kostas Kloudas created FLINK-5317: - Summary: Make the continuous file processing code backwards compatible, Key: FLINK-5317 URL: https://issues.apache.org/jira/browse/FLINK-5317 Project: Flink

[jira] [Created] (FLINK-5316) Make the GenericWriteAheadSink backwards compatible.

2016-12-12 Thread Kostas Kloudas (JIRA)
Kostas Kloudas created FLINK-5316: - Summary: Make the GenericWriteAheadSink backwards compatible. Key: FLINK-5316 URL: https://issues.apache.org/jira/browse/FLINK-5316 Project: Flink Issue Ty

Re: [VOTE] Release Apache Flink 1.1.4 (RC2)

2016-12-12 Thread Gyula Fóra
Hi Ufuk, Thanks for the RC and all the effort that went into the bugfixes :) I have tested the following things: - Built from source (mvn clean install) - Tested savepoint/restore on both YARN (+HDFS) and standalone - Ran heavy production jobs with pretty much the same build (- the last few co

Re: [DISCUSS] Schedule and Scope for Flink 1.2

2016-12-12 Thread Vijay
Max and Ufuk, I respect your concerns and fully understand the importance of the network layer stack in Flink code base. Will you be comfortable to merge the code if I remove the Netty layer changes and leave the rest of the code. We can address the Netty code changes post 1.2 release? Regards,

Re: How to merge messages from all partitions

2016-12-12 Thread Till Rohrmann
Hi Naveen, you would have to apply an all reduce after you’ve aggregated the values by key. metrics... timeWindowAll(Time.seconds(30)).fold(..., ...); However, be aware that this all window fold operation will be executed by a single operator (parallelism 1). You could also do this all window o

Re: [DISCUSS] Schedule and Scope for Flink 1.2

2016-12-12 Thread Ufuk Celebi
On 12 December 2016 at 12:30:31, Maximilian Michels (m...@apache.org) wrote: > > It seems like we lack the resources for now to properly to take > care > of your pull request before the release. Unless someone from > the > community is really eager to help out here, I would be in favor > of >

Re: [DISCUSS] Schedule and Scope for Flink 1.2

2016-12-12 Thread Maximilian Michels
Hi Vijay, Thank you for updating the pull request. I appreciate your work in the security realm of Flink and value your contributions so far. It would be great to merge the authorization pull request for the release. However, I don't feel comfortable about the network stack (i.e. Netty) related ch

Re: Reg. custom sinks in Flink

2016-12-12 Thread Till Rohrmann
Hi Meghashyam, 1. You can perform initializations in the open method of the RichSinkFunction interface. The open method will be called once for every sub task when initializing it. If you want to share the resource across multiple sub tasks running in the same JVM you can also stor

Re: [DISCUSS] FLIP-14: Loops API and Termination

2016-12-12 Thread Paris Carbone
Hi again folks, It has been almost a month so I would like to remind you about the loop redesign which many of you have checked already. Anyone else excited about iterative streams and wants to add a comment or simply +1 this? It would be more than welcome :) You can find the description in the

[VOTE] Release Apache Flink 1.1.4 (RC2)

2016-12-12 Thread Ufuk Celebi
Dear Flink community, Please vote on releasing the following candidate as Apache Flink version 1.1.4. The commit to be voted on: a587fe9 (http://git-wip-us.apache.org/repos/asf/flink/commit/a587fe9) Branch: release-1.1.4-rc2 (https://git1-us-west.apache.org/repos/asf/flink/repo?p=flink.git;a=sho