Re: Map Reduce Sorting

2016-08-02 Thread Hilmi Yildirim
Hi Fabian, thank you very much! This answers my question. BR, Hilmi Am 01.08.2016 um 22:29 schrieb Fabian Hueske: Hi Hilmi, the results of the combiner are usually not completely sorted and if they are this property is not leveraged. This is due to the following reasons: 1) a sort-combiner

Re: [VOTE] Release Apache Flink 1.1.0 (RC1)

2016-08-02 Thread Till Rohrmann
I can confirm Aljoscha's findings concerning building Flink with Hadoop version 2.6.0 using Maven 3.3.9. Aljoscha is right that it is indeed a Maven 3.3 issue. If you build flink-runtime twice, then everything goes through because the shaded curator Flink dependency is installed in during the first

Re: [VOTE] Release Apache Flink 1.1.0 (RC1)

2016-08-02 Thread Ufuk Celebi
Dear community, I would like to vote +1, but during testing I've noted that we should have reverted FLINK-4154 (correction of murmur hash) for this release. We had a wrong murmur hash implementation for 1.0, which was fixed for 1.1. We reverted that fix, because we thought that it broke savepoint

[jira] [Created] (FLINK-4304) Jar names that contain whitespace cause problems in web client

2016-08-02 Thread Timo Walther (JIRA)
Timo Walther created FLINK-4304: --- Summary: Jar names that contain whitespace cause problems in web client Key: FLINK-4304 URL: https://issues.apache.org/jira/browse/FLINK-4304 Project: Flink I

Re: [DISCUSS] FLIP-3 - Organization of Documentation

2016-08-02 Thread Ufuk Celebi
If there are no objections, I would like to work on this in the next days. I would like to only do the restructuring and don't add any new content (e.g. we would have a few empty pages in the beginning). On Wed, Jul 20, 2016 at 9:57 PM, Stephan Ewen wrote: > I added to the "Application Developmen

[jira] [Created] (FLINK-4305) Build pull requests with Jenkins

2016-08-02 Thread Maximilian Michels (JIRA)
Maximilian Michels created FLINK-4305: - Summary: Build pull requests with Jenkins Key: FLINK-4305 URL: https://issues.apache.org/jira/browse/FLINK-4305 Project: Flink Issue Type: Task

Re: [DISCUSS] FLIP-3 - Organization of Documentation

2016-08-02 Thread Stephan Ewen
+1, thanks :-) On Tue, Aug 2, 2016 at 11:39 AM, Ufuk Celebi wrote: > If there are no objections, I would like to work on this in the next > days. I would like to only do the restructuring and don't add any new > content (e.g. we would have a few empty pages in the beginning). > > On Wed, Jul 20,

Re: [DISCUSS] FLIP-3 - Organization of Documentation

2016-08-02 Thread Till Rohrmann
+1 :-) On Tue, Aug 2, 2016 at 6:09 PM, Stephan Ewen wrote: > +1, thanks :-) > > On Tue, Aug 2, 2016 at 11:39 AM, Ufuk Celebi wrote: > > > If there are no objections, I would like to work on this in the next > > days. I would like to only do the restructuring and don't add any new > > content (e

Re: [VOTE] Release Apache Flink 1.1.0 (RC1)

2016-08-02 Thread Stephan Ewen
+1 from my side Create a new RC that differs only in the hash function commit. I would support to carry forward the vote thread (extend it for one additional day), because virtually all test results should apply to the new RC as well. We certainly need to redo: - signature validation - Build

Re: [VOTE] Release Apache Flink 1.1.0 (RC1)

2016-08-02 Thread Maximilian Michels
I agree with Ufuk and Stephan that we could forward most of the testing if we only included the hash function fix in the new RC. There are some other minor issues we could merge as well, but they are involved enough that they would set us back to redoing the testing. So +1 for a new RC with the has

Re: [VOTE] Release Apache Flink 1.1.0 (RC1)

2016-08-02 Thread Ufuk Celebi
I just saw that we changed the behaviour of ListState and FoldingState. They used to return the default value given to the state descriptor, but have been changed to return null now (in [1]). Furthermore ValueState still returns the default value instead of null. Gyula noticed another inconsistency

Re: [VOTE] Release Apache Flink 1.1.0 (RC1)

2016-08-02 Thread Stephan Ewen
@Ufuk - I agree, this looks quite dubious. Need to resolve that before proceeding with the release... On Tue, Aug 2, 2016 at 1:45 PM, Ufuk Celebi wrote: > I just saw that we changed the behaviour of ListState and > FoldingState. They used to return the default value given to the state > descri

[jira] [Created] (FLINK-4306) Fix Flink and Storm dependencies in flink-storm and flink-storm-examples

2016-08-02 Thread Stephan Ewen (JIRA)
Stephan Ewen created FLINK-4306: --- Summary: Fix Flink and Storm dependencies in flink-storm and flink-storm-examples Key: FLINK-4306 URL: https://issues.apache.org/jira/browse/FLINK-4306 Project: Flink

Re: [DISCUSS] FLIP-7 Expose metrics to WebInterface

2016-08-02 Thread Stephan Ewen
Hi! Thanks for writing this up. I think it looks quite reasonable (I hope I understood that design correctly) There is one point of confusions left for me, though: The MetricDumper and MetricSnapshot: I think it is just the names that confuse me here. It looks like they define a way to query the

Re: [DISCUSS] FLIP-7 Expose metrics to WebInterface

2016-08-02 Thread Till Rohrmann
The metrics transfer design document looks good to me. Thanks for your work Chesnay :-) I think the benefit of registering the metrics at the MetricDumper is that we don't have to walk through the hierarchy of metric groups to collect the metric values. Indeed, this comes with increased costs at s

Re: [DISCUSS] FLIP-7 Expose metrics to WebInterface

2016-08-02 Thread Chesnay Schepler
Thank you for your feedback :) Regarding names: The Dumper does not create a MetricSnapshot. The Dumper creates a list of key-value pairs; metric_name:value. A (single) MetricSnapshot exists in the WebRuntimeMonitor, into which the dumped list is inserted. So the dumper creates a

Re: [DISCUSS] FLIP-7 Expose metrics to WebInterface

2016-08-02 Thread Stephan Ewen
Regarding transfer: I think objects are fine, as long as they are not user-defined objects. We can limit it to String and subclasses of Number. Regarding traversal of groups: I am still thinking here in terms of the paradigm that the metrics should impact the regular system as little as possible.

Re: [DISCUSS] FLIP-7 Expose metrics to WebInterface

2016-08-02 Thread Chesnay Schepler
Names are only constructed once; when the dumper is notified of the new metric. We don't have to use scopeFormats; we can hard-code the scope assembly into the ComponentMetricGroups. This way we don't need special behavior in the dumper. On 02.08.2016 16:29, Stephan Ewen wrote: Regarding tr

Re: [DISCUSS] FLIP-3 - Organization of Documentation

2016-08-02 Thread Aljoscha Krettek
+1 On Tue, 2 Aug 2016 at 03:15 Till Rohrmann wrote: > +1 :-) > > On Tue, Aug 2, 2016 at 6:09 PM, Stephan Ewen wrote: > > > +1, thanks :-) > > > > On Tue, Aug 2, 2016 at 11:39 AM, Ufuk Celebi wrote: > > > > > If there are no objections, I would like to work on this in the next > > > days. I wou

[jira] [Created] (FLINK-4307) Broken user-facing API for ListState

2016-08-02 Thread Stephan Ewen (JIRA)
Stephan Ewen created FLINK-4307: --- Summary: Broken user-facing API for ListState Key: FLINK-4307 URL: https://issues.apache.org/jira/browse/FLINK-4307 Project: Flink Issue Type: Bug Co

[jira] [Created] (FLINK-4308) Allow uploaded jar directory to be configurable

2016-08-02 Thread Zhenzhong Xu (JIRA)
Zhenzhong Xu created FLINK-4308: --- Summary: Allow uploaded jar directory to be configurable Key: FLINK-4308 URL: https://issues.apache.org/jira/browse/FLINK-4308 Project: Flink Issue Type: Impr

[RESULT] [VOTE] Release Apache Flink 1.1.0 (RC1)

2016-08-02 Thread Ufuk Celebi
This vote has been cancelled in favour of RC2. On Tue, Aug 2, 2016 at 1:51 PM, Stephan Ewen wrote: > @Ufuk - I agree, this looks quite dubious. > > Need to resolve that before proceeding with the release... > > > On Tue, Aug 2, 2016 at 1:45 PM, Ufuk Celebi wrote: > >> I just saw that we changed

[VOTE] Release Apache Flink 1.1.0 (RC2)

2016-08-02 Thread Ufuk Celebi
Dear Flink community, Please vote on releasing the following candidate as Apache Flink version 1.1.0. The commit to be voted on: 45f7825 (http://git-wip-us.apache.org/repos/asf/flink/commit/45f7825) Branch: release-1.1.0-rc2 (https://git1-us-west.apache.org/repos/asf/flink/repo?p=flink.git;a=sho

[jira] [Created] (FLINK-4309) Potential null pointer dereference in DelegatingConfiguration#keySet()

2016-08-02 Thread Ted Yu (JIRA)
Ted Yu created FLINK-4309: - Summary: Potential null pointer dereference in DelegatingConfiguration#keySet() Key: FLINK-4309 URL: https://issues.apache.org/jira/browse/FLINK-4309 Project: Flink Issue