[GitHub] flink pull request #4939: [FLINK-4228][yarn/s3a] fix yarn resource upload s3...

2017-11-02 Thread NicoK
GitHub user NicoK opened a pull request: https://github.com/apache/flink/pull/4939 [FLINK-4228][yarn/s3a] fix yarn resource upload s3a defaultFs ## What is the purpose of the change If YARN is configured to use the `s3a` default file system, upload of the Flink jars will

[GitHub] flink pull request #4939: [FLINK-4228][yarn/s3a] fix yarn resource upload s3...

2017-11-03 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4939#discussion_r148756295 --- Diff: flink-filesystems/flink-s3-fs-hadoop/src/test/java/org/apache/flink/fs/s3hadoop/HadoopS3FileSystemITCase.java --- @@ -57,11 +62,52

[GitHub] flink pull request #4939: [FLINK-4228][yarn/s3a] fix yarn resource upload s3...

2017-11-03 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4939#discussion_r148756933 --- Diff: flink-filesystems/flink-s3-fs-hadoop/pom.xml --- @@ -182,6 +182,21 @@ under the License. ${project.version

[GitHub] flink pull request #4759: [FLINK-7746][network] move ResultPartitionWriter#w...

2017-11-04 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4759#discussion_r148944363 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/ResultPartition.java --- @@ -293,6 +293,27 @@ public void add(Buffer

[GitHub] flink issue #4939: [FLINK-4228][yarn/s3a] fix yarn resource upload s3a defau...

2017-11-04 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/4939 ok, finally, this seems to be in shape My try to fool Travis in running for every commit though resulted in all those failed runs (which should really be read as "cancelled" since

[GitHub] flink pull request #4950: [hotfix][s3] fix HadoopS3FileSystemITCase leaving ...

2017-11-06 Thread NicoK
GitHub user NicoK opened a pull request: https://github.com/apache/flink/pull/4950 [hotfix][s3] fix HadoopS3FileSystemITCase leaving test directories behind in S3 ## What is the purpose of the change This PR fixes `HadoopS3FileSystemITCase` leaving a test directory behind

[GitHub] flink pull request #4951: [FLINK-7989][yarn] do not deploy the dist.jar file...

2017-11-06 Thread NicoK
GitHub user NicoK opened a pull request: https://github.com/apache/flink/pull/4951 [FLINK-7989][yarn] do not deploy the dist.jar file twice ## What is the purpose of the change We always add the dist.jar ourselves, but it could also be inside a shipped folder such as the

[GitHub] flink pull request #4952: [FLINK-7992][docs] extend the PR template asking f...

2017-11-06 Thread NicoK
GitHub user NicoK opened a pull request: https://github.com/apache/flink/pull/4952 [FLINK-7992][docs] extend the PR template asking for any S3 relation ## What is the purpose of the change This adds a question to the PR template whether a PR is S3-related because this

[GitHub] flink pull request #4952: [FLINK-7992][docs] extend the PR template asking f...

2017-11-06 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4952#discussion_r149070730 --- Diff: .github/PULL_REQUEST_TEMPLATE.md --- @@ -63,9 +63,9 @@ This change added tests and can be verified as follows: - The serializers: (yes / no

[GitHub] flink pull request #4951: [FLINK-7989][yarn] do not deploy the dist.jar file...

2017-11-06 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4951#discussion_r149071640 --- Diff: flink-yarn/src/main/java/org/apache/flink/yarn/AbstractYarnClusterDescriptor.java --- @@ -186,6 +186,8 @@ public void addShipFiles(List shipFiles

[GitHub] flink pull request #4939: [FLINK-4228][yarn/s3a] fix yarn resource upload s3...

2017-11-06 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4939#discussion_r149076237 --- Diff: flink-yarn/pom.xml --- @@ -99,6 +99,13 @@ under the License. + org.apache.flink

[GitHub] flink pull request #4939: [FLINK-4228][yarn/s3a] fix yarn resource upload s3...

2017-11-06 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4939#discussion_r149077620 --- Diff: flink-yarn/src/main/java/org/apache/flink/yarn/Utils.java --- @@ -117,27 +118,50 @@ public static void setupYarnClassPath(Configuration conf, Map ap

[GitHub] flink pull request #4939: [FLINK-4228][yarn/s3a] fix yarn resource upload s3...

2017-11-06 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4939#discussion_r149077720 --- Diff: flink-yarn/src/main/java/org/apache/flink/yarn/Utils.java --- @@ -117,27 +118,50 @@ public static void setupYarnClassPath(Configuration conf, Map ap

[GitHub] flink pull request #4939: [FLINK-4228][yarn/s3a] fix yarn resource upload s3...

2017-11-06 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4939#discussion_r149078183 --- Diff: flink-yarn/src/main/java/org/apache/flink/yarn/Utils.java --- @@ -408,10 +437,12 @@ static ContainerLaunchContext createTaskExecutorContext

[GitHub] flink issue #4759: [FLINK-7746][network] move ResultPartitionWriter#writeBuf...

2017-11-06 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/4759 agreed - we could merge it after the other two PRs are ready to merge ---

[GitHub] flink pull request #4961: [FLINK-7973] fix shading and relocating Hhadoop fo...

2017-11-06 Thread NicoK
GitHub user NicoK opened a pull request: https://github.com/apache/flink/pull/4961 [FLINK-7973] fix shading and relocating Hhadoop for the S3 filesystems ## What is the purpose of the change The current shading of the `flink-s3-fs-hadoop` and `flink-s3-fs-presto` projects

[GitHub] flink issue #4939: [FLINK-4228][yarn/s3a] fix yarn resource upload s3a defau...

2017-11-06 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/4939 I addresses the PR comments in my latest commit but I will also have to adapt the S3 test to not use our S3 implementation and instead the one in the Hadoop version YARN uses. Stay tuned ;) ---

[GitHub] flink pull request #4961: [FLINK-7973] fix shading and relocating Hadoop for...

2017-11-07 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4961#discussion_r149449571 --- Diff: flink-filesystems/flink-s3-fs-hadoop/pom.xml --- @@ -33,6 +33,7 @@ under the License. jar + --- End diff

[GitHub] flink issue #4961: [FLINK-7973] fix shading and relocating Hadoop for the S3...

2017-11-08 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/4961 looks like I created a new word: `fute` ;) - fixed the typos both of you found ---

[GitHub] flink issue #4939: [FLINK-4228][yarn/s3a] fix yarn resource upload s3a defau...

2017-11-08 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/4939 @tillrohrmann can you have a second look at the changes? (Travis already gave a green light on my branch with S3 tests enabled, the PR results will give the results without S3 tests due to missing

[GitHub] flink issue #4951: [FLINK-7989][yarn] do not deploy the dist.jar file twice

2017-11-08 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/4951 (rebased onto the newest #4939 which should fix the test failures shown by Travis on the previous build here) ---

[GitHub] flink issue #4961: [FLINK-7973] fix shading and relocating Hadoop for the S3...

2017-11-08 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/4961 unfortunately, it seems we cannot minimize the shaded `jar` or it will not contain non-imported/dynamically-loaded classes like this one ---

[GitHub] flink pull request #4594: [FLINK-7517][network] let NettyBufferPool extend P...

2017-11-09 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4594#discussion_r149889426 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/netty/NettyBufferPool.java --- @@ -52,51 +48,61 @@ /** Configured chunk

[GitHub] flink issue #4939: [FLINK-4228][yarn/s3a] fix yarn resource upload s3a defau...

2017-11-10 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/4939 1) yes, I have tested the changes on a YARN cluster as described in the PR comment (before the changes on the unit tests) 2) regarding the `pom.xml` changes: I actually took those from

[GitHub] flink pull request #4939: [FLINK-4228][yarn/s3a] fix yarn resource upload s3...

2017-11-10 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4939#discussion_r150275161 --- Diff: flink-yarn/pom.xml --- @@ -153,6 +159,63 @@ under the License

[GitHub] flink pull request #4939: [FLINK-4228][yarn/s3a] fix yarn resource upload s3...

2017-11-10 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4939#discussion_r150275387 --- Diff: flink-yarn/pom.xml --- @@ -153,6 +159,63 @@ under the License

[GitHub] flink issue #4961: [FLINK-7973] fix shading and relocating Hadoop for the S3...

2017-11-10 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/4961 I did include your end-to-end tests (with some fixes) and the fixes for the errors they found. Should be fine now, let's see what travis says... ---

[GitHub] flink issue #4961: [FLINK-7973] fix shading and relocating Hadoop for the S3...

2017-11-13 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/4961 Fixed on master in 0e5fb0b78cd0a3ccb144071a47579eb6c3d0570a e9e7c3372189db7e933ff59114b9ec6245838eda Fixed on release-1.4 in 25a28ab32609c45fb8c40f717148e32fb453d2fc

[GitHub] flink pull request #4961: [FLINK-7973] fix shading and relocating Hadoop for...

2017-11-13 Thread NicoK
Github user NicoK closed the pull request at: https://github.com/apache/flink/pull/4961 ---

[GitHub] flink pull request #5007: [FLINK-7845] Make NettyMessage public

2017-11-13 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/5007#discussion_r150615019 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/netty/NettyMessage.java --- @@ -52,8 +52,10 @@ /** * A simple and

[GitHub] flink pull request #5007: [FLINK-7845] Make NettyMessage public

2017-11-14 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/5007#discussion_r150775237 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/netty/NettyMessage.java --- @@ -52,8 +52,10 @@ /** * A simple and

[GitHub] flink pull request #5013: [FLINK-7973] disable JNI bridge for relocated hado...

2017-11-14 Thread NicoK
GitHub user NicoK opened a pull request: https://github.com/apache/flink/pull/5013 [FLINK-7973] disable JNI bridge for relocated hadoop classes in s3-fs-* ## What is the purpose of the change If some Hadoop's JNI library is in the classpath, it will be loaded by our s

[GitHub] flink issue #5013: [FLINK-7973] disable JNI bridge for relocated hadoop clas...

2017-11-14 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/5013 This would be difficult to test - the most reasonable test would be to put the Hadoop JNI into the classpath and then run the end-to-end test on that but we don't want to include the pla

[GitHub] flink issue #5013: [FLINK-7973] disable JNI bridge for relocated hadoop clas...

2017-11-17 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/5013 Fixed on master in 0e5fb0b78cd0a3ccb144071a47579eb6c3d0570a e9e7c3372189db7e933ff59114b9ec6245838eda Fixed on release-1.4 in 25a28ab32609c45fb8c40f717148e32fb453d2fc

[GitHub] flink pull request #5013: [FLINK-7973] disable JNI bridge for relocated hado...

2017-11-17 Thread NicoK
Github user NicoK closed the pull request at: https://github.com/apache/flink/pull/5013 ---

[GitHub] flink issue #4939: [FLINK-4228][yarn/s3a] fix yarn resource upload s3a defau...

2017-11-17 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/4939 I'm actually with Chesnay on the part of using exclusions (and letting AWS pull them in with the right version) rather than adding explicit dependencies because it will be easier to debug th

[GitHub] flink pull request #5029: [FLINK-7841][docs] update AWS docs with respect to...

2017-11-17 Thread NicoK
GitHub user NicoK opened a pull request: https://github.com/apache/flink/pull/5029 [FLINK-7841][docs] update AWS docs with respect to S3 file system changes ## What is the purpose of the change This updates the S3 file system configuration section in the AWS documentation

[GitHub] flink issue #4951: [FLINK-7989][yarn] do not deploy the dist.jar file twice

2017-11-20 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/4951 #4939 has been merged now - I rebased this PR now so we can go ahead ---

[GitHub] flink issue #5029: [FLINK-7841][docs] update AWS docs with respect to S3 fil...

2017-11-20 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/5029 apparently, the `aws-java-sdk-.jar` has been split into multiple (separate) jar files now - I'll update this part, too, in a moment... ---

[GitHub] flink issue #4951: [FLINK-7989][yarn] do not deploy the dist.jar file twice

2017-11-21 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/4951 FYI: travis errors are unrelated (ERROR 404 during download of http://mirror.netcologne.de/apache.org/kafka/0.10.2.0/kafka_2.11-0.10.2.0.tgz) ---

[GitHub] flink issue #5029: [FLINK-7841][docs] update AWS docs with respect to S3 fil...

2017-11-21 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/5029 FYI: travis errors are unrelated (ERROR 404 during download of http://mirror.netcologne.de/apache.org/kafka/0.10.2.0/kafka_2.11-0.10.2.0.tgz) ---

[GitHub] flink pull request #4509: [FLINK-7406][network] Implement Netty receiver inc...

2017-11-21 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4509#discussion_r152245491 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannel.java --- @@ -240,15 +245,13 @@ void

[GitHub] flink pull request #4509: [FLINK-7406][network] Implement Netty receiver inc...

2017-11-21 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4509#discussion_r152280350 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannelTest.java --- @@ -301,81 +306,388 @@ public

[GitHub] flink pull request #4509: [FLINK-7406][network] Implement Netty receiver inc...

2017-11-21 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4509#discussion_r152239228 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannel.java --- @@ -462,4 +554,42 @@ public String

[GitHub] flink pull request #4509: [FLINK-7406][network] Implement Netty receiver inc...

2017-11-21 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4509#discussion_r152242064 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannel.java --- @@ -134,9 +138,9 @@ void

[GitHub] flink pull request #4509: [FLINK-7406][network] Implement Netty receiver inc...

2017-11-21 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4509#discussion_r152294000 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannelTest.java --- @@ -301,81 +306,388 @@ public

[GitHub] flink pull request #4509: [FLINK-7406][network] Implement Netty receiver inc...

2017-11-21 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4509#discussion_r152239190 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannel.java --- @@ -462,4 +554,42 @@ public String

[GitHub] flink pull request #4509: [FLINK-7406][network] Implement Netty receiver inc...

2017-11-21 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4509#discussion_r152252414 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannel.java --- @@ -306,62 +314,79 @@ public void

[GitHub] flink pull request #4509: [FLINK-7406][network] Implement Netty receiver inc...

2017-11-21 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4509#discussion_r152291537 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannelTest.java --- @@ -301,81 +306,388 @@ public

[GitHub] flink pull request #4509: [FLINK-7406][network] Implement Netty receiver inc...

2017-11-21 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4509#discussion_r152292548 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannelTest.java --- @@ -301,81 +306,388 @@ public

[GitHub] flink pull request #4509: [FLINK-7406][network] Implement Netty receiver inc...

2017-11-21 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4509#discussion_r152239410 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannel.java --- @@ -462,4 +554,42 @@ public String

[GitHub] flink pull request #4509: [FLINK-7406][network] Implement Netty receiver inc...

2017-11-21 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4509#discussion_r152255026 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannel.java --- @@ -82,17 +84,19 @@ /** The

[GitHub] flink pull request #4509: [FLINK-7406][network] Implement Netty receiver inc...

2017-11-21 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4509#discussion_r152255421 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannel.java --- @@ -394,7 +419,63 @@ public

[GitHub] flink pull request #4509: [FLINK-7406][network] Implement Netty receiver inc...

2017-11-21 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4509#discussion_r152290197 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannelTest.java --- @@ -301,81 +306,388 @@ public

[GitHub] flink pull request #4509: [FLINK-7406][network] Implement Netty receiver inc...

2017-11-21 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4509#discussion_r152279228 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannel.java --- @@ -394,7 +419,63 @@ public

[GitHub] flink pull request #4509: [FLINK-7406][network] Implement Netty receiver inc...

2017-11-21 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4509#discussion_r152234362 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannel.java --- @@ -287,17 +290,22 @@ void

[GitHub] flink pull request #4581: [FLINK-7499][io] also let AsynchronousBufferFileWr...

2017-11-21 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4581#discussion_r152342364 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/iomanager/AsynchronousBufferFileWriter.java --- @@ -31,9 +31,26 @@ protected

[GitHub] flink pull request #4581: [FLINK-7499][io] also let AsynchronousBufferFileWr...

2017-11-21 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4581#discussion_r152342531 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/SpillableSubpartitionView.java --- @@ -108,11 +108,7 @@ int releaseMemory

[GitHub] flink issue #4481: [FLINK-7316][network] always use off-heap network buffers

2017-11-21 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/4481 rebased again - this should be good to go. @StefanRRichter can you continue with this? ---

[GitHub] flink pull request #4581: [FLINK-7499][io] also let AsynchronousBufferFileWr...

2017-11-23 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4581#discussion_r152813605 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/SpillableSubpartitionTest.java --- @@ -192,21 +198,39 @@ public void

[GitHub] flink pull request #4581: [FLINK-7499][io] also let AsynchronousBufferFileWr...

2017-11-23 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4581#discussion_r152813679 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/SpillableSubpartitionTest.java --- @@ -192,21 +198,39 @@ public void

[GitHub] flink pull request #4581: [FLINK-7499][io] also let AsynchronousBufferFileWr...

2017-11-23 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4581#discussion_r152813796 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/SpillableSubpartitionTest.java --- @@ -192,21 +198,39 @@ public void

[GitHub] flink pull request #4581: [FLINK-7499][io] also let AsynchronousBufferFileWr...

2017-11-23 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4581#discussion_r152813899 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/SpillableSubpartitionTest.java --- @@ -192,21 +198,39 @@ public void

[GitHub] flink issue #4581: [FLINK-7499][io] also let AsynchronousBufferFileWriter#wr...

2017-11-23 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/4581 ok, I extracted two hotfix commits not directly related to this fix (one for the added test checks you found, another one for fixing additional test cases) and made a follow-up commit with the changes

[GitHub] flink issue #4761: [FLINK-7748][network] properly use the TaskEventDispatche...

2017-11-23 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/4761 alright - didn't even need to do anything here since git auto-merged them :) ---

[GitHub] flink pull request #4581: [FLINK-7499][io] fix double buffer release in Spil...

2017-11-23 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4581#discussion_r152841030 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/ResultPartition.java --- @@ -267,28 +268,29 @@ public ResultPartitionType

[GitHub] flink pull request #4581: [FLINK-7499][io] fix double buffer release in Spil...

2017-11-23 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4581#discussion_r152842686 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/SpillableSubpartition.java --- @@ -95,20 +95,23 @@ public boolean add

[GitHub] flink pull request #4581: [FLINK-7499][io] fix double buffer release in Spil...

2017-11-23 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4581#discussion_r152845922 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/ResultPartitionTest.java --- @@ -145,6 +165,45 @@ protected void

[GitHub] flink pull request #4581: [FLINK-7499][io] fix double buffer release in Spil...

2017-11-23 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4581#discussion_r152847273 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/SpillableSubpartitionTest.java --- @@ -300,6 +315,230 @@ public void

[GitHub] flink pull request #4581: [FLINK-7499][io] fix double buffer release in Spil...

2017-11-23 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4581#discussion_r152849113 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/SpillableSubpartitionTest.java --- @@ -300,6 +315,230 @@ public void

[GitHub] flink pull request #4581: [FLINK-7499][io] fix double buffer release in Spil...

2017-11-23 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4581#discussion_r152849820 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/SpillableSubpartitionTest.java --- @@ -300,6 +315,230 @@ public void

[GitHub] flink pull request #4581: [FLINK-7499][io] fix double buffer release in Spil...

2017-11-23 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4581#discussion_r152850290 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/SpillableSubpartitionTest.java --- @@ -320,4 +559,40 @@ void

[GitHub] flink issue #4581: [FLINK-7499][io] fix double buffer release in SpillableSu...

2017-11-23 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/4581 alright - using fixup commits now for you ;) FYI: all `[FLINK-7499]` belong together and can be squashed ---

[GitHub] flink pull request #4509: [FLINK-7406][network] Implement Netty receiver inc...

2017-11-23 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4509#discussion_r152854762 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannelTest.java --- @@ -306,51 +306,88 @@ public

[GitHub] flink pull request #4509: [FLINK-7406][network] Implement Netty receiver inc...

2017-11-23 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4509#discussion_r152859784 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannelTest.java --- @@ -715,4 +686,58 @@ private

[GitHub] flink pull request #4509: [FLINK-7406][network] Implement Netty receiver inc...

2017-11-23 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4509#discussion_r152860104 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannelTest.java --- @@ -301,81 +306,388 @@ public

[GitHub] flink pull request #4509: [FLINK-7406][network] Implement Netty receiver inc...

2017-11-23 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4509#discussion_r152853208 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannel.java --- @@ -572,20 +560,53 @@ void

[GitHub] flink pull request #4509: [FLINK-7406][network] Implement Netty receiver inc...

2017-11-23 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4509#discussion_r152859069 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannelTest.java --- @@ -301,81 +306,388 @@ public

[GitHub] flink pull request #4509: [FLINK-7406][network] Implement Netty receiver inc...

2017-11-23 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4509#discussion_r152857580 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannelTest.java --- @@ -306,51 +306,88 @@ public

[GitHub] flink pull request #4509: [FLINK-7406][network] Implement Netty receiver inc...

2017-11-23 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4509#discussion_r152836741 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannel.java --- @@ -572,20 +560,53 @@ void

[GitHub] flink pull request #4509: [FLINK-7406][network] Implement Netty receiver inc...

2017-11-23 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4509#discussion_r152852133 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannel.java --- @@ -572,20 +560,53 @@ void

[GitHub] flink pull request #4509: [FLINK-7406][network] Implement Netty receiver inc...

2017-11-23 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4509#discussion_r152856498 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannelTest.java --- @@ -306,51 +306,88 @@ public

[GitHub] flink pull request #4509: [FLINK-7406][network] Implement Netty receiver inc...

2017-11-23 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4509#discussion_r152855823 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannelTest.java --- @@ -306,51 +306,88 @@ public

[GitHub] flink pull request #4509: [FLINK-7406][network] Implement Netty receiver inc...

2017-11-23 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4509#discussion_r152859769 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannelTest.java --- @@ -715,4 +686,58 @@ private

[GitHub] flink pull request #4509: [FLINK-7406][network] Implement Netty receiver inc...

2017-11-23 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4509#discussion_r152858602 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannelTest.java --- @@ -306,51 +306,88 @@ public

[GitHub] flink pull request #4509: [FLINK-7406][network] Implement Netty receiver inc...

2017-11-23 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4509#discussion_r152859775 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannelTest.java --- @@ -715,4 +686,58 @@ private

[GitHub] flink pull request #4509: [FLINK-7406][network] Implement Netty receiver inc...

2017-11-23 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4509#discussion_r152860980 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannelTest.java --- @@ -506,36 +540,22 @@ public

[GitHub] flink pull request #4509: [FLINK-7406][network] Implement Netty receiver inc...

2017-11-23 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4509#discussion_r152861029 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannelTest.java --- @@ -440,21 +476,25 @@ public

[GitHub] flink pull request #4509: [FLINK-7406][network] Implement Netty receiver inc...

2017-11-23 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4509#discussion_r152861040 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannelTest.java --- @@ -465,7 +505,7 @@ public Void

[GitHub] flink pull request #4509: [FLINK-7406][network] Implement Netty receiver inc...

2017-11-23 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4509#discussion_r152862469 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannelTest.java --- @@ -475,16 +515,10 @@ public

[GitHub] flink pull request #4509: [FLINK-7406][network] Implement Netty receiver inc...

2017-11-23 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4509#discussion_r152860672 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannelTest.java --- @@ -301,81 +306,388 @@ public

[GitHub] flink issue #4581: [FLINK-7499][io] fix double buffer release in SpillableSu...

2017-11-23 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/4581 ok, now I'm using the buffer incorrectly in `SpillableSubpartition#add`...let me re-think it once more ---

[GitHub] flink issue #4581: [FLINK-7499][io] fix double buffer release in SpillableSu...

2017-11-23 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/4581 ok, fixed ---

[GitHub] flink pull request #4581: [FLINK-7499][io] fix double buffer release in Spil...

2017-11-24 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4581#discussion_r152944166 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/SpillableSubpartitionTest.java --- @@ -320,4 +559,40 @@ void

[GitHub] flink pull request #5064: [FLINK-8145][tests] fix various IOManagerAsync ins...

2017-11-24 Thread NicoK
GitHub user NicoK opened a pull request: https://github.com/apache/flink/pull/5064 [FLINK-8145][tests] fix various IOManagerAsync instances not being shut down ## What is the purpose of the change Fix several unit tests using `IOManagerAsync` but not shutting it down

[GitHub] flink pull request #4509: [FLINK-7406][network] Implement Netty receiver inc...

2017-11-24 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4509#discussion_r152961281 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannel.java --- @@ -572,20 +560,53 @@ void

[GitHub] flink pull request #4509: [FLINK-7406][network] Implement Netty receiver inc...

2017-11-24 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4509#discussion_r152972414 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannelTest.java --- @@ -715,4 +686,58 @@ private

[GitHub] flink pull request #4509: [FLINK-7406][network] Implement Netty receiver inc...

2017-11-24 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4509#discussion_r152969860 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannelTest.java --- @@ -330,64 +332,120 @@ public

[GitHub] flink pull request #4509: [FLINK-7406][network] Implement Netty receiver inc...

2017-11-24 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4509#discussion_r152963546 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannelTest.java --- @@ -330,64 +332,120 @@ public

[GitHub] flink pull request #4509: [FLINK-7406][network] Implement Netty receiver inc...

2017-11-24 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4509#discussion_r152962580 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannelTest.java --- @@ -330,64 +332,120 @@ public

<    1   2   3   4   5   6   7   8   9   10   >