[GitHub] flink pull request #4464: [FLINK-7354][tests] ignore "initialSeedUniquifierG...

2017-08-02 Thread NicoK
GitHub user NicoK opened a pull request: https://github.com/apache/flink/pull/4464 [FLINK-7354][tests] ignore "initialSeedUniquifierGenerator" thread in thread list ## What is the purpose of the change Netty may spawn a thread in its `ThreadLocalRandom` becau

[GitHub] flink issue #3816: [FLINK-5720] Deprecate DataStream#fold()

2017-08-03 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/3816 Actually, `KeyedStream.aggregate` is protected and thus not accessible. You may, however, access `WindowedStream#aggregate(AggregateFunction) ` and `AllWindowedStream#aggregate

[GitHub] flink pull request #4477: [FLINK-7370][docs] rework the operator documentati...

2017-08-04 Thread NicoK
GitHub user NicoK opened a pull request: https://github.com/apache/flink/pull/4477 [FLINK-7370][docs] rework the operator documentation structure ## What is the purpose of the change Improve the docs by re-structuring the operators documentation. ## Brief change

[GitHub] flink pull request #4478: [hotfix][docs] add documentation for `taskmanager....

2017-08-04 Thread NicoK
GitHub user NicoK opened a pull request: https://github.com/apache/flink/pull/4478 [hotfix][docs] add documentation for `taskmanager.exit-on-fatal-akka-error` ## What is the purpose of the change When the quarantine monitor was added as of FLINK-3347, documentation for

[GitHub] flink issue #4445: [FLINK-7310][core] always use the HybridMemorySegment

2017-08-04 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/4445 in a non-exhaustive mini benchmark, I ran `HashVsSortMiniBenchmark` and got the following results: # Best out of 5 (in ms) Test | `master` | `Flink-7310

[GitHub] flink pull request #4481: [FLINK-7316][network] always use off-heap network ...

2017-08-04 Thread NicoK
GitHub user NicoK opened a pull request: https://github.com/apache/flink/pull/4481 [FLINK-7316][network] always use off-heap network buffers ## What is the purpose of the change For now, network buffers may be on-heap or off-heap along with Flink memory settings. As a step

[GitHub] flink issue #3085: [FLINK-5178] allow BlobCache to use a distributed file sy...

2017-08-04 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/3085 no, this is not part of FLIP-19 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] flink pull request #3085: [FLINK-5178] allow BlobCache to use a distributed ...

2017-08-04 Thread NicoK
Github user NicoK closed the pull request at: https://github.com/apache/flink/pull/3085 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] flink pull request #4234: [FLINK-7053][blob] improve code quality in some te...

2017-08-04 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4234#discussion_r131431731 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/blob/BlobClientSslTest.java --- @@ -107,195 +96,89 @@ public static void stopServers() throws

[GitHub] flink pull request #4234: [FLINK-7053][blob] improve code quality in some te...

2017-08-04 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4234#discussion_r131431720 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/blob/BlobCacheSuccessTest.java --- @@ -92,15 +100,15 @@ private void uploadFileGetTest(final

[GitHub] flink pull request #4234: [FLINK-7053][blob] improve code quality in some te...

2017-08-04 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4234#discussion_r131431976 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/blob/BlobClientSslTest.java --- @@ -107,195 +96,89 @@ public static void stopServers() throws

[GitHub] flink issue #4367: [FLINK-4499] [build] Add spotbugs plugin

2017-08-07 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/4367 looks like the build times increase by 5-10 minutes but I'd say this is worth it +1 from my side --- If your project is set up for it, you can reply to this email and have your reply appe

[GitHub] flink pull request #4481: [FLINK-7316][network] always use off-heap network ...

2017-08-07 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4481#discussion_r131672655 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/buffer/NetworkBufferPool.java --- @@ -274,7 +260,7 @@ private void

[GitHub] flink issue #4481: [FLINK-7316][network] always use off-heap network buffers

2017-08-08 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/4481 actually, I need to fix the test failures in `ContaineredTaskManagerParametersTest` and some failure in the `flink-yarn-tests` first... --- If your project is set up for it, you can reply to this

[GitHub] flink issue #4481: [FLINK-7316][network] always use off-heap network buffers

2017-08-08 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/4481 ok, one test fixed, the other is not so simple but maybe @tillrohrmann can help with it: Inside `ContaineredTaskManagerParameters#create()`, we calculate the amount of off-heap space that we

[GitHub] flink pull request #4506: [FLINK-7400][cluster] fix off-heap limits set to c...

2017-08-09 Thread NicoK
GitHub user NicoK opened a pull request: https://github.com/apache/flink/pull/4506 [FLINK-7400][cluster] fix off-heap limits set to conservatively in cluster environments ## What is the purpose of the change Inside `ContaineredTaskManagerParameters`, since #3648, the

[GitHub] flink issue #4447: [FLINK-7312][checkstyle] activate checkstyle for flink/co...

2017-08-10 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/4447 :) in here, however, I had to add an exception for these `final` keywords which may be removed when #4458 is merged. --- If your project is set up for it, you can reply to this email and have your

[GitHub] flink pull request #4346: [FLINK-7199] [gelly] Graph simplification does not...

2017-08-10 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4346#discussion_r132400522 --- Diff: flink-libraries/flink-gelly-examples/src/test/java/org/apache/flink/graph/drivers/EdgeListITCase.java --- @@ -56,214 +56,304 @@ public void

[GitHub] flink pull request #4346: [FLINK-7199] [gelly] Graph simplification does not...

2017-08-10 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4346#discussion_r132400996 --- Diff: flink-libraries/flink-gelly-examples/src/test/java/org/apache/flink/graph/drivers/EdgeListITCase.java --- @@ -56,214 +56,304 @@ public void

[GitHub] flink pull request #4346: [FLINK-7199] [gelly] Graph simplification does not...

2017-08-10 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4346#discussion_r132401190 --- Diff: flink-libraries/flink-gelly-examples/src/test/java/org/apache/flink/graph/drivers/EdgeListITCase.java --- @@ -56,214 +56,304 @@ public void

[GitHub] flink pull request #4346: [FLINK-7199] [gelly] Graph simplification does not...

2017-08-10 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4346#discussion_r132401256 --- Diff: flink-libraries/flink-gelly-examples/src/test/java/org/apache/flink/graph/drivers/EdgeListITCase.java --- @@ -56,214 +56,304 @@ public void

[GitHub] flink pull request #4346: [FLINK-7199] [gelly] Graph simplification does not...

2017-08-10 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4346#discussion_r132401061 --- Diff: flink-libraries/flink-gelly-examples/src/test/java/org/apache/flink/graph/drivers/EdgeListITCase.java --- @@ -56,214 +56,304 @@ public void

[GitHub] flink pull request #4346: [FLINK-7199] [gelly] Graph simplification does not...

2017-08-10 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4346#discussion_r132401474 --- Diff: flink-libraries/flink-gelly-examples/src/test/java/org/apache/flink/graph/drivers/EdgeListITCase.java --- @@ -56,214 +56,304 @@ public void

[GitHub] flink pull request #4346: [FLINK-7199] [gelly] Graph simplification does not...

2017-08-10 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4346#discussion_r132402963 --- Diff: flink-libraries/flink-gelly-examples/src/main/java/org/apache/flink/graph/drivers/parameter/LongParameter.java --- @@ -52,16 +52,6 @@ public

[GitHub] flink pull request #4346: [FLINK-7199] [gelly] Graph simplification does not...

2017-08-10 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4346#discussion_r132401144 --- Diff: flink-libraries/flink-gelly-examples/src/test/java/org/apache/flink/graph/drivers/EdgeListITCase.java --- @@ -56,214 +56,304 @@ public void

[GitHub] flink pull request #4346: [FLINK-7199] [gelly] Graph simplification does not...

2017-08-10 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4346#discussion_r132400879 --- Diff: flink-libraries/flink-gelly-examples/src/test/java/org/apache/flink/graph/drivers/EdgeListITCase.java --- @@ -56,214 +56,304 @@ public void

[GitHub] flink pull request #4346: [FLINK-7199] [gelly] Graph simplification does not...

2017-08-10 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4346#discussion_r132401371 --- Diff: flink-libraries/flink-gelly-examples/src/test/java/org/apache/flink/graph/drivers/EdgeListITCase.java --- @@ -56,214 +56,304 @@ public void

[GitHub] flink pull request #4346: [FLINK-7199] [gelly] Graph simplification does not...

2017-08-10 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4346#discussion_r132400768 --- Diff: flink-libraries/flink-gelly-examples/src/test/java/org/apache/flink/graph/drivers/EdgeListITCase.java --- @@ -56,214 +56,304 @@ public void

[GitHub] flink issue #4481: [FLINK-7316][network] always use off-heap network buffers

2017-08-10 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/4481 by cherry-picking the commits from #4506, plus some fixes for code which was changed in the wrong way previously, the failing yarn tests should now be fixed --- If your project is set up for it, you

[GitHub] flink pull request #4346: [FLINK-7199] [gelly] Graph simplification does not...

2017-08-10 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4346#discussion_r132452312 --- Diff: flink-libraries/flink-gelly-examples/src/main/java/org/apache/flink/graph/drivers/parameter/LongParameter.java --- @@ -52,16 +52,6 @@ public

[GitHub] flink pull request #4517: [FLINK-7411][network] minor performance improvemen...

2017-08-10 Thread NicoK
GitHub user NicoK opened a pull request: https://github.com/apache/flink/pull/4517 [FLINK-7411][network] minor performance improvements in NettyMessage ## What is the purpose of the change This PR adds some (minor) performance improvements to `NettyMessage` which I came

[GitHub] flink pull request #4518: [FLINK-7412][network] optimise NettyMessage.TaskEv...

2017-08-10 Thread NicoK
GitHub user NicoK opened a pull request: https://github.com/apache/flink/pull/4518 [FLINK-7412][network] optimise NettyMessage.TaskEventRequest#readFrom() to read from netty buffers directly ## What is the purpose of the change `NettyMessage.TaskEventRequest#readFrom

[GitHub] flink pull request #4528: [FLINK-7427][network] integrate PartitionRequestPr...

2017-08-11 Thread NicoK
GitHub user NicoK opened a pull request: https://github.com/apache/flink/pull/4528 [FLINK-7427][network] integrate PartitionRequestProtocol into NettyProtocol ## What is the purpose of the change This is to make the network protocol classes more understandable by removing

[GitHub] flink issue #4445: [FLINK-7310][core] always use the HybridMemorySegment

2017-08-11 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/4445 FYI: I just rebased this PR onto current `master` to make this mergable and support further extensions --- If your project is set up for it, you can reply to this email and have your reply appear on

[GitHub] flink issue #4447: [FLINK-7312][checkstyle] activate checkstyle for flink/co...

2017-08-11 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/4447 FYI: I just rebased this PR onto current `master` to make this mergable and support further extensions --- If your project is set up for it, you can reply to this email and have your reply appear on

[GitHub] flink issue #4481: [FLINK-7316][network] always use off-heap network buffers

2017-08-11 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/4481 FYI: I just rebased this PR onto current `master` to make this mergable and support further extensions --- If your project is set up for it, you can reply to this email and have your reply appear on

[GitHub] flink pull request #4529: [FLINK-7428][network] avoid buffer copies when rec...

2017-08-11 Thread NicoK
GitHub user NicoK opened a pull request: https://github.com/apache/flink/pull/4529 [FLINK-7428][network] avoid buffer copies when receiving messages ## What is the purpose of the change The `LengthFieldBasedFrameDecoder` used so far creates one additional copy in its

[GitHub] flink pull request #4485: [FLINK-7378][core]Implement the FixedBufferPool fo...

2017-08-11 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4485#discussion_r132670989 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/buffer/FixedBufferPool.java --- @@ -0,0 +1,280 @@ +/* + * Licensed to the

[GitHub] flink pull request #4485: [FLINK-7378][core]Implement the FixedBufferPool fo...

2017-08-11 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4485#discussion_r132655594 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/buffer/BufferPool.java --- @@ -79,4 +79,12 @@ * Returns the number of used

[GitHub] flink pull request #4485: [FLINK-7378][core]Implement the FixedBufferPool fo...

2017-08-11 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4485#discussion_r132673916 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/buffer/NetworkBufferPool.java --- @@ -221,13 +228,44 @@ public BufferPool

[GitHub] flink pull request #4485: [FLINK-7378][core]Implement the FixedBufferPool fo...

2017-08-11 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4485#discussion_r13267 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/buffer/BufferPoolFactory.java --- @@ -39,6 +39,17 @@ BufferPool

[GitHub] flink pull request #4485: [FLINK-7378][core]Implement the FixedBufferPool fo...

2017-08-11 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4485#discussion_r132655177 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/buffer/FixedBufferPool.java --- @@ -0,0 +1,280 @@ +/* + * Licensed to the

[GitHub] flink pull request #4485: [FLINK-7378][core]Implement the FixedBufferPool fo...

2017-08-11 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4485#discussion_r132655478 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/buffer/BufferPoolListener.java --- @@ -0,0 +1,34 @@ +/* + * Licensed to the

[GitHub] flink issue #4529: [FLINK-7428][network] avoid buffer copies when receiving ...

2017-08-11 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/4529 FYI: - newer netty 4.0.xx versions also use `buffer.slice(index, length)` inside their `extractFrame()` method (see https://github.com/netty/netty/commit/891be30a28c6dc5a1edf1cb5a3690644cf4ff66e

[GitHub] flink issue #4529: [FLINK-7428][network] avoid buffer copies when receiving ...

2017-08-14 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/4529 we also need to double-check this against the previous issue of d92e422ec7089376583a8f57043274d236c340a4 which may be solved by the way I am using the `LengthFieldBasedFrameDecoder` now compared to

[GitHub] flink issue #4238: [FLINK-7057][blob] move BLOB ref-counting from LibraryCac...

2017-08-16 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/4238 rebased, squashed, and fixed the new issue found by the spotbugs plugin since Flink is moving fast, though, there's a conflict again - should I rebase again? --- If your project is s

[GitHub] flink pull request #4558: [FLINK-7057][tests][hotfix] make sure the client a...

2017-08-17 Thread NicoK
GitHub user NicoK opened a pull request: https://github.com/apache/flink/pull/4558 [FLINK-7057][tests][hotfix] make sure the client and a created InputStream are closed ## What is the purpose of the change This fixes some stalling tests in the following cases: if the

[GitHub] flink pull request #4558: [FLINK-7056][tests][hotfix] make sure the client a...

2017-08-17 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4558#discussion_r133675837 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/blob/BlobClientTest.java --- @@ -139,30 +139,35 @@ private static BlobKey prepareTestFile(File

[GitHub] flink pull request #4238: [FLINK-7057][blob] move BLOB ref-counting from Lib...

2017-08-17 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4238#discussion_r133685671 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/JobMaster.java --- @@ -148,7 +149,10 @@ /** Service to contend for and

[GitHub] flink pull request #4238: [FLINK-7057][blob] move BLOB ref-counting from Lib...

2017-08-17 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4238#discussion_r133687826 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/blob/BlobCacheCleanupTest.java --- @@ -0,0 +1,324 @@ +/* + * Licensed to the Apache

[GitHub] flink pull request #4238: [FLINK-7057][blob] move BLOB ref-counting from Lib...

2017-08-17 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4238#discussion_r133688433 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/jobmanager/JobManagerCleanupITCase.java --- @@ -0,0 +1,298 @@ +/* + * Licensed to the

[GitHub] flink pull request #4238: [FLINK-7057][blob] move BLOB ref-counting from Lib...

2017-08-17 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4238#discussion_r133688568 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/testtasks/FailingBlockingInvokable.java --- @@ -0,0 +1,48 @@ +/* + * Licensed to the

[GitHub] flink issue #4238: [FLINK-7057][blob] move BLOB ref-counting from LibraryCac...

2017-08-17 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/4238 ok, rebased again but upon #4558 which adds one small fix that could cause hanging tests. I also addressed the comments you had, thanks. --- If your project is set up for it, you can reply to this

[GitHub] flink issue #4358: [FLINK-7068][blob] change BlobService sub-classes for per...

2017-08-18 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/4358 Rebased onto `master`, but had to drag in #4402 early to fix the end-to-end tests failing due to spurious warnings. The test failure you observed was actually a test instability introduced with #4238

[GitHub] flink pull request #4562: [FLINK-7402] Fix ineffective null check in NettyMe...

2017-08-21 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4562#discussion_r134166175 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/netty/NettyMessage.java --- @@ -220,6 +220,10 @@ void releaseBuffer

[GitHub] flink issue #4238: [FLINK-7057][blob] move BLOB ref-counting from LibraryCac...

2017-08-21 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/4238 Thanks, @tedyu both things should be changed - I'll create a hotfix for that --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as wel

[GitHub] flink pull request #4568: [FLINK-7483][blob] prevent cleanup of re-registere...

2017-08-21 Thread NicoK
GitHub user NicoK opened a pull request: https://github.com/apache/flink/pull/4568 [FLINK-7483][blob] prevent cleanup of re-registered jobs ## What is the purpose of the change Since #4238, when a job is registered but was released before and the ref count hit `0`, its

[GitHub] flink issue #4568: [FLINK-7483][blob] prevent cleanup of re-registered jobs

2017-08-21 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/4568 @tedyu this fixes the two issues you found in #4238. Could you have a quick look, also? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well

[GitHub] flink issue #4358: [FLINK-7068][blob] change BlobService sub-classes for per...

2017-08-21 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/4358 sorry for the mess, but let me also drag in #4568 and adapt the code in here (which is moved from `BlobCache` to `PermanentBlobCache` by this PR) --- If your project is set up for it, you can reply

[GitHub] flink issue #4568: [FLINK-7483][blob] prevent cleanup of re-registered jobs

2017-08-21 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/4568 FYI: the test instability of `JobManagerCleanupITCase` will be fixed by #4358, the next BLOB-PR in line - I don't want to mess up the following PRs anymore (again) by integrating it separately or

[GitHub] flink pull request #4571: [FLINK-7487][tests] fix ClassLoaderITCase#testDisp...

2017-08-22 Thread NicoK
GitHub user NicoK opened a pull request: https://github.com/apache/flink/pull/4571 [FLINK-7487][tests] fix ClassLoaderITCase#testDisposeSavepointWithCustomKvState not self-contained ## What is the purpose of the change The cancellation of the job started in

[GitHub] flink pull request #4562: [FLINK-7402] Fix ineffective null check in NettyMe...

2017-08-23 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4562#discussion_r134699645 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/netty/NettyMessage.java --- @@ -220,9 +220,7 @@ void releaseBuffer

[GitHub] flink pull request #4562: [FLINK-7402] Fix ineffective null check in NettyMe...

2017-08-23 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4562#discussion_r134711249 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/netty/NettyMessage.java --- @@ -18,6 +18,12 @@ package

[GitHub] flink pull request #4581: [FLINK-7499][io] also let AsynchronousBufferFileWr...

2017-08-24 Thread NicoK
GitHub user NicoK opened a pull request: https://github.com/apache/flink/pull/4581 [FLINK-7499][io] also let AsynchronousBufferFileWriter#writeBlock() recycle the buffer in case of failures ## What is the purpose of the change `SpillableSubpartitionView#releaseMemory

[GitHub] flink pull request #4590: [FLINK-7513][tests] remove TestBufferFactory#MOCK_...

2017-08-25 Thread NicoK
GitHub user NicoK opened a pull request: https://github.com/apache/flink/pull/4590 [FLINK-7513][tests] remove TestBufferFactory#MOCK_BUFFER ## What is the purpose of the change The static `TestBufferFactory#MOCK_BUFFER` buffer did not allow proper reference counting and we

[GitHub] flink pull request #4591: [FLINK-7514][tests] fix BackPressureStatsTrackerIT...

2017-08-25 Thread NicoK
GitHub user NicoK opened a pull request: https://github.com/apache/flink/pull/4591 [FLINK-7514][tests] fix BackPressureStatsTrackerITCase releasing buffers twice ## What is the purpose of the change `BackPressureStatsTrackerITCase#testBackPressuredProducer()` is releasing

[GitHub] flink pull request #4592: [FLINK-7515][network] allow actual 0-length conten...

2017-08-25 Thread NicoK
GitHub user NicoK opened a pull request: https://github.com/apache/flink/pull/4592 [FLINK-7515][network] allow actual 0-length content in NettyMessage#allocateBuffer() ## What is the purpose of the change Previously, length "0" meant "unknown content length

[GitHub] flink pull request #4593: [FLINK-7516][memory] do not allow copies into a re...

2017-08-25 Thread NicoK
GitHub user NicoK opened a pull request: https://github.com/apache/flink/pull/4593 [FLINK-7516][memory] do not allow copies into a read-only ByteBuffer ## What is the purpose of the change `HybridMemorySegment#get(int, ByteBuffer, int)` allows writing into a read-only

[GitHub] flink pull request #4594: [FLINK-7517][network] let NettyBufferPool extend P...

2017-08-25 Thread NicoK
GitHub user NicoK opened a pull request: https://github.com/apache/flink/pull/4594 [FLINK-7517][network] let NettyBufferPool extend PooledByteBufAllocator ## What is the purpose of the change `NettyBufferPool` wraps `PooledByteBufAllocator` but due to this, any allocated

[GitHub] flink issue #4529: [FLINK-7428][network] avoid buffer copies when receiving ...

2017-08-25 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/4529 Thanks, @greghogan - I put most of them under FLINK-7315 now (at least the ones which are loosely related). Regarding the PR itself, I still need to investigate whether the previous issues

[GitHub] flink issue #4485: [FLINK-7378][core]Create a fix size (non rebalancing) buf...

2017-08-25 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/4485 Reviewed 9 of 13 files at r2. Review status: all files reviewed at latest revision, 7 unresolved discussions, some commit checks failed. --- *[a discussion

[GitHub] flink pull request #4485: [FLINK-7378][core]Create a fix size (non rebalanci...

2017-08-28 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4485#discussion_r135480975 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/buffer/NetworkBufferPool.java --- @@ -131,6 +133,50 @@ public void recycle

[GitHub] flink pull request #4485: [FLINK-7378][core]Create a fix size (non rebalanci...

2017-08-28 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4485#discussion_r135481583 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/SingleInputGate.java --- @@ -259,17 +267,72 @@ public int

[GitHub] flink issue #4485: [FLINK-7378][core]Create a fix size (non rebalancing) buf...

2017-08-28 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/4485 Hi @zhijiangW, regarding the buffer pool implementation, I was just curious about why it was done that way. But it is fine to keep the logic in `RemoteInputChannel` if you make sure, that a

[GitHub] flink pull request #4613: [FLINK-7520][network] let our Buffer class extend ...

2017-08-29 Thread NicoK
GitHub user NicoK opened a pull request: https://github.com/apache/flink/pull/4613 [FLINK-7520][network] let our Buffer class extend from Netty's buffer class ## What is the purpose of the change With this PR, we extend out own `Buffer` class to extend from Ne

[GitHub] flink pull request #4615: [FLINK-7518][network] pass our own NetworkBuffer t...

2017-08-29 Thread NicoK
GitHub user NicoK opened a pull request: https://github.com/apache/flink/pull/4615 [FLINK-7518][network] pass our own NetworkBuffer to Netty ## What is the purpose of the change With this PR, based on #4613, we finally pass our own `NetworkBuffer` class to Netty and remove

[GitHub] flink pull request #4485: [FLINK-7378][core]Create a fix size (non rebalanci...

2017-08-30 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4485#discussion_r136021056 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/io/network/buffer/NetworkBufferPoolTest.java --- @@ -168,4 +171,45 @@ public void

[GitHub] flink pull request #4485: [FLINK-7378][core]Create a fix size (non rebalanci...

2017-08-30 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4485#discussion_r136019424 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/io/network/NetworkEnvironmentTest.java --- @@ -144,7 +150,7 @@ private static ResultPartition

[GitHub] flink pull request #4485: [FLINK-7378][core]Create a fix size (non rebalanci...

2017-08-30 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4485#discussion_r136020758 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/io/network/buffer/NetworkBufferPoolTest.java --- @@ -168,4 +171,45 @@ public void

[GitHub] flink pull request #4485: [FLINK-7378][core]Create a fix size (non rebalanci...

2017-08-30 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4485#discussion_r136023768 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/SingleInputGateTest.java --- @@ -372,6 +375,52 @@ public void

[GitHub] flink pull request #4485: [FLINK-7378][core]Create a fix size (non rebalanci...

2017-08-30 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4485#discussion_r136018729 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/buffer/NetworkBufferPool.java --- @@ -131,6 +135,59 @@ public void recycle

[GitHub] flink pull request #4485: [FLINK-7378][core]Create a fix size (non rebalanci...

2017-08-30 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4485#discussion_r136022209 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/SingleInputGateTest.java --- @@ -372,6 +375,52 @@ public void

[GitHub] flink pull request #4485: [FLINK-7378][core]Create a fix size (non rebalanci...

2017-08-30 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4485#discussion_r136021108 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/io/network/buffer/NetworkBufferPoolTest.java --- @@ -168,4 +171,45 @@ public void

[GitHub] flink pull request #4485: [FLINK-7378][core]Create a fix size (non rebalanci...

2017-08-30 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4485#discussion_r136020966 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/io/network/buffer/NetworkBufferPoolTest.java --- @@ -168,4 +171,45 @@ public void

[GitHub] flink pull request #4653: [FLINK-7598][travis] fix ineffective shaded artifa...

2017-09-07 Thread NicoK
GitHub user NicoK opened a pull request: https://github.com/apache/flink/pull/4653 [FLINK-7598][travis] fix ineffective shaded artifacts checks ## What is the purpose of the change This fixes the Netty shaded dependencies check and makes all of the checks more robust

[GitHub] flink pull request #4506: [FLINK-7400][cluster] fix off-heap limits set to c...

2017-10-30 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4506#discussion_r147656471 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/clusterframework/ContaineredTaskManagerParameters.java --- @@ -143,7 +143,8 @@ public static

[GitHub] flink pull request #4506: [FLINK-7400][cluster] fix off-heap limits set to c...

2017-10-30 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4506#discussion_r147657947 --- Diff: flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionCapacitySchedulerITCase.java --- @@ -132,6 +132,36 @@ public void perJobYarnCluster

[GitHub] flink issue #4506: [FLINK-7400][cluster] fix off-heap limits set to conserva...

2017-10-30 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/4506 I just tested this on a real yarn cluster with `./bin/flink run -m yarn-cluster -yn 1 -ys 2 -yjm 768 -ytm 1024 -yD taskmanager.memory.off-heap=true -yD taskmanager.memory.size=260 -yD

[GitHub] flink pull request #4774: [FLINK-6495] Fix Akka's default value for heartbea...

2017-10-30 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4774#discussion_r147743853 --- Diff: flink-runtime/src/main/scala/org/apache/flink/runtime/akka/AkkaUtils.scala --- @@ -257,6 +257,20 @@ object AkkaUtils

[GitHub] flink pull request #4925: [FLINK-7950][build] add flink-queryable-state-runt...

2017-10-30 Thread NicoK
GitHub user NicoK opened a pull request: https://github.com/apache/flink/pull/4925 [FLINK-7950][build] add flink-queryable-state-runtime as a dependency to flink-dist ## What is the purpose of the change Since #4906, `flink-queryable-state-runtime`'s jar file was put

[GitHub] flink pull request #4593: [FLINK-7516][memory] do not allow copies into a re...

2017-11-01 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4593#discussion_r148209947 --- Diff: flink-core/src/main/java/org/apache/flink/core/memory/HybridMemorySegment.java --- @@ -306,6 +307,9 @@ public final void get(int offset, ByteBuffer

[GitHub] flink issue #4506: [FLINK-7400][cluster] fix off-heap limits set to conserva...

2017-11-01 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/4506 rebased successfully, waiting for Travis now... ---

[GitHub] flink pull request #4592: [FLINK-7515][network] allow actual 0-length conten...

2017-11-01 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4592#discussion_r148212023 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/netty/NettyMessage.java --- @@ -64,12 +65,53

[GitHub] flink issue #4592: [FLINK-7515][network] allow actual 0-length content in Ne...

2017-11-01 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/4592 of course, the last change added a conflict...rebased now ---

[GitHub] flink pull request #4758: [FLINK-7745][tests] add tests for ensuring Network...

2017-11-01 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4758#discussion_r148400245 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/io/network/buffer/BufferPoolFactoryTest.java --- @@ -53,9 +64,89 @@ public void

[GitHub] flink pull request #4758: [FLINK-7745][tests] add tests for ensuring Network...

2017-11-01 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4758#discussion_r148401163 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/io/network/buffer/BufferPoolFactoryTest.java --- @@ -53,9 +64,89 @@ public void

[GitHub] flink pull request #4758: [FLINK-7745][tests] add tests for ensuring Network...

2017-11-01 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4758#discussion_r148402248 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/io/network/buffer/BufferPoolFactoryTest.java --- @@ -53,9 +64,89 @@ public void

[GitHub] flink pull request #4758: [FLINK-7745][tests] add tests for ensuring Network...

2017-11-01 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/4758#discussion_r148402375 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/io/network/buffer/BufferPoolFactoryTest.java --- @@ -53,9 +64,89 @@ public void

[GitHub] flink issue #4758: [FLINK-7745][tests] add tests for ensuring NetworkBufferP...

2017-11-01 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/4758 Added some comments and also found out that the `testOverprovisioned()` test wasn't really testing what it was supposed to test :( - also fixed now. ---

[GitHub] flink issue #4733: [FLINK-7701][network] really fix watermark configuration ...

2017-11-01 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/4733 yes, unfortunately this was for netty 4.1 only (it changed the API)... I rebased the PR to fix the conflict (which I did not have during the rebase though...) ---

<    1   2   3   4   5   6   7   8   9   10   >