[GitHub] flink issue #4626: [FLINK-7561][streaming] Implement PreAggregationOperator

2017-09-14 Thread pnowojski
Github user pnowojski commented on the issue: https://github.com/apache/flink/pull/4626 @mproch you might be interested in this one :) ---

[GitHub] flink pull request #3511: [Flink-5734] code generation for normalizedkey sor...

2017-09-21 Thread pnowojski
Github user pnowojski commented on a diff in the pull request: https://github.com/apache/flink/pull/3511#discussion_r140258675 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/operators/sort/NormalizedKeySorter.java --- @@ -47,61 +47,61 @@ public

[GitHub] flink pull request #3511: [Flink-5734] code generation for normalizedkey sor...

2017-09-21 Thread pnowojski
Github user pnowojski commented on a diff in the pull request: https://github.com/apache/flink/pull/3511#discussion_r140258895 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/operators/sort/NormalizedKeySorter.java --- @@ -309,7 +309,7 @@ public boolean write(T

[GitHub] flink pull request #4722: [FLINK-7683] Iterate over keys in KeyedStateBacken...

2017-09-25 Thread pnowojski
GitHub user pnowojski opened a pull request: https://github.com/apache/flink/pull/4722 [FLINK-7683] Iterate over keys in KeyedStateBackend ## What is the purpose of the change This is required to make possible preserving backward compatibility while changing state

[GitHub] flink pull request #4723: [FLINK-7684] Avoid data copies in MergingWindowSet

2017-09-25 Thread pnowojski
GitHub user pnowojski opened a pull request: https://github.com/apache/flink/pull/4723 [FLINK-7684] Avoid data copies in MergingWindowSet ## What is the purpose of the change Previously MergingWindowSet uses ListState of tuples to persists it's mapping. This is ineffi

[GitHub] flink issue #4609: [FLINK-7541] Refactor StateAssignmentOperation

2017-09-26 Thread pnowojski
Github user pnowojski commented on the issue: https://github.com/apache/flink/pull/4609 Sorry @tillrohrmann I have missed that :( @StefanRRichter I pushed a fixed version (but lets wait for travis to pass) ---

[GitHub] flink pull request #4722: [FLINK-7683] Iterate over keys in KeyedStateBacken...

2017-09-26 Thread pnowojski
Github user pnowojski commented on a diff in the pull request: https://github.com/apache/flink/pull/4722#discussion_r141100065 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/state/heap/CopyOnWriteStateTable.java --- @@ -287,6 +289,14 @@ public S get(K key, N

[GitHub] flink pull request #4722: [FLINK-7683] Iterate over keys in KeyedStateBacken...

2017-09-26 Thread pnowojski
Github user pnowojski commented on a diff in the pull request: https://github.com/apache/flink/pull/4722#discussion_r141090682 --- Diff: flink-contrib/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBKeyedStateBackend.java --- @@ -252,6

[GitHub] flink pull request #4722: [FLINK-7683] Iterate over keys in KeyedStateBacken...

2017-09-26 Thread pnowojski
Github user pnowojski commented on a diff in the pull request: https://github.com/apache/flink/pull/4722#discussion_r141083304 --- Diff: flink-contrib/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBKeyedStateBackend.java --- @@ -252,6

[GitHub] flink issue #4609: [FLINK-7541] Refactor StateAssignmentOperation

2017-09-26 Thread pnowojski
Github user pnowojski commented on the issue: https://github.com/apache/flink/pull/4609 Surprisingly even kafka tests have passed this time oO ---

[GitHub] flink pull request #4722: [FLINK-7683] Iterate over keys in KeyedStateBacken...

2017-09-27 Thread pnowojski
Github user pnowojski commented on a diff in the pull request: https://github.com/apache/flink/pull/4722#discussion_r141258536 --- Diff: flink-contrib/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBKeyedStateBackend.java --- @@ -1946,4

[GitHub] flink issue #4722: [FLINK-7683] Iterate over keys in KeyedStateBackend

2017-09-27 Thread pnowojski
Github user pnowojski commented on the issue: https://github.com/apache/flink/pull/4722 We have got carried away with those concurrent modifications and snapshots. To rewrite keyed state variable from an old to a new one, old is used as a read only. Thus `getKeys` for the the old one

[GitHub] flink pull request #4748: [hotfix][tests] Use G1GC for tests

2017-09-29 Thread pnowojski
GitHub user pnowojski opened a pull request: https://github.com/apache/flink/pull/4748 [hotfix][tests] Use G1GC for tests We are using G1GC for running TaskManager, I think we should do it so also for tests. You can merge this pull request into a Git repository by running

[GitHub] flink pull request #4749: [FLINK-7739][tests] Properly shutdown resources in...

2017-09-29 Thread pnowojski
GitHub user pnowojski opened a pull request: https://github.com/apache/flink/pull/4749 [FLINK-7739][tests] Properly shutdown resources in tests This is a fixup of tests, without touching the production code. You can merge this pull request into a Git repository by running

[GitHub] flink pull request #4751: [FLINK-7739][kafka-tests] Throttle down data produ...

2017-09-29 Thread pnowojski
GitHub user pnowojski opened a pull request: https://github.com/apache/flink/pull/4751 [FLINK-7739][kafka-tests] Throttle down data producing thread Minor tests improvement in tests to avoid busy loop You can merge this pull request into a Git repository by running: $ git

[GitHub] flink issue #4748: [hotfix][tests] Use G1GC for tests

2017-10-04 Thread pnowojski
Github user pnowojski commented on the issue: https://github.com/apache/flink/pull/4748 Mostly for consistency, however using Serial GC on it's own doesn't seem like a better idea compared t G1GC (avoiding long pauses) ---

[GitHub] flink issue #4239: [FLINK-6988] flink-connector-kafka-0.11 with exactly-once...

2017-10-04 Thread pnowojski
Github user pnowojski commented on the issue: https://github.com/apache/flink/pull/4239 @aljoscha rebased on latest master and integrated your changes ---

[GitHub] flink pull request #4774: [FLINK-6495] Fix Akka's default value for heartbea...

2017-10-04 Thread pnowojski
GitHub user pnowojski opened a pull request: https://github.com/apache/flink/pull/4774 [FLINK-6495] Fix Akka's default value for heartbeat pause ## Brief change log This PR consists of two hotfixes regarding akka's heartbeat pause. The critical one is reve

[GitHub] flink pull request #4775: [FLINK-7739] Fix KafkaXXITCase tests stability

2017-10-04 Thread pnowojski
GitHub user pnowojski opened a pull request: https://github.com/apache/flink/pull/4775 [FLINK-7739] Fix KafkaXXITCase tests stability ## What is the purpose of the change This change fixes Kafka*ITCase tests stability. Main fix is excluding `netty` dependency from

[GitHub] flink issue #4775: [FLINK-7739] Fix KafkaXXITCase tests stability

2017-10-05 Thread pnowojski
Github user pnowojski commented on the issue: https://github.com/apache/flink/pull/4775 I have run ~500 Kafka09 tests on travis and problem with `TaskManager` was lost/no more resources is gone. However in those 500 runs twice I have seen `at-least-once` test failure ( @tzulitai is

[GitHub] flink issue #4774: [FLINK-6495] Fix Akka's default value for heartbeat pause

2017-10-05 Thread pnowojski
Github user pnowojski commented on the issue: https://github.com/apache/flink/pull/4774 I have added runtime check for that. To be clear, this was not the reason for Kafka tests instabilities and I'm not aware if this was causing any issues. But it definitely could and shou

[GitHub] flink pull request #4777: Convergence

2017-10-05 Thread pnowojski
GitHub user pnowojski opened a pull request: https://github.com/apache/flink/pull/4777 Convergence ## What is the purpose of the change This pull request enables dependency convergence for couple of modules. Enabling it for all of them at once is unfortunately too

[GitHub] flink pull request #4779: [FLINK-6988] Add additional tests coverage for Kaf...

2017-10-06 Thread pnowojski
GitHub user pnowojski opened a pull request: https://github.com/apache/flink/pull/4779 [FLINK-6988] Add additional tests coverage for Kafka 0.11 connector This PR only adds additional tests coverage for Kafka 0.11 connector. Please check individual commit messages for change log

[GitHub] flink pull request #4775: [FLINK-7739] Fix KafkaXXITCase tests stability

2017-10-06 Thread pnowojski
Github user pnowojski commented on a diff in the pull request: https://github.com/apache/flink/pull/4775#discussion_r143162433 --- Diff: flink-connectors/flink-connector-kafka-base/src/test/java/org/apache/flink/streaming/connectors/kafka/KafkaTestBase.java --- @@ -121,10 +122,13

[GitHub] flink issue #4775: [FLINK-7739] Fix KafkaXXITCase tests stability

2017-10-09 Thread pnowojski
Github user pnowojski commented on the issue: https://github.com/apache/flink/pull/4775 @StephanEwen thanks for merging. We can do that. ---

[GitHub] flink pull request #4774: [FLINK-6495] Fix Akka's default value for heartbea...

2017-10-09 Thread pnowojski
Github user pnowojski commented on a diff in the pull request: https://github.com/apache/flink/pull/4774#discussion_r143416489 --- Diff: flink-core/src/main/java/org/apache/flink/configuration/IllegalConfigurationException.java --- @@ -42,6 +42,17 @@ public

[GitHub] flink pull request #4774: [FLINK-6495] Fix Akka's default value for heartbea...

2017-10-09 Thread pnowojski
Github user pnowojski commented on a diff in the pull request: https://github.com/apache/flink/pull/4774#discussion_r143416770 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/restart/FixedDelayRestartStrategy.java --- @@ -82,7 +82,7 @@ public static

[GitHub] flink issue #4774: [FLINK-6495] Fix Akka's default value for heartbeat pause

2017-10-09 Thread pnowojski
Github user pnowojski commented on the issue: https://github.com/apache/flink/pull/4774 Made requested changes to fixed delay strategy and added one more hot fix regarding akka's documentation (last commit). ---

[GitHub] flink issue #4749: [FLINK-7739][tests] Properly shutdown resources in tests

2017-10-09 Thread pnowojski
Github user pnowojski commented on the issue: https://github.com/apache/flink/pull/4749 Users shouldn't be able to call `shutdown()` and if they are, they shouldn't be using it (it would be incorrect). `shutdown()` was a protected method and shouldn't be used the users

[GitHub] flink pull request #4777: [FLINK-7739] Enable dependency convergence

2017-10-09 Thread pnowojski
Github user pnowojski commented on a diff in the pull request: https://github.com/apache/flink/pull/4777#discussion_r143480344 --- Diff: flink-core/pom.xml --- @@ -77,6 +88,12 @@ under the License. org.apache.commons

[GitHub] flink issue #4239: [FLINK-6988] flink-connector-kafka-0.11 with exactly-once...

2017-10-09 Thread pnowojski
Github user pnowojski commented on the issue: https://github.com/apache/flink/pull/4239 Thanks :) ---

[GitHub] flink pull request #4239: [FLINK-6988] flink-connector-kafka-0.11 with exact...

2017-10-09 Thread pnowojski
Github user pnowojski closed the pull request at: https://github.com/apache/flink/pull/4239 ---

[GitHub] flink pull request #4779: [FLINK-6988] Add additional tests coverage for Kaf...

2017-10-10 Thread pnowojski
Github user pnowojski closed the pull request at: https://github.com/apache/flink/pull/4779 ---

[GitHub] flink issue #4779: [FLINK-6988] Add additional tests coverage for Kafka 0.11...

2017-10-10 Thread pnowojski
Github user pnowojski commented on the issue: https://github.com/apache/flink/pull/4779 Merged alongside #4239 ---

[GitHub] flink pull request #4801: [FLINK-7812] Log system resources metrics

2017-10-11 Thread pnowojski
GitHub user pnowojski opened a pull request: https://github.com/apache/flink/pull/4801 [FLINK-7812] Log system resources metrics ## What is the purpose of the change This PR adds various system resources metrics, useful for analysing issues on machines/clusters for which

[GitHub] flink pull request #4533: [FLINK-7416][network] Implement Netty receiver out...

2017-10-11 Thread pnowojski
Github user pnowojski commented on a diff in the pull request: https://github.com/apache/flink/pull/4533#discussion_r144014064 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/netty/CreditBasedClientHandler.java --- @@ -37,18 +43,29 @@ import

[GitHub] flink pull request #4533: [FLINK-7416][network] Implement Netty receiver out...

2017-10-11 Thread pnowojski
Github user pnowojski commented on a diff in the pull request: https://github.com/apache/flink/pull/4533#discussion_r144022698 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/netty/CreditBasedClientHandler.java --- @@ -272,4 +316,53 @@ private void

[GitHub] flink pull request #4533: [FLINK-7416][network] Implement Netty receiver out...

2017-10-11 Thread pnowojski
Github user pnowojski commented on a diff in the pull request: https://github.com/apache/flink/pull/4533#discussion_r144013921 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/netty/CreditBasedClientHandler.java --- @@ -37,18 +43,29 @@ import

[GitHub] flink pull request #4533: [FLINK-7416][network] Implement Netty receiver out...

2017-10-11 Thread pnowojski
Github user pnowojski commented on a diff in the pull request: https://github.com/apache/flink/pull/4533#discussion_r144025089 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/io/network/netty/PartitionRequestClientHandlerTest.java --- @@ -206,6 +209,53 @@ public

[GitHub] flink pull request #4533: [FLINK-7416][network] Implement Netty receiver out...

2017-10-11 Thread pnowojski
Github user pnowojski commented on a diff in the pull request: https://github.com/apache/flink/pull/4533#discussion_r144016699 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/netty/CreditBasedClientHandler.java --- @@ -104,8 +130,8 @@ public void

[GitHub] flink pull request #4533: [FLINK-7416][network] Implement Netty receiver out...

2017-10-11 Thread pnowojski
Github user pnowojski commented on a diff in the pull request: https://github.com/apache/flink/pull/4533#discussion_r144052410 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/netty/PartitionRequestClientHandler.java --- @@ -330,6 +330,10 @@ else if

[GitHub] flink pull request #4533: [FLINK-7416][network] Implement Netty receiver out...

2017-10-11 Thread pnowojski
Github user pnowojski commented on a diff in the pull request: https://github.com/apache/flink/pull/4533#discussion_r144025139 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/io/network/netty/PartitionRequestClientHandlerTest.java --- @@ -206,6 +209,53 @@ public

[GitHub] flink pull request #4533: [FLINK-7416][network] Implement Netty receiver out...

2017-10-12 Thread pnowojski
Github user pnowojski commented on a diff in the pull request: https://github.com/apache/flink/pull/4533#discussion_r144214324 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/netty/CreditBasedClientHandler.java --- @@ -272,4 +316,53 @@ private void

[GitHub] flink issue #4533: [FLINK-7416][network] Implement Netty receiver outgoing p...

2017-10-12 Thread pnowojski
Github user pnowojski commented on the issue: https://github.com/apache/flink/pull/4533 Ok thanks for the clarification. Previously I didn't understand how this is supposed to work, but after saying this: > Actually `CreditBasedClientHandler` would replace the

[GitHub] flink issue #4801: [FLINK-7812] Log system resources metrics

2017-10-13 Thread pnowojski
Github user pnowojski commented on the issue: https://github.com/apache/flink/pull/4801 As on dev mailing list discussion, this feature uses https://github.com/oshi/oshi library licensed under EPL 1.0. It seems to be compatible with ours: https://www.apache.org/legal/resolved.html

[GitHub] flink pull request #4840: [FLINK-7368][metrics] Make MetricStore ThreadSafe ...

2017-10-16 Thread pnowojski
GitHub user pnowojski opened a pull request: https://github.com/apache/flink/pull/4840 [FLINK-7368][metrics] Make MetricStore ThreadSafe class Remove external synchronisation on MetricStore ## What is the purpose of the change This is a refactor that makes

[GitHub] flink pull request #4841: [FLINK-7368][metrics] Backport synchronization fix...

2017-10-16 Thread pnowojski
GitHub user pnowojski opened a pull request: https://github.com/apache/flink/pull/4841 [FLINK-7368][metrics] Backport synchronization fix for MetricStore ## What is the purpose of the change This backports adding missing synchronization on MetricStore ## Verifying

[GitHub] flink pull request #4705: [FLINK-6444] [build] Add a check that '@VisibleFor...

2017-10-17 Thread pnowojski
Github user pnowojski commented on a diff in the pull request: https://github.com/apache/flink/pull/4705#discussion_r145125538 --- Diff: flink-tests/src/test/java/org/apache/flink/test/manual/CheckVisibleForTestingUsage.java --- @@ -0,0 +1,65 @@ +/* + * Licensed to the

[GitHub] flink pull request #4705: [FLINK-6444] [build] Add a check that '@VisibleFor...

2017-10-17 Thread pnowojski
Github user pnowojski commented on a diff in the pull request: https://github.com/apache/flink/pull/4705#discussion_r145133848 --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamExecutionEnvironment.java --- @@ -439,7 +439,7 @@ public

[GitHub] flink pull request #4705: [FLINK-6444] [build] Add a check that '@VisibleFor...

2017-10-17 Thread pnowojski
Github user pnowojski commented on a diff in the pull request: https://github.com/apache/flink/pull/4705#discussion_r145125984 --- Diff: flink-tests/src/test/java/org/apache/flink/test/manual/CheckVisibleForTestingUsage.java --- @@ -0,0 +1,65 @@ +/* + * Licensed to the

[GitHub] flink pull request #4705: [FLINK-6444] [build] Add a check that '@VisibleFor...

2017-10-17 Thread pnowojski
Github user pnowojski commented on a diff in the pull request: https://github.com/apache/flink/pull/4705#discussion_r145132472 --- Diff: flink-tests/src/test/java/org/apache/flink/test/manual/CheckVisibleForTestingUsage.java --- @@ -0,0 +1,65 @@ +/* + * Licensed to the

[GitHub] flink pull request #4705: [FLINK-6444] [build] Add a check that '@VisibleFor...

2017-10-17 Thread pnowojski
Github user pnowojski commented on a diff in the pull request: https://github.com/apache/flink/pull/4705#discussion_r145126703 --- Diff: flink-tests/src/test/java/org/apache/flink/test/manual/CheckVisibleForTestingUsage.java --- @@ -0,0 +1,65 @@ +/* + * Licensed to the

[GitHub] flink issue #4841: [FLINK-7368][metrics] Backport synchronization fix for Me...

2017-10-18 Thread pnowojski
Github user pnowojski commented on the issue: https://github.com/apache/flink/pull/4841 @zentol indeed, thanks for pointing out. There was one more place `MutableIOMetrics.java`. ---

[GitHub] flink pull request #4851: [FLINK-7623][tests] Add tests to make sure operato...

2017-10-18 Thread pnowojski
GitHub user pnowojski opened a pull request: https://github.com/apache/flink/pull/4851 [FLINK-7623][tests] Add tests to make sure operator is never restored when using new operator id ## What is the purpose of the change This PR adds tests coverage for correct behaviour of

[GitHub] flink issue #4841: [FLINK-7368][metrics] Backport synchronization fix for Me...

2017-10-18 Thread pnowojski
Github user pnowojski commented on the issue: https://github.com/apache/flink/pull/4841 Yes sure, thanks for merging ---

[GitHub] flink pull request #4841: [FLINK-7368][metrics] Backport synchronization fix...

2017-10-18 Thread pnowojski
Github user pnowojski closed the pull request at: https://github.com/apache/flink/pull/4841 ---

[GitHub] flink pull request #4840: [FLINK-7368][metrics] Make MetricStore ThreadSafe ...

2017-10-18 Thread pnowojski
Github user pnowojski commented on a diff in the pull request: https://github.com/apache/flink/pull/4840#discussion_r145611565 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/rest/handler/legacy/metrics/MetricStore.java --- @@ -38,29 +43,136 @@ import static

[GitHub] flink issue #4705: [FLINK-6444] [build] Add a check that '@VisibleForTesting...

2017-10-19 Thread pnowojski
Github user pnowojski commented on the issue: https://github.com/apache/flink/pull/4705 Hmmm, I looked a little bit and it seems like there are tools doing this: https://github.com/google/guava/issues/1640 https://github.com/WorksApplications/findbugs-plugin https

[GitHub] flink issue #4751: [FLINK-7739][kafka-tests] Throttle down data producing th...

2017-10-19 Thread pnowojski
Github user pnowojski commented on the issue: https://github.com/apache/flink/pull/4751 But this is not supposed to be a stress test, so why make it more computation demanding/longer to execute? ---

[GitHub] flink pull request #4533: [FLINK-7416][network] Implement Netty receiver out...

2017-10-19 Thread pnowojski
Github user pnowojski commented on a diff in the pull request: https://github.com/apache/flink/pull/4533#discussion_r144244479 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannelTest.java --- @@ -304,12 +304,21 @@ public

[GitHub] flink pull request #4533: [FLINK-7416][network] Implement Netty receiver out...

2017-10-19 Thread pnowojski
Github user pnowojski commented on a diff in the pull request: https://github.com/apache/flink/pull/4533#discussion_r144244569 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannelTest.java --- @@ -378,32 +387,47 @@ public

[GitHub] flink pull request #4533: [FLINK-7416][network] Implement Netty receiver out...

2017-10-19 Thread pnowojski
Github user pnowojski commented on a diff in the pull request: https://github.com/apache/flink/pull/4533#discussion_r145697953 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/netty/CreditBasedClientHandler.java --- @@ -37,18 +43,29 @@ import

[GitHub] flink pull request #4533: [FLINK-7416][network] Implement Netty receiver out...

2017-10-19 Thread pnowojski
Github user pnowojski commented on a diff in the pull request: https://github.com/apache/flink/pull/4533#discussion_r145697819 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/io/network/netty/PartitionRequestClientHandlerTest.java --- @@ -208,6 +211,53 @@ public

[GitHub] flink pull request #4533: [FLINK-7416][network] Implement Netty receiver out...

2017-10-19 Thread pnowojski
Github user pnowojski commented on a diff in the pull request: https://github.com/apache/flink/pull/4533#discussion_r145697848 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/io/network/netty/PartitionRequestClientHandlerTest.java --- @@ -208,6 +211,53 @@ public

[GitHub] flink pull request #4533: [FLINK-7416][network] Implement Netty receiver out...

2017-10-19 Thread pnowojski
Github user pnowojski commented on a diff in the pull request: https://github.com/apache/flink/pull/4533#discussion_r145697924 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/netty/CreditBasedClientHandler.java --- @@ -37,18 +43,29 @@ import

[GitHub] flink pull request #4533: [FLINK-7416][network] Implement Netty receiver out...

2017-10-19 Thread pnowojski
Github user pnowojski commented on a diff in the pull request: https://github.com/apache/flink/pull/4533#discussion_r145692507 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/netty/CreditBasedClientHandler.java --- @@ -269,4 +315,49 @@ private void

[GitHub] flink pull request #4851: [FLINK-7623][tests] Add tests to make sure operato...

2017-10-20 Thread pnowojski
Github user pnowojski commented on a diff in the pull request: https://github.com/apache/flink/pull/4851#discussion_r145917513 --- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/tasks/RestoreStreamTaskTest.java --- @@ -0,0 +1,325

[GitHub] flink pull request #4851: [FLINK-7623][tests] Add tests to make sure operato...

2017-10-20 Thread pnowojski
Github user pnowojski commented on a diff in the pull request: https://github.com/apache/flink/pull/4851#discussion_r145915808 --- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/tasks/RestoreStreamTaskTest.java --- @@ -0,0 +1,325

[GitHub] flink pull request #4851: [FLINK-7623][tests] Add tests to make sure operato...

2017-10-20 Thread pnowojski
Github user pnowojski commented on a diff in the pull request: https://github.com/apache/flink/pull/4851#discussion_r145903139 --- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/tasks/RestoreStreamTaskTest.java --- @@ -0,0 +1,325

[GitHub] flink pull request #4851: [FLINK-7623][tests] Add tests to make sure operato...

2017-10-20 Thread pnowojski
Github user pnowojski commented on a diff in the pull request: https://github.com/apache/flink/pull/4851#discussion_r145903567 --- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/tasks/RestoreStreamTaskTest.java --- @@ -0,0 +1,325

[GitHub] flink pull request #4851: [FLINK-7623][tests] Add tests to make sure operato...

2017-10-20 Thread pnowojski
Github user pnowojski commented on a diff in the pull request: https://github.com/apache/flink/pull/4851#discussion_r145917932 --- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/tasks/RestoreStreamTaskTest.java --- @@ -0,0 +1,325

[GitHub] flink pull request #4851: [FLINK-7623][tests] Add tests to make sure operato...

2017-10-20 Thread pnowojski
Github user pnowojski commented on a diff in the pull request: https://github.com/apache/flink/pull/4851#discussion_r145902939 --- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/tasks/AcknowledgeStreamMockEnvironment.java --- @@ -0,0 +1,65

[GitHub] flink issue #4851: [FLINK-7623][tests] Add tests to make sure operator is ne...

2017-10-20 Thread pnowojski
Github user pnowojski commented on the issue: https://github.com/apache/flink/pull/4851 It appears that current behaviour is as you wished @StefanRRichter: - Operator participated in checkpoint, data written -> `isRestored == true` - Operator participated in checkpo

[GitHub] flink issue #4777: [FLINK-7765] Enable dependency convergence

2017-10-20 Thread pnowojski
Github user pnowojski commented on the issue: https://github.com/apache/flink/pull/4777 I created subtasks in JIRA for the follow ups, however I have bundled them pretty arbitrarily (creating a subtask per module would be tiresome) ---

[GitHub] flink pull request #4533: [FLINK-7416][network] Implement Netty receiver out...

2017-10-20 Thread pnowojski
Github user pnowojski commented on a diff in the pull request: https://github.com/apache/flink/pull/4533#discussion_r145978661 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/netty/CreditBasedClientHandler.java --- @@ -269,4 +315,49 @@ private void

[GitHub] flink pull request #4533: [FLINK-7416][network] Implement Netty receiver out...

2017-10-20 Thread pnowojski
Github user pnowojski commented on a diff in the pull request: https://github.com/apache/flink/pull/4533#discussion_r145980535 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/io/network/netty/PartitionRequestClientHandlerTest.java --- @@ -208,6 +211,53 @@ public

[GitHub] flink issue #4875: [FLINK-7889] Enable dependency convergence in flink-strea...

2017-10-20 Thread pnowojski
Github user pnowojski commented on the issue: https://github.com/apache/flink/pull/4875 Doesn't this require fixing/enabling convergence in flink-shaded-hadoop and flink-runtime? ---

[GitHub] flink pull request #4876: [FLINK-7737][filesystem] Add syncOnFlush flag to S...

2017-10-20 Thread pnowojski
GitHub user pnowojski opened a pull request: https://github.com/apache/flink/pull/4876 [FLINK-7737][filesystem] Add syncOnFlush flag to StreamWriterBase ## What is the purpose of the change It depends whether to call hsync or hflush on the underlying file system and user

[GitHub] flink pull request #3951: [FLINK-6461] Replace usages of deprecated web port...

2017-06-19 Thread pnowojski
Github user pnowojski commented on a diff in the pull request: https://github.com/apache/flink/pull/3951#discussion_r122701964 --- Diff: flink-runtime/src/main/scala/org/apache/flink/runtime/jobmanager/JobManager.scala --- @@ -173,7 +173,7 @@ class JobManager( * to run in

[GitHub] flink pull request #3844: [FLINK-5892] Enable 1.2 keyed state test

2017-06-19 Thread pnowojski
Github user pnowojski commented on a diff in the pull request: https://github.com/apache/flink/pull/3844#discussion_r122718852 --- Diff: flink-tests/src/test/java/org/apache/flink/test/state/operator/restore/keyed/KeyedJob.java --- @@ -98,10 +98,9 @@ public static void main

[GitHub] flink pull request #4034: [FLINK-6786] [metrics] Deduplicate QueryScopeIntoT...

2017-06-19 Thread pnowojski
Github user pnowojski commented on a diff in the pull request: https://github.com/apache/flink/pull/4034#discussion_r122725538 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/metrics/dump/QueryScopeInfoTest.java --- @@ -20,58 +20,140 @@ import

[GitHub] flink pull request #3844: [FLINK-5892] Enable 1.2 keyed state test

2017-06-20 Thread pnowojski
Github user pnowojski commented on a diff in the pull request: https://github.com/apache/flink/pull/3844#discussion_r122897990 --- Diff: flink-tests/src/test/java/org/apache/flink/test/state/operator/restore/keyed/KeyedJob.java --- @@ -98,10 +98,9 @@ public static void main

[GitHub] flink issue #3844: [FLINK-5892] Enable 1.2 keyed state test

2017-06-20 Thread pnowojski
Github user pnowojski commented on the issue: https://github.com/apache/flink/pull/3844 Are you sure that errors in travis are intermittent or unrelated to your change? One is already reported here: https://issues.apache.org/jira/browse/FLINK-6843 but second one I'm not

[GitHub] flink pull request #4034: [FLINK-6786] [metrics] Deduplicate QueryScopeIntoT...

2017-06-20 Thread pnowojski
Github user pnowojski commented on a diff in the pull request: https://github.com/apache/flink/pull/4034#discussion_r122986064 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/metrics/dump/QueryScopeInfoTest.java --- @@ -20,58 +20,140 @@ import

[GitHub] flink pull request #4454: [hotfix][docs] Add section in docs about writing u...

2017-08-01 Thread pnowojski
GitHub user pnowojski opened a pull request: https://github.com/apache/flink/pull/4454 [hotfix][docs] Add section in docs about writing unit/integration tests Improved documentation. CC @twalthr You can merge this pull request into a Git repository by running: $ git pull

[GitHub] flink pull request #4456: [FLINK-6996][kafka] Increase Xmx for tests

2017-08-01 Thread pnowojski
GitHub user pnowojski opened a pull request: https://github.com/apache/flink/pull/4456 [FLINK-6996][kafka] Increase Xmx for tests As reported by @NicoK, sometimes 1000m was not enough memory to run at-least-once tests with broker failures on Travis. I remember having the same

[GitHub] flink issue #4456: [FLINK-7343][kafka] Increase Xmx for tests

2017-08-03 Thread pnowojski
Github user pnowojski commented on the issue: https://github.com/apache/flink/pull/4456 @NicoK sounds reasonable however this will drop `-UseGCOverheadLimit` flag. It should be fine when at the same time we increase the `Xmx`. Btw, having `UseGCOverheadLimit` seems a little bit fishy

[GitHub] flink pull request #4470: [FLINK-7343] Simulate network failures in kafka at...

2017-08-03 Thread pnowojski
GitHub user pnowojski opened a pull request: https://github.com/apache/flink/pull/4470 [FLINK-7343] Simulate network failures in kafka at-least-once test We shouldn't fail KafkaServers directly, because they might not be able to flush the data (`log.flush.interval.***` prope

[GitHub] flink pull request #4368: [FLINK-7210] Introduce TwoPhaseCommitSinkFunction

2017-08-04 Thread pnowojski
Github user pnowojski commented on a diff in the pull request: https://github.com/apache/flink/pull/4368#discussion_r131330545 --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/sink/TwoPhaseCommitSinkFunction.java --- @@ -51,39 +48,52

[GitHub] flink pull request #4368: [FLINK-7210] Introduce TwoPhaseCommitSinkFunction

2017-08-04 Thread pnowojski
Github user pnowojski commented on a diff in the pull request: https://github.com/apache/flink/pull/4368#discussion_r131332988 --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/sink/TwoPhaseCommitSinkFunction.java --- @@ -0,0 +1,311

[GitHub] flink pull request #4368: [FLINK-7210] Introduce TwoPhaseCommitSinkFunction

2017-08-04 Thread pnowojski
Github user pnowojski commented on a diff in the pull request: https://github.com/apache/flink/pull/4368#discussion_r131334260 --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/sink/TwoPhaseCommitSinkFunction.java --- @@ -0,0 +1,311

[GitHub] flink pull request #4454: [hotfix][docs] Add section in docs about writing u...

2017-08-04 Thread pnowojski
Github user pnowojski commented on a diff in the pull request: https://github.com/apache/flink/pull/4454#discussion_r131342944 --- Diff: docs/dev/testing.md --- @@ -0,0 +1,102 @@ +--- +title: "Testing" +nav-parent_id: dev +nav-pos: 110 +--- + +

[GitHub] flink pull request #4454: [hotfix][docs] Add section in docs about writing u...

2017-08-04 Thread pnowojski
Github user pnowojski commented on a diff in the pull request: https://github.com/apache/flink/pull/4454#discussion_r131337294 --- Diff: docs/dev/testing.md --- @@ -0,0 +1,102 @@ +--- +title: "Testing" +nav-parent_id: dev +nav-pos: 110 +--- + +

[GitHub] flink pull request #4454: [hotfix][docs] Add section in docs about writing u...

2017-08-04 Thread pnowojski
Github user pnowojski commented on a diff in the pull request: https://github.com/apache/flink/pull/4454#discussion_r131343119 --- Diff: docs/dev/testing.md --- @@ -0,0 +1,102 @@ +--- +title: "Testing" +nav-parent_id: dev +nav-pos: 110 +--- + +

[GitHub] flink pull request #4454: [hotfix][docs] Add section in docs about writing u...

2017-08-04 Thread pnowojski
Github user pnowojski commented on a diff in the pull request: https://github.com/apache/flink/pull/4454#discussion_r131344197 --- Diff: docs/dev/testing.md --- @@ -0,0 +1,102 @@ +--- +title: "Testing" +nav-parent_id: dev +nav-pos: 110 +--- + +

[GitHub] flink issue #4454: [hotfix][docs] Add section in docs about writing unit/int...

2017-08-04 Thread pnowojski
Github user pnowojski commented on the issue: https://github.com/apache/flink/pull/4454 @tzulitai maybe I will add a link to `TwoPhaseCommitSinkFunctionTest` for how to test `exactly-once`? --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] flink issue #4368: [FLINK-7210] Introduce TwoPhaseCommitSinkFunction

2017-08-07 Thread pnowojski
Github user pnowojski commented on the issue: https://github.com/apache/flink/pull/4368 Thanks! :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if

[GitHub] flink pull request #4464: [FLINK-7354][tests] ignore "initialSeedUniquifierG...

2017-08-07 Thread pnowojski
Github user pnowojski commented on a diff in the pull request: https://github.com/apache/flink/pull/4464#discussion_r131633276 --- Diff: flink-tests/src/test/java/org/apache/flink/test/runtime/minicluster/LocalFlinkMiniClusterITCase.java --- @@ -49,7 +49,7 @@ */ public

[GitHub] flink issue #4456: [FLINK-7343][kafka] Increase Xmx for tests

2017-08-07 Thread pnowojski
Github user pnowojski commented on the issue: https://github.com/apache/flink/pull/4456 Thanks :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if

[GitHub] flink pull request #4470: [FLINK-7343] Simulate network failures in kafka at...

2017-08-07 Thread pnowojski
Github user pnowojski commented on a diff in the pull request: https://github.com/apache/flink/pull/4470#discussion_r131648638 --- Diff: flink-test-utils-parent/flink-test-utils/src/main/java/org/apache/flink/networking/NetworkFailureHandler.java --- @@ -0,0 +1,176

  1   2   3   4   5   6   7   8   9   10   >