GitHub user NicoK opened a pull request:
https://github.com/apache/flink/pull/5255
[FLINK-8221][network-benchmarks] Define latency network benchmarks in Flink
project
## What is the purpose of the change
This PR is based on #5134 and adds tools to setup simple micro-benchmarks
for verifying the latency of the network stack. It also adds one latency
benchmark itself.
## Brief change log
- add latency micro-benchmark tools in `StreamNetworkPointToPointBenchmark`
- add a simple point-to-point latency benchmark in
`StreamNetworkPointToPointBenchmarkTest`
## Verifying this change
The defined benchmarks can be run manually and are auto-executed by
flink-benchmarks (dataArtisans/flink-benchmarks#3).
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): **no**
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: **no**
- The serializers: **no**
- The runtime per-record code paths (performance sensitive): **no**
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Yarn/Mesos, ZooKeeper: **no**
- The S3 file system connector: **no**
## Documentation
- Does this pull request introduce a new feature? **no**
- If yes, how is the feature documented? **not applicable**
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/NicoK/flink flink-8221
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/5255.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5255
----
commit 82e3fa72af108c92a179ea441bf67e8b2e494d23
Author: Zhijiang <wangzhijiang999@...>
Date: 2017-08-10T05:29:13Z
[FLINK-7406][network] Implement Netty receiver incoming pipeline for
credit-based
commit 7c364f84df0969cc135c5a9835681c9d2d8e678a
Author: Zhijiang <wangzhijiang999@...>
Date: 2017-09-28T15:39:26Z
[FLINK-7416][network] Implement Netty receiver outgoing pipeline for
credit-based
commit cfe85ced9a4c5bbc342e527d8b59e17ecf1cdad6
Author: Piotr Nowojski <piotr.nowojski@...>
Date: 2017-12-18T14:26:20Z
[hotfix][network] Drop redundant this reference usages
commit 69fe14ce70dee11f8eed173f4ad9af7616de5d76
Author: Piotr Nowojski <piotr.nowojski@...>
Date: 2017-11-28T07:41:40Z
[hotfix][test] Add timeout for joining with CheckedThread
commit ab954dc1ba1d254a584e5d294ed2b99193964828
Author: Piotr Nowojski <piotr.nowojski@...>
Date: 2017-11-28T07:42:19Z
[hotfix][util] Added suppressExceptions for lambda functions
commit b9ddbac5da1ebdfbcda4d0c57427c59d4ba9410b
Author: Piotr Nowojski <piotr.nowojski@...>
Date: 2017-11-28T15:49:37Z
[FLINK-8178][network] Introduce not threadsafe write only BufferBuilder
While Buffer class is used in multithreaded context it requires
synchronisation.
Previously it was miss-leading and unclear, suggesting that
RecordSerializer should
take into account synchronisation of the Buffer that's holding. With
NotThreadSafe
BufferBuilder there is now clear separation between single-threaded
writing/creating
a BufferBuilder and multithreaded Buffer handling/retaining/recycling.
This increases throughput of network stack by factor of 2, because
previously
method getMemorySegment() was called twice per record and it is a
synchronized
method on recycleLock, while RecordSerializer is sole owner of the Buffer at
this point, so synchronisation is not needed.
commit 1c015dd3eebe7171606fa1d90045733eac2c6b75
Author: Piotr Nowojski <piotr.nowojski@...>
Date: 2017-12-01T10:14:06Z
[FLINK-8208][network-tests] Reduce mockito usage in RecordWriterTest
commit 8101dc8f422a815bead8cedf92bd6a361d27ec44
Author: Piotr Nowojski <piotr.nowojski@...>
Date: 2017-12-04T14:08:12Z
[hotfix][network-tests] Simplify TestPooledBufferProvider
commit bdcce927bd0b15739fa7be3da6f5f656fbb36790
Author: Piotr Nowojski <piotr.nowojski@...>
Date: 2017-12-04T14:13:06Z
[FLINK-8209][network-tests] Make LocalBufferPoolDestroyTest less
implementation dependent
commit dc5427696476d898da6ece19c7e05f6063ed98ae
Author: Piotr Nowojski <piotr.nowojski@...>
Date: 2017-12-04T14:13:57Z
[FLINK-8207][network-tests] Unify TestInfiniteBufferProvider and
TestPooledBufferProvider
commit 9e55e2c2f1bad338825e9847cbfe64a2de5b6172
Author: Piotr Nowojski <piotr.nowojski@...>
Date: 2017-12-05T08:36:55Z
[FLINK-8210][network-tests] Collect results into proper mock in
MockEnvironment
commit b3752d2b113ed6e4aa9d78cd0aaf3115cfe7776a
Author: Piotr Nowojski <piotr.nowojski@...>
Date: 2017-12-05T08:42:19Z
[hotfix][runtime] Remove unused methods
commit 52bd79408433821d5ca1c79e1afd09e046ed6e2b
Author: Piotr Nowojski <piotr.nowojski@...>
Date: 2017-12-06T15:02:26Z
[FLINK-8214][streaming-tests] Collect results into proper mock in
StreamMockEnvironment
commit abeddc9bfd2d279cc042bbd1b7aeb28c275e9db2
Author: Piotr Nowojski <piotr.nowojski@...>
Date: 2017-12-07T09:03:32Z
[FLINK-8220][network-benchmarks] Define network benchmarks in Flink project
commit daed2c7ad7a4407951233688dd0e8501b80ec0b9
Author: Nico Kruber <nico@...>
Date: 2017-12-07T09:03:49Z
[FLINK-8221][network-benchmarks] Define latency network benchmarks in Flink
project
----
---
