Yingjie Cao created FLINK-15455:
-----------------------------------
Summary: Enable TCP connection reuse across multiple jobs.
Key: FLINK-15455
URL: https://issues.apache.org/jira/browse/FLINK-15455
Project: Flink
Issue Type: Improvement
Components: Runtime / Network
Reporter: Yingjie Cao
Currently, tcp connections can be only reuse by tasks residing in the same
TaskManager and consumes the same IntermediateResult. And after job finish or
failover, the TCP connections are closed and new connections must be setup
latter.
As an improvement, we can make tcp connections a cluster level resource which
can be reused by multi jobs. The advantages are as follows:
# Reduce the number of TCP connections so we can save some resources.
# Reduce the overhead of connection setup and close so restarted jobs after
failover and latter jobs submitted to the same session cluster can reuse the
previous connections.
We use Flink session cluster as a service for ad-hoc queries and the users can
produce some statistics or create some statements and reports at any time. Most
of the queries finish in 2s and we find tcp connection reuse help a lot to
reduce the average execution time which means more queries can be processed
using the same resource and time with even better user experience.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
