[
https://issues.apache.org/jira/browse/FLINK-10462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhijiang updated FLINK-10462:
-----------------------------
Description:
Every {{IntermediateResult}} generates a random {{ConnectionIndex}} which will
be included in {{ConnectionID}}.
The {{RemoteInputChannel}} requests to establish tcp connection via
{{ConnectionID}}. That means one tcp connection may be shared by multiple
{{RemoteInputChannel}} {{s which have the same ConnectionID}}. To do so, we can
reduce the physical connections between two {{TaskManager}}s, and it brings
benefits for large scale jobs.
But this sharing is limited only for the same {{IntermediateResult}}, and I
think it is mainly because we may temporarily switch off {{autoread}} for the
channel during back pressure in previous network flow control. For credit-based
mode, the channel is always open for transporting different intermediate data,
so we can further share the tcp connection for different
{{IntermediateResults}} to remove the limit.
was:
Every {{IntermediateResult}} generates a random {{ConnectionIndex}} which will
be included in {{ConnectionID}}.
The {{RemoteInputChannel}} requests to establish tcp connection via
{{ConnectionID}}. That means one tcp connection may be shared by multiple
{{RemoteInputChannel}} {{s which have the same ConnectionID}}. To do so, we can
reduce the physical connections between two {{TaskManager}} s, and it brings
benefits for large scale jobs.
But this sharing is limited only for the same {{IntermediateResult}}, and I
think it is mainly because we may temporarily switch off {{autoread}} for the
channel during back pressure in previous network flow control. For credit-based
mode, the channel is always open for transporting different intermediate data,
so we can further share the tcp connection for different
{{IntermediateResults}} to remove the limit.
> Remove ConnectionIndex for further sharing tcp connection in credit-based
> mode
> -------------------------------------------------------------------------------
>
> Key: FLINK-10462
> URL: https://issues.apache.org/jira/browse/FLINK-10462
> Project: Flink
> Issue Type: Improvement
> Components: Network
> Affects Versions: 1.5.0, 1.5.1, 1.5.2, 1.5.3, 1.6.0, 1.6.1, 1.5.4
> Reporter: zhijiang
> Assignee: zhijiang
> Priority: Minor
>
> Every {{IntermediateResult}} generates a random {{ConnectionIndex}} which
> will be included in {{ConnectionID}}.
> The {{RemoteInputChannel}} requests to establish tcp connection via
> {{ConnectionID}}. That means one tcp connection may be shared by multiple
> {{RemoteInputChannel}} {{s which have the same ConnectionID}}. To do so, we
> can reduce the physical connections between two {{TaskManager}}s, and it
> brings benefits for large scale jobs.
> But this sharing is limited only for the same {{IntermediateResult}}, and I
> think it is mainly because we may temporarily switch off {{autoread}} for the
> channel during back pressure in previous network flow control. For
> credit-based mode, the channel is always open for transporting different
> intermediate data, so we can further share the tcp connection for different
> {{IntermediateResults}} to remove the limit.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
