[
https://issues.apache.org/jira/browse/FLINK-1275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16778988#comment-16778988
]
zhijiang commented on FLINK-1275:
---------------------------------
[~uce], thanks for creating this valuable ticket before and I think compression
feature is necessary to be configured as an option for enhancing the network
stack in some scenarios.
Actually in Alibaba Blink branch, we already realized the compression function
mainly used in large-scale batch jobs especially for SATA disk environment. In
detail, the {{ResultPartitionWriter}} outputs the data into local persistent
files in compression mode to save disk spaces for large data, furthermore it
could also improve the whole shuffle performance because the total transferred
data via network is less than before. Considering the cost of network bandwidth
mentioned above, the compression might be also a proper way for streaming job.
I think the compression feature would be contributed into Flink future,
especially based on the motion of unified engine for both streaming and batch
jobs. But the priority seems not very high currently and I might focus on it
after the pluggable shuffle manager
[https://issues.apache.org/jira/browse/FLINK-10653] is done.
> Add support to compress network I/O
> -----------------------------------
>
> Key: FLINK-1275
> URL: https://issues.apache.org/jira/browse/FLINK-1275
> Project: Flink
> Issue Type: Improvement
> Components: Network
> Affects Versions: 0.8.0
> Reporter: Ufuk Celebi
> Priority: Minor
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
