[
https://issues.apache.org/jira/browse/FLINK-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16983338#comment-16983338
]
Yingjie Cao commented on FLINK-14845:
-------------------------------------
> Would it be more complicate to request a buffer from the network pool?
We have to consider how to handle the condition when there is no free buffers
in the buffer pool, and there is may deadlock problem if we choose to wait in a
blocking way. In my opinion, we should guarantee the upper layer of network
stack don't hold more than one buffer (the min number of exclusive buffer per
channel) to avoid deadlock, let us consider an extreme situation, the upper
layer of network stack always request buffer but never recycle them, then it is
doom to lead to the exhaustion of network buffer thus result in deadlock
problem, then how does the upper layer of network know how many buffer it can
request, I think restricting the number to only one buffer is reasonable, and
if the upper layer wants to cache data for some reason, then it should copy the
data to a new place (maybe buffer from MemoryManager or other unpooled buffer)
and recycle the network buffer to network stack.
> Also, could you list the dependencies that you would like to add to the
> runtime for the default compression (if any)?
Currently, only one dependency (org.lz4:lz4-java) is added. Do you think it
acceptable if we don't make it pluggable in this version? If so, we only need
to add one config option which is used to enable/disable the compression, after
there is only one compress algorithm at the moment. We can make it pluggable in
the future. I am asking this mainly because of the limited time. If we make it
pluggable in this, we may need to decide the plugin interface (can be used by
both table and runtime) which may take some time. Currently, I use the
interface copied from table module directly in my POC implementation.
> I was mainly thinking if you want to expose to the user some compression
>tweaking parameters (affecting compression speed/ratio). I would prefer to
>avoid that, at least in the first version.
I totally agree with you, we should avoid to add too many config options in the
first version. We may add some in the future if there are demands from users.
BTW, do you have any other concerns? [~pnowojski]
> Introduce data compression to blocking shuffle.
> -----------------------------------------------
>
> Key: FLINK-14845
> URL: https://issues.apache.org/jira/browse/FLINK-14845
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Network
> Reporter: Yingjie Cao
> Assignee: Yingjie Cao
> Priority: Major
>
> Currently, blocking shuffle writer writes raw output data to disk without
> compression. For IO bounded scenario, this can be optimized by compressing
> the output data. It is better to introduce a compression mechanism and offer
> users a config option to let the user decide whether to compress the shuffle
> data. Actually, we hava implemented compression in our inner Flink version
> and here are some key points:
> 1. Where to compress/decompress?
> Compressing at upstream and decompressing at downstream.
> 2. Which thread do compress/decompress?
> Task threads do compress/decompress.
> 3. Data compression granularity.
> Per buffer.
> 4. How to handle that when data size become even bigger after compression?
> Give up compression in this case and introduce an extra flag to identify if
> the data was compressed, that is, the output may be a mixture of compressed
> and uncompressed data.
>
> We'd like to introduce blocking shuffle data compression to Flink if there
> are interests.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
