[
https://issues.apache.org/jira/browse/BEAM-14534?focusedWorklogId=776961&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-776961
]
ASF GitHub Bot logged work on BEAM-14534:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 01/Jun/22 15:35
Start Date: 01/Jun/22 15:35
Worklog Time Spent: 10m
Work Description: lukecwik commented on PR #17783:
URL: https://github.com/apache/beam/pull/17783#issuecomment-1143765738
I see that BEAM-14534 does specifically mention using a wrapping coder to
compress the data and I think it would still be a better solution since it
would allow you to:
* plugin compression into more places such as compressing the shuffle key,
user state and side inputs
* works for all runners and not just Dataflow runner v1. (Note that this
solution won't work for any portable runners like Dataflow runner v2 since you
can't edit the shuffle code)
* the compression can be configured per GroupByKey since in some places it
won't make sense since the values are not easily compressible
You can also create a PipelineVisitor that walks all transforms and replaces
the `KV<KeyCoder, ValueCoder>` with `KV<KeyCoder,
CompressingCoder<ValueCoder>>` so that users don't need to do this themselves
which would give similar behavior as specifying the shuffle compressor.
The other part of the change makes sense to reduce byte[] copies by using
ByteString.
CC: @tudorm
Issue Time Tracking
-------------------
Worklog Id: (was: 776961)
Time Spent: 1h 10m (was: 1h)
> Add an interface to allow users to compress values being written to shuffle
> ---------------------------------------------------------------------------
>
> Key: BEAM-14534
> URL: https://issues.apache.org/jira/browse/BEAM-14534
> Project: Beam
> Issue Type: Improvement
> Components: runner-dataflow
> Reporter: Steve Niemitz
> Assignee: Steve Niemitz
> Priority: P2
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Frequently values being shuffled are large and compressible, while users can
> compress them on their own by using a coder that compresses the data, it
> would be nice to be able to do so globally for all values.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
