Hi,
In the future we could add new compression algorithms by simply
extending /StreamCompressionDecorator/. For now there is only 2
extensions: /UncompressedStreamCompressionDecorator/ and
/SnappyStreamCompressionDecorator/.
But I agree, I'd stick to /SnappyStreamCompressionDecorator /which is in
use for keyed state compression.
I'll add a comment on the FLIP saying that.
Best
Etienne
Le 18/01/2023 à 16:36, Dawid Wysakowicz a écrit :
For the compression formats I'd stick with what is supported for the
keyed state, which is either enable or disable compression. The format
used for keyed state backend compression is snappy.
Best,
Dawid
On 18/01/2023 15:33, ConradJam wrote:
Thanks for driver this flip,I think have some question about this flip
- Can we select the compression format, for example？ such as
snappy or
lz0
- If compression format is supported, I think the metadata will
probably
have to include which compression format
and I agree idea with Dawid
Dawid Wysakowicz <dwysakow...@apache.org> 于2023年1月18日周三 20:55写道：
It makes sense from my side.
Could you, just for completeness, extend it with the info what will be
the compression unit? If understand it correctly you envision each
state
to be compressed separately.
Best,
Dawid
On 17/01/2023 15:47, Etienne Chauchot wrote:
Hi everyone,
I just published a new FLIP to introduce operator state compression
(1). This feature is pretty straightforward to implement, so a PR is
already under review (2) but it appeared during the review that to be
able to insure backward compatibility of the operator state snapshots,
we needed to modify the snapshot format hence the FLIP.
Please tell me if you have any thoughts about it.
[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-290+Operator+state+compression
[2] https://github.com/apache/flink/pull/21636
Best,
Etienne
