Re: [DISCUSS] FLIP-290: Operator state compression (FLINK-30113)

[Dawid Wysakowicz]

I would not unnecessarily extend the scope of the FLIP. From what I've gathered having a compression for operator state is good enough.

I'd start a vote on this FLIP as we've gathered some opinions already.

Best,

Dawid

On 20/01/2023 09:33, Yun Tang wrote:

If requests for more compression formats come I'd start it as a separate effort as it requires more changes both on the operator and keyed state side, including e.g. new configuration options etc. I see that as an independent story that does not need to block this FLIP.

I think this proposal is reasonable. And the community actually had a 
discussion on supporting more compression formats for checkpoints several years 
ago [1].

The previous ticket is not done due to the priority is not high and the lack of 
developers to drive. I think we might continue this work considering some 
developers willing to introduce operator state compression.


[1] https://issues.apache.org/jira/browse/FLINK-11313

Best
Yun Tang

________________________________
From: Yuan Mei <yuanmei.w...@gmail.com>
Sent: Friday, January 20, 2023 12:56
To: dev@flink.apache.org <dev@flink.apache.org>
Subject: Re: [DISCUSS] FLIP-290: Operator state compression (FLINK-30113)

The proposal reads quite reasonable!

I do not have additional comments as long as the change can insure backward
compatibility. And many thanks to Dawid for catching this!

Best

Yuan

On Thu, Jan 19, 2023 at 6:03 PM Piotr Nowojski <pnowoj...@apache.org> wrote:

Hi,

The idea sounds like a nice improvement to complete the feature. I don't
have any comments on top of what has been written already above.

Bets,
Piotrek

czw., 19 sty 2023 o 09:57 Etienne Chauchot <echauc...@apache.org>
napisał(a):

Hi,

In the future we could add new compression algorithms by simply
extending /StreamCompressionDecorator/. For now there is only 2
extensions: /UncompressedStreamCompressionDecorator/ and
/SnappyStreamCompressionDecorator/.

But I agree, I'd stick to /SnappyStreamCompressionDecorator /which is in
use for keyed state compression.

I'll add a comment on the FLIP saying that.

Best

Etienne

Le 18/01/2023 à 16:36, Dawid Wysakowicz a écrit :

For the compression formats I'd stick with what is supported for the
keyed state, which is either enable or disable compression. The format
used for keyed state backend compression is snappy.

Best,

Dawid

On 18/01/2023 15:33, ConradJam wrote:

Thanks for driver this flip,I think have some question about this flip

     - Can we select the compression format, for example？ such as
snappy or
     lz0
     - If compression format is supported, I think the metadata will
probably
     have to include which compression format

and I agree idea with Dawid

Dawid Wysakowicz <dwysakow...@apache.org> 于2023年1月18日周三 20:55写道：

It makes sense from my side.

Could you, just for completeness, extend it with the info what will

be

the compression unit? If understand it correctly you envision each
state
to be compressed separately.

Best,

Dawid

On 17/01/2023 15:47, Etienne Chauchot wrote:

Hi everyone,

I just published a new FLIP to introduce operator state compression
(1). This feature is pretty straightforward to implement, so a PR is
already under review (2) but it appeared during the review that to

be

able to insure backward compatibility of the operator state

snapshots,

we needed to modify the snapshot format hence the FLIP.

Please tell me if you have any thoughts about it.


[1]

https://cwiki.apache.org/confluence/display/FLINK/FLIP-290+Operator+state+compression

[2] https://github.com/apache/flink/pull/21636


Best,

Etienne

OpenPGP_0x31D2DD10BFC15A2D.asc
Description: OpenPGP public key

OpenPGP_signature
Description: OpenPGP digital signature