I think it's fine to start the vote now considering received positive
options already.
The reason I replied with an older ticket about compression formats
aims to provide some existing discussion for information. I think
supporting operator state compression is a good start if we really
want to introduce more compression formats in the future.
Best
Yun Tang
------------------------------------------------------------------------
*From:* Dawid Wysakowicz
*Sent:* Friday, January 20, 2023 16:41
*To:* dev@flink.apache.org; Yun Tang
*Subject:* Re: [DISCUSS] FLIP-290: Operator state compression
(FLINK-30113)
I would not unnecessarily extend the scope of the FLIP. From what I've
gathered having a compression for operator state is good enough.
If requests for more compression formats come I'd start it as a separate
effort as it requires more changes both on the operator and keyed state
side, including e.g. new configuration options etc. I see that as an
independent story that does not need to block this FLIP.
I'd start a vote on this FLIP as we've gathered some opinions already.
Best,
Dawid
On 20/01/2023 09:33, Yun Tang wrote:
> I think this proposal is reasonable. And the community actually had
a discussion on supporting more compression formats for checkpoints
several years ago [1].
>
> The previous ticket is not done due to the priority is not high and
the lack of developers to drive. I think we might continue this work
considering some developers willing to introduce operator state
compression.
>
>
> [1] https://issues.apache.org/jira/browse/FLINK-11313
>
> Best
> Yun Tang
>
> ________________________________
> From: Yuan Mei <yuanmei.w...@gmail.com>
> Sent: Friday, January 20, 2023 12:56
> To: dev@flink.apache.org <dev@flink.apache.org>
> Subject: Re: [DISCUSS] FLIP-290: Operator state compression
(FLINK-30113)
>
> The proposal reads quite reasonable!
>
> I do not have additional comments as long as the change can insure
backward
> compatibility. And many thanks to Dawid for catching this!
>
> Best
>
> Yuan
>
> On Thu, Jan 19, 2023 at 6:03 PM Piotr Nowojski
<pnowoj...@apache.org> wrote:
>
>> Hi,
>>
>> The idea sounds like a nice improvement to complete the feature. I
don't
>> have any comments on top of what has been written already above.
>>
>> Bets,
>> Piotrek
>>
>> czw., 19 sty 2023 o 09:57 Etienne Chauchot <echauc...@apache.org>
>> napisał(a):
>>
>>> Hi,
>>>
>>> In the future we could add new compression algorithms by simply
>>> extending /StreamCompressionDecorator/. For now there is only 2
>>> extensions: /UncompressedStreamCompressionDecorator/ and
>>> /SnappyStreamCompressionDecorator/.
>>>
>>> But I agree, I'd stick to /SnappyStreamCompressionDecorator
/which is in
>>> use for keyed state compression.
>>>
>>> I'll add a comment on the FLIP saying that.
>>>
>>> Best
>>>
>>> Etienne
>>>
>>> Le 18/01/2023 à 16:36, Dawid Wysakowicz a écrit :
>>>> For the compression formats I'd stick with what is supported for the
>>>> keyed state, which is either enable or disable compression. The
format
>>>> used for keyed state backend compression is snappy.
>>>>
>>>> Best,
>>>>
>>>> Dawid
>>>>
>>>> On 18/01/2023 15:33, ConradJam wrote:
>>>>> Thanks for driver this flip,I think have some question about
this flip
>>>>>
>>>>> - Can we select the compression format, for example？ such as
>>>>> snappy or
>>>>> lz0
>>>>> - If compression format is supported, I think the metadata
will
>>>>> probably
>>>>> have to include which compression format
>>>>>
>>>>> and I agree idea with Dawid
>>>>>
>>>>> Dawid Wysakowicz <dwysakow...@apache.org> 于2023年1月18日周三
20:55写道：
>>>>>
>>>>>
>>>>>> It makes sense from my side.
>>>>>>
>>>>>> Could you, just for completeness, extend it with the info what
will
>> be
>>>>>> the compression unit? If understand it correctly you envision each
>>>>>> state
>>>>>> to be compressed separately.
>>>>>>
>>>>>> Best,
>>>>>>
>>>>>> Dawid
>>>>>>
>>>>>> On 17/01/2023 15:47, Etienne Chauchot wrote:
>>>>>>> Hi everyone,
>>>>>>>
>>>>>>> I just published a new FLIP to introduce operator state
compression
>>>>>>> (1). This feature is pretty straightforward to implement, so
a PR is
>>>>>>> already under review (2) but it appeared during the review
that to
>> be
>>>>>>> able to insure backward compatibility of the operator state
>> snapshots,
>>>>>>> we needed to modify the snapshot format hence the FLIP.
>>>>>>>
>>>>>>> Please tell me if you have any thoughts about it.
>>>>>>>
>>>>>>>
>>>>>>> [1]
>>>>>>>
>>
https://cwiki.apache.org/confluence/display/FLINK/FLIP-290+Operator+state+compression
>>>>>>> [2] https://github.com/apache/flink/pull/21636
>>>>>>>
>>>>>>>
>>>>>>> Best,
>>>>>>>
>>>>>>> Etienne
>>>>>>>
