The reason I replied with an older ticket about compression formats aims to provide some existing discussion for information. I think supporting operator state compression is a good start if we really want to introduce more compression formats in the future.
Makes sense! Thanks for that! On 20/01/2023 10:55, Yun Tang wrote:
I think it's fine to start the vote now considering received positive options already.The reason I replied with an older ticket about compression formats aims to provide some existing discussion for information. I think supporting operator state compression is a good start if we really want to introduce more compression formats in the future.Best Yun Tang ------------------------------------------------------------------------ *From:* Dawid Wysakowicz *Sent:* Friday, January 20, 2023 16:41 *To:* dev@flink.apache.org; Yun Tang*Subject:* Re: [DISCUSS] FLIP-290: Operator state compression (FLINK-30113)I would not unnecessarily extend the scope of the FLIP. From what I've gathered having a compression for operator state is good enough. If requests for more compression formats come I'd start it as a separate effort as it requires more changes both on the operator and keyed state side, including e.g. new configuration options etc. I see that as an independent story that does not need to block this FLIP. I'd start a vote on this FLIP as we've gathered some opinions already. Best, Dawid On 20/01/2023 09:33, Yun Tang wrote:> I think this proposal is reasonable. And the community actually had a discussion on supporting more compression formats for checkpoints several years ago [1].>> The previous ticket is not done due to the priority is not high and the lack of developers to drive. I think we might continue this work considering some developers willing to introduce operator state compression.> > > [1] https://issues.apache.org/jira/browse/FLINK-11313 > > Best > Yun Tang > > ________________________________ > From: Yuan Mei <yuanmei.w...@gmail.com> > Sent: Friday, January 20, 2023 12:56 > To: dev@flink.apache.org <dev@flink.apache.org>> Subject: Re: [DISCUSS] FLIP-290: Operator state compression (FLINK-30113)> > The proposal reads quite reasonable! >> I do not have additional comments as long as the change can insure backward> compatibility. And many thanks to Dawid for catching this! > > Best > > Yuan >> On Thu, Jan 19, 2023 at 6:03 PM Piotr Nowojski <pnowoj...@apache.org> wrote:> >> Hi, >>>> The idea sounds like a nice improvement to complete the feature. I don't>> have any comments on top of what has been written already above. >> >> Bets, >> Piotrek >> >> czw., 19 sty 2023 o 09:57 Etienne Chauchot <echauc...@apache.org> >> napisał(a): >> >>> Hi, >>> >>> In the future we could add new compression algorithms by simply >>> extending /StreamCompressionDecorator/. For now there is only 2 >>> extensions: /UncompressedStreamCompressionDecorator/ and >>> /SnappyStreamCompressionDecorator/. >>>>>> But I agree, I'd stick to /SnappyStreamCompressionDecorator /which is in>>> use for keyed state compression. >>> >>> I'll add a comment on the FLIP saying that. >>> >>> Best >>> >>> Etienne >>> >>> Le 18/01/2023 à 16:36, Dawid Wysakowicz a écrit : >>>> For the compression formats I'd stick with what is supported for the>>>> keyed state, which is either enable or disable compression. The format>>>> used for keyed state backend compression is snappy. >>>> >>>> Best, >>>> >>>> Dawid >>>> >>>> On 18/01/2023 15:33, ConradJam wrote:>>>>> Thanks for driver this flip,I think have some question about this flip>>>>> >>>>> - Can we select the compression format, for example？ such as >>>>> snappy or >>>>> lz0 >>>>> - If compression format is supported, I think the metadata will >>>>> probably >>>>> have to include which compression format >>>>> >>>>> and I agree idea with Dawid >>>>>>>>>> Dawid Wysakowicz <dwysakow...@apache.org> 于2023年1月18日周三 20:55写道：>>>>> >>>>> >>>>>> It makes sense from my side. >>>>>> >>>>>> Could you, just for completeness, extend it with the info what will >> be >>>>>> the compression unit? If understand it correctly you envision each >>>>>> state >>>>>> to be compressed separately. >>>>>> >>>>>> Best, >>>>>> >>>>>> Dawid >>>>>> >>>>>> On 17/01/2023 15:47, Etienne Chauchot wrote: >>>>>>> Hi everyone, >>>>>>>>>>>>>> I just published a new FLIP to introduce operator state compression >>>>>>> (1). This feature is pretty straightforward to implement, so a PR is>>>>>>> already under review (2) but it appeared during the review that to >> be >>>>>>> able to insure backward compatibility of the operator state >> snapshots, >>>>>>> we needed to modify the snapshot format hence the FLIP. >>>>>>> >>>>>>> Please tell me if you have any thoughts about it. >>>>>>> >>>>>>> >>>>>>> [1] >>>>>>>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-290+Operator+state+compression>>>>>>> [2] https://github.com/apache/flink/pull/21636 >>>>>>> >>>>>>> >>>>>>> Best, >>>>>>> >>>>>>> Etienne >>>>>>>
OpenPGP_0x31D2DD10BFC15A2D.asc
Description: OpenPGP public key
OpenPGP_signature
Description: OpenPGP digital signature
