Hi everyone, Good catch! I update my proposal on https://github.com/apache/pulsar/issues/14529, and the compatibility part has been appended:
1. The compression is disabled by default 2. We need to consider how to migrate the old data when this compression has been enabled. If the cursor data header is compressed format, we will parse the bytes data by compressed format, otherwise we will parse the cursor data directly by the original way Zixuan Liu <node...@gmail.com> 于2022年3月7日周一 15:11写道： > Hi PengHui, > > Sorry, the correct URL: https://github.com/apache/pulsar/issues/14529. > > :( Because of the problem of subscription, the email here is very > confusing. > > > PengHui Li <peng...@apache.org> 于2022年3月7日周一 12:39写道： > >> Hi Zixuan, >> >> Looks like you have added the wrong link for the proposal? >> https://github.com/apache/pulsar/issues/14395 is for PIP-44 >> >> Penghui >> >> On Mon, Mar 7, 2022 at 12:37 PM PengHui Li <peng...@apache.org> wrote: >> >> > > This is a global setting now. But I wonder if we should compress it >> only >> > if the size >> > is over a threshold? >> > >> > +1 >> > >> > Penghui >> > >> > On Sun, Mar 6, 2022 at 6:57 PM Enrico Olivelli <eolive...@gmail.com> >> > wrote: >> > >> >> Il Dom 6 Mar 2022, 05:04 Haiting Jiang <jianghait...@apache.org> ha >> >> scritto: >> >> >> >> > This is a global setting now. But I wonder if we should compress it >> only >> >> > if the size >> >> > is over a threshold? >> >> >> >> >> >> Good idea >> >> >> >> Enrico >> >> >> >> >> >> Because: >> >> > 1. It's not easy for us to notice some managed cursor info is too >> large >> >> in >> >> > advance, normally it would be found only if it have actual impact. >> But >> >> if >> >> > we enable this compression in advance, it will took some extra >> computing >> >> > resources. >> >> > 2. It seems that it won't be a common case that this managed cursor >> info >> >> > is too large (only if there are a lot individualDeletedMessages and >> >> > batchedEntryDeletionIndexInfo). So not quite necessary to compress >> all >> >> > managed cursor info. >> >> > >> >> > Regards, >> >> > Haiting >> >> > >> >> > >> >> > On 2022/03/02 04:41:16 Zixuan Liu wrote: >> >> > > Hi Pulsar Community, >> >> > > >> >> > > >> >> > > I create a proposal that support ManagedCursorInfo compression. >> >> > > >> >> > > The proposal can be found: >> >> https://github.com/apache/pulsar/issues/14395 >> >> > > >> >> > > >> >> > > Motivation >> >> > > >> >> > > The cursor data is managed by ZooKeeper/etcd metadata store. When >> >> > > cursor data becomes more and more, the data size will increase and >> >> > > will take a lot of time to pull the data. Therefore, it is >> necessary >> >> > > to add compression for the cursor, which can reduce the size of >> data >> >> > > and reduce the time of pulling data. >> >> > > Goal >> >> > > >> >> > > Support use the LZ4/ZLIB/ZSTD/SNAPPY to compress the >> >> ManagedCursorInfo. >> >> > > Implementation >> >> > > >> >> > > - Cursor compression format >> >> > > [MAGIC_NUMBER] + [METADATA_SIZE] + [METADATA_PAYLOAD] + >> >> > > [MANAGED_CURSOR_INFO_PAYLOAD] >> >> > > >> >> > > >> >> > > - >> >> > > >> >> > > MAGIC_NUMBER >> >> > > Ox4779 >> >> > > - >> >> > > >> >> > > METADATA >> >> > > Add a named ManagedCursorInfoMetadata message to >> >> MLDataFormats.proto: >> >> > > message ManagedCursorInfoMetadata { >> >> > > required CompressionType compressionType = 1; >> >> > > required int32 uncompressedSize = 2; >> >> > > } >> >> > > >> >> > > Currently, these compressions have been supported, we only need to >> >> > > deal with compression and decompression of the ManagedCursorInfo >> data: >> >> > > >> >> > > - >> >> > > >> >> > > Get CursorInfo from the metadata store >> >> > > We will check the cursor data header, if it is compressed, we >> will >> >> > > parse the bytes data by compressed format, otherwise by the >> original >> >> > > way. >> >> > > - >> >> > > >> >> > > Add/Update CursorInfo to the metadata store >> >> > > The default is to use compression if the compression type is >> >> > specified. >> >> > > >> >> > > >> >> > > Thanks, >> >> > > Zixuan >> >> > > >> >> > >> >> >> > >> >
