Il Lun 7 Mar 2022, 09:16 r...@apache.org <ranxiaolong...@gmail.com> ha scritto:
> Hi Zixuan: > > Here I am more concerned about whether this feature will break backward > compatibility, for historical data or old clusters, how do we use this > feature. > It is disabled by default. New code will be able to read u compressed data. So you can safely update Pulsar. Once you enable the feature then you cannot rollback to previous versions. The fact that we are going to compress only big values will allow you to have more chances to be able to downgrade even if you enabled the feature Enrico Enrico > -- > Thanks > Xiaolong Ran > > Zixuan Liu <node...@gmail.com> 于2022年3月7日周一 15:14写道： > > > Hi everyone, > > > > Good catch! I update my proposal on > > https://github.com/apache/pulsar/issues/14529, and the compatibility > part > > has been appended: > > > > 1. The compression is disabled by default > > 2. We need to consider how to migrate the old data when this compression > > has been enabled. If the cursor data header is compressed format, we will > > parse the bytes data by compressed format, otherwise we will parse the > > cursor data directly by the original way > > > > Zixuan Liu <node...@gmail.com> 于2022年3月7日周一 15:11写道： > > > > > Hi PengHui, > > > > > > Sorry, the correct URL: https://github.com/apache/pulsar/issues/14529. > > > > > > :( Because of the problem of subscription, the email here is very > > > confusing. > > > > > > > > > PengHui Li <peng...@apache.org> 于2022年3月7日周一 12:39写道： > > > > > >> Hi Zixuan, > > >> > > >> Looks like you have added the wrong link for the proposal? > > >> https://github.com/apache/pulsar/issues/14395 is for PIP-44 > > >> > > >> Penghui > > >> > > >> On Mon, Mar 7, 2022 at 12:37 PM PengHui Li <peng...@apache.org> > wrote: > > >> > > >> > > This is a global setting now. But I wonder if we should compress > it > > >> only > > >> > if the size > > >> > is over a threshold? > > >> > > > >> > +1 > > >> > > > >> > Penghui > > >> > > > >> > On Sun, Mar 6, 2022 at 6:57 PM Enrico Olivelli <eolive...@gmail.com > > > > >> > wrote: > > >> > > > >> >> Il Dom 6 Mar 2022, 05:04 Haiting Jiang <jianghait...@apache.org> > ha > > >> >> scritto: > > >> >> > > >> >> > This is a global setting now. But I wonder if we should compress > it > > >> only > > >> >> > if the size > > >> >> > is over a threshold? > > >> >> > > >> >> > > >> >> Good idea > > >> >> > > >> >> Enrico > > >> >> > > >> >> > > >> >> Because: > > >> >> > 1. It's not easy for us to notice some managed cursor info is too > > >> large > > >> >> in > > >> >> > advance, normally it would be found only if it have actual > impact. > > >> But > > >> >> if > > >> >> > we enable this compression in advance, it will took some extra > > >> computing > > >> >> > resources. > > >> >> > 2. It seems that it won't be a common case that this managed > cursor > > >> info > > >> >> > is too large (only if there are a lot individualDeletedMessages > and > > >> >> > batchedEntryDeletionIndexInfo). So not quite necessary to > compress > > >> all > > >> >> > managed cursor info. > > >> >> > > > >> >> > Regards, > > >> >> > Haiting > > >> >> > > > >> >> > > > >> >> > On 2022/03/02 04:41:16 Zixuan Liu wrote: > > >> >> > > Hi Pulsar Community, > > >> >> > > > > >> >> > > > > >> >> > > I create a proposal that support ManagedCursorInfo compression. > > >> >> > > > > >> >> > > The proposal can be found: > > >> >> https://github.com/apache/pulsar/issues/14395 > > >> >> > > > > >> >> > > > > >> >> > > Motivation > > >> >> > > > > >> >> > > The cursor data is managed by ZooKeeper/etcd metadata store. > When > > >> >> > > cursor data becomes more and more, the data size will increase > > and > > >> >> > > will take a lot of time to pull the data. Therefore, it is > > >> necessary > > >> >> > > to add compression for the cursor, which can reduce the size of > > >> data > > >> >> > > and reduce the time of pulling data. > > >> >> > > Goal > > >> >> > > > > >> >> > > Support use the LZ4/ZLIB/ZSTD/SNAPPY to compress the > > >> >> ManagedCursorInfo. > > >> >> > > Implementation > > >> >> > > > > >> >> > > - Cursor compression format > > >> >> > > [MAGIC_NUMBER] + [METADATA_SIZE] + [METADATA_PAYLOAD] + > > >> >> > > [MANAGED_CURSOR_INFO_PAYLOAD] > > >> >> > > > > >> >> > > > > >> >> > > - > > >> >> > > > > >> >> > > MAGIC_NUMBER > > >> >> > > Ox4779 > > >> >> > > - > > >> >> > > > > >> >> > > METADATA > > >> >> > > Add a named ManagedCursorInfoMetadata message to > > >> >> MLDataFormats.proto: > > >> >> > > message ManagedCursorInfoMetadata { > > >> >> > > required CompressionType compressionType = 1; > > >> >> > > required int32 uncompressedSize = 2; > > >> >> > > } > > >> >> > > > > >> >> > > Currently, these compressions have been supported, we only need > > to > > >> >> > > deal with compression and decompression of the > ManagedCursorInfo > > >> data: > > >> >> > > > > >> >> > > - > > >> >> > > > > >> >> > > Get CursorInfo from the metadata store > > >> >> > > We will check the cursor data header, if it is compressed, > we > > >> will > > >> >> > > parse the bytes data by compressed format, otherwise by the > > >> original > > >> >> > > way. > > >> >> > > - > > >> >> > > > > >> >> > > Add/Update CursorInfo to the metadata store > > >> >> > > The default is to use compression if the compression type is > > >> >> > specified. > > >> >> > > > > >> >> > > > > >> >> > > Thanks, > > >> >> > > Zixuan > > >> >> > > > > >> >> > > > >> >> > > >> > > > >> > > > > > >
