Hi PengHui, Sorry, the correct URL: https://github.com/apache/pulsar/issues/14529.
:( Because of the problem of subscription, the email here is very confusing. PengHui Li <peng...@apache.org> 于2022年3月7日周一 12:39写道： > Hi Zixuan, > > Looks like you have added the wrong link for the proposal? > https://github.com/apache/pulsar/issues/14395 is for PIP-44 > > Penghui > > On Mon, Mar 7, 2022 at 12:37 PM PengHui Li <peng...@apache.org> wrote: > > > > This is a global setting now. But I wonder if we should compress it > only > > if the size > > is over a threshold? > > > > +1 > > > > Penghui > > > > On Sun, Mar 6, 2022 at 6:57 PM Enrico Olivelli <eolive...@gmail.com> > > wrote: > > > >> Il Dom 6 Mar 2022, 05:04 Haiting Jiang <jianghait...@apache.org> ha > >> scritto: > >> > >> > This is a global setting now. But I wonder if we should compress it > only > >> > if the size > >> > is over a threshold? > >> > >> > >> Good idea > >> > >> Enrico > >> > >> > >> Because: > >> > 1. It's not easy for us to notice some managed cursor info is too > large > >> in > >> > advance, normally it would be found only if it have actual impact. > But > >> if > >> > we enable this compression in advance, it will took some extra > computing > >> > resources. > >> > 2. It seems that it won't be a common case that this managed cursor > info > >> > is too large (only if there are a lot individualDeletedMessages and > >> > batchedEntryDeletionIndexInfo). So not quite necessary to compress all > >> > managed cursor info. > >> > > >> > Regards, > >> > Haiting > >> > > >> > > >> > On 2022/03/02 04:41:16 Zixuan Liu wrote: > >> > > Hi Pulsar Community, > >> > > > >> > > > >> > > I create a proposal that support ManagedCursorInfo compression. > >> > > > >> > > The proposal can be found: > >> https://github.com/apache/pulsar/issues/14395 > >> > > > >> > > > >> > > Motivation > >> > > > >> > > The cursor data is managed by ZooKeeper/etcd metadata store. When > >> > > cursor data becomes more and more, the data size will increase and > >> > > will take a lot of time to pull the data. Therefore, it is necessary > >> > > to add compression for the cursor, which can reduce the size of data > >> > > and reduce the time of pulling data. > >> > > Goal > >> > > > >> > > Support use the LZ4/ZLIB/ZSTD/SNAPPY to compress the > >> ManagedCursorInfo. > >> > > Implementation > >> > > > >> > > - Cursor compression format > >> > > [MAGIC_NUMBER] + [METADATA_SIZE] + [METADATA_PAYLOAD] + > >> > > [MANAGED_CURSOR_INFO_PAYLOAD] > >> > > > >> > > > >> > > - > >> > > > >> > > MAGIC_NUMBER > >> > > Ox4779 > >> > > - > >> > > > >> > > METADATA > >> > > Add a named ManagedCursorInfoMetadata message to > >> MLDataFormats.proto: > >> > > message ManagedCursorInfoMetadata { > >> > > required CompressionType compressionType = 1; > >> > > required int32 uncompressedSize = 2; > >> > > } > >> > > > >> > > Currently, these compressions have been supported, we only need to > >> > > deal with compression and decompression of the ManagedCursorInfo > data: > >> > > > >> > > - > >> > > > >> > > Get CursorInfo from the metadata store > >> > > We will check the cursor data header, if it is compressed, we > will > >> > > parse the bytes data by compressed format, otherwise by the original > >> > > way. > >> > > - > >> > > > >> > > Add/Update CursorInfo to the metadata store > >> > > The default is to use compression if the compression type is > >> > specified. > >> > > > >> > > > >> > > Thanks, > >> > > Zixuan > >> > > > >> > > >> > > >
