Il Dom 6 Mar 2022, 05:04 Haiting Jiang <jianghait...@apache.org> ha scritto:
> This is a global setting now. But I wonder if we should compress it only > if the size > is over a threshold? Good idea Enrico Because: > 1. It's not easy for us to notice some managed cursor info is too large in > advance, normally it would be found only if it have actual impact. But if > we enable this compression in advance, it will took some extra computing > resources. > 2. It seems that it won't be a common case that this managed cursor info > is too large (only if there are a lot individualDeletedMessages and > batchedEntryDeletionIndexInfo). So not quite necessary to compress all > managed cursor info. > > Regards, > Haiting > > > On 2022/03/02 04:41:16 Zixuan Liu wrote: > > Hi Pulsar Community, > > > > > > I create a proposal that support ManagedCursorInfo compression. > > > > The proposal can be found: https://github.com/apache/pulsar/issues/14395 > > > > > > Motivation > > > > The cursor data is managed by ZooKeeper/etcd metadata store. When > > cursor data becomes more and more, the data size will increase and > > will take a lot of time to pull the data. Therefore, it is necessary > > to add compression for the cursor, which can reduce the size of data > > and reduce the time of pulling data. > > Goal > > > > Support use the LZ4/ZLIB/ZSTD/SNAPPY to compress the ManagedCursorInfo. > > Implementation > > > > - Cursor compression format > > [MAGIC_NUMBER] + [METADATA_SIZE] + [METADATA_PAYLOAD] + > > [MANAGED_CURSOR_INFO_PAYLOAD] > > > > > > - > > > > MAGIC_NUMBER > > Ox4779 > > - > > > > METADATA > > Add a named ManagedCursorInfoMetadata message to MLDataFormats.proto: > > message ManagedCursorInfoMetadata { > > required CompressionType compressionType = 1; > > required int32 uncompressedSize = 2; > > } > > > > Currently, these compressions have been supported, we only need to > > deal with compression and decompression of the ManagedCursorInfo data: > > > > - > > > > Get CursorInfo from the metadata store > > We will check the cursor data header, if it is compressed, we will > > parse the bytes data by compressed format, otherwise by the original > > way. > > - > > > > Add/Update CursorInfo to the metadata store > > The default is to use compression if the compression type is > specified. > > > > > > Thanks, > > Zixuan > > >
