This is a global setting now. But I wonder if we should compress it only if the size is over a threshold? Because: 1. It's not easy for us to notice some managed cursor info is too large in advance, normally it would be found only if it have actual impact. But if we enable this compression in advance, it will took some extra computing resources. 2. It seems that it won't be a common case that this managed cursor info is too large (only if there are a lot individualDeletedMessages and batchedEntryDeletionIndexInfo). So not quite necessary to compress all managed cursor info.
Regards, Haiting On 2022/03/02 04:41:16 Zixuan Liu wrote: > Hi Pulsar Community, > > > I create a proposal that support ManagedCursorInfo compression. > > The proposal can be found: https://github.com/apache/pulsar/issues/14395 > > > Motivation > > The cursor data is managed by ZooKeeper/etcd metadata store. When > cursor data becomes more and more, the data size will increase and > will take a lot of time to pull the data. Therefore, it is necessary > to add compression for the cursor, which can reduce the size of data > and reduce the time of pulling data. > Goal > > Support use the LZ4/ZLIB/ZSTD/SNAPPY to compress the ManagedCursorInfo. > Implementation > > - Cursor compression format > [MAGIC_NUMBER] + [METADATA_SIZE] + [METADATA_PAYLOAD] + > [MANAGED_CURSOR_INFO_PAYLOAD] > > > - > > MAGIC_NUMBER > Ox4779 > - > > METADATA > Add a named ManagedCursorInfoMetadata message to MLDataFormats.proto: > message ManagedCursorInfoMetadata { > required CompressionType compressionType = 1; > required int32 uncompressedSize = 2; > } > > Currently, these compressions have been supported, we only need to > deal with compression and decompression of the ManagedCursorInfo data: > > - > > Get CursorInfo from the metadata store > We will check the cursor data header, if it is compressed, we will > parse the bytes data by compressed format, otherwise by the original > way. > - > > Add/Update CursorInfo to the metadata store > The default is to use compression if the compression type is specified. > > > Thanks, > Zixuan >
