Good proposal. It is important that this is disabled by default otherwise we cannot easily support the rollback
Apart from that I don't have other comments Enrico Il Sab 5 Mar 2022, 11:22 PengHui Li <peng...@apache.org> ha scritto: > Hi Zixuan, > > We should add the compatibility part to the proposal. > And should also provide steps to roll back to the old version which > enabled the compression in the new version. > > I don't have objections to the proposal, and we have done > the same enhancement for topic metadata > https://github.com/apache/pulsar/pull/11490, > and this proposal also follows the same way. > > Thanks, > Penghui > > On Thu, Mar 3, 2022 at 10:26 AM Zixuan Liu <node...@gmail.com> wrote: > > > Thank you for your feedback. > > > > Forward compatibility is required: > > > > 1. Get CursorInfo from the metadata store: We will check the cursor data > > header, if it is compressed, we will parse the bytes data by compressed > > format, otherwise we will parse the cursor data directly by the original > > way. > > > > 2. Add/Update CursorInfo to the metadata store: The default is to use > > compression if the compression type is specified, otherwise we will put > > this data to the metadata store directly. > > > > > > > > > 2022年3月2日 下午11:48，Zike Yang <zky...@streamnative.io.INVALID> 写道： > > > > > > Hi, Zixuan > > > Thanks for creating this PIP. Here are my thoughts. > > > > > >> CursorInfo compression format > > >> > > >> [MAGIC_NUMBER] + [METADATA_SIZE] + [METADATA_PAYLOAD] + > > [MANAGED_CURSOR_INFO_PAYLOAD] > > >> > > >> MAGIC_NUMBER: Ox4779 > > > > > > Since we change the ManagedCursorInfo data format here. How do we > > > handle the old data format in the ZK. Could you explain the > > > compatibility for this PIP? > > > > > > Thanks, > > > Zike Yang > > > > > > On Wed, Mar 2, 2022 at 3:34 PM Zixuan Liu <node...@gmail.com <mailto: > > node...@gmail.com>> wrote: > > >> > > >> Hi Pulsar Community, > > >> > > >> I create a proposal that ManagedCursorInfo compression. The proposal > > can be found: https://github.com/apache/pulsar/issues/14529 < > > https://github.com/apache/pulsar/issues/14529> < > > https://github.com/apache/pulsar/issues/14529 < > > https://github.com/apache/pulsar/issues/14529>> > > >> > > >> Thanks, > > >> Zixuan > > >> > > >> ------------------ > > >> > > >> Motivation > > >> > > >> The cursor data is managed by ZooKeeper/etcd metadata store. When > > cursor data becomes more and more, the data size will increase and will > > take a lot of time to pull the data. Therefore, it is necessary to add > > compression for the cursor, which can reduce the size of data and reduce > > the time of pulling data. > > >> > > >> Goal > > >> > > >> Support use the LZ4/ZLIB/ZSTD/SNAPPY to compress the > ManagedCursorInfo. > > >> > > >> Implementation > > >> > > >> CursorInfo compression format > > >> > > >> [MAGIC_NUMBER] + [METADATA_SIZE] + [METADATA_PAYLOAD] + > > [MANAGED_CURSOR_INFO_PAYLOAD] > > >> > > >> MAGIC_NUMBER: Ox4779 > > >> > > >> METADATA > > >> Add a named ManagedCursorInfoMetadata message to MLDataFormats.proto > > >> > > >> message ManagedCursorInfoMetadata { > > >> required CompressionType compressionType = 1; > > >> required int32 uncompressedSize = 2; > > >> } > > >> CursorInfo compression and decompression design > > >> > > >> Currently, these compressions types have been defined and implemented > > by Pulsar, we only need to deal with compression and decompression of the > > ManagedCursorInfo data: > > >> > > >> Get CursorInfo from the metadata store > > >> We will check the cursor data header, if it is compressed, we will > > parse the bytes data by compressed format, otherwise by the original way. > > >> > > >> Add/Update CursorInfo to the metadata store > > >> The default is to use compression if the compression type is > specified. > > >> > > >> CursorInfo compression type configuration > > >> > > >> Add managedCursorInfoCompressionType in > > org.apache.pulsar.broker.ServiceConfiguration and > > org.apache.bookkeeper.mledger.ManagedLedgerFactoryConfig. > > >> > > >> > > > > > > > > > -- > > > Zike Yang > > > > >
