Hi Enrico, Could you review https://github.com/apache/pulsar/pull/14542? Thanks for your time!
Thanks, Zixuan Enrico Olivelli <eolive...@gmail.com> 于2022年3月6日周日 04:55写道： > Good proposal. > It is important that this is disabled by default otherwise we cannot easily > support the rollback > > Apart from that I don't have other comments > > > > Enrico > > Il Sab 5 Mar 2022, 11:22 PengHui Li <peng...@apache.org> ha scritto: > > > Hi Zixuan, > > > > We should add the compatibility part to the proposal. > > And should also provide steps to roll back to the old version which > > enabled the compression in the new version. > > > > I don't have objections to the proposal, and we have done > > the same enhancement for topic metadata > > https://github.com/apache/pulsar/pull/11490, > > and this proposal also follows the same way. > > > > Thanks, > > Penghui > > > > On Thu, Mar 3, 2022 at 10:26 AM Zixuan Liu <node...@gmail.com> wrote: > > > > > Thank you for your feedback. > > > > > > Forward compatibility is required: > > > > > > 1. Get CursorInfo from the metadata store: We will check the cursor > data > > > header, if it is compressed, we will parse the bytes data by compressed > > > format, otherwise we will parse the cursor data directly by the > original > > > way. > > > > > > 2. Add/Update CursorInfo to the metadata store: The default is to use > > > compression if the compression type is specified, otherwise we will put > > > this data to the metadata store directly. > > > > > > > > > > > > > 2022年3月2日 下午11:48，Zike Yang <zky...@streamnative.io.INVALID> 写道： > > > > > > > > Hi, Zixuan > > > > Thanks for creating this PIP. Here are my thoughts. > > > > > > > >> CursorInfo compression format > > > >> > > > >> [MAGIC_NUMBER] + [METADATA_SIZE] + [METADATA_PAYLOAD] + > > > [MANAGED_CURSOR_INFO_PAYLOAD] > > > >> > > > >> MAGIC_NUMBER: Ox4779 > > > > > > > > Since we change the ManagedCursorInfo data format here. How do we > > > > handle the old data format in the ZK. Could you explain the > > > > compatibility for this PIP? > > > > > > > > Thanks, > > > > Zike Yang > > > > > > > > On Wed, Mar 2, 2022 at 3:34 PM Zixuan Liu <node...@gmail.com > <mailto: > > > node...@gmail.com>> wrote: > > > >> > > > >> Hi Pulsar Community, > > > >> > > > >> I create a proposal that ManagedCursorInfo compression. The proposal > > > can be found: https://github.com/apache/pulsar/issues/14529 < > > > https://github.com/apache/pulsar/issues/14529> < > > > https://github.com/apache/pulsar/issues/14529 < > > > https://github.com/apache/pulsar/issues/14529>> > > > >> > > > >> Thanks, > > > >> Zixuan > > > >> > > > >> ------------------ > > > >> > > > >> Motivation > > > >> > > > >> The cursor data is managed by ZooKeeper/etcd metadata store. When > > > cursor data becomes more and more, the data size will increase and will > > > take a lot of time to pull the data. Therefore, it is necessary to add > > > compression for the cursor, which can reduce the size of data and > reduce > > > the time of pulling data. > > > >> > > > >> Goal > > > >> > > > >> Support use the LZ4/ZLIB/ZSTD/SNAPPY to compress the > > ManagedCursorInfo. > > > >> > > > >> Implementation > > > >> > > > >> CursorInfo compression format > > > >> > > > >> [MAGIC_NUMBER] + [METADATA_SIZE] + [METADATA_PAYLOAD] + > > > [MANAGED_CURSOR_INFO_PAYLOAD] > > > >> > > > >> MAGIC_NUMBER: Ox4779 > > > >> > > > >> METADATA > > > >> Add a named ManagedCursorInfoMetadata message to MLDataFormats.proto > > > >> > > > >> message ManagedCursorInfoMetadata { > > > >> required CompressionType compressionType = 1; > > > >> required int32 uncompressedSize = 2; > > > >> } > > > >> CursorInfo compression and decompression design > > > >> > > > >> Currently, these compressions types have been defined and > implemented > > > by Pulsar, we only need to deal with compression and decompression of > the > > > ManagedCursorInfo data: > > > >> > > > >> Get CursorInfo from the metadata store > > > >> We will check the cursor data header, if it is compressed, we will > > > parse the bytes data by compressed format, otherwise by the original > way. > > > >> > > > >> Add/Update CursorInfo to the metadata store > > > >> The default is to use compression if the compression type is > > specified. > > > >> > > > >> CursorInfo compression type configuration > > > >> > > > >> Add managedCursorInfoCompressionType in > > > org.apache.pulsar.broker.ServiceConfiguration and > > > org.apache.bookkeeper.mledger.ManagedLedgerFactoryConfig. > > > >> > > > >> > > > > > > > > > > > > -- > > > > Zike Yang > > > > > > > > >
