Great work!
I have no other comments other than the compatibility everybody mentioned. Best, Mattison > On Mar 6, 2022, at 4:55 AM, Enrico Olivelli <eolive...@gmail.com> wrote: > > Good proposal. > It is important that this is disabled by default otherwise we cannot easily > support the rollback > > Apart from that I don't have other comments > > > > Enrico > > Il Sab 5 Mar 2022, 11:22 PengHui Li <peng...@apache.org> ha scritto: > >> Hi Zixuan, >> >> We should add the compatibility part to the proposal. >> And should also provide steps to roll back to the old version which >> enabled the compression in the new version. >> >> I don't have objections to the proposal, and we have done >> the same enhancement for topic metadata >> https://github.com/apache/pulsar/pull/11490, >> and this proposal also follows the same way. >> >> Thanks, >> Penghui >> >> On Thu, Mar 3, 2022 at 10:26 AM Zixuan Liu <node...@gmail.com> wrote: >> >>> Thank you for your feedback. >>> >>> Forward compatibility is required: >>> >>> 1. Get CursorInfo from the metadata store: We will check the cursor data >>> header, if it is compressed, we will parse the bytes data by compressed >>> format, otherwise we will parse the cursor data directly by the original >>> way. >>> >>> 2. Add/Update CursorInfo to the metadata store: The default is to use >>> compression if the compression type is specified, otherwise we will put >>> this data to the metadata store directly. >>> >>> >>> >>>> 2022年3月2日 下午11:48，Zike Yang <zky...@streamnative.io.INVALID> 写道： >>>> >>>> Hi, Zixuan >>>> Thanks for creating this PIP. Here are my thoughts. >>>> >>>>> CursorInfo compression format >>>>> >>>>> [MAGIC_NUMBER] + [METADATA_SIZE] + [METADATA_PAYLOAD] + >>> [MANAGED_CURSOR_INFO_PAYLOAD] >>>>> >>>>> MAGIC_NUMBER: Ox4779 >>>> >>>> Since we change the ManagedCursorInfo data format here. How do we >>>> handle the old data format in the ZK. Could you explain the >>>> compatibility for this PIP? >>>> >>>> Thanks, >>>> Zike Yang >>>> >>>> On Wed, Mar 2, 2022 at 3:34 PM Zixuan Liu <node...@gmail.com <mailto: >>> node...@gmail.com>> wrote: >>>>> >>>>> Hi Pulsar Community, >>>>> >>>>> I create a proposal that ManagedCursorInfo compression. The proposal >>> can be found: https://github.com/apache/pulsar/issues/14529 < >>> https://github.com/apache/pulsar/issues/14529> < >>> https://github.com/apache/pulsar/issues/14529 < >>> https://github.com/apache/pulsar/issues/14529>> >>>>> >>>>> Thanks, >>>>> Zixuan >>>>> >>>>> ------------------ >>>>> >>>>> Motivation >>>>> >>>>> The cursor data is managed by ZooKeeper/etcd metadata store. When >>> cursor data becomes more and more, the data size will increase and will >>> take a lot of time to pull the data. Therefore, it is necessary to add >>> compression for the cursor, which can reduce the size of data and reduce >>> the time of pulling data. >>>>> >>>>> Goal >>>>> >>>>> Support use the LZ4/ZLIB/ZSTD/SNAPPY to compress the >> ManagedCursorInfo. >>>>> >>>>> Implementation >>>>> >>>>> CursorInfo compression format >>>>> >>>>> [MAGIC_NUMBER] + [METADATA_SIZE] + [METADATA_PAYLOAD] + >>> [MANAGED_CURSOR_INFO_PAYLOAD] >>>>> >>>>> MAGIC_NUMBER: Ox4779 >>>>> >>>>> METADATA >>>>> Add a named ManagedCursorInfoMetadata message to MLDataFormats.proto >>>>> >>>>> message ManagedCursorInfoMetadata { >>>>> required CompressionType compressionType = 1; >>>>> required int32 uncompressedSize = 2; >>>>> } >>>>> CursorInfo compression and decompression design >>>>> >>>>> Currently, these compressions types have been defined and implemented >>> by Pulsar, we only need to deal with compression and decompression of the >>> ManagedCursorInfo data: >>>>> >>>>> Get CursorInfo from the metadata store >>>>> We will check the cursor data header, if it is compressed, we will >>> parse the bytes data by compressed format, otherwise by the original way. >>>>> >>>>> Add/Update CursorInfo to the metadata store >>>>> The default is to use compression if the compression type is >> specified. >>>>> >>>>> CursorInfo compression type configuration >>>>> >>>>> Add managedCursorInfoCompressionType in >>> org.apache.pulsar.broker.ServiceConfiguration and >>> org.apache.bookkeeper.mledger.ManagedLedgerFactoryConfig. >>>>> >>>>> >>>> >>>> >>>> -- >>>> Zike Yang >>> >>> >>
