Re: [DISCUSSION][PIP-146] ManagedCursorInfo compression

Hi Enrico, Could you review https://github.com/apache/pulsar/pull/14542? Thanks for your time! Thanks, Zixuan Enrico Olivelli 于2022年3月6日周日 04:55写道： > Good proposal. > It is important that this is disabled by default otherwise we cannot easily > support the rollback > > Apart from that I don't

Re: [DISCUSSION][PIP-146] ManagedCursorInfo compression

Good catch! PengHui Li 于2022年3月8日周二 13:37写道： > > We don't support rollback the data of the previous version Once you > enable > this feature. > > If you want to roll back to an old version, need to disable the cursor > compression, > wait a while, or restart the broker first to make sure the cur

Re: [DISCUSSION][PIP-146] ManagedCursorInfo compression

> We don't support rollback the data of the previous version Once you enable this feature. If you want to roll back to an old version, need to disable the cursor compression, wait a while, or restart the broker first to make sure the cursor data can be flush to the cursor ledger. And then roll bac

Re: [DISCUSSION][PIP-146] ManagedCursorInfo compression

Hi Xiaolong, It is disabled by default. Once you enable this feature: When reading your data, we will check your data header, if it is compressed data, we will parse this data by compression format, otherwise parse it by the original way. When updating your data, we will compress your data by the

Re: [DISCUSSION][PIP-146] ManagedCursorInfo compression

Il Lun 7 Mar 2022, 09:16 r...@apache.org ha scritto: > Hi Zixuan: > > Here I am more concerned about whether this feature will break backward > compatibility, for historical data or old clusters, how do we use this > feature. > It is disabled by default. New code will be able to read u compresse

Re: [DISCUSSION][PIP-146] ManagedCursorInfo compression

Hi Zixuan: Here I am more concerned about whether this feature will break backward compatibility, for historical data or old clusters, how do we use this feature. -- Thanks Xiaolong Ran Zixuan Liu 于2022年3月7日周一 15:14写道： > Hi everyone, > > Good catch! I update my proposal on > https://github.com

Re: [DISCUSSION][PIP-146] ManagedCursorInfo compression

Hi everyone, Good catch! I update my proposal on https://github.com/apache/pulsar/issues/14529, and the compatibility part has been appended: 1. The compression is disabled by default 2. We need to consider how to migrate the old data when this compression has been enabled. If the cursor data hea

Re: [DISCUSSION][PIP-146] ManagedCursorInfo compression

Hi PengHui, Sorry, the correct URL: https://github.com/apache/pulsar/issues/14529. :( Because of the problem of subscription, the email here is very confusing. PengHui Li 于2022年3月7日周一 12:39写道： > Hi Zixuan, > > Looks like you have added the wrong link for the proposal? > https://github.com/apa

Re: [DISCUSSION][PIP-146] ManagedCursorInfo compression

Hi Haiting, Good catch! I can add a threshold to decide to compress or not. Best, Zixuan Haiting Jiang 于2022年3月6日周日 12:04写道： > This is a global setting now. But I wonder if we should compress it only > if the size > is over a threshold? Because: > 1. It's not easy for us to notice some manag

Re: [DISCUSSION][PIP-146] ManagedCursorInfo compression

Hi Mattison, Thanks for your feedback! I think using two configurations is more flexible, and users can set up different compression types. Best, Zixuan mattison chao 于2022年3月6日周日 08:41写道： > Hi, Zi Xuan > > After deep think, I have another question: > > Why don't we combine ledger compaction

Re: [DISCUSSION][PIP-146] ManagedCursorInfo compression

Hi Zixuan, Looks like you have added the wrong link for the proposal? https://github.com/apache/pulsar/issues/14395 is for PIP-44 Penghui On Mon, Mar 7, 2022 at 12:37 PM PengHui Li wrote: > > This is a global setting now. But I wonder if we should compress it only > if the size > is over a thr

Re: [DISCUSSION][PIP-146] ManagedCursorInfo compression

> This is a global setting now. But I wonder if we should compress it only if the size is over a threshold? +1 Penghui On Sun, Mar 6, 2022 at 6:57 PM Enrico Olivelli wrote: > Il Dom 6 Mar 2022, 05:04 Haiting Jiang ha > scritto: > > > This is a global setting now. But I wonder if we should com

Re: [DISCUSSION][PIP-146] ManagedCursorInfo compression

Il Dom 6 Mar 2022, 05:04 Haiting Jiang ha scritto: > This is a global setting now. But I wonder if we should compress it only > if the size > is over a threshold? Good idea Enrico Because: > 1. It's not easy for us to notice some managed cursor info is too large in > advance, normally it

Re: [DISCUSSION][PIP-146] ManagedCursorInfo compression

This is a global setting now. But I wonder if we should compress it only if the size is over a threshold? Because: 1. It's not easy for us to notice some managed cursor info is too large in advance, normally it would be found only if it have actual impact. But if we enable this compression in

Re: [DISCUSSION][PIP-146] ManagedCursorInfo compression

Hi, Zi Xuan After deep think, I have another question: Why don't we combine ledger compaction and cursor compaction into one configuration switch? I’m not sure, do we have users who need to set the compression configuration for ledger and cursor separately? I think if they were to be set, they

Re: [DISCUSSION][PIP-146] ManagedCursorInfo compression

Great work! I have no other comments other than the compatibility everybody mentioned. Best, Mattison > On Mar 6, 2022, at 4:55 AM, Enrico Olivelli wrote: > > Good proposal. > It is important that this is disabled by default otherwise we cannot easily > support the rollback > > Apart from t

Re: [DISCUSSION][PIP-146] ManagedCursorInfo compression

Good proposal. It is important that this is disabled by default otherwise we cannot easily support the rollback Apart from that I don't have other comments Enrico Il Sab 5 Mar 2022, 11:22 PengHui Li ha scritto: > Hi Zixuan, > > We should add the compatibility part to the proposal. > And shou

Re: [DISCUSSION][PIP-146] ManagedCursorInfo compression

Hi Zixuan, We should add the compatibility part to the proposal. And should also provide steps to roll back to the old version which enabled the compression in the new version. I don't have objections to the proposal, and we have done the same enhancement for topic metadata https://github.com/apa

Re: [DISCUSSION][PIP-146] ManagedCursorInfo compression

Thank you for your feedback. Forward compatibility is required: 1. Get CursorInfo from the metadata store: We will check the cursor data header, if it is compressed, we will parse the bytes data by compressed format, otherwise we will parse the cursor data directly by the original way. 2. Add

Re: [DISCUSSION][PIP-146] ManagedCursorInfo compression

Update proposal link: https://github.com/apache/pulsar/issues/14529 Zixuan Liu 于2022年3月2日周三 12:41写道： > Hi Pulsar Community, > > > I create a proposal that support ManagedCursorInfo compression. > > The proposal can be found: https://github.com/apache/pulsar/issues/14395 > > > Motivation > > The

[DISCUSSION][PIP-146] ManagedCursorInfo compression

Hi Pulsar Community, I create a proposal that support ManagedCursorInfo compression. The proposal can be found: https://github.com/apache/pulsar/issues/14395 Motivation The cursor data is managed by ZooKeeper/etcd metadata store. When cursor data becomes more and more, the data size will incr

Re: [DISCUSSION][PIP-146] ManagedCursorInfo compression

Hi, Zixuan Thanks for creating this PIP. Here are my thoughts. > CursorInfo compression format > > [MAGIC_NUMBER] + [METADATA_SIZE] + [METADATA_PAYLOAD] + > [MANAGED_CURSOR_INFO_PAYLOAD] > > MAGIC_NUMBER: Ox4779 Since we change the ManagedCursorInfo data format here. How do we handle the old da

[DISCUSSION][PIP-146] ManagedCursorInfo compression

Hi Pulsar Community, I create a proposal that ManagedCursorInfo compression. The proposal can be found: https://github.com/apache/pulsar/issues/14529 Thanks, Zixuan -- Motivation The cursor data is managed by ZooKeeper/etcd meta