date:20220305

Re: [DISCUSS] PIP-139 : Support Broker send command to real close producer/consumer.

2022-03-05 Thread PengHui Li

> Great point. I was focused on deleting namespaces and missed the case
where the user wants to delete a topic from a replicated namespace. I
agree that we should make it possible to delete these topics without
removing the namespace replication.

Oh, sorry. I thought again, after
https://github.com/apache/pulsar/pull/12136
introduced topic level geo-replication configuration, users can disable for
a topic even if the namespace enabled geo-replication.

And topic level policy is introduced in 2.6.0. I think It's time to make it
one of
the features turned on by default.

So, we can follow the steps that stop the replication first and then delete
the
topic from clusters.

> Note that it is already possible
to delete these topics by deleting with force or by configuring
inactive topic deletion.

Hmm, I think both of them do not work for now, because the producer of
the replicator will always be active.

> Considering users can delete replicated topics by deleting with force,
are you saying we need to provide a non-force way to delete these
topics?

Yes, if the topic doesn't have users' producers/consumers, users don't need
to force delete the topic, it's a normal deletion.

--
@mattison @jiwei could you please check if we can use
https://github.com/apache/pulsar/pull/12136
to stop the geo-replication first, and then delete the topic?
If it works, we don't need to change the protocol.

Thanks,
Penghui



On Thu, Mar 3, 2022 at 7:12 AM Michael Marshall 
wrote:

> > The geo-replication's configuration can be centralized by using
> > one configuration store. But that doesn't change anything, we should
> > provide the same behavior for both centralized and decentralized
> > configuration store.
>
> Perhaps I misused the word decentralized. I meant that the challenge
> comes because geo-replication configurations for clusters are
> independent from each other.
>
> > The current challenge is
> > users usually set replicated clusters for a namespace, if remove
> > the replication configuration, the entire namespace will be affected.
> > We have supported setting the replicated configuration for a topic [1],
> > only for 2.10.0 or later.
>
> Great point. I was focused on deleting namespaces and missed the case
> where the user wants to delete a topic from a replicated namespace. I
> agree that we should make it possible to delete these topics without
> removing the namespace replication. Note that it is already possible
> to delete these topics by deleting with force or by configuring
> inactive topic deletion.
>
> > we should provided steps for how to delete an unused replicated
> > topic and what are the effects of not removing properly
>
> I agree, and I think documentation coupled with helpful error logs
> will be very helpful here.
>
> Considering users can delete replicated topics by deleting with force,
> are you saying we need to provide a non-force way to delete these
> topics?
>
> If we want to provide a path to normal deletion, we already have an
> `isRemote` flag on producers, and the inactive topic deletion code
> uses this flag to determine if there are non-replication producers
> connected to a topic. We could modify the deletion logic for a global
> topic so that it can be deleted as long as the only producers
> connected are remote producers. My main concern is that normal
> deletion could allow users to miss the nuance that they must also
> delete the topic in the other cluster. This might be an overly
> cautious concern, though.
>
> Thanks,
> Michael
>
>
>
> On Tue, Mar 1, 2022 at 1:55 AM PengHui Li  wrote:
> >
> > > I agree with you that it'd be nice to provide the same deletion
> > behavior. However, because geo-replication's configuration is
> > decentralized, I think namespace or topic deletion is more complicated
> > than unreplicated deletion. Note that users cannot currently delete
> > namespaces that are configured with remote replication clusters.
> >
> > The geo-replication's configuration can be centralized by using
> > one configuration store. But that doesn't change anything, we should
> > provide the same behavior for both centralized and decentralized
> > configuration store.
> >
> > > Note that while a user cannot explicitly delete a replicated topic,
> > they can remove the replication configuration in cluster A and cluster
> > B, and then they are left with unreplicated namespaces and topics,
> > which can be deleted.
> >
> > Yes, stopping the replication first is a more elegant approach,
> > and it is also very easy to understand for users. The current challenge
> is
> > users usually set replicated clusters for a namespace, if remove
> > the replication configuration, the entire namespace will be affected.
> > We have supported setting the replicated configuration for a topic [1],
> > only for 2.10.0 or later.
> >
> > > In this case, the specific problem stems from a remote cluster
> > targeting the l

Re: [DISCUSS] Releasing pulsar-client-go 0.8.1

2022-03-05 Thread PengHui Li

+1

Penghui

On Sat, Mar 5, 2022 at 4:58 AM Matteo Merli  wrote:

> +1 Thanks Rui, we should eliminate the GPL dependency ASAP.
>
>
>
> --
> Matteo Merli
> 
>
> On Thu, Mar 3, 2022 at 2:08 AM Rui Fu  wrote:
> >
> > Hi everyone,
> >
> > I would like to start a discussion here about starting a new release of
> > pulsar-client-go v0.8.1. Recently we have some of dependencies updated
> PRs
> > from the community, [1] is bumping `github.com/beefsack/go-rate`
>  to the
> > latest version, which migrates the license from GPL to MIT. [2] is
> bumping `
> > github.com/prometheus/client_golang`
>  to address the
> CVE-2022-21698. For
> > more details, please check the links below.
> >
> > As the v0.8.0 was just released weeks ago and the next release will start
> > about 2 month later, I think we should start the release of v0.8.1.
> >
> > [1]: https://github.com/apache/pulsar-client-go/pull/735
> > [2]: https://github.com/apache/pulsar-client-go/pull/738
> >
> > --
> >
> > Best Regards,
> >
> > Rui Fu
>

Re: [DISCUSSION][PIP-146] ManagedCursorInfo compression

2022-03-05 Thread PengHui Li

Hi Zixuan,

We should add the compatibility part to the proposal.
And should also provide steps to roll back to the old version which
enabled the compression in the new version.

I don't have objections to the proposal, and we have done
the same enhancement for topic metadata
https://github.com/apache/pulsar/pull/11490,
and this proposal also follows the same way.

Thanks,
Penghui

On Thu, Mar 3, 2022 at 10:26 AM Zixuan Liu  wrote:

> Thank you for your feedback.
>
> Forward compatibility is required:
>
> 1. Get CursorInfo from the metadata store: We will check the cursor data
> header, if it is compressed, we will parse the bytes data by compressed
> format, otherwise we will parse the cursor data directly by the original
> way.
>
> 2. Add/Update CursorInfo to the metadata store: The default is to use
> compression if the compression type is specified, otherwise we will put
> this data to the metadata store directly.
>
>
>
> > 2022年3月2日 下午11:48，Zike Yang  写道：
> >
> > Hi, Zixuan
> > Thanks for creating this PIP.  Here are my thoughts.
> >
> >> CursorInfo compression format
> >>
> >> [MAGIC_NUMBER] + [METADATA_SIZE] + [METADATA_PAYLOAD] +
> [MANAGED_CURSOR_INFO_PAYLOAD]
> >>
> >> MAGIC_NUMBER: Ox4779
> >
> > Since we change the ManagedCursorInfo data format here. How do we
> > handle the old data format in the ZK. Could you explain the
> > compatibility for this PIP?
> >
> > Thanks,
> > Zike Yang
> >
> > On Wed, Mar 2, 2022 at 3:34 PM Zixuan Liu  node...@gmail.com>> wrote:
> >>
> >> Hi Pulsar Community,
> >>
> >> I create a proposal that ManagedCursorInfo compression. The proposal
> can be found: https://github.com/apache/pulsar/issues/14529 <
> https://github.com/apache/pulsar/issues/14529> <
> https://github.com/apache/pulsar/issues/14529 <
> https://github.com/apache/pulsar/issues/14529>>
> >>
> >> Thanks,
> >> Zixuan
> >>
> >> --
> >>
> >> Motivation
> >>
> >> The cursor data is managed by ZooKeeper/etcd metadata store. When
> cursor data becomes more and more, the data size will increase and will
> take a lot of time to pull the data. Therefore, it is necessary to add
> compression for the cursor, which can reduce the size of data and reduce
> the time of pulling data.
> >>
> >> Goal
> >>
> >> Support use the LZ4/ZLIB/ZSTD/SNAPPY to compress the ManagedCursorInfo.
> >>
> >> Implementation
> >>
> >> CursorInfo compression format
> >>
> >> [MAGIC_NUMBER] + [METADATA_SIZE] + [METADATA_PAYLOAD] +
> [MANAGED_CURSOR_INFO_PAYLOAD]
> >>
> >> MAGIC_NUMBER: Ox4779
> >>
> >> METADATA
> >> Add a named ManagedCursorInfoMetadata message to MLDataFormats.proto
> >>
> >> message ManagedCursorInfoMetadata {
> >>required CompressionType compressionType = 1;
> >>required int32 uncompressedSize = 2;
> >> }
> >> CursorInfo compression and decompression design
> >>
> >> Currently, these compressions types have been defined and implemented
> by Pulsar, we only need to deal with compression and decompression of the
> ManagedCursorInfo data:
> >>
> >> Get CursorInfo from the metadata store
> >> We will check the cursor data header, if it is compressed, we will
> parse the bytes data by compressed format, otherwise by the original way.
> >>
> >> Add/Update CursorInfo to the metadata store
> >> The default is to use compression if the compression type is specified.
> >>
> >> CursorInfo compression type configuration
> >>
> >> Add managedCursorInfoCompressionType in
> org.apache.pulsar.broker.ServiceConfiguration and
> org.apache.bookkeeper.mledger.ManagedLedgerFactoryConfig.
> >>
> >>
> >
> >
> > --
> > Zike Yang
>
>

[GitHub] [pulsar-site] dave2wave opened a new pull request #9: Adjust .htaccess

2022-03-05 Thread GitBox



dave2wave opened a new pull request #9:
URL: https://github.com/apache/pulsar-site/pull/9


   Adjustments to .htaccess.
   
   Also going to move the `.htaccess` file to the top level


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-site] dave2wave merged pull request #9: Adjust .htaccess

2022-03-05 Thread GitBox



dave2wave merged pull request #9:
URL: https://github.com/apache/pulsar-site/pull/9


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [DISCUSSION][PIP-146] ManagedCursorInfo compression

2022-03-05 Thread Enrico Olivelli

Good proposal.
It is important that this is disabled by default otherwise we cannot easily
support the rollback

Apart from that I don't have other comments



Enrico

Il Sab 5 Mar 2022, 11:22 PengHui Li  ha scritto:

> Hi Zixuan,
>
> We should add the compatibility part to the proposal.
> And should also provide steps to roll back to the old version which
> enabled the compression in the new version.
>
> I don't have objections to the proposal, and we have done
> the same enhancement for topic metadata
> https://github.com/apache/pulsar/pull/11490,
> and this proposal also follows the same way.
>
> Thanks,
> Penghui
>
> On Thu, Mar 3, 2022 at 10:26 AM Zixuan Liu  wrote:
>
> > Thank you for your feedback.
> >
> > Forward compatibility is required:
> >
> > 1. Get CursorInfo from the metadata store: We will check the cursor data
> > header, if it is compressed, we will parse the bytes data by compressed
> > format, otherwise we will parse the cursor data directly by the original
> > way.
> >
> > 2. Add/Update CursorInfo to the metadata store: The default is to use
> > compression if the compression type is specified, otherwise we will put
> > this data to the metadata store directly.
> >
> >
> >
> > > 2022年3月2日 下午11:48，Zike Yang  写道：
> > >
> > > Hi, Zixuan
> > > Thanks for creating this PIP.  Here are my thoughts.
> > >
> > >> CursorInfo compression format
> > >>
> > >> [MAGIC_NUMBER] + [METADATA_SIZE] + [METADATA_PAYLOAD] +
> > [MANAGED_CURSOR_INFO_PAYLOAD]
> > >>
> > >> MAGIC_NUMBER: Ox4779
> > >
> > > Since we change the ManagedCursorInfo data format here. How do we
> > > handle the old data format in the ZK. Could you explain the
> > > compatibility for this PIP?
> > >
> > > Thanks,
> > > Zike Yang
> > >
> > > On Wed, Mar 2, 2022 at 3:34 PM Zixuan Liu  > node...@gmail.com>> wrote:
> > >>
> > >> Hi Pulsar Community,
> > >>
> > >> I create a proposal that ManagedCursorInfo compression. The proposal
> > can be found: https://github.com/apache/pulsar/issues/14529 <
> > https://github.com/apache/pulsar/issues/14529> <
> > https://github.com/apache/pulsar/issues/14529 <
> > https://github.com/apache/pulsar/issues/14529>>
> > >>
> > >> Thanks,
> > >> Zixuan
> > >>
> > >> --
> > >>
> > >> Motivation
> > >>
> > >> The cursor data is managed by ZooKeeper/etcd metadata store. When
> > cursor data becomes more and more, the data size will increase and will
> > take a lot of time to pull the data. Therefore, it is necessary to add
> > compression for the cursor, which can reduce the size of data and reduce
> > the time of pulling data.
> > >>
> > >> Goal
> > >>
> > >> Support use the LZ4/ZLIB/ZSTD/SNAPPY to compress the
> ManagedCursorInfo.
> > >>
> > >> Implementation
> > >>
> > >> CursorInfo compression format
> > >>
> > >> [MAGIC_NUMBER] + [METADATA_SIZE] + [METADATA_PAYLOAD] +
> > [MANAGED_CURSOR_INFO_PAYLOAD]
> > >>
> > >> MAGIC_NUMBER: Ox4779
> > >>
> > >> METADATA
> > >> Add a named ManagedCursorInfoMetadata message to MLDataFormats.proto
> > >>
> > >> message ManagedCursorInfoMetadata {
> > >>required CompressionType compressionType = 1;
> > >>required int32 uncompressedSize = 2;
> > >> }
> > >> CursorInfo compression and decompression design
> > >>
> > >> Currently, these compressions types have been defined and implemented
> > by Pulsar, we only need to deal with compression and decompression of the
> > ManagedCursorInfo data:
> > >>
> > >> Get CursorInfo from the metadata store
> > >> We will check the cursor data header, if it is compressed, we will
> > parse the bytes data by compressed format, otherwise by the original way.
> > >>
> > >> Add/Update CursorInfo to the metadata store
> > >> The default is to use compression if the compression type is
> specified.
> > >>
> > >> CursorInfo compression type configuration
> > >>
> > >> Add managedCursorInfoCompressionType in
> > org.apache.pulsar.broker.ServiceConfiguration and
> > org.apache.bookkeeper.mledger.ManagedLedgerFactoryConfig.
> > >>
> > >>
> > >
> > >
> > > --
> > > Zike Yang
> >
> >
>

Re: [VOTE] [PIP-143] Support split bundle by specified boundaries

2022-03-05 Thread mattison chao

+1

Best
Mattison 

> On Mar 3, 2022, at 11:08 AM, r...@apache.org  wrote:
> 
> +1
> 
> Thanks
> Xiaolong Ran
> 
> PengHui Li  于2022年2月27日周日 13:04写道：
> 
>> +1
>> 
>> Thanks,
>> Penghui
>> 
>> On Sat, Feb 26, 2022 at 11:59 AM Haiting Jiang 
>> wrote:
>> 
>>> +1
>>> 
>>> Thanks,
>>> Haiting
>>> 
>>> On 2022/02/26 02:53:23 Aloys Zhang wrote:
 Hi Pulsar Community,
 
 I would like to start a VOTE on  "Support split bundle by specified
 boundaries."(PIP-143)
 
 Here is the design detail
>> https://github.com/apache/pulsar/issues/13761
 
 and the discussion thread
 https://lists.apache.org/thread/xz80vz4j1yv7zm5bh4pfxgm8p2rg289z
 
 
 
 Thanks,
 Aloys
 
>>> 
>>

Re: [DISCUSSION][PIP-146] ManagedCursorInfo compression

2022-03-05 Thread mattison chao



Great work!

I have no other comments other than the compatibility everybody mentioned.

Best,
Mattison

> On Mar 6, 2022, at 4:55 AM, Enrico Olivelli  wrote:
> 
> Good proposal.
> It is important that this is disabled by default otherwise we cannot easily
> support the rollback
> 
> Apart from that I don't have other comments
> 
> 
> 
> Enrico
> 
> Il Sab 5 Mar 2022, 11:22 PengHui Li  ha scritto:
> 
>> Hi Zixuan,
>> 
>> We should add the compatibility part to the proposal.
>> And should also provide steps to roll back to the old version which
>> enabled the compression in the new version.
>> 
>> I don't have objections to the proposal, and we have done
>> the same enhancement for topic metadata
>> https://github.com/apache/pulsar/pull/11490,
>> and this proposal also follows the same way.
>> 
>> Thanks,
>> Penghui
>> 
>> On Thu, Mar 3, 2022 at 10:26 AM Zixuan Liu  wrote:
>> 
>>> Thank you for your feedback.
>>> 
>>> Forward compatibility is required:
>>> 
>>> 1. Get CursorInfo from the metadata store: We will check the cursor data
>>> header, if it is compressed, we will parse the bytes data by compressed
>>> format, otherwise we will parse the cursor data directly by the original
>>> way.
>>> 
>>> 2. Add/Update CursorInfo to the metadata store: The default is to use
>>> compression if the compression type is specified, otherwise we will put
>>> this data to the metadata store directly.
>>> 
>>> 
>>> 
 2022年3月2日 下午11:48，Zike Yang  写道：
 
 Hi, Zixuan
 Thanks for creating this PIP.  Here are my thoughts.
 
> CursorInfo compression format
> 
> [MAGIC_NUMBER] + [METADATA_SIZE] + [METADATA_PAYLOAD] +
>>> [MANAGED_CURSOR_INFO_PAYLOAD]
> 
> MAGIC_NUMBER: Ox4779
 
 Since we change the ManagedCursorInfo data format here. How do we
 handle the old data format in the ZK. Could you explain the
 compatibility for this PIP?
 
 Thanks,
 Zike Yang
 
 On Wed, Mar 2, 2022 at 3:34 PM Zixuan Liu >> node...@gmail.com>> wrote:
> 
> Hi Pulsar Community,
> 
> I create a proposal that ManagedCursorInfo compression. The proposal
>>> can be found: https://github.com/apache/pulsar/issues/14529 <
>>> https://github.com/apache/pulsar/issues/14529> <
>>> https://github.com/apache/pulsar/issues/14529 <
>>> https://github.com/apache/pulsar/issues/14529>>
> 
> Thanks,
> Zixuan
> 
> --
> 
> Motivation
> 
> The cursor data is managed by ZooKeeper/etcd metadata store. When
>>> cursor data becomes more and more, the data size will increase and will
>>> take a lot of time to pull the data. Therefore, it is necessary to add
>>> compression for the cursor, which can reduce the size of data and reduce
>>> the time of pulling data.
> 
> Goal
> 
> Support use the LZ4/ZLIB/ZSTD/SNAPPY to compress the
>> ManagedCursorInfo.
> 
> Implementation
> 
> CursorInfo compression format
> 
> [MAGIC_NUMBER] + [METADATA_SIZE] + [METADATA_PAYLOAD] +
>>> [MANAGED_CURSOR_INFO_PAYLOAD]
> 
> MAGIC_NUMBER: Ox4779
> 
> METADATA
> Add a named ManagedCursorInfoMetadata message to MLDataFormats.proto
> 
> message ManagedCursorInfoMetadata {
>   required CompressionType compressionType = 1;
>   required int32 uncompressedSize = 2;
> }
> CursorInfo compression and decompression design
> 
> Currently, these compressions types have been defined and implemented
>>> by Pulsar, we only need to deal with compression and decompression of the
>>> ManagedCursorInfo data:
> 
> Get CursorInfo from the metadata store
> We will check the cursor data header, if it is compressed, we will
>>> parse the bytes data by compressed format, otherwise by the original way.
> 
> Add/Update CursorInfo to the metadata store
> The default is to use compression if the compression type is
>> specified.
> 
> CursorInfo compression type configuration
> 
> Add managedCursorInfoCompressionType in
>>> org.apache.pulsar.broker.ServiceConfiguration and
>>> org.apache.bookkeeper.mledger.ManagedLedgerFactoryConfig.
> 
> 
 
 
 --
 Zike Yang
>>> 
>>> 
>>

Re: [DISCUSSION][PIP-146] ManagedCursorInfo compression

2022-03-05 Thread mattison chao

Hi, Zi Xuan

After deep think, I have another question:

Why don't we combine ledger compaction and cursor compaction into one 
configuration switch?

I’m not sure, do we have users who need to set the compression configuration 
for ledger and cursor separately? I think if they were to be set, they would 
all be set to the same.


We could add new configurations ``xxxCompressionType``  to control the 
ledger and cursor compress and deprecate the 
``managedLedgerInfoCompressionType`` for compatibility.

After that, if users set ``managedLedgerInfoCompressionType``, we just compress 
the ledger, if the user set another configuration ``xxxCompressionType `` 
we will compress the ledger and cursor.


It’s just a question or suggestion. You can feel free to go ahead.

Best,
Mattison



> On Mar 6, 2022, at 8:22 AM, mattison chao  wrote:
> 
> 
> Great work!
> 
> I have no other comments other than the compatibility everybody mentioned.
> 
> Best,
> Mattison
> 
>> On Mar 6, 2022, at 4:55 AM, Enrico Olivelli  wrote:
>> 
>> Good proposal.
>> It is important that this is disabled by default otherwise we cannot easily
>> support the rollback
>> 
>> Apart from that I don't have other comments
>> 
>> 
>> 
>> Enrico
>> 
>> Il Sab 5 Mar 2022, 11:22 PengHui Li  ha scritto:
>> 
>>> Hi Zixuan,
>>> 
>>> We should add the compatibility part to the proposal.
>>> And should also provide steps to roll back to the old version which
>>> enabled the compression in the new version.
>>> 
>>> I don't have objections to the proposal, and we have done
>>> the same enhancement for topic metadata
>>> https://github.com/apache/pulsar/pull/11490,
>>> and this proposal also follows the same way.
>>> 
>>> Thanks,
>>> Penghui
>>> 
>>> On Thu, Mar 3, 2022 at 10:26 AM Zixuan Liu  wrote:
>>> 
 Thank you for your feedback.
 
 Forward compatibility is required:
 
 1. Get CursorInfo from the metadata store: We will check the cursor data
 header, if it is compressed, we will parse the bytes data by compressed
 format, otherwise we will parse the cursor data directly by the original
 way.
 
 2. Add/Update CursorInfo to the metadata store: The default is to use
 compression if the compression type is specified, otherwise we will put
 this data to the metadata store directly.
 
 
 
> 2022年3月2日 下午11:48，Zike Yang  写道：
> 
> Hi, Zixuan
> Thanks for creating this PIP.  Here are my thoughts.
> 
>> CursorInfo compression format
>> 
>> [MAGIC_NUMBER] + [METADATA_SIZE] + [METADATA_PAYLOAD] +
 [MANAGED_CURSOR_INFO_PAYLOAD]
>> 
>> MAGIC_NUMBER: Ox4779
> 
> Since we change the ManagedCursorInfo data format here. How do we
> handle the old data format in the ZK. Could you explain the
> compatibility for this PIP?
> 
> Thanks,
> Zike Yang
> 
> On Wed, Mar 2, 2022 at 3:34 PM Zixuan Liu >>> node...@gmail.com>> wrote:
>> 
>> Hi Pulsar Community,
>> 
>> I create a proposal that ManagedCursorInfo compression. The proposal
 can be found: https://github.com/apache/pulsar/issues/14529 <
 https://github.com/apache/pulsar/issues/14529> <
 https://github.com/apache/pulsar/issues/14529 <
 https://github.com/apache/pulsar/issues/14529>>
>> 
>> Thanks,
>> Zixuan
>> 
>> --
>> 
>> Motivation
>> 
>> The cursor data is managed by ZooKeeper/etcd metadata store. When
 cursor data becomes more and more, the data size will increase and will
 take a lot of time to pull the data. Therefore, it is necessary to add
 compression for the cursor, which can reduce the size of data and reduce
 the time of pulling data.
>> 
>> Goal
>> 
>> Support use the LZ4/ZLIB/ZSTD/SNAPPY to compress the
>>> ManagedCursorInfo.
>> 
>> Implementation
>> 
>> CursorInfo compression format
>> 
>> [MAGIC_NUMBER] + [METADATA_SIZE] + [METADATA_PAYLOAD] +
 [MANAGED_CURSOR_INFO_PAYLOAD]
>> 
>> MAGIC_NUMBER: Ox4779
>> 
>> METADATA
>> Add a named ManagedCursorInfoMetadata message to MLDataFormats.proto
>> 
>> message ManagedCursorInfoMetadata {
>>  required CompressionType compressionType = 1;
>>  required int32 uncompressedSize = 2;
>> }
>> CursorInfo compression and decompression design
>> 
>> Currently, these compressions types have been defined and implemented
 by Pulsar, we only need to deal with compression and decompression of the
 ManagedCursorInfo data:
>> 
>> Get CursorInfo from the metadata store
>> We will check the cursor data header, if it is compressed, we will
 parse the bytes data by compressed format, otherwise by the original way.
>> 
>> Add/Update CursorInfo to the metadata store
>> The default is to use compression if the compression type is
>>> specified.
>> 
>> CursorInfo compression type configuration
>> 
>> Add

[GitHub] [pulsar-site] dave2wave merged pull request #11: Suggested change to rewrite rules

2022-03-05 Thread GitBox



dave2wave merged pull request #11:
URL: https://github.com/apache/pulsar-site/pull/11


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [DISCUSSION][PIP-146] ManagedCursorInfo compression

2022-03-05 Thread Haiting Jiang

This is a global setting now. But I wonder if we should compress it only if the 
size 
is over a threshold?  Because:
1. It's not easy for us to notice some managed cursor info is too large in 
advance,  normally it would be found only if it have actual impact. But if we 
enable this compression in advance, it will took some extra computing resources.
2. It seems that it won't be a common case that this managed cursor info is too 
large (only if there are a lot individualDeletedMessages and 
batchedEntryDeletionIndexInfo). So not quite necessary to compress all managed 
cursor info.

Regards,
Haiting


On 2022/03/02 04:41:16 Zixuan Liu wrote:
> Hi Pulsar Community,
> 
> 
> I create a proposal that support ManagedCursorInfo compression.
> 
> The proposal can be found: https://github.com/apache/pulsar/issues/14395
> 
> 
> Motivation
> 
> The cursor data is managed by ZooKeeper/etcd metadata store. When
> cursor data becomes more and more, the data size will increase and
> will take a lot of time to pull the data. Therefore, it is necessary
> to add compression for the cursor, which can reduce the size of data
> and reduce the time of pulling data.
> Goal
> 
> Support use the LZ4/ZLIB/ZSTD/SNAPPY to compress the ManagedCursorInfo.
> Implementation
> 
>- Cursor compression format
>[MAGIC_NUMBER] + [METADATA_SIZE] + [METADATA_PAYLOAD] +
> [MANAGED_CURSOR_INFO_PAYLOAD]
> 
> 
>-
> 
>MAGIC_NUMBER
>Ox4779
>-
> 
>METADATA
>Add a named ManagedCursorInfoMetadata message to MLDataFormats.proto:
>message ManagedCursorInfoMetadata {
>   required CompressionType compressionType = 1;
>   required int32 uncompressedSize = 2;
>}
> 
> Currently, these compressions have been supported, we only need to
> deal with compression and decompression of the ManagedCursorInfo data:
> 
>-
> 
>Get CursorInfo from the metadata store
>We will check the cursor data header, if it is compressed, we will
> parse the bytes data by compressed format, otherwise by the original
> way.
>-
> 
>Add/Update CursorInfo to the metadata store
>The default is to use compression if the compression type is specified.
> 
> 
> Thanks,
> Zixuan
>

[GitHub] [pulsar-site] eolivelli merged pull request #10: Fix Copyright Footer

2022-03-05 Thread GitBox



eolivelli merged pull request #10:
URL: https://github.com/apache/pulsar-site/pull/10


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [DISCUSS] PIP-139 : Support Broker send command to real close producer/consumer.

Re: [DISCUSS] Releasing pulsar-client-go 0.8.1

Re: [DISCUSSION][PIP-146] ManagedCursorInfo compression

[GitHub] [pulsar-site] dave2wave opened a new pull request #9: Adjust .htaccess

[GitHub] [pulsar-site] dave2wave merged pull request #9: Adjust .htaccess

Re: [DISCUSSION][PIP-146] ManagedCursorInfo compression

Re: [VOTE] [PIP-143] Support split bundle by specified boundaries

Re: [DISCUSSION][PIP-146] ManagedCursorInfo compression

Re: [DISCUSSION][PIP-146] ManagedCursorInfo compression

[GitHub] [pulsar-site] dave2wave merged pull request #11: Suggested change to rewrite rules

Re: [DISCUSSION][PIP-146] ManagedCursorInfo compression

[GitHub] [pulsar-site] eolivelli merged pull request #10: Fix Copyright Footer

12 matches

Site Navigation

Mail list logo

Footer information