[DISCUSS] [PIP-153] Optimize metadataPositions in MLPendingAckStore

[xiangying Meng]

Hi Pulsar community,


I create a PIP to optimize metadataPosition in MLPendingAckStore to use less 
memory and avoid OOM.
The proposal can be 
found:  https://github.com/apache/pulsar/issues/15073 ;




Thanks,
xiangying Meng

Re: [DISCUSS] [PIP-160] Batch writing ledger for transaction operation

[Xiangying Meng]

I think this is a nice optimization of transaction internal components.
But I have two little questions:
1. I noticed you write conf transactionMetadataStoreProviderClassName is
configured in broker.conf. Should this be added to `Configuration Changes`?
2. This proposal will add a batch mechanism for pending ack, but this
proposal is no detailed description for pending ack similar to the
description of metadata log batch.

On Fri, Jun 10, 2022 at 7:15 PM Enrico Olivelli  wrote:

> I have read the PIP, and overall I agree with the design.
> Good work !
>
> I am not sure I understand the part of making it configurable via a
> classname.
> I believe it is better to simply have a flag
> "transactionEnableBatchWrites".
> Otherwise the matrix of possible implementations will grow without limits.
>
> Enrico
>
> Il giorno ven 10 giu 2022 alle ore 11:35 Yubiao Feng
>  ha scritto:
> >
> > Hi Pulsar community:
> >
> > I open a pip to discuss "Batch writing ledger for transaction operation"
> >
> > Proposal Link: https://github.com/apache/pulsar/issues/15370
> >
> > ## Motivation
> >
> > Before reading the background, I suggest you read section “Transaction
> > Flow” of [PIP-31: Transactional Streaming](
> >
> https://docs.google.com/document/d/145VYp09JKTw9jAT-7yNyFU255FptB2_B2Fye100ZXDI/edit#heading=h.bm5ainqxosrx
> > )
> >
> > ###  Normal Flow vs. Transaction Flow
> 
> > ![MG3](
> >
> https://user-images.githubusercontent.com/25195800/172985866-25e496a4-ea93-42ec-aa0d-e6a02aa0635e.jpeg
> > )
> > In *Figure 1. Normal Flow vs. Transaction Flow*:
> > - The gray square boxes represent logical components.
> > - All the blue boxes represent logs. The logs are usually Managed ledger
> > - Each arrow represents the request flow or message flow. These
> operations
> > occur in sequence indicated by the numbers next to each arrow.
> > - The black arrows indicate those shared by transaction and normal flow.
> > - The blue arrows represent normal-message-specific flow.
> > - The orange arrows represent transaction-message-specific flow.
> > - The sections below are numbered to match the operations showed in the
> > diagram(differ from [PIP-31: Transactional Streaming](
> >
> https://docs.google.com/document/d/145VYp09JKTw9jAT-7yNyFU255FptB2_B2Fye100ZXDI/edit#heading=h.bm5ainqxosrx
> > ))
> >
> >
> >  2.4a Write logs to ledger which Acknowledgement State is PENDING_ACK
> > [Acknowledgement State Machine](
> >
> https://docs.google.com/document/d/145VYp09JKTw9jAT-7yNyFU255FptB2_B2Fye100ZXDI/edit#bookmark=id.4bikq6sjiy8u
> )
> > tells about the changes of the Acknowledge State and why we need
> persistent
> > “The Log which the Acknowledgement State is PENDING_ACK”.
> >  2.4a’ Mark messages is no longer useful with current subscription
> > Update `Cursor` to mark the messages as DELETED. So they can be deleted.
> >  3.2b Mark messages is no longer useful with current subscription
> > The implementation here is exactly the same as 2.4a’, except that the
> > execution is triggered later, after the Transaction has been committed.
> >
> >
> > ### Analyze the performance cost of transaction
> > As you can see Figure 1. Normal Flow
> vs.
> > Transaction Flow]: 2.4a 'and 3.2b are exactly the same logic, so the
> > remaining orange arrows are the additional performance overhead of all
> > transactions.
> > In terms of whether or not each transaction is executed multiple times,
> we
> > can split the flow into two classes(Optimizing a process that is executed
> > multiple times will yield more benefits):
> > - Executed once each transaction: flow-1.x and flow-3.x
> > - Executed multiple times each transaction: flow-2.x
> >
> > So optimizing the flow 2.x with a lot of execution is a good choice.
> Let's
> > split flow-2.x into two groups: those that cost more and those that cost
> > less:
> > - No disk written: flow-2.1 and fow-2.3
> > - Disk written: fow-2.1a, fow-2.3a, flow-2.4a
> >
> > From the previous analysis, we found that optimizing flow-2.1a,
> flow-2.3a,
> > flow-2.4a would bring the most benefits, and batch writes would be an
> > excellent solution for multiple disk writes. Flow-2.1a and Flow-2.3a are
> > both manipulations written into the transaction log, we can combine them
> in
> > one batch; 2.4a is the operation of writing pending ACK log, we combine
> > multiple 2.4a's into one batch for processing.
> > As we can see from “Transaction Flow” of [PIP-31: Transactional
> Streaming](
> >
> https://docs.google.com/document/d/145VYp09JKTw9jAT-7yNyFU255FptB2_B2Fye100ZXDI/edit#heading=h.bm5ainqxosrx
> ),
> > these instructions are strictly sequential (guaranteed by the client):
> > - flow-1.x end before flow-2.x start
> > - flow-2.x end before flow-3.x start
> > - flow-3.1a end before flow-3.3a start
> >
> > Therefore, the broker does not need to worry about the dependency of
> these
> > flows, we can also put flow-1a flow-31a and flow-3.3a into The
> Transaction
> > Log Batch too.
> >
> > ## Goal
> > Provide a mechan

[DISCUSS][PIP-176] PIP176:Multiple snapshots for transaction buffer snapshot

[Xiangying Meng]

Hi Pulsar Community,
I have opened a proposal to implement multiple snapshots for the
transaction buffer.
The details can be found here
<https://github.com/apache/pulsar/issues/16042>.

Thanks,
Xiangying Meng

Re: [VOTE] PIP-165: Auto release client useless connections

[Xiangying Meng]

+1

On Tue, Jun 14, 2022 at 3:52 PM Gavin Gao  wrote:

> +1
>
>
> On 2022/06/01 07:11:51 Yubiao Feng wrote:
> > Hi Pulsar Community:
> >
> > There were some mistakes in the last email, so I have corrected them.
> >
> > I would like to start a VOTE on "Auto release client useless connections"
> > (PIP-165).
> >
> > Proposal Link: [PIP-165] Auto release client useless connections · Issue
> > #15516 · apache/pulsar (github.com)
> > 
> >
> > Discuss Link: [DISCUSS] [PIP-165] Auto release client useless
> > connections-Apache Mail Archives
> > 
> >
> > Voting will stay open for at least 48h.
> > Thanks, Yubiao Feng
> >
> > On Wed, Jun 1, 2022 at 2:40 PM Yubiao Feng 
> > wrote:
> >
> > > [VOTE] PIP-165: Auto release client useless connectionsHi Pulsar
> > > Community, I would like to start a VOTE on "Auto release client useless
> > > connections" (PIP-165). The proposal can be read at
> > > https://github.com/apache/pulsar/issues/15516
> > >  and the discussion
> thead
> > > is available at
> > > https://lists.apache.org/thread/t6h98qs2coc56z06tw38hdlljl67ft4n
> > >   Voting will stay open for at least 48h. Thanks, Yubiao Feng
> > >
> >
>

Re: Pull requests potentially blocked by the Documentation Bot workflow

[Xiangying Meng]

You can add a doc-label-missing and then remove it to run the label test.
I have tried this solution several times.

On Wed, Jul 6, 2022 at 7:41 PM Nicolò Boschi  wrote:

> Hi all,
>
> There are some pull requests blocked by the Doc bot that never ends.
> I left a comment here https://github.com/apache/pulsar/pull/16368 because
> I
> think this pull is the culprit
>
> If we don't find a quick solution, I'll push a revert to unblock the master
> branch
>
> Thanks,
> Nicolò Boschi
>

Re: [ANNOUNCE] Micheal Marshall as a new PMC member in Pulsar

[Xiangying Meng]

Congratulations, Michael!

BR,
xiangying Meng

On Wed, Jul 27, 2022 at 10:11 AM Zike Yang  wrote:

> Congratulations, Michael!
>
> BR
> Zike Yang
>
> On Wed, Jul 27, 2022 at 10:04 AM guo jiwei  wrote:
> >
> > Congrats Michael!
> >
> >
> > Regards
> > Jiwei Guo (Tboy)
> >
> >
> > On Wed, Jul 27, 2022 at 9:56 AM Haiting Jiang 
> > wrote:
> >
> > > Congratulations, Michael!
> > >
> > > BR,
> > > Haiting
> > >
> > > On 2022/07/26 15:21:59 Enrico Olivelli wrote:
> > > > I am glad to announce that the Apache Pulsar PMC invited Micheal to
> > > > join the PMC and he accepted.
> > > >
> > > > Micheal is doing a great job in stewarding our community
> > > >
> > > > Please join me and celebrate !
> > > >
> > > > Enrico Olivelli
> > > >
> > >
>

[DISCUSSION] PiP196 TransactionBuffer Multiple-snapshots

[Xiangying Meng]

Hi, Pulsar community,
I`d like to start a discussion about transaction multiple-snapshot.
In order to get rid of the capacity limitation of the bookkeeper entry, we
plan to use multiple snapshots. More details can be found here
<https://github.com/apache/pulsar/issues/16913>.

Yours sincerely,
Xiangying Meng

Re: [Vote] PIP 198: Standardize PR Naming Convention using GitHub Actions

[Xiangying Meng]

+1(non-binding)

yours sincerely,
xiangying Meng

On Thu, Aug 4, 2022 at 4:13 PM Yu  wrote:

> Hi team,
>
> It has been 4 months since we discussed the [Guideline] Pulsar PR Naming
> Convention [1].
>
> Nowadays, when reading the PR list [2], you’ll find more and more people
> follow and get used to this rule.
>
> It improves collaboration efficiency, that is great!
>
> This makes us think about moving the rule forward, that is, standardizing
> PR title naming using GitHub Actions, which is a more efficient way.
>
> So we'd like to start a vote on PIP 198: Standardize PR Naming Convention
> using GitHub Actions [3].
>
>
> This proposal contains:
>
> - Why do this?
>
> - How do this?
>
> - Pre-discussions and other thoughts
>
> Feel free to comment, thank you!
>
> [1] https://lists.apache.org/thread/sk9ops3t94jmzc5tndk08y9khf7pj6so
>
> [2] https://github.com/apache/pulsar/pulls
>
> [3]
>
> https://docs.google.com/document/d/1sJlUNAHnYAbvu9UtEgCrn_oVTnVc1M5nHC19x1bFab4/edit?pli=1#
>
>
> Yu, Max, mangoGoForward
>

Re: [DISCUSSION] PiP196 TransactionBuffer Multiple-snapshots

[Xiangying Meng]

Hi, yubiao,
First of all, thanks for the attention and questions. Then for your three
questions:
1.
 > Does the merge take place in memory or in BK?
The snapshot will merge in BK. For specific details, you can see detailed
instructions in the* ### Merge snapshot section.*
2.
>How do we ensure the atomicity of the two writes, I suggest adding a check
We do not guarantee their atomicity. The position of the snapshot is
generally unchanged, so the previous index is also valid. If the index
write fails after a snapshot is written, the final result is that the
snapshot write fails this time. There will be no other worse results, and
no dirty data will be introduced due to compression.
3.
>Clean up unused aborts data
Snapshot cleanup can be found in *take snapshot # How*.
The cleanup of the index is done automatically by the compressor. I will
add it at *### Snapshot index topic.*

yours sincerely,
Xiangying Meng




On Mon, Aug 15, 2022 at 3:56 PM Yubiao Feng
 wrote:

> Hi Xiangying
>
> I think Multiple-snapshots for TB is a good idea. And I have these
> questions:
>
>
> > The number of the transactions in a snapshot can be configured, and we
> hope it is small, then we can merge the small snapshots into a large
> snapshot when it reaches a configured number.
>
> Does the merge take place in memory or in BK?
>
> - If we merge small-snapshot in memory, can we just use large-snapshot?
> - If we merge small-snapshot in BK, how to do it?
>
>
>
> > The index is written after each multiple-snapshot is written.
>
> Snapshot and index are stored in different topics, right?
>
> How do we ensure the atomicity of the two writes, I suggest adding a check
> mechanism that snapshot not recorded in the index is invalid.
>
>
>
> >  Clean up unused aborts data
>
> Now, this section only has instructions for clear snapshots.
> I think we should add this: how to delete/override the index data.
>
> Thanks
> Yubiao Feng
>
> On Thu, Aug 4, 2022 at 10:27 AM Xiangying Meng 
> wrote:
>
> > Hi, Pulsar community,
> > I`d like to start a discussion about transaction multiple-snapshot.
> > In order to get rid of the capacity limitation of the bookkeeper entry,
> we
> > plan to use multiple snapshots. More details can be found here
> > <https://github.com/apache/pulsar/issues/16913>.
> >
> > Yours sincerely,
> > Xiangying Meng
> >
>

Re: [DISCUSSION] PiP196 TransactionBuffer Multiple-snapshots

[Xiangying Meng]

Hi, yubiao,
I think it's an implementation detail that shouldn't be exposed to the user
at all.
Yours sincerely,
Xiangying Meng

On Mon, Aug 15, 2022 at 8:02 PM Yubiao Feng
 wrote:

> Hi Xiangying
>
> Thank you for your reply. Sorry, I have one more question:
>
> > If these operations are failed at operation 2, the old snapshots will be
> covered by the new large snapshot when compact due to they have the same
> sequence ID.
>
> Can the sequence id generation strategy be added to the doc？
>
> On Mon, Aug 15, 2022 at 6:35 PM Xiangying Meng 
> wrote:
>
> > Hi, yubiao,
> > First of all, thanks for the attention and questions. Then for your three
> > questions:
> > 1.
> >  > Does the merge take place in memory or in BK?
> > The snapshot will merge in BK. For specific details, you can see detailed
> > instructions in the* ### Merge snapshot section.*
> > 2.
> > >How do we ensure the atomicity of the two writes, I suggest adding a
> check
> > We do not guarantee their atomicity. The position of the snapshot is
> > generally unchanged, so the previous index is also valid. If the index
> > write fails after a snapshot is written, the final result is that the
> > snapshot write fails this time. There will be no other worse results, and
> > no dirty data will be introduced due to compression.
> > 3.
> > >Clean up unused aborts data
> > Snapshot cleanup can be found in *take snapshot ##### How*.
> > The cleanup of the index is done automatically by the compressor. I will
> > add it at *### Snapshot index topic.*
> >
> > yours sincerely,
> > Xiangying Meng
> >
> >
> >
> >
> > On Mon, Aug 15, 2022 at 3:56 PM Yubiao Feng
> >  wrote:
> >
> > > Hi Xiangying
> > >
> > > I think Multiple-snapshots for TB is a good idea. And I have these
> > > questions:
> > >
> > >
> > > > The number of the transactions in a snapshot can be configured, and
> we
> > > hope it is small, then we can merge the small snapshots into a large
> > > snapshot when it reaches a configured number.
> > >
> > > Does the merge take place in memory or in BK?
> > >
> > > - If we merge small-snapshot in memory, can we just use large-snapshot?
> > > - If we merge small-snapshot in BK, how to do it?
> > >
> > >
> > >
> > > > The index is written after each multiple-snapshot is written.
> > >
> > > Snapshot and index are stored in different topics, right?
> > >
> > > How do we ensure the atomicity of the two writes, I suggest adding a
> > check
> > > mechanism that snapshot not recorded in the index is invalid.
> > >
> > >
> > >
> > > >  Clean up unused aborts data
> > >
> > > Now, this section only has instructions for clear snapshots.
> > > I think we should add this: how to delete/override the index data.
> > >
> > > Thanks
> > > Yubiao Feng
> > >
> > > On Thu, Aug 4, 2022 at 10:27 AM Xiangying Meng 
> > > wrote:
> > >
> > > > Hi, Pulsar community,
> > > > I`d like to start a discussion about transaction multiple-snapshot.
> > > > In order to get rid of the capacity limitation of the bookkeeper
> entry,
> > > we
> > > > plan to use multiple snapshots. More details can be found here
> > > > <https://github.com/apache/pulsar/issues/16913>.
> > > >
> > > > Yours sincerely,
> > > > Xiangying Meng
> > > >
> > >
> >
>

Re: [DISCUSS] Move PIPs to the codebase?

[Xiangying Meng]

Hi Penghui
Thanks for you start this discussion. IMO, It is also a good way for
beginners to learn the design and implementation of each module of Pulsar.
> 3. /wiki/pip/accepted - for PIPs that have been accepted and are works in
progress
> 4. /wiki/pip/complete - for PIPs that have been completed.
> 5. /wiki/pip/rejected - for PIPs that were proposed, but then rejected or
abandoned.
We can classify the pips under these folders according to the pulsar
modules, instead of just placing these pips under these folders in an
incrementing sequence number.

In this way, readers can create a new local branch dedicated to reading and
annotating proposals for themselves to read proposals they are interested
in and write their own understanding and comments anytime and anywhere.
Thanks,
Xiangying Meng

On Fri, Aug 26, 2022 at 12:23 AM PengHui Li  wrote:

> Hi Dave,
>
> > Let’s outline how PIPs are currently working and then discuss changes.
>
> Yes, I will prepare for the changes.
> This is the documentation for how PIPs are currently working:
>
> https://github.com/apache/pulsar/pull/17270/files#diff-a56445d038f8a3df4c74724076c62437915f091437b4e26a1c5aada7184dcffd
> The mailing list discussion:
> https://lists.apache.org/thread/m8dr0hz7qn7rkd48bcp430lcq2q3674g
>
> Anyway, I will start a new discussion with the new changes to the current
> process.
>
> > I’m not sure what is meant by putting the PIP into the “codebase”.
> > Is the proposal to create PIPs here?
> https://github.com/apache/pulsar/tree/master/wiki
>
> Just move out from https://github.com/apache/pulsar/tree/master/wiki to
> Pulsar codebase /wiki/proposals
>
> > I think that a directory structure / convention could be used for pip
> status:
>
> > 1. /wiki/pip/discussion - for PIPs being discussed and specified.
> > 2. /wiki/pip/proposed - for PIPs ready to be formally DISCUSSED and VOTED
> > 3. /wiki/pip/accepted - for PIPs that have been accepted and are works in
> progress
> > 4. /wiki/pip/complete - for PIPs that have been completed.
> > 5. /wiki/pip/rejected - for PIPs that were proposed, but then rejected or
> abandoned.
>
> I think it's a good point, I don't see any obvious cons.
>
> Thanks,
> Penghui
>
> On Thu, Aug 25, 2022 at 11:40 PM Dave Fisher  wrote:
>
> >
> > > On Aug 23, 2022, at 10:22 AM, Rajan Dhabalia 
> > wrote:
> > >
> > > Hi,
> > >
> > >>>> I think we can move all the PIPs to the codebase and the new
> proposal
> > > and proposal without any reviews should happen with a PR first. So that
> > we
> > > can review and comment easily.
> > >
> > > I didn't understand this part. You want one to create a PR before
> > > submitting a proposal? That's clearly not a good idea because if the
> PIP
> > > approach will change then the entire development effort will be wasted
> > and
> > > that's the whole purpose of PIP. I guess creating PIP into an issue and
> > > discussing the issue is definitely working and it's an easier way to
> > > discuss quickly rather than discussing over email threads.
> > >
> > > Let's not change this practice without good discussion and agreement
> from
> > > the community.
> >
> > Agreed let’s have a PIP Discussion here to carefully outline how the PIP
> > process will change. I don’t think that a new PIP should be overly
> planned
> > or implemented before the idea is more fully discussed and accepted. The
> > Apache Way always works best with small incremental and reversible steps.
> >
> > Let’s outline how PIPs are currently working and then discuss changes.
> I’m
> > not sure what is meant by putting the PIP into the “codebase”.
> >
> > Is the proposal to create PIPs here?
> > https://github.com/apache/pulsar/tree/master/wiki
> >
> > I think that a directory structure / convention could be used for pip
> > status:
> >
> > 1. /wiki/pip/discussion - for PIPs being discussed and specified.
> > 2. /wiki/pip/proposed - for PIPs ready to be formally DISCUSSED and VOTED
> > 3. /wiki/pip/accepted - for PIPs that have been accepted and are works in
> > progress
> > 4. /wiki/pip/complete - for PIPs that have been completed.
> > 5. /wiki/pip/rejected - for PIPs that were proposed, but then rejected or
> > abandoned.
> >
> > Regards,
> > Dave
> >
> > >
> > > Thanks,
> > > Rajan
> > >
> > > On Thu, Aug 18, 2022 at 8:27 AM PengHui Li  wrote:
> > >
> > >> Hi all,
> > >>
> > >> Currently, the new

[VOTE] PIP-196 Segmented transaction buffer snapshot

[Xiangying Meng]

Hi, community
I,d like to start a vote for the PIP-196
 Segmented transaction
buffer snapshot.
And the discussion can be found here
.

Sincerely,
Xiangying

[DISCUSS]PIP-296 Introduce the `getLastMessageIds` API to Reader

[Xiangying Meng]

Hi, community,

I would like to bring attention to the current absence of the
`getLastMessageIds` method within the Reader interface, which has
caused various inconveniences. To address this issue, I have prepared
a proposal [1] to incorporate this API into the Reader interface.
Please review the proposal and provide valuable feedback.

Best regards,
Xiangying

[1] Proposal link: [https://github.com/apache/pulsar/pull/21052]

Re: [DISCUSS]PIP-295: Fixing Chunk Message Duplication Issue

[Xiangying Meng]

Hi Zike
Thank you for your attention.
>For the existing behavior, the consumer assembles messages 3,4,5 into the 
>original large message. But the changes brought about by this PIP will cause 
>the consumer to use messages 1,2,5 for assembly. There is no guarantee that 
>the The producer will split the message in the same way twice before and 
>after. For example, the producer's maxMessageSize may be different. This may 
>cause the consumer to receive a corrupt message.

For the previous behavior, if duplication is not enabled, messages 1,
2, 3, 4, and 5 will all be persisted. When the consumer receives
message 3  (sequenceID: 0, ChunkID: 0), it will find that the message
is out of order and rewind the cursor. Loop this operation, and
discard this message after it expires instead of assembling 3, 4, 5
into a message.
If duplication is enabled, the chunk messages 2, 3, 4, and 5 will be
filtered. The message also will be discarded.

>I think a simple better approach is to only check the deduplication for the 
>last chunk of the large message.

This solution also cannot solve the out-of-order messages inside the
chunks. For example, the above five messages will still be persisted.

On Wed, Aug 23, 2023 at 12:34 PM Zike Yang  wrote:
>
> Hi, xiangying,
>
> Thanks for your PIP.
>
> IIUC, this may change the existing behavior and may introduce inconsistencies.
> Suppose that we have a large message with 3 chunks. But the producer
> crashes and resends the message after sending the chunk-1. It will
> send a total of 5 messages to the Pulsar topic:
>
> 1. SequenceID: 0, ChunkID: 0
> 2. SequenceID: 0, ChunkID: 1
> 3. SequenceID: 0, ChunkID: 0   -> This message will be dropped
> 4. SequenceID: 0, ChunkID: 1-> Will also be dropped
> 5. SequenceID: 0, ChunkID: 2-> The last chunk of the message
>
> For the existing behavior, the consumer assembles messages 3,4,5 into
> the original large message. But the changes brought about by this PIP
> will cause the consumer to use messages 1,2,5 for assembly. There is
> no guarantee that the producer will split the message in the same way
> twice before and after. For example, the producer's maxMessageSize may
> be different. This may cause the consumer to receive a corrupt
> message.
>
> Also, this PIP increases the complexity of handling chunks on the
> broker side. Brokers should, in general, treat the chunk as a normal
> message.
>
> I think a simple better approach is to only check the deduplication
> for the last chunk of the large message. The consumer only gets the
> whole message after receiving the last chunk. We don't need to check
> the deduplication for all previous chunks. Also by doing this we only
> need bug fixes, we don't need to introduce a new PIP.
>
> BR,
> Zike Yang
>
> On Fri, Aug 18, 2023 at 7:54 PM Xiangying Meng  wrote:
> >
> > Dear Community,
> >
> > I hope this email finds you well. I'd like to address an important
> > issue related to Apache Pulsar and discuss a solution I've proposed on
> > GitHub. The problem pertains to the handling of Chunk Messages after
> > enabling deduplication.
> >
> > In the current version of Apache Pulsar, all chunks of a Chunk Message
> > share the same sequence ID. However, enabling the depublication
> > feature results in an inability to send Chunk Messages. To tackle this
> > problem, I've proposed a solution [1] that ensures messages are not
> > duplicated throughout end-to-end delivery. While this fix addresses
> > the duplication issue for end-to-end messages, there remains a
> > possibility of duplicate chunks within topics.
> >
> > To address this concern, I believe we should introduce a "Chunk ID
> > map" at the Broker level, similar to the existing "sequence ID map",
> > to facilitate effective filtering. However, implementing this has led
> > to a challenge: a producer requires storage for two Long values
> > simultaneously (sequence ID and chunk ID). Because the snapshot of the
> > sequence ID map is stored through the properties of the cursor
> > (Map), so in order to satisfy the storage of two Longs
> > (sequence ID, chunk ID) corresponding to one producer, we hope to add
> > a mark DeleteProperties (Map) String, String>) to
> > replace the properties (Map) field. To resolve this,
> > I've proposed an alternative proposal [2] involving the introduction
> > of a "mark DeleteProperties" (Map) to replace the
> > current properties (Map) field.
> >
> > I'd appreciate it if you carefully review both PRs and share your
> > valuable feedback and insights. Thank you immensely for your time and
> > attention. I eagerly anticipate your valuable opinions and
> > recommendations.
> >
> > Warm regards,
> > Xiangying
> >
> > [1] https://github.com/apache/pulsar/pull/20948
> > [2] https://github.com/apache/pulsar/pull/21027

Re: [DISCUSS]PIP-295: Fixing Chunk Message Duplication Issue

[Xiangying Meng]

gt; > > > about by this PIP
> > > > will cause the consumer to use messages 1,2,5 for
> > > > assembly. There is
> > > > no guarantee that the producer will split the message
> > > > in the same way
> > > > twice before and after. For example, the producer's
> > > > maxMessageSize may
> > > > be different. This may cause the consumer to
> > > > receive a corrupt
> > > > message.
> > >
> > > Good point.
> > >
> > >
> > > Thanks
> > > Yubiao Feng
> > >
> > > On Wed, Aug 23, 2023 at 12:34 PM Zike Yang  wrote:
> > >
> > > > Hi, xiangying,
> > > >
> > > > Thanks for your PIP.
> > > >
> > > > IIUC, this may change the existing behavior and may introduce
> > > > inconsistencies.
> > > > Suppose that we have a large message with 3 chunks. But the producer
> > > > crashes and resends the message after sending the chunk-1. It will
> > > > send a total of 5 messages to the Pulsar topic:
> > > >
> > > > 1. SequenceID: 0, ChunkID: 0
> > > > 2. SequenceID: 0, ChunkID: 1
> > > > 3. SequenceID: 0, ChunkID: 0   -> This message will be dropped
> > > > 4. SequenceID: 0, ChunkID: 1-> Will also be dropped
> > > > 5. SequenceID: 0, ChunkID: 2-> The last chunk of the message
> > > >
> > > > For the existing behavior, the consumer assembles messages 3,4,5 into
> > > > the original large message. But the changes brought about by this PIP
> > > > will cause the consumer to use messages 1,2,5 for assembly. There is
> > > > no guarantee that the producer will split the message in the same way
> > > > twice before and after. For example, the producer's maxMessageSize may
> > > > be different. This may cause the consumer to receive a corrupt
> > > > message.
> > > >
> > > > Also, this PIP increases the complexity of handling chunks on the
> > > > broker side. Brokers should, in general, treat the chunk as a normal
> > > > message.
> > > >
> > > > I think a simple better approach is to only check the deduplication
> > > > for the last chunk of the large message. The consumer only gets the
> > > > whole message after receiving the last chunk. We don't need to check
> > > > the deduplication for all previous chunks. Also by doing this we only
> > > > need bug fixes, we don't need to introduce a new PIP.
> > > >
> > > > BR,
> > > > Zike Yang
> > > >
> > > > On Fri, Aug 18, 2023 at 7:54 PM Xiangying Meng 
> > > > wrote:
> > > > >
> > > > > Dear Community,
> > > > >
> > > > > I hope this email finds you well. I'd like to address an important
> > > > > issue related to Apache Pulsar and discuss a solution I've proposed on
> > > > > GitHub. The problem pertains to the handling of Chunk Messages after
> > > > > enabling deduplication.
> > > > >
> > > > > In the current version of Apache Pulsar, all chunks of a Chunk Message
> > > > > share the same sequence ID. However, enabling the depublication
> > > > > feature results in an inability to send Chunk Messages. To tackle this
> > > > > problem, I've proposed a solution [1] that ensures messages are not
> > > > > duplicated throughout end-to-end delivery. While this fix addresses
> > > > > the duplication issue for end-to-end messages, there remains a
> > > > > possibility of duplicate chunks within topics.
> > > > >
> > > > > To address this concern, I believe we should introduce a "Chunk ID
> > > > > map" at the Broker level, similar to the existing "sequence ID map",
> > > > > to facilitate effective filtering. However, implementing this has led
> > > > > to a challenge: a producer requires storage for two Long values
> > > > > simultaneously (sequence ID and chunk ID). Because the snapshot of the
> > > > > sequence ID map is stored through the properties of the cursor
> > > > > (Map), so in order to satisfy the storage of two Longs
> > > > > (sequence ID, chunk ID) corresponding to one producer, we hope to add
> > > > > a mark DeleteProperties (Map) String, String>) to
> > > > > replace the properties (Map) field. To resolve this,
> > > > > I've proposed an alternative proposal [2] involving the introduction
> > > > > of a "mark DeleteProperties" (Map) to replace the
> > > > > current properties (Map) field.
> > > > >
> > > > > I'd appreciate it if you carefully review both PRs and share your
> > > > > valuable feedback and insights. Thank you immensely for your time and
> > > > > attention. I eagerly anticipate your valuable opinions and
> > > > > recommendations.
> > > > >
> > > > > Warm regards,
> > > > > Xiangying
> > > > >
> > > > > [1] https://github.com/apache/pulsar/pull/20948
> > > > > [2] https://github.com/apache/pulsar/pull/21027
> > > >

Re: [DISCUSS]PIP-295: Fixing Chunk Message Duplication Issue

[Xiangying Meng]

>IIUC, this may change the existing behavior and may introduce inconsistencies.
>Suppose that we have a large message with 3 chunks. But the producer
>crashes and resends the message after sending the chunk-1. It will
>send a total of 5 messages to the Pulsar topic:
>
>1. SequenceID: 0, ChunkID: 0
>2. SequenceID: 0, ChunkID: 1
>3. SequenceID: 0, ChunkID: 0   -> This message will be dropped
>4. SequenceID: 0, ChunkID: 1-> Will also be dropped
>5. SequenceID: 0, ChunkID: 2-> The last chunk of the message

Hi Zike
There is another point. The resend of the chunk message has a bug that
I shared with you, and you fixed in PR [0]. It will make this case
happen in another way.
Sample description for the  bug:
Because the chunk message uses the same message metadata, if the chunk
is not sent out immediately. Then, when resending, all chunks of the
same chunk message use the chunk ID of the last chunk.
In this case, It should happen as:
1. SequenceID: 0, ChunkID: 0 (Put op1 into `pendingMessages` and send)
2. SequenceID: 0, ChunkID: 1 (Put op2 into `pendingMessages` and send)
3. SequenceID: 0, ChunkID: 2   -> (Put op3 into `pendingMessages`)
4. SequenceID: 0, ChunkID: 2   -> (Resend op1)
5. SequenceID: 0, ChunkID: 2   -> (Resend op2)
6. SequenceID: 0, ChunkID: 2   -> (Send op3)


BR,
Xiangying

[0] - https://github.com/apache/pulsar/pull/21048

On Thu, Aug 24, 2023 at 8:09 PM Xiangying Meng  wrote:
>
> >> This solution also cannot solve the out-of-order messages inside the
> >>chunks. For example, the above five messages will still be persisted.
> >The consumer already handles this case. The above 5 messages will all
> >be persisted but the consumer will skip message 1 and 2.
> >For messages 3, 4, and 5. The producer can guarantee these chunks are in 
> >order.
>
> The rewind operation is seen in the test log. Every time an incorrect
> chunk message is received, it will rewind, and the code has yet to be
> studied in depth.
> If it does not call rewind, then this case is considered a workable
> case. Let's look at another case.
> 1. SequenceID: 0, ChunkID: 0
> 2. SequenceID: 0, ChunkID: 1
> 3. SequenceID: 0, ChunkID: 1
> 4. SequenceID: 0, ChunkID: 2
> Such four chunks cannot be processed correctly by the consumer.
>
> In fact, this solution is my original idea. The PR I mentioned in the
> first email above uses a similar solution and modifies the logic on
> the consumer side.
> Also, as I mentioned in the first email, this solution can only solve
> the problem of end-to-end duplication. But chunks 1, 2, 3, and 4 are
> still persisted in the topic.
>
> On Thu, Aug 24, 2023 at 3:00 PM Zike Yang  wrote:
> >
> > Hi Heesung,
> >
> > > I believe in this PIP "similar to the existing "sequence ID map",
> > to facilitate effective filtering" actually means tracking the last
> > chunkId(not all chunk ids) on the broker side.
> >
> > With this simple solution, I think we don't need to track the
> > (sequenceID, chunkID) on the broker side at all. The broker just needs
> > to apply the deduplication logic to the last chunk instead of all
> > previous chunks. This PIP actually could do that, but it will
> > introduce a new data format and compatibility issue.
> >
> > > This is still a behavior change(deduping chunk messages on the broker),
> > and I believe we need to discuss this addition as a PIP.
> >
> > Actually, we didn't specifically state the deduping chunk message
> > behavior before. The chunked message should be equally applicable to
> > the de-duplication logic as a regular message. Therefore, I think it
> > should be considered as a bug fix. But if this FIX is worth discussing
> > in depth. I have no objection to it being a new PIP.
> >
> > > I think brokers can track the last chunkMaxMessageSize for
> > each producer.
> >
> > Using different chunkMaxMessageSize is just one of the aspects. In
> > PIP-132 [0], we have included the message metadata size when checking
> > maxMessageSize.
> > The message metadata can be changed after splitting the chunks. We are
> > still uncertain about the way the chunked message is split, even using
> > the same ss chunkMaxMessageSize.
> >
> > > then the brokers can assume that the producer is resending the chunks from
> > the beginning with a different scheme(restarted with a different
> > chunkMaxMessageSize) and accept those new chunks from the beginning.
> >
> > Regarding this, it seems like we are implementing dynamic
> > configuration for the chunkMaxMessageSize. I'm afraid that this would
> > change the expected behavior and introduc

[VOTE] PIP 296: Introduce the `getLastMessageIds` API to Reader

[Xiangying Meng]

Hi Pulsar Community,

This is the vote thread for PIP 296:
https://github.com/apache/pulsar/pull/21052

This PIP will help to improve the flexibility of Reader usage.

Thanks,
Xiangying

Re: [DISCUSS]PIP-295: Fixing Chunk Message Duplication Issue

[Xiangying Meng]

Hi Zike,

>How would this happen to get two duplicated and consecutive ChunkID-1
>messages? The producer should guarantee to retry the whole chunked
>messages instead of some parts of the chunks.

If the producer guarantees to retry the whole chunked messages instead
of some parts of the chunks,
Why doesn't the bug of the producer retry chunk messages in the PR [0] appear?
And why do you need to set `chunkId` in `op.rePopulate`?
It will be rested when split chunk messages again if the producer
guarantees to retry the whole chunked messages.
```
final MessageMetadata finalMsgMetadata = msgMetadata;
op.rePopulate = () -> {
if (msgMetadata.hasChunkId()) {
// The message metadata is shared between all chunks in a large message
// We need to reset the chunk id for each call of this method
// It's safe to do that because there is only 1 thread to manipulate
this message metadata
finalMsgMetadata.setChunkId(chunkId);
}
op.cmd = sendMessage(producerId, sequenceId, numMessages, messageId,
finalMsgMetadata,
encryptedPayload);
};

```

>> But chunks 1, 2, 3, and 4 are still persisted in the topic.
>
>I think it's OK to persist them all on the topic. Is there any issue
>with doing that?

This is just one scenario. Whether only check the sequence ID of the
first chunk (as I used in PR[1]) or check the sequence ID of the last
chunk (as you suggested), in reality, neither of these methods can
deduplicate chunks on the broker side because the broker cannot know
the chunk ID of the previous message.

However, if combined with the optimization of consumer-side logic,
end-to-end deduplication can be completed.
This is also a less-than-perfect solution I mentioned in my first
email and implemented in PR[1].

The reason I propose this proposal is not to solve the end-to-end
deduplication of chunk messages between producers and consumers. That
aspect has essentially been addressed in PR[1] and is still undergoing
review.

This proposal aims to ensure that no corrupt data exists within the
topic, as our data might be offloaded and used elsewhere in scenarios
where consumer logic is not optimized.

BR,
Xiangying

[0] https://github.com/apache/pulsar/pull/21048
[1] https://github.com/apache/pulsar/pull/20948

On Fri, Aug 25, 2023 at 5:18 PM Zike Yang  wrote:
>
> HI xiangying
>
> > The rewind operation is seen in the test log.
>
> That seems weird. Not sure if this rewind is related to the chunk consuming.
>
> > 1. SequenceID: 0, ChunkID: 0
> 2. SequenceID: 0, ChunkID: 1
> 3. SequenceID: 0, ChunkID: 1
> 4. SequenceID: 0, ChunkID: 2
> Such four chunks cannot be processed correctly by the consumer.
>
> How would this happen to get two duplicated and consecutive ChunkID-1
> messages? The producer should guarantee to retry the whole chunked
> messages instead of some parts of the chunks.
>
> > But chunks 1, 2, 3, and 4 are still persisted in the topic.
>
> I think it's OK to persist them all in the topic. Is there any issue
> with doing that?
>
> > There is another point. The resend of the chunk message has a bug that
> I shared with you, and you fixed in PR [0]. It will make this case
> happen in another way.
>
> If the user sets the sequence ID manually, the case could be reproduced.
>
> BR,
> Zike Yang
>
> On Thu, Aug 24, 2023 at 8:48 PM Xiangying Meng  wrote:
> >
> > >IIUC, this may change the existing behavior and may introduce 
> > >inconsistencies.
> > >Suppose that we have a large message with 3 chunks. But the producer
> > >crashes and resends the message after sending the chunk-1. It will
> > >send a total of 5 messages to the Pulsar topic:
> > >
> > >1. SequenceID: 0, ChunkID: 0
> > >2. SequenceID: 0, ChunkID: 1
> > >3. SequenceID: 0, ChunkID: 0   -> This message will be dropped
> > >4. SequenceID: 0, ChunkID: 1-> Will also be dropped
> > >5. SequenceID: 0, ChunkID: 2-> The last chunk of the message
> >
> > Hi Zike
> > There is another point. The resend of the chunk message has a bug that
> > I shared with you, and you fixed in PR [0]. It will make this case
> > happen in another way.
> > Sample description for the  bug:
> > Because the chunk message uses the same message metadata, if the chunk
> > is not sent out immediately. Then, when resending, all chunks of the
> > same chunk message use the chunk ID of the last chunk.
> > In this case, It should happen as:
> > 1. SequenceID: 0, ChunkID: 0 (Put op1 into `pendingMessages` and send)
> > 2. SequenceID: 0, ChunkID: 1 (Put op2 into `pendingMessages` and send)
> > 3. SequenceID: 0, ChunkID: 2   -> (Put op3 into `pendingMessages`)
> > 4. SequenceID: 0, ChunkID: 2   -> (Resend op1)
> > 5. SequenceID: 0, Chunk

Re: [DISCUSS]PIP-295: Fixing Chunk Message Duplication Issue

[Xiangying Meng]

Hi Heesung

It is a good idea to cover this incompatibility case if the producer
splits the chunk message again when retrying.

But in fact, the producer only resents the chunks that are assembled
to `OpSendMsg` instead of splitting the chunk message again.
So, there is no incompatibility issue of resenting the chunk message
by splitting the chunk message again.

The logic of sending chunk messages can be found here:
https://github.com/apache/pulsar/blob/e0c481e5f8d7fa5534d3327785928a234376789e/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java#L533

The logic of resending the message can be found here:
https://github.com/apache/pulsar/blob/e0c481e5f8d7fa5534d3327785928a234376789e/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java#L1892

BR,
Xiangying







On Sat, Aug 26, 2023 at 2:24 AM Heesung Sohn
 wrote:
>
> >> I think brokers can track the last chunkMaxMessageSize for each producer.
>
> > Using different chunkMaxMessageSize is just one of the aspects. In
> PIP-132 [0], we have included the message metadata size when checking
> maxMessageSize.The message metadata can be changed after splitting the
> chunks. We are still uncertain about the way the chunked message is split,
> even using the same ss chunkMaxMessageSize.
>
> This sounds like we need to revisit chunking uuid logic.
> Like I commented here,
> https://github.com/apache/pulsar/pull/20948/files#r1305997883
> Why don't we add a chunk session id suffix to identify the ongoing chunking
> uniquely?
>
> Currently,
>
> chunking uuid = producer + sequence_id
>
> Proposal
> chunking  uuid = producer + sequence_id + chunkingSessionId
>
> * chunkingSessionId could be a timestamp when the chunking started.
>
>
>
> On Fri, Aug 25, 2023 at 6:02 AM Xiangying Meng  wrote:
>
> > Hi Zike,
> >
> > >How would this happen to get two duplicated and consecutive ChunkID-1
> > >messages? The producer should guarantee to retry the whole chunked
> > >messages instead of some parts of the chunks.
> >
> > If the producer guarantees to retry the whole chunked messages instead
> > of some parts of the chunks,
> > Why doesn't the bug of the producer retry chunk messages in the PR [0]
> > appear?
> > And why do you need to set `chunkId` in `op.rePopulate`?
> > It will be rested when split chunk messages again if the producer
> > guarantees to retry the whole chunked messages.
> > ```
> > final MessageMetadata finalMsgMetadata = msgMetadata;
> > op.rePopulate = () -> {
> > if (msgMetadata.hasChunkId()) {
> > // The message metadata is shared between all chunks in a large message
> > // We need to reset the chunk id for each call of this method
> > // It's safe to do that because there is only 1 thread to manipulate
> > this message metadata
> > finalMsgMetadata.setChunkId(chunkId);
> > }
> > op.cmd = sendMessage(producerId, sequenceId, numMessages, messageId,
> > finalMsgMetadata,
> > encryptedPayload);
> > };
> >
> > ```
> >
> > >> But chunks 1, 2, 3, and 4 are still persisted in the topic.
> > >
> > >I think it's OK to persist them all on the topic. Is there any issue
> > >with doing that?
> >
> > This is just one scenario. Whether only check the sequence ID of the
> > first chunk (as I used in PR[1]) or check the sequence ID of the last
> > chunk (as you suggested), in reality, neither of these methods can
> > deduplicate chunks on the broker side because the broker cannot know
> > the chunk ID of the previous message.
> >
> > However, if combined with the optimization of consumer-side logic,
> > end-to-end deduplication can be completed.
> > This is also a less-than-perfect solution I mentioned in my first
> > email and implemented in PR[1].
> >
> > The reason I propose this proposal is not to solve the end-to-end
> > deduplication of chunk messages between producers and consumers. That
> > aspect has essentially been addressed in PR[1] and is still undergoing
> > review.
> >
> > This proposal aims to ensure that no corrupt data exists within the
> > topic, as our data might be offloaded and used elsewhere in scenarios
> > where consumer logic is not optimized.
> >
> > BR,
> > Xiangying
> >
> > [0] https://github.com/apache/pulsar/pull/21048
> > [1] https://github.com/apache/pulsar/pull/20948
> >
> > On Fri, Aug 25, 2023 at 5:18 PM Zike Yang  wrote:
> > >
> > > HI xiangying
> > >
> > > > The rewind operation is seen in the test log.
> > >
> > > That seem

Re: [DISCUSS]PIP-295: Fixing Chunk Message Duplication Issue

[Xiangying Meng]

Hi Heesung

It is a good point.
Assume the producer application jvm restarts in the middle of chunking and
resends the message chunks from the beginning with the previous sequence id.

For the previous version, it should be:

Producer send:
1. SequenceID: 0, ChunkID: 0
2. SequenceID: 0, ChunkID: 1
3. SequenceID: 0, ChunkID: 0
4. SequenceID: 0, ChunkID: 1
5. SequenceID: 0, ChunkID: 2

Consumer receive:
1. SequenceID: 0, ChunkID: 0
2. SequenceID: 0, ChunkID: 1
3. SequenceID: 0, ChunkID: 0 // chunk ID out of order. Release this
chunk and recycle its `chunkedMsgCtx`.
4. SequenceID: 0, ChunkID: 1  // chunkedMsgCtx == null Release it.
5. SequenceID: 0, ChunkID: 2  // chunkedMsgCtx == null Release it.

Therefore, for the previous version, this chunk message can not be
received by the consumer. It is not an incompatibility issue.

However, the solution of optimizing the `uuid` is valuable to the new
implementation.
I will modify this in the PR[0]. Thank you very much for your reminder
and the provided UUID optimization solution.

BR,
Xiangying

[0] https://github.com/apache/pulsar/pull/20948

On Sat, Aug 26, 2023 at 8:48 AM Heesung Sohn
 wrote:
>
> Hi, I meant
>
> What if the producer application jvm restarts in the middle of chunking and
> resends the message chunks from the beginning with the previous sequence id?
>
>
>
> On Fri, Aug 25, 2023 at 5:15 PM Xiangying Meng  wrote:
>
> > Hi Heesung
> >
> > It is a good idea to cover this incompatibility case if the producer
> > splits the chunk message again when retrying.
> >
> > But in fact, the producer only resents the chunks that are assembled
> > to `OpSendMsg` instead of splitting the chunk message again.
> > So, there is no incompatibility issue of resenting the chunk message
> > by splitting the chunk message again.
> >
> > The logic of sending chunk messages can be found here:
> >
> > https://github.com/apache/pulsar/blob/e0c481e5f8d7fa5534d3327785928a234376789e/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java#L533
> >
> > The logic of resending the message can be found here:
> >
> > https://github.com/apache/pulsar/blob/e0c481e5f8d7fa5534d3327785928a234376789e/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java#L1892
> >
> > BR,
> > Xiangying
> >
> >
> >
> >
> >
> >
> >
> > On Sat, Aug 26, 2023 at 2:24 AM Heesung Sohn
> >  wrote:
> > >
> > > >> I think brokers can track the last chunkMaxMessageSize for each
> > producer.
> > >
> > > > Using different chunkMaxMessageSize is just one of the aspects. In
> > > PIP-132 [0], we have included the message metadata size when checking
> > > maxMessageSize.The message metadata can be changed after splitting the
> > > chunks. We are still uncertain about the way the chunked message is
> > split,
> > > even using the same ss chunkMaxMessageSize.
> > >
> > > This sounds like we need to revisit chunking uuid logic.
> > > Like I commented here,
> > > https://github.com/apache/pulsar/pull/20948/files#r1305997883
> > > Why don't we add a chunk session id suffix to identify the ongoing
> > chunking
> > > uniquely?
> > >
> > > Currently,
> > >
> > > chunking uuid = producer + sequence_id
> > >
> > > Proposal
> > > chunking  uuid = producer + sequence_id + chunkingSessionId
> > >
> > > * chunkingSessionId could be a timestamp when the chunking started.
> > >
> > >
> > >
> > > On Fri, Aug 25, 2023 at 6:02 AM Xiangying Meng 
> > wrote:
> > >
> > > > Hi Zike,
> > > >
> > > > >How would this happen to get two duplicated and consecutive ChunkID-1
> > > > >messages? The producer should guarantee to retry the whole chunked
> > > > >messages instead of some parts of the chunks.
> > > >
> > > > If the producer guarantees to retry the whole chunked messages instead
> > > > of some parts of the chunks,
> > > > Why doesn't the bug of the producer retry chunk messages in the PR [0]
> > > > appear?
> > > > And why do you need to set `chunkId` in `op.rePopulate`?
> > > > It will be rested when split chunk messages again if the producer
> > > > guarantees to retry the whole chunked messages.
> > > > ```
> > > > final MessageMetadata finalMsgMetadata = msgMetadata;
> > > > op.rePopulate = () -> {
> > > > if (msgMetadata.hasChunkId()) {
> > > > // The message metadata is shared be

Re: [DISCUSS]PIP-295: Fixing Chunk Message Duplication Issue

[Xiangying Meng]

Hi Heesung,

>I think this means, for the PIP, the broker side's chunk deduplication.
>I think brokers probably need to track map to dedup

What is the significance of doing this?
My understanding is that if the client crashes and restarts after
sending half of a chunk message and then it resends the previous chunk
message, the resent chunk message should be treated as a new message
since it calls the producer's API again.
If deduplication is enabled, users should ensure that their customized
sequence ID is not lower than the previous sequence ID.
I have considered this scenario and added a warning log in PR[0]. (I'm
not sure whether an error log should be added or an exception thrown.)
If deduplication is not enabled, on the consumer side, there should be
an incomplete chunk message received alongside another complete chunk
message, each with a different UUID, and they will not interfere with
each other.

My main point is that every message sent using
`producer.newMessage().send()` should be treated as a new message.
UUID is solely used for the consumer side to identify different chunk messages.

BR
Xiangying

[0] https://github.com/apache/pulsar/pull/21047

On Sat, Aug 26, 2023 at 9:34 AM Heesung Sohn
 wrote:
>
> I think this means, for the PIP, the broker side's chunk deduplication.
> I think brokers probably need to track map to dedup
> chunks on the broker side.
>
>
>
>
> On Fri, Aug 25, 2023 at 6:16 PM Xiangying Meng  wrote:
>
> > Hi Heesung
> >
> > It is a good point.
> > Assume the producer application jvm restarts in the middle of chunking and
> > resends the message chunks from the beginning with the previous sequence
> > id.
> >
> > For the previous version, it should be:
> >
> > Producer send:
> > 1. SequenceID: 0, ChunkID: 0
> > 2. SequenceID: 0, ChunkID: 1
> > 3. SequenceID: 0, ChunkID: 0
> > 4. SequenceID: 0, ChunkID: 1
> > 5. SequenceID: 0, ChunkID: 2
> >
> > Consumer receive:
> > 1. SequenceID: 0, ChunkID: 0
> > 2. SequenceID: 0, ChunkID: 1
> > 3. SequenceID: 0, ChunkID: 0 // chunk ID out of order. Release this
> > chunk and recycle its `chunkedMsgCtx`.
> > 4. SequenceID: 0, ChunkID: 1  // chunkedMsgCtx == null Release it.
> > 5. SequenceID: 0, ChunkID: 2  // chunkedMsgCtx == null Release it.
> >
> > Therefore, for the previous version, this chunk message can not be
> > received by the consumer. It is not an incompatibility issue.
> >
> > However, the solution of optimizing the `uuid` is valuable to the new
> > implementation.
> > I will modify this in the PR[0]. Thank you very much for your reminder
> > and the provided UUID optimization solution.
> >
> > BR,
> > Xiangying
> >
> > [0] https://github.com/apache/pulsar/pull/20948
> >
> > On Sat, Aug 26, 2023 at 8:48 AM Heesung Sohn
> >  wrote:
> > >
> > > Hi, I meant
> > >
> > > What if the producer application jvm restarts in the middle of chunking
> > and
> > > resends the message chunks from the beginning with the previous sequence
> > id?
> > >
> > >
> > >
> > > On Fri, Aug 25, 2023 at 5:15 PM Xiangying Meng 
> > wrote:
> > >
> > > > Hi Heesung
> > > >
> > > > It is a good idea to cover this incompatibility case if the producer
> > > > splits the chunk message again when retrying.
> > > >
> > > > But in fact, the producer only resents the chunks that are assembled
> > > > to `OpSendMsg` instead of splitting the chunk message again.
> > > > So, there is no incompatibility issue of resenting the chunk message
> > > > by splitting the chunk message again.
> > > >
> > > > The logic of sending chunk messages can be found here:
> > > >
> > > >
> > https://github.com/apache/pulsar/blob/e0c481e5f8d7fa5534d3327785928a234376789e/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java#L533
> > > >
> > > > The logic of resending the message can be found here:
> > > >
> > > >
> > https://github.com/apache/pulsar/blob/e0c481e5f8d7fa5534d3327785928a234376789e/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java#L1892
> > > >
> > > > BR,
> > > > Xiangying
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Sat, Aug 26, 2023 at 2:24 AM Heesung Sohn
> > > >  wrote:
> > > > >
> > > > > >> I think broker

Re: [DISCUSS]PIP-295: Fixing Chunk Message Duplication Issue

[Xiangying Meng]

Hi Heesung,

In this case, the consumer only can receive m1.

But it has the same content as the previous case: What should we do if
the user sends messages with the sequence ID that was used previously?

I am afraid to introduce the incompatibility in this case, so I only
added a warning log in the PR[0] instead of throwing an exception.
Regarding this matter, what do you think? Should we throw an exception
or add error logs?

I'm looking forward to hearing your viewpoint.

Thanks,
Xiangying

[0] https://github.com/apache/pulsar/pull/21047

On Sat, Aug 26, 2023 at 10:58 AM Heesung Sohn
 wrote:
>
> Actually, can we think about this case too?
>
> What happens if the cx sends the same chunked msg with the same seq id when
> dedup is enabled?
>
> // user send a chunked msg, m1
> s1, c0
> s1, c1
> s1, c2 // complete
>
> // user resend the duplicate msg, m2
> s1, c0
> s1, c1
> s1, c2 //complete
>
> Do consumers receive m1 and m2(no dedup)?
>
>
>
> On Fri, Aug 25, 2023 at 6:55 PM Xiangying Meng  wrote:
>
> > Hi Heesung,
> >
> > >I think this means, for the PIP, the broker side's chunk deduplication.
> > >I think brokers probably need to track map to dedup
> >
> > What is the significance of doing this?
> > My understanding is that if the client crashes and restarts after
> > sending half of a chunk message and then it resends the previous chunk
> > message, the resent chunk message should be treated as a new message
> > since it calls the producer's API again.
> > If deduplication is enabled, users should ensure that their customized
> > sequence ID is not lower than the previous sequence ID.
> > I have considered this scenario and added a warning log in PR[0]. (I'm
> > not sure whether an error log should be added or an exception thrown.)
> > If deduplication is not enabled, on the consumer side, there should be
> > an incomplete chunk message received alongside another complete chunk
> > message, each with a different UUID, and they will not interfere with
> > each other.
> >
> > My main point is that every message sent using
> > `producer.newMessage().send()` should be treated as a new message.
> > UUID is solely used for the consumer side to identify different chunk
> > messages.
> >
> > BR
> > Xiangying
> >
> > [0] https://github.com/apache/pulsar/pull/21047
> >
> > On Sat, Aug 26, 2023 at 9:34 AM Heesung Sohn
> >  wrote:
> > >
> > > I think this means, for the PIP, the broker side's chunk deduplication.
> > > I think brokers probably need to track map to dedup
> > > chunks on the broker side.
> > >
> > >
> > >
> > >
> > > On Fri, Aug 25, 2023 at 6:16 PM Xiangying Meng 
> > wrote:
> > >
> > > > Hi Heesung
> > > >
> > > > It is a good point.
> > > > Assume the producer application jvm restarts in the middle of chunking
> > and
> > > > resends the message chunks from the beginning with the previous
> > sequence
> > > > id.
> > > >
> > > > For the previous version, it should be:
> > > >
> > > > Producer send:
> > > > 1. SequenceID: 0, ChunkID: 0
> > > > 2. SequenceID: 0, ChunkID: 1
> > > > 3. SequenceID: 0, ChunkID: 0
> > > > 4. SequenceID: 0, ChunkID: 1
> > > > 5. SequenceID: 0, ChunkID: 2
> > > >
> > > > Consumer receive:
> > > > 1. SequenceID: 0, ChunkID: 0
> > > > 2. SequenceID: 0, ChunkID: 1
> > > > 3. SequenceID: 0, ChunkID: 0 // chunk ID out of order. Release this
> > > > chunk and recycle its `chunkedMsgCtx`.
> > > > 4. SequenceID: 0, ChunkID: 1  // chunkedMsgCtx == null Release it.
> > > > 5. SequenceID: 0, ChunkID: 2  // chunkedMsgCtx == null Release it.
> > > >
> > > > Therefore, for the previous version, this chunk message can not be
> > > > received by the consumer. It is not an incompatibility issue.
> > > >
> > > > However, the solution of optimizing the `uuid` is valuable to the new
> > > > implementation.
> > > > I will modify this in the PR[0]. Thank you very much for your reminder
> > > > and the provided UUID optimization solution.
> > > >
> > > > BR,
> > > > Xiangying
> > > >
> > > > [0] https://github.com/apache/pulsar/pull/20948
> > > >
> > > > On Sat, Aug 26, 2023 at 8:48 AM Heesung Sohn
> > > >  wrote:
> > > > >
> > >

Re: [DISCUSS]PIP-295: Fixing Chunk Message Duplication Issue

[Xiangying Meng]

Hi Heesung,

Maybe we only need to maintain the last chunk ID in a map.
Map map1.
And we already have a map maintaining the last sequence ID.
Map map2.

If we do not throw an exception when users use the same sequence to
send the message.

For any incoming msg, m :
chunk ID = -1;
If m is a chunk message:
chunk ID = m.chunkID.
  If m.currentSeqid < LastSeqId, dedup.
  If m.currentSeqid > LastSeqId && m.chunk ID = 0, nodedup
if chunk ID exists in the map.
   Update it and log an error. This means there is an
incomplete chunk message.
If chunk ID does not exist in the map.
   Put it on the map.
  If m.currentSeqid == LastSeqId,
   1. if m.chunk ID == -1 || m.chunk ID == 0. dedup.
   2. If 1 <= m.chunkID <= total chunk.
  1. If chunk ID does not exist in the map. dedup.
  2. If chunk ID exists in the map. dedup. Check the order
of the chunkID to determine whether dedup;
   3. If m.chunkID == total chunk, persistent the chunk and
remove the chunkID in the map.


If we throw an exception when users use the same sequence to send the message.

For any incoming msg, m :
chunk ID = 0;
If m is a chunk message:
chunk ID = m.chunkID.
   If m.currentSeqid < LastSeqId, dedup.
   If m.currentSeqid == LastSeqId.
   If chunkID > 0, Check the last chunkID to determine whether to dedup.
If chunkID == 1, put chunkID into the map if absent.
   IF chunkID = 0, dedup.

BR,
xiangying

On Sat, Aug 26, 2023 at 11:53 AM Heesung Sohn
 wrote:
>
> However, what If the producer jvm gets restarted after the broker persists
> the m1 (but before updating their sequence id in their persistence
> storage), and the producer is trying to resend the same msg(so m2) with the
> same sequence id after restarting?
>
>
>
>
>
> On Fri, Aug 25, 2023 at 8:22 PM Xiangying Meng  wrote:
>
> > Hi Heesung,
> >
> > In this case, the consumer only can receive m1.
> >
> > But it has the same content as the previous case: What should we do if
> > the user sends messages with the sequence ID that was used previously?
> >
> > I am afraid to introduce the incompatibility in this case, so I only
> > added a warning log in the PR[0] instead of throwing an exception.
> > Regarding this matter, what do you think? Should we throw an exception
> > or add error logs?
> >
> > I'm looking forward to hearing your viewpoint.
> >
> > Thanks,
> > Xiangying
> >
> > [0] https://github.com/apache/pulsar/pull/21047
> >
> > On Sat, Aug 26, 2023 at 10:58 AM Heesung Sohn
> >  wrote:
> > >
> > > Actually, can we think about this case too?
> > >
> > > What happens if the cx sends the same chunked msg with the same seq id
> > when
> > > dedup is enabled?
> > >
> > > // user send a chunked msg, m1
> > > s1, c0
> > > s1, c1
> > > s1, c2 // complete
> > >
> > > // user resend the duplicate msg, m2
> > > s1, c0
> > > s1, c1
> > > s1, c2 //complete
> > >
> > > Do consumers receive m1 and m2(no dedup)?
> > >
> > >
> > >
> > > On Fri, Aug 25, 2023 at 6:55 PM Xiangying Meng 
> > wrote:
> > >
> > > > Hi Heesung,
> > > >
> > > > >I think this means, for the PIP, the broker side's chunk
> > deduplication.
> > > > >I think brokers probably need to track map to
> > dedup
> > > >
> > > > What is the significance of doing this?
> > > > My understanding is that if the client crashes and restarts after
> > > > sending half of a chunk message and then it resends the previous chunk
> > > > message, the resent chunk message should be treated as a new message
> > > > since it calls the producer's API again.
> > > > If deduplication is enabled, users should ensure that their customized
> > > > sequence ID is not lower than the previous sequence ID.
> > > > I have considered this scenario and added a warning log in PR[0]. (I'm
> > > > not sure whether an error log should be added or an exception thrown.)
> > > > If deduplication is not enabled, on the consumer side, there should be
> > > > an incomplete chunk message received alongside another complete chunk
> > > > message, each with a different UUID, and they will not interfere with
> > > > each other.
> > > >
> > > > My main point is that every message sent using
> > > > `producer.newMessage().send()` should be treated as a new message.
&

Re: [DISCUSS]PIP-295: Fixing Chunk Message Duplication Issue

[Xiangying Meng]

>Regarding this comment, can you explain how the consumer only receives m1?
>Here, m1's and m2's uuid and msgId will be different(if we suffix with a
>chunkingSessionId), although the sequence ID is the same.

They have the same sequence ID, so the first chunk will be dropped
because the sequence ID <= the last sequence ID. The other sequence ID
will be dropped because there is no chunk ID in the chunk ID map.
The detailed process can be found in my last email.


>> If we throw an exception when users use the same sequence to send the
>message.
>Do You mean `If "producers" throw an exception when users use the same
>sequence to send the message.`.
>Again, when the producers restart, they lose the last sequence id sent.

When producers restart, they will get the last sequence ID sent from the broker.
There are 2 sequence IDs for each producer:  seq 1, seq2. The seq1 is
updated before persisting messages, and the seq2 is updated after
persisting successfully.
If the client crushes and the producer rebuilds, the producer will get
seq1 from the broker.

So, the different messages should only have the same sequence ID if
the user mistakenly sets it.

>> If we do not throw an exception when users use the same sequence to
> Send the message.
>
>For this logic, how do we handle if the producer suddenly resends the
>chunked message with a different chunking scheme(e.g. maxMessageSize)?
>uuid=1, sid=0, cid=0
>uuid=1, sid=0, cid=1
>uuid=2, sid=0, cid=0
>uuid=2, sid=0, cid=1

They have different uuid, so they are different messages. But they
have the same sequence ID. The sequence ID should be recognized as
mistakenly set by users.
If we do not throw an exception, it should be deduplicated by the broker.

 > We could refine what to track and algo logic on the broker side more, but
> do we agree that the broker chunk dedup logic is needed?

I agree.
We have two issues that need to be clarified:
1. Can we allow some dirty data in the topic? This relates to whether
this proposal is necessary.
2. When the user's manually set sequence ID is greater than the
sequence ID of a previously sent message, should we throw an
exception?

On Sat, Aug 26, 2023 at 1:38 PM Heesung Sohn
 wrote:
>
> > In this case, the consumer only can receive m1.
>
> Regarding this comment, can you explain how the consumer only receives m1?
> Here, m1's and m2's uuid and msgId will be different(if we suffix with a
> chunkingSessionId), although the sequence id is the same.
>
> > If we throw an exception when users use the same sequence to send the
> message.
> Do You mean `If "producers" throw an exception when users use the same
> sequence to send the message.`.
> Again, when the producers restart, they lose the last sequence id sent.
>
>
> > If we do not throw an exception when users use the same sequence to
> send the message.
>
> For this logic, how do we handle if the producer suddenly resends the
> chunked message with a different chunking scheme(e.g. maxMessageSize) ?
> uuid=1, sid=0, cid=0
> uuid=1, sid=0, cid=1
> uuid=2, sid=0, cid=0
> uuid=2, sid=0, cid=1
>
> We could refine what to track and algo logic on the broker side more, but
> do we agree that the broker chunk dedup logic is needed?
>
> I will continue to think more next week. Have a nice weekend.
>
>
>
>
> On Fri, Aug 25, 2023 at 9:14 PM Xiangying Meng  wrote:
>
> > Hi Heesung,
> >
> > Maybe we only need to maintain the last chunk ID in a map.
> > Map map1.
> > And we already have a map maintaining the last sequence ID.
> > Map map2.
> >
> > If we do not throw an exception when users use the same sequence to
> > send the message.
> >
> > For any incoming msg, m :
> > chunk ID = -1;
> > If m is a chunk message:
> > chunk ID = m.chunkID.
> >   If m.currentSeqid < LastSeqId, dedup.
> >   If m.currentSeqid > LastSeqId && m.chunk ID = 0, nodedup
> > if chunk ID exists in the map.
> >Update it and log an error. This means there is an
> > incomplete chunk message.
> > If chunk ID does not exist in the map.
> >Put it on the map.
> >   If m.currentSeqid == LastSeqId,
> >1. if m.chunk ID == -1 || m.chunk ID == 0. dedup.
> >2. If 1 <= m.chunkID <= total chunk.
> >   1. If chunk ID does not exist in the map. dedup.
> >   2. If chunk ID exists in the map. dedup. Check the order
> > of the chunkID to determine whether dedup;
> >3. If m.chunkID == total chunk, persistent the chunk and
> > remove the chunkID in the map.
> >
> >
&g

Re: [DISCUSS]PIP-295: Fixing Chunk Message Duplication Issue

[Xiangying Meng]

>> Consumer receive:
>1. SequenceID: 0, ChunkID: 0
>2. SequenceID: 0, ChunkID: 1
>3. SequenceID: 0, ChunkID: 0 // chunk ID out of order. Release this
>chunk and recycle its `chunkedMsgCtx`.
>4. SequenceID: 0, ChunkID: 1  // chunkedMsgCtx == null Release it.
>5. SequenceID: 0, ChunkID: 2  // chunkedMsgCtx == null Release it.
>
>I think this case is wrong. For the current implementation, the
>message 3,4,5 will be assembled as a original large message.

Hi Zike
You can see the processMessageChunk method of the ConsumerImpl.

```

ChunkedMessageCtx chunkedMsgCtx = chunkedMessagesMap.get(msgMetadata.getUuid());

if (msgMetadata.getChunkId() == 0 && chunkedMsgCtx == null) {
//assemble a chunkedMsgCtx and put into
pendingChunkedMessageUuidQueue and chunkedMessagesMap.
}

if (chunkedMsgCtx == null || chunkedMsgCtx.chunkedMsgBuffer == null
|| msgMetadata.getChunkId() !=
(chunkedMsgCtx.lastChunkedMessageId + 1)) {
if (chunkedMsgCtx != null) {
if (chunkedMsgCtx.chunkedMsgBuffer != null) {
ReferenceCountUtil.safeRelease(chunkedMsgCtx.chunkedMsgBuffer);
}
chunkedMsgCtx.recycle();
}
chunkedMessagesMap.remove(msgMetadata.getUuid());
compressedPayload.release();
increaseAvailablePermits(cnx);
}

```

On Sat, Aug 26, 2023 at 3:48 PM Zike Yang  wrote:
>
> > Consumer receive:
> 1. SequenceID: 0, ChunkID: 0
> 2. SequenceID: 0, ChunkID: 1
> 3. SequenceID: 0, ChunkID: 0 // chunk ID out of order. Release this
> chunk and recycle its `chunkedMsgCtx`.
> 4. SequenceID: 0, ChunkID: 1  // chunkedMsgCtx == null Release it.
> 5. SequenceID: 0, ChunkID: 2  // chunkedMsgCtx == null Release it.
>
> I think this case is wrong. For the current implementation, the
> message 3,4,5 will be assembled as a original large message.
>
> HI, Heesung
>
>
> > I think brokers probably need to track map to dedup
>
> I propose a simpler solution in this mail thread earlier, which
> doesn't need to introduce map :
>
> > I think a simple better approach is to only check the deduplication
> for the last chunk of the large message. The consumer only gets the
> whole message after receiving the last chunk. We don't need to check
> the deduplication for all previous chunks. Also by doing this we only
> need bug fixes, we don't need to introduce a new PIP.
>
> Could you explain or show a case in what cases would lead to this
> simpler solution not working?
>
> Thanks,
> Zike Yang
>
> On Sat, Aug 26, 2023 at 1:38 PM Heesung Sohn
>  wrote:
> >
> > > In this case, the consumer only can receive m1.
> >
> > Regarding this comment, can you explain how the consumer only receives m1?
> > Here, m1's and m2's uuid and msgId will be different(if we suffix with a
> > chunkingSessionId), although the sequence id is the same.
> >
> > > If we throw an exception when users use the same sequence to send the
> > message.
> > Do You mean `If "producers" throw an exception when users use the same
> > sequence to send the message.`.
> > Again, when the producers restart, they lose the last sequence id sent.
> >
> >
> > > If we do not throw an exception when users use the same sequence to
> > send the message.
> >
> > For this logic, how do we handle if the producer suddenly resends the
> > chunked message with a different chunking scheme(e.g. maxMessageSize) ?
> > uuid=1, sid=0, cid=0
> > uuid=1, sid=0, cid=1
> > uuid=2, sid=0, cid=0
> > uuid=2, sid=0, cid=1
> >
> > We could refine what to track and algo logic on the broker side more, but
> > do we agree that the broker chunk dedup logic is needed?
> >
> > I will continue to think more next week. Have a nice weekend.
> >
> >
> >
> >
> > On Fri, Aug 25, 2023 at 9:14 PM Xiangying Meng  wrote:
> >
> > > Hi Heesung,
> > >
> > > Maybe we only need to maintain the last chunk ID in a map.
> > > Map map1.
> > > And we already have a map maintaining the last sequence ID.
> > > Map map2.
> > >
> > > If we do not throw an exception when users use the same sequence to
> > > send the message.
> > >
> > > For any incoming msg, m :
> > > chunk ID = -1;
> > > If m is a chunk message:
> > > chunk ID = m.chunkID.
> > >   If m.currentSeqid < LastSeqId, dedup.
> > >   If m.currentSeqid > LastSeqId && m.chunk ID = 0, nodedup
> > > if chunk ID exists in the map.
> > >Update it and log an error. This means there is an
> > > incomplete chunk message.
> > >

Re: [DISCUSS]PIP-295: Fixing Chunk Message Duplication Issue

[Xiangying Meng]

Hi Zike,

PR [0] has already fixed this bug and won't introduce compatibility issues.
PR [1] is unnecessary and can be closed. However, I still greatly
appreciate the information you provided.

[0] https://github.com/apache/pulsar/pull/20948
[1] https://github.com/apache/pulsar/pull/21070

On Sat, Aug 26, 2023 at 4:49 PM Zike Yang  wrote:
>
> > Hi Zike
> You can see the processMessageChunk method of the ConsumerImpl.
>
> Ah. That seems like a regression bug introduced by
> https://github.com/apache/pulsar/pull/18511. I have pushed a PR to fix
> it: https://github.com/apache/pulsar/pull/21070
>
> For the behavior before Pulsar 3.0.0. The consumer should assemble the
> message using 3,4,5.
>
> Thanks for pointing this out.
>
> BR,
> Zike Yang
>
> On Sat, Aug 26, 2023 at 3:58 PM Xiangying Meng  wrote:
> >
> > >> Consumer receive:
> > >1. SequenceID: 0, ChunkID: 0
> > >2. SequenceID: 0, ChunkID: 1
> > >3. SequenceID: 0, ChunkID: 0 // chunk ID out of order. Release this
> > >chunk and recycle its `chunkedMsgCtx`.
> > >4. SequenceID: 0, ChunkID: 1  // chunkedMsgCtx == null Release it.
> > >5. SequenceID: 0, ChunkID: 2  // chunkedMsgCtx == null Release it.
> > >
> > >I think this case is wrong. For the current implementation, the
> > >message 3,4,5 will be assembled as a original large message.
> >
> > Hi Zike
> > You can see the processMessageChunk method of the ConsumerImpl.
> >
> > ```
> >
> > ChunkedMessageCtx chunkedMsgCtx = 
> > chunkedMessagesMap.get(msgMetadata.getUuid());
> >
> > if (msgMetadata.getChunkId() == 0 && chunkedMsgCtx == null) {
> > //assemble a chunkedMsgCtx and put into
> > pendingChunkedMessageUuidQueue and chunkedMessagesMap.
> > }
> >
> > if (chunkedMsgCtx == null || chunkedMsgCtx.chunkedMsgBuffer == null
> > || msgMetadata.getChunkId() !=
> > (chunkedMsgCtx.lastChunkedMessageId + 1)) {
> > if (chunkedMsgCtx != null) {
> > if (chunkedMsgCtx.chunkedMsgBuffer != null) {
> > ReferenceCountUtil.safeRelease(chunkedMsgCtx.chunkedMsgBuffer);
> > }
> > chunkedMsgCtx.recycle();
> > }
> > chunkedMessagesMap.remove(msgMetadata.getUuid());
> > compressedPayload.release();
> > increaseAvailablePermits(cnx);
> > }
> >
> > ```
> >
> > On Sat, Aug 26, 2023 at 3:48 PM Zike Yang  wrote:
> > >
> > > > Consumer receive:
> > > 1. SequenceID: 0, ChunkID: 0
> > > 2. SequenceID: 0, ChunkID: 1
> > > 3. SequenceID: 0, ChunkID: 0 // chunk ID out of order. Release this
> > > chunk and recycle its `chunkedMsgCtx`.
> > > 4. SequenceID: 0, ChunkID: 1  // chunkedMsgCtx == null Release it.
> > > 5. SequenceID: 0, ChunkID: 2  // chunkedMsgCtx == null Release it.
> > >
> > > I think this case is wrong. For the current implementation, the
> > > message 3,4,5 will be assembled as a original large message.
> > >
> > > HI, Heesung
> > >
> > >
> > > > I think brokers probably need to track map to dedup
> > >
> > > I propose a simpler solution in this mail thread earlier, which
> > > doesn't need to introduce map :
> > >
> > > > I think a simple better approach is to only check the deduplication
> > > for the last chunk of the large message. The consumer only gets the
> > > whole message after receiving the last chunk. We don't need to check
> > > the deduplication for all previous chunks. Also by doing this we only
> > > need bug fixes, we don't need to introduce a new PIP.
> > >
> > > Could you explain or show a case in what cases would lead to this
> > > simpler solution not working?
> > >
> > > Thanks,
> > > Zike Yang
> > >
> > > On Sat, Aug 26, 2023 at 1:38 PM Heesung Sohn
> > >  wrote:
> > > >
> > > > > In this case, the consumer only can receive m1.
> > > >
> > > > Regarding this comment, can you explain how the consumer only receives 
> > > > m1?
> > > > Here, m1's and m2's uuid and msgId will be different(if we suffix with a
> > > > chunkingSessionId), although the sequence id is the same.
> > > >
> > > > > If we throw an exception when users use the same sequence to send the
> > > > message.
> > > > Do You mean `If "producers" throw an exception when users use the same
> > > > sequence to send the message.`.
>

Re: [DISCUSS]PIP-295: Fixing Chunk Message Duplication Issue

[Xiangying Meng]

Share more information: Even for versions before 3.0.0, the approach
doesn't assemble chunks 3, 4, and 5 together.

Please note this line of code:

```java
chunkedMsgCtx = chunkedMessagesMap.computeIfAbsent(msgMetadata.getUuid(),
(key) -> ChunkedMessageCtx.get(totalChunks,
chunkedMsgBuffer));
```

And the new solution we adopted in the PR [0] is to add a timestamp in
the uuid. Thank Heesung for providing this idea again.

[0]  https://github.com/apache/pulsar/pull/20948


On Sat, Aug 26, 2023 at 5:20 PM Xiangying Meng  wrote:
>
> Hi Zike,
>
> PR [0] has already fixed this bug and won't introduce compatibility issues.
> PR [1] is unnecessary and can be closed. However, I still greatly
> appreciate the information you provided.
>
> [0] https://github.com/apache/pulsar/pull/20948
> [1] https://github.com/apache/pulsar/pull/21070
>
> On Sat, Aug 26, 2023 at 4:49 PM Zike Yang  wrote:
> >
> > > Hi Zike
> > You can see the processMessageChunk method of the ConsumerImpl.
> >
> > Ah. That seems like a regression bug introduced by
> > https://github.com/apache/pulsar/pull/18511. I have pushed a PR to fix
> > it: https://github.com/apache/pulsar/pull/21070
> >
> > For the behavior before Pulsar 3.0.0. The consumer should assemble the
> > message using 3,4,5.
> >
> > Thanks for pointing this out.
> >
> > BR,
> > Zike Yang
> >
> > On Sat, Aug 26, 2023 at 3:58 PM Xiangying Meng  wrote:
> > >
> > > >> Consumer receive:
> > > >1. SequenceID: 0, ChunkID: 0
> > > >2. SequenceID: 0, ChunkID: 1
> > > >3. SequenceID: 0, ChunkID: 0 // chunk ID out of order. Release this
> > > >chunk and recycle its `chunkedMsgCtx`.
> > > >4. SequenceID: 0, ChunkID: 1  // chunkedMsgCtx == null Release it.
> > > >5. SequenceID: 0, ChunkID: 2  // chunkedMsgCtx == null Release it.
> > > >
> > > >I think this case is wrong. For the current implementation, the
> > > >message 3,4,5 will be assembled as a original large message.
> > >
> > > Hi Zike
> > > You can see the processMessageChunk method of the ConsumerImpl.
> > >
> > > ```
> > >
> > > ChunkedMessageCtx chunkedMsgCtx = 
> > > chunkedMessagesMap.get(msgMetadata.getUuid());
> > >
> > > if (msgMetadata.getChunkId() == 0 && chunkedMsgCtx == null) {
> > > //assemble a chunkedMsgCtx and put into
> > > pendingChunkedMessageUuidQueue and chunkedMessagesMap.
> > > }
> > >
> > > if (chunkedMsgCtx == null || chunkedMsgCtx.chunkedMsgBuffer == null
> > > || msgMetadata.getChunkId() !=
> > > (chunkedMsgCtx.lastChunkedMessageId + 1)) {
> > > if (chunkedMsgCtx != null) {
> > > if (chunkedMsgCtx.chunkedMsgBuffer != null) {
> > > 
> > > ReferenceCountUtil.safeRelease(chunkedMsgCtx.chunkedMsgBuffer);
> > > }
> > > chunkedMsgCtx.recycle();
> > > }
> > > chunkedMessagesMap.remove(msgMetadata.getUuid());
> > > compressedPayload.release();
> > > increaseAvailablePermits(cnx);
> > > }
> > >
> > > ```
> > >
> > > On Sat, Aug 26, 2023 at 3:48 PM Zike Yang  wrote:
> > > >
> > > > > Consumer receive:
> > > > 1. SequenceID: 0, ChunkID: 0
> > > > 2. SequenceID: 0, ChunkID: 1
> > > > 3. SequenceID: 0, ChunkID: 0 // chunk ID out of order. Release this
> > > > chunk and recycle its `chunkedMsgCtx`.
> > > > 4. SequenceID: 0, ChunkID: 1  // chunkedMsgCtx == null Release it.
> > > > 5. SequenceID: 0, ChunkID: 2  // chunkedMsgCtx == null Release it.
> > > >
> > > > I think this case is wrong. For the current implementation, the
> > > > message 3,4,5 will be assembled as a original large message.
> > > >
> > > > HI, Heesung
> > > >
> > > >
> > > > > I think brokers probably need to track map to 
> > > > > dedup
> > > >
> > > > I propose a simpler solution in this mail thread earlier, which
> > > > doesn't need to introduce map :
> > > >
> > > > > I think a simple better approach is to only check the deduplication
> > > > for the last chunk of the large message. The consumer only gets the
> > > > whole message after receiving the last chunk. We don't need to check
> > > > the deduplication for all previous chunks. Also by doing this we only
> > > > 

Re: [DISCUSS]PIP-295: Fixing Chunk Message Duplication Issue

[Xiangying Meng]

Hi Penghui,

>From my understanding.
>The message deduplication should only check the last chunk of the message.

Yes, I agree. If we only check the first chunk, the third chunk will be dropped.

Thanks,
Xiangying

On Mon, Aug 28, 2023 at 10:08 AM PengHui Li  wrote:
>
> Hi Xiangying,
>
> Thanks for driving the proposal.
> From my understanding.
> The message deduplication should only check the last chunk of the message.
> It doesn't need to care about whether each chunk is duplicated.
> The client side should handle issues like duplicated chunks.
>
> For the example that you have discussed:
>
> ```
> Producer sent:
> 1. SequenceID: 0, ChunkID: 0
> 2. SequenceID: 0, ChunkID: 1
> 3. SequenceID: 0, ChunkID: 0
> 4. SequenceID: 0, ChunkID: 1
> 5. SequenceID: 0, ChunkID: 2
> ```
>
> The consumer should give up 1 and 2. And start to build the chunk message
> from 3 to 5.
> Because 1 and 2 belong to an incomplete chunk message.
>
> For the deduplication. If the chunkId 2 is the last chunk of the message.
> We should put it into the persistence map in the deduplication once it has
> been persistent.
> Any subsequent messages with the same sequence ID and producer name will be
> treated as
> duplicates, no matter whether the sequence ID is generated by the producer
> or specified by users.
>
> Regards,
> Penghui
>
> On Sat, Aug 26, 2023 at 5:55 PM Xiangying Meng  wrote:
>
> > Share more information: Even for versions before 3.0.0, the approach
> > doesn't assemble chunks 3, 4, and 5 together.
> >
> > Please note this line of code:
> >
> > ```java
> > chunkedMsgCtx = chunkedMessagesMap.computeIfAbsent(msgMetadata.getUuid(),
> > (key) -> ChunkedMessageCtx.get(totalChunks,
> > chunkedMsgBuffer));
> > ```
> >
> > And the new solution we adopted in the PR [0] is to add a timestamp in
> > the uuid. Thank Heesung for providing this idea again.
> >
> > [0]  https://github.com/apache/pulsar/pull/20948
> >
> >
> > On Sat, Aug 26, 2023 at 5:20 PM Xiangying Meng 
> > wrote:
> > >
> > > Hi Zike,
> > >
> > > PR [0] has already fixed this bug and won't introduce compatibility
> > issues.
> > > PR [1] is unnecessary and can be closed. However, I still greatly
> > > appreciate the information you provided.
> > >
> > > [0] https://github.com/apache/pulsar/pull/20948
> > > [1] https://github.com/apache/pulsar/pull/21070
> > >
> > > On Sat, Aug 26, 2023 at 4:49 PM Zike Yang  wrote:
> > > >
> > > > > Hi Zike
> > > > You can see the processMessageChunk method of the ConsumerImpl.
> > > >
> > > > Ah. That seems like a regression bug introduced by
> > > > https://github.com/apache/pulsar/pull/18511. I have pushed a PR to fix
> > > > it: https://github.com/apache/pulsar/pull/21070
> > > >
> > > > For the behavior before Pulsar 3.0.0. The consumer should assemble the
> > > > message using 3,4,5.
> > > >
> > > > Thanks for pointing this out.
> > > >
> > > > BR,
> > > > Zike Yang
> > > >
> > > > On Sat, Aug 26, 2023 at 3:58 PM Xiangying Meng 
> > wrote:
> > > > >
> > > > > >> Consumer receive:
> > > > > >1. SequenceID: 0, ChunkID: 0
> > > > > >2. SequenceID: 0, ChunkID: 1
> > > > > >3. SequenceID: 0, ChunkID: 0 // chunk ID out of order. Release this
> > > > > >chunk and recycle its `chunkedMsgCtx`.
> > > > > >4. SequenceID: 0, ChunkID: 1  // chunkedMsgCtx == null Release it.
> > > > > >5. SequenceID: 0, ChunkID: 2  // chunkedMsgCtx == null Release it.
> > > > > >
> > > > > >I think this case is wrong. For the current implementation, the
> > > > > >message 3,4,5 will be assembled as a original large message.
> > > > >
> > > > > Hi Zike
> > > > > You can see the processMessageChunk method of the ConsumerImpl.
> > > > >
> > > > > ```
> > > > >
> > > > > ChunkedMessageCtx chunkedMsgCtx =
> > chunkedMessagesMap.get(msgMetadata.getUuid());
> > > > >
> > > > > if (msgMetadata.getChunkId() == 0 && chunkedMsgCtx == null) {
> > > > > //assemble a chunkedMsgCtx and put into
> > > > > pendingChunkedMessageUuidQueue and chunkedMessagesMap.
> > > > > }
> > > > >
> > > 

Re: [DISCUSS]PIP-295: Fixing Chunk Message Duplication Issue

[Xiangying Meng]

Hi community,

After internal discussions and agreement with Penghui and Zike, we've
come to a consensus that a small amount of dirty data within the topic
can be tolerated. As a result, this proposal has lost its
significance. We are now closing this discussion. Thanks to all
participants for their contributions. This has been a successful
collaboration.

Best regards,
Xiangying

On Mon, Aug 28, 2023 at 10:13 AM Xiangying Meng  wrote:
>
> Hi Penghui,
>
> >From my understanding.
> >The message deduplication should only check the last chunk of the message.
>
> Yes, I agree. If we only check the first chunk, the third chunk will be 
> dropped.
>
> Thanks,
> Xiangying
>
> On Mon, Aug 28, 2023 at 10:08 AM PengHui Li  wrote:
> >
> > Hi Xiangying,
> >
> > Thanks for driving the proposal.
> > From my understanding.
> > The message deduplication should only check the last chunk of the message.
> > It doesn't need to care about whether each chunk is duplicated.
> > The client side should handle issues like duplicated chunks.
> >
> > For the example that you have discussed:
> >
> > ```
> > Producer sent:
> > 1. SequenceID: 0, ChunkID: 0
> > 2. SequenceID: 0, ChunkID: 1
> > 3. SequenceID: 0, ChunkID: 0
> > 4. SequenceID: 0, ChunkID: 1
> > 5. SequenceID: 0, ChunkID: 2
> > ```
> >
> > The consumer should give up 1 and 2. And start to build the chunk message
> > from 3 to 5.
> > Because 1 and 2 belong to an incomplete chunk message.
> >
> > For the deduplication. If the chunkId 2 is the last chunk of the message.
> > We should put it into the persistence map in the deduplication once it has
> > been persistent.
> > Any subsequent messages with the same sequence ID and producer name will be
> > treated as
> > duplicates, no matter whether the sequence ID is generated by the producer
> > or specified by users.
> >
> > Regards,
> > Penghui
> >
> > On Sat, Aug 26, 2023 at 5:55 PM Xiangying Meng  wrote:
> >
> > > Share more information: Even for versions before 3.0.0, the approach
> > > doesn't assemble chunks 3, 4, and 5 together.
> > >
> > > Please note this line of code:
> > >
> > > ```java
> > > chunkedMsgCtx = chunkedMessagesMap.computeIfAbsent(msgMetadata.getUuid(),
> > >     (key) -> ChunkedMessageCtx.get(totalChunks,
> > > chunkedMsgBuffer));
> > > ```
> > >
> > > And the new solution we adopted in the PR [0] is to add a timestamp in
> > > the uuid. Thank Heesung for providing this idea again.
> > >
> > > [0]  https://github.com/apache/pulsar/pull/20948
> > >
> > >
> > > On Sat, Aug 26, 2023 at 5:20 PM Xiangying Meng 
> > > wrote:
> > > >
> > > > Hi Zike,
> > > >
> > > > PR [0] has already fixed this bug and won't introduce compatibility
> > > issues.
> > > > PR [1] is unnecessary and can be closed. However, I still greatly
> > > > appreciate the information you provided.
> > > >
> > > > [0] https://github.com/apache/pulsar/pull/20948
> > > > [1] https://github.com/apache/pulsar/pull/21070
> > > >
> > > > On Sat, Aug 26, 2023 at 4:49 PM Zike Yang  wrote:
> > > > >
> > > > > > Hi Zike
> > > > > You can see the processMessageChunk method of the ConsumerImpl.
> > > > >
> > > > > Ah. That seems like a regression bug introduced by
> > > > > https://github.com/apache/pulsar/pull/18511. I have pushed a PR to fix
> > > > > it: https://github.com/apache/pulsar/pull/21070
> > > > >
> > > > > For the behavior before Pulsar 3.0.0. The consumer should assemble the
> > > > > message using 3,4,5.
> > > > >
> > > > > Thanks for pointing this out.
> > > > >
> > > > > BR,
> > > > > Zike Yang
> > > > >
> > > > > On Sat, Aug 26, 2023 at 3:58 PM Xiangying Meng 
> > > wrote:
> > > > > >
> > > > > > >> Consumer receive:
> > > > > > >1. SequenceID: 0, ChunkID: 0
> > > > > > >2. SequenceID: 0, ChunkID: 1
> > > > > > >3. SequenceID: 0, ChunkID: 0 // chunk ID out of order. Release this
> > > > > > >chunk and recycle its `chunkedMsgCtx`.
> > > > > > >4. SequenceID: 0, ChunkID: 1  // chunkedMsgCtx == null Release it.
> 

Re: [VOTE] PIP 296: Introduce the `getLastMessageIds` API to Reader

[Xiangying Meng]

Close this vote with 4 binding
- Penghui
- Mattison
- Hang
- tison

On Mon, Aug 28, 2023 at 10:41 AM Zili Chen  wrote:
>
> +1 (binding)
>
> Thanks for driving the proposal!
>
> On 2023/08/25 06:52:38 Xiangying Meng wrote:
> > Hi Pulsar Community,
> >
> > This is the vote thread for PIP 296:
> > https://github.com/apache/pulsar/pull/21052
> >
> > This PIP will help to improve the flexibility of Reader usage.
> >
> > Thanks,
> > Xiangying
> >

[DISCUSS] PIP-302 Add new API readAllExistingMessages for TableView

[Xiangying Meng]

Hi dev,
I proposed a PIP, accessible via
https://github.com/apache/pulsar/pull/21166, to introduce an API that
allows us to wait until all data has been fully retrieved before
accessing the value corresponding to the desired key.
Please take a look and give your feedback.

Best Regards,
Xiangying

Re: [DISSCUSS] PIP-298: Consumer supports specifying consumption isolation level

[Xiangying Meng]

Hi Dave,
This is an external request. Paimon has added support for Kafka but
has not yet incorporated support for Pulsar. Therefore, the Paimon
community desires to integrate Pulsar.

Furthermore, when integrating Pulsar into Paimon, it is desired to
enable the ability to configure isolation levels, similar to Kafka, to
support reading uncommitted transaction logs.

Additional context can be found in the following link:
https://github.com/apache/incubator-paimon/issues/765

Sincerely,
Xiangying

On Mon, Sep 18, 2023 at 10:30 AM Dave Fisher  wrote:
>
> My concern is that this pip allows consumers to change the processing rules 
> for transactions in ways that a producer might find unexpected.
>
> I think if this proceeds then the scope needs to be expanded to 
> producers/admins needing to proactively allow transactions to be consumed 
> uncommitted.
>
> I am also interested in the use case that motivates this change.
>
> Best,
> Dave
>
> Sent from my iPhone
>
> > On Sep 17, 2023, at 8:50 AM, hzh0425  wrote:
> >
> > Hi, all
> >
> > This PR contributed to pip 298: https://github.com/apache/pulsar/pull/21114
> >
> >
> >
> >
> > This pip is to implement Read Committed and Read Uncommitted isolation 
> > levels for Pulsar transactions, allow consumers to configure isolation 
> > levels during the building process.
> >
> > For more details, please refer to pip-298.md
> >
> >
> >
> >
> > I hope everyone can help review and discuss this pip and enter the discuss 
> > stage.
> >
> > Thanks
> > Zhangheng Huang
>

Re: [DISSCUSS] PIP-298: Consumer supports specifying consumption isolation level

[Xiangying Meng]

Hi Dave,

I greatly appreciate your perspective, yet it leaves me with some
uncertainties that I am eager to address. Why would the introduction
of isolation levels constitute an insecure action?

>I think if this proceeds then the scope needs to be expanded to 
>producers/admins needing to proactively allow transactions to be consumed 
>uncommitted.

 We are merely presenting an option to the users. The notion of
establishing isolation levels for producers and administrators is, in
my view, devoid of necessity, and I am not inclined to implement it,
for it is devoid of substance.

Sincerely,
Xiangying

On Mon, Sep 18, 2023 at 10:58 PM Dave Fisher  wrote:
>
> Thanks. So, this is to support exfiltration of uncommitted transaction data? 
> This is IMO wrong and a security risk.
>
> Pulsar already supports CDC through IO Connectors.
>
> Kafka can be wrong about these isolation levels.
>
> There is really no information in those Paimon issues. How is Paimon’s 
> ability to support Pulsar broken by this edge case?
>
> Best,
> Dave
>
> Sent from my iPhone
>
> > On Sep 18, 2023, at 7:26 AM, Xiangying Meng  wrote:
> >
> > Hi Dave,
> > This is an external request. Paimon has added support for Kafka but
> > has not yet incorporated support for Pulsar. Therefore, the Paimon
> > community desires to integrate Pulsar.
> >
> > Furthermore, when integrating Pulsar into Paimon, it is desired to
> > enable the ability to configure isolation levels, similar to Kafka, to
> > support reading uncommitted transaction logs.
> >
> > Additional context can be found in the following link:
> > https://github.com/apache/incubator-paimon/issues/765
> >
> > Sincerely,
> > Xiangying
> >
> >> On Mon, Sep 18, 2023 at 10:30 AM Dave Fisher  wrote:
> >>
> >> My concern is that this pip allows consumers to change the processing 
> >> rules for transactions in ways that a producer might find unexpected.
> >>
> >> I think if this proceeds then the scope needs to be expanded to 
> >> producers/admins needing to proactively allow transactions to be consumed 
> >> uncommitted.
> >>
> >> I am also interested in the use case that motivates this change.
> >>
> >> Best,
> >> Dave
> >>
> >> Sent from my iPhone
> >>
> >>>> On Sep 17, 2023, at 8:50 AM, hzh0425  wrote:
> >>>
> >>> Hi, all
> >>>
> >>> This PR contributed to pip 298: 
> >>> https://github.com/apache/pulsar/pull/21114
> >>>
> >>>
> >>>
> >>>
> >>> This pip is to implement Read Committed and Read Uncommitted isolation 
> >>> levels for Pulsar transactions, allow consumers to configure isolation 
> >>> levels during the building process.
> >>>
> >>> For more details, please refer to pip-298.md
> >>>
> >>>
> >>>
> >>>
> >>> I hope everyone can help review and discuss this pip and enter the 
> >>> discuss stage.
> >>>
> >>> Thanks
> >>> Zhangheng Huang
> >>
>

Re: [DISSCUSS] PIP-298: Consumer supports specifying consumption isolation level

[Xiangying Meng]

Hi, all,

Let's consider another example:

**System**: Financial Transaction System

**Operations**: Large volume of deposit and withdrawal operations, a
small number of transfer operations.

**Roles**:

- **Client A1**
- **Client A2**
- **User Account B1**
- **User Account B2**
- **Request Topic C**
- **Real-time Monitoring System D**
- **Business Processing System E**

**Client Operations**:

- **Withdrawal**: Client A1 decreases the deposit amount from User
Account B1 or B2.
- **Deposit**: Client A1 increases the deposit amount in User Account B1 or B2.
- **Transfer**: Client A2 decreases the deposit amount from User
Account B1 and increases it in User Account B2. Or vice versa.

**Real-time Monitoring System D**: Obtains the latest data from
Request Topic C as quickly as possible to monitor transaction data and
changes in bank reserves in real-time. This is necessary for the
timely detection of anomalies and real-time decision-making.

**Business Processing System E**: Reads data from Request Topic C,
then actually operates User Accounts B1, B2.

**User Scenario**: Client A1 sends a large number of deposit and
withdrawal requests to Request Topic C. Client A2 writes a small
number of transfer requests to Request Topic C.

In this case, Business Processing System E needs a read-committed
isolation level to ensure operation consistency and Exactly Once
semantics. The real-time monitoring system does not care if a small
number of transfer requests are incomplete (dirty data). What it
cannot tolerate is a situation where a large number of deposit and
withdrawal requests cannot be presented in real time due to a small
number of transfer requests (the current situation is that uncommitted
transaction messages can block the reading of committed transaction
messages).

In this case, it is necessary to set different isolation levels for
different consumers/subscriptions.

Thanks,
Xiangying

On Tue, Sep 19, 2023 at 11:35 PM 杨国栋  wrote:
>
> Hi Dave and Xiangying,
> Thanks for all your support.
>
> Let me add some background.
>
> Apache Paimon take message queue as External Log Systems and changelog of
> Paimon can also be consumed from message queue.
> By default, change-log of message queue in Paimon are visible to consumers
> only after a snapshot. Snapshot have a same life cycle as message queue
> transactions.
> However, users can immediately consume change-log by read uncommited
> message without waiting for the next snapshot.
> This behavior reduces the latency of changelog, but it relies on reading
> uncommited message in Kafka or other message queue.
> So we hope Pulsar can support Read Uncommitted isolation level.
>
> Put aside the application scenarios of Paimon. Let's discuss Read
> Uncommitted isolation level itself.
>
> Read Uncommitted isolation will bring certain security risks, but will also
> make the message immediately readable.
> Reading submitted data can ensure accuracy, and reading uncommitted data
> can ensure real-time performance (there may be some repeated message or
> dirty message).
> Real-time performance is what users need. How to handle dirty message
> should be considered by the application side.
>
> We can still get complete and accurate data from Read Committed isolation
> level.
>
> Sincerely yours.

[VOTE] PIP-302 Add new API readAllExistingMessages for TableView

[Xiangying Meng]

Hi dev,
   This thread is to start a vote for PIP-302 Add new API
readAllExistingMessages for TableView.
Discuss thread: https://lists.apache.org/thread/o085y2314o0fymvx0x8pojmgjwcwn59q
PIP: https://github.com/apache/pulsar/pull/21166

BR,
Xiangying

Re: [DISCUSS] Unload Rate Limiting during Graceful Shutdown of Pulsar

[Xiangying Meng]

Hi Donglai, Heesung

>brokerShutdownMaxNumberOfGracefulBundleUnloadPerMinute=60 is the same as
brokerShutdownMaxNumberOfGracefulBundleUnloadPerSec=1 So, the "per-min"
config can be more granular.

I have some doubts about introducing the
`brokerShutdownMaxNumberOfGracefulBundleUnloadPerMinute`
configuration.

Firstly, I also think that a minute is too long. If we want the
maximum concurrency per second to be 1, and set the maximum
concurrency per minute to 60, then the actual maximum concurrency per
second could be up to 60, which is 60 times larger than our expected
maximum concurrency. Moreover, if the unload requests are concentrated
in the last 10 seconds of the previous minute and the first 10 seconds
of the next minute, then the concurrency during this period will
exceed our configuration. Such fluctuations are inevitable, but the
larger the time span we set, the greater the distortion of the
configuration.

Secondly, we already have the `maxConcurrentUnloadPerSec`
configuration, which is provided to the user in the CLI. Adding a
similar `brokerShutdownMaxNumberOfGracefulBundleUnloadPerMinute`
configuration might confuse users. When designing configuration
parameters, we should try to keep it simple and consistent, and avoid
introducing unnecessary complexity.

Thanks,
Xiangying

On Mon, Sep 25, 2023 at 12:14 PM Yubiao Feng
 wrote:
>
> Hi Donglai, Mattison
>
> I agree with @Mattison
>
> Thanks
> Yubiao Feng
>
> On Mon, Aug 21, 2023 at 8:50 PM  wrote:
>
> >
> > Hi,
> >
> > I agree with this change to improve the stability of the pulsar cluster.
> >
> > Just one concern. it's better to move this discussion to a new PIP.
> > because you wanna introduce a new broker configuration.
> > `brokerShutdownMaxNumberOfGracefulBundleUnloadPerMinute`
> >
> > FYI: https://github.com/apache/pulsar/blob/master/pip/README.md
> >
> > Looking forward this change and thanks for your contribution. :)
> >
> > Best,
> > Mattison
> >
> >
> >
> > On 7 Jul 2023 at 15:30 +0800, labuladong , wrote:
> > > Thanks you guys.
> > >
> > >
> > > I agree that per-minute is better than per-second, which is more
> > flexible. 
> > >
> > >
> > > I open an issue here:
> > >
> > >
> > > https://github.com/apache/pulsar/issues/20753
> > >
> > >
> > > Regards,
> > > donglai
> >

Re: [VOTE] PIP-302 Add new API readAllExistingMessages for TableView

[Xiangying Meng]

Thank you for your reminder. In our discussion, there were several
changes to the specific plan and method names, which resulted in the
PR title not being updated promptly. This was my oversight. The email
title for the vote was not modified to match the titles of the
discussed emails.

Regarding my proposal itself, would you happen to have any other questions?

BR,
Xiangying

On Mon, Sep 25, 2023 at 6:03 PM Zike Yang  wrote:
>
> Hi, Xiangying
>
> This PIP is a little confusing to me.
> This mail title states that we are introducing `readAllExistingMessages`.
> But actually, we only introduced `refreshAsync` in the PIP:
> https://github.com/apache/pulsar/pull/21166/files#diff-45c655583d6c0c73d87afd3df3fe67f77caadbf1bd691cf8f8211cc89728a1ceR34-R36
> And the PR title doesn't seem relevant. “PIP-302 Add alwaysRefresh
> Configuration Option for TableView to Read Latest Values”
>
> BR,
> Zike Yang
>
> On Mon, Sep 25, 2023 at 3:25 PM Xiangying Meng  wrote:
> >
> > Hi dev,
> >This thread is to start a vote for PIP-302 Add new API
> > readAllExistingMessages for TableView.
> > Discuss thread: 
> > https://lists.apache.org/thread/o085y2314o0fymvx0x8pojmgjwcwn59q
> > PIP: https://github.com/apache/pulsar/pull/21166
> >
> > BR,
> > Xiangying

Re: [DISCUSS] Unload Rate Limiting during Graceful Shutdown of Pulsar

[Xiangying Meng]

>I think the RateLimiter can handle it:
https://github.com/apache/pulsar/blob/a1405ea006f175b1bd0b9d28b9444d592fb4a010/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java#L965-L968

See here. Do you mean we make `maxConcurrentUnloadPerSec` and
`brokerShutdownMaxNumberOfGracefulBundleUnloadPerMinute` work
together? What is the meaning of this?
https://github.com/apache/pulsar/blob/a1405ea006f175b1bd0b9d28b9444d592fb4a010/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/BrokersBase.java#L564-L568

>`maxConcurrentUnloadPerSec ` is for the admin and CLI usage. This
proposal is to add
`brokerShutdownMaxNumberOfGracefulBundleUnloadPerMinute ` to the
broker configuration.

Yes, we are adding a broker configuration. That's not an issue, but
adding `brokerShutdownMaxNumberOfGracefulBundleUnloadPerMinute` with a
minute-based interval and expecting it to work in conjunction with
`maxConcurrentUnloadPerSec`, which is only used in the CLI, doesn't
make sense.

I understand that what we want to achieve is to have a broker
configuration that limits the concurrency of unloads even when
`maxConcurrentUnloadPerSec` is not set. Instead of adding
`brokerShutdownMaxNumberOfGracefulBundleUnloadPerMinute` when
`maxConcurrentUnloadPerSec` is already controlling the concurrency

Thanks,
Xiangying

On Mon, Sep 25, 2023 at 6:45 PM Zike Yang  wrote:
>
> > If we want the
> maximum concurrency per second to be 1, and set the maximum
> concurrency per minute to 60, then the actual maximum concurrency per
> second could be up to 60, which is 60 times larger than our expected
> maximum concurrency.
>
> I think the RateLimiter can handle it:
> https://github.com/apache/pulsar/blob/a1405ea006f175b1bd0b9d28b9444d592fb4a010/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java#L965-L968
>
> > Secondly, we already have the `maxConcurrentUnloadPerSec`
> configuration, which is provided to the user in the CLI. Adding a
> similar `brokerShutdownMaxNumberOfGracefulBundleUnloadPerMinute`
> configuration might confuse users.
>
> `maxConcurrentUnloadPerSec ` is for the admin and CLI usage. This
> proposal is to add
> `brokerShutdownMaxNumberOfGracefulBundleUnloadPerMinute ` to the
> broker configuration.
>
>
> Overall, I'm +1 for this proposal. And I agree that we need a new PIP
> for this change.
>
> BR,
> Zike Yang
>
> On Mon, Sep 25, 2023 at 3:54 PM Xiangying Meng  wrote:
> >
> > Hi Donglai, Heesung
> >
> > >brokerShutdownMaxNumberOfGracefulBundleUnloadPerMinute=60 is the same as
> > brokerShutdownMaxNumberOfGracefulBundleUnloadPerSec=1 So, the "per-min"
> > config can be more granular.
> >
> > I have some doubts about introducing the
> > `brokerShutdownMaxNumberOfGracefulBundleUnloadPerMinute`
> > configuration.
> >
> > Firstly, I also think that a minute is too long. If we want the
> > maximum concurrency per second to be 1, and set the maximum
> > concurrency per minute to 60, then the actual maximum concurrency per
> > second could be up to 60, which is 60 times larger than our expected
> > maximum concurrency. Moreover, if the unload requests are concentrated
> > in the last 10 seconds of the previous minute and the first 10 seconds
> > of the next minute, then the concurrency during this period will
> > exceed our configuration. Such fluctuations are inevitable, but the
> > larger the time span we set, the greater the distortion of the
> > configuration.
> >
> > Secondly, we already have the `maxConcurrentUnloadPerSec`
> > configuration, which is provided to the user in the CLI. Adding a
> > similar `brokerShutdownMaxNumberOfGracefulBundleUnloadPerMinute`
> > configuration might confuse users. When designing configuration
> > parameters, we should try to keep it simple and consistent, and avoid
> > introducing unnecessary complexity.
> >
> > Thanks,
> > Xiangying
> >
> > On Mon, Sep 25, 2023 at 12:14 PM Yubiao Feng
> >  wrote:
> > >
> > > Hi Donglai, Mattison
> > >
> > > I agree with @Mattison
> > >
> > > Thanks
> > > Yubiao Feng
> > >
> > > On Mon, Aug 21, 2023 at 8:50 PM  wrote:
> > >
> > > >
> > > > Hi,
> > > >
> > > > I agree with this change to improve the stability of the pulsar cluster.
> > > >
> > > > Just one concern. it's better to move this discussion to a new PIP.
> > > > because you wanna introduce a new broker configuration.
> > > > `brokerShutdownMaxNumberOfGracefulBundleUnloadPerMinute`
> > > >
> > > > 

Re: [DISSCUSS] PIP-298: Consumer supports specifying consumption isolation level

[Xiangying Meng]

Hi Dave,
The uncommitted transactions do not impact actual users' bank accounts.
Business Processing System E only reads committed transactional
messages and operates users' accounts. It needs Exactly-once semantic.
Real-time Monitoring System D reads uncommitted transactional
messages. It does not need Exactly-once semantic.

They use different subscriptions and choose different isolation
levels. One needs transaction, one does not.
In general, multiple subscriptions of the same topic do not all
require transaction guarantees.
Some want low latency without the exact-once semantic guarantee, and
some must require the exactly-once guarantee.
We just provide a new option for different subscriptions. This should
not be a breaking change,right?

Looking forward to your reply.

Thanks,
Xiangying

On Tue, Sep 26, 2023 at 4:09 AM Dave Fisher  wrote:
>
>
>
> > On Sep 20, 2023, at 12:50 AM, Xiangying Meng  wrote:
> >
> > Hi, all,
> >
> > Let's consider another example:
> >
> > **System**: Financial Transaction System
> >
> > **Operations**: Large volume of deposit and withdrawal operations, a
> > small number of transfer operations.
> >
> > **Roles**:
> >
> > - **Client A1**
> > - **Client A2**
> > - **User Account B1**
> > - **User Account B2**
> > - **Request Topic C**
> > - **Real-time Monitoring System D**
> > - **Business Processing System E**
> >
> > **Client Operations**:
> >
> > - **Withdrawal**: Client A1 decreases the deposit amount from User
> > Account B1 or B2.
> > - **Deposit**: Client A1 increases the deposit amount in User Account B1 or 
> > B2.
> > - **Transfer**: Client A2 decreases the deposit amount from User
> > Account B1 and increases it in User Account B2. Or vice versa.
> >
> > **Real-time Monitoring System D**: Obtains the latest data from
> > Request Topic C as quickly as possible to monitor transaction data and
> > changes in bank reserves in real-time. This is necessary for the
> > timely detection of anomalies and real-time decision-making.
> >
> > **Business Processing System E**: Reads data from Request Topic C,
> > then actually operates User Accounts B1, B2.
> >
> > **User Scenario**: Client A1 sends a large number of deposit and
> > withdrawal requests to Request Topic C. Client A2 writes a small
> > number of transfer requests to Request Topic C.
> >
> > In this case, Business Processing System E needs a read-committed
> > isolation level to ensure operation consistency and Exactly Once
> > semantics. The real-time monitoring system does not care if a small
> > number of transfer requests are incomplete (dirty data). What it
> > cannot tolerate is a situation where a large number of deposit and
> > withdrawal requests cannot be presented in real time due to a small
> > number of transfer requests (the current situation is that uncommitted
> > transaction messages can block the reading of committed transaction
> > messages).
>
> So you are willing to let uncommitted transactions impact actual users bank 
> accounts? Are you sure that there is not another way to bypass uncommitted 
> records? Letting uncommitted records through is not Exactly once.
>
> Are you ready to rewrite Pulsar’s documentation to explain how normal users 
> can avoid allowing this?
>
> Best,
> Dave
>
>
> >
> > In this case, it is necessary to set different isolation levels for
> > different consumers/subscriptions.
> >
> > Thanks,
> > Xiangying
> >
> > On Tue, Sep 19, 2023 at 11:35 PM 杨国栋  wrote:
> >>
> >> Hi Dave and Xiangying,
> >> Thanks for all your support.
> >>
> >> Let me add some background.
> >>
> >> Apache Paimon take message queue as External Log Systems and changelog of
> >> Paimon can also be consumed from message queue.
> >> By default, change-log of message queue in Paimon are visible to consumers
> >> only after a snapshot. Snapshot have a same life cycle as message queue
> >> transactions.
> >> However, users can immediately consume change-log by read uncommited
> >> message without waiting for the next snapshot.
> >> This behavior reduces the latency of changelog, but it relies on reading
> >> uncommited message in Kafka or other message queue.
> >> So we hope Pulsar can support Read Uncommitted isolation level.
> >>
> >> Put aside the application scenarios of Paimon. Let's discuss Read
> >> Uncommitted isolation level itself.
> >>
> >> Read Uncommitted isolation will bring certain security risks, but will also
> >> make the message immediately readable.
> >> Reading submitted data can ensure accuracy, and reading uncommitted data
> >> can ensure real-time performance (there may be some repeated message or
> >> dirty message).
> >> Real-time performance is what users need. How to handle dirty message
> >> should be considered by the application side.
> >>
> >> We can still get complete and accurate data from Read Committed isolation
> >> level.
> >>
> >> Sincerely yours.
>

Re: [DISCUSS] PIP-300: Add custom dynamic configuration for plugins

[Xiangying Meng]

Hi Zixuan,

This is really a great feature. I support it.

Regarding cherry-pick, as far as I know, we have cherry-picked some
configuration items and interfaces into branch-2.10.
But that should be mentioned in a separate discussion and provide
sufficient reasons why we have to do it. Cherry-pick is performed
after the vote passes. So, I suggest that feature and cherry-pick can
be discussed separately through different emails.

Sincerely,
Xiangying

On Mon, Sep 25, 2023 at 6:51 PM Zike Yang  wrote:
>
> Hi, Zixuan
>
> Thanks for your proposal. I'm +1 for it.
>
> >  This is a feature I need. If cherry-pick is not allowed, then it will
> only be kept in 3.2+.
>
> This is a new feature, and I also think that we couldn't cherry-pick
> it. What about cherry-picking this change to your fork repo and
> building the pulsar for your own to meet this need? Does it make sense
> to you?
>
> BR,
> Zike Yang
>
> On Mon, Sep 25, 2023 at 12:23 AM mattison chao  wrote:
> >
> > Hi, Zixuan
> >
> > I am afraid I can't support you in cherry-picking this feature for all of 
> > the active branches by the current fact. Because this is a new feature and 
> > it's not a bug fix or security issue.
> >
> > We can wait for other contributor's ideas.  WDYT?
> >
> > Thanks,
> > Mattison
> > On 19 Sep 2023 at 10:42 +0800, Zixuan Liu , wrote:
> > > > 1. When you want to cherry-pick a PR, it should be cherry-picked for 
> > > > all branches after the target branch. Isn't it?
> > >
> > > I think we should cherry-pick this PR to Pulsar 2.10, 2.11, 3.0.
> > >
> > > > 2. I've got a slight concern about cherry-picking. Do you have any 
> > > > issue or feature request in the community that can demonstrate this 
> > > > feature is required to cherry-pick?
> > >
> > > This is a feature I need. If cherry-pick is not allowed, then it will
> > > only be kept in 3.2+.
> > >
> > > Thanks,
> > > Zixuan
> > >
> > > mattison chao  于2023年9月18日周一 09:42写道：
> > >
> > > >
> > > > HI, Zixuan
> > > >
> > > > Thanks for your discussion. It's a great feature. But I've got several 
> > > > questions I want to discuss here.
> > > >
> > > > 1. When you want to cherry-pick a PR, it should be cherry-picked for 
> > > > all branches after the target branch. Isn't it?
> > > > 2. I've got a slight concern about cherry-picking. Do you have any 
> > > > issue or feature request in the community that can demonstrate this 
> > > > feature is required to cherry-pick?
> > > >
> > > >
> > > > Best,
> > > > Mattison
> > > > On 12 Sep 2023 at 11:25 +0800, Zixuan Liu , wrote:
> > > > > > BTW, I think we can cherry-pick this feature to the Pulsar 2.10. As
> > > > > > far as I know, the Pulsar 2.10 is a stable/main version, because 
> > > > > > many
> > > > > > users are using it.
> > > > > >
> > > > > > Thanks,
> > > > > > Zixuan
> > > > > >
> > > > > > Zixuan Liu  于2023年9月5日周二 17:09写道：
> > > > > > > >
> > > > > > > > Hi Pulsar Community,
> > > > > > > >
> > > > > > > > Discuss for PIP-300: https://github.com/apache/pulsar/pull/21127
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Zixuan

Re: [DISSCUSS] PIP-298: Consumer supports specifying consumption isolation level

[Xiangying Meng]

Hi Dave,

Thanks for your support.
I also think this should only be for the master branch.

Thanks,
Xiangying

On Tue, Sep 26, 2023 at 9:34 AM Dave Fisher  wrote:
>
> Hi -
>
> OK. I’ll agree, but I think the PIP ought to include documentation. There 
> should also be clear communication about this use case and how to use it.
>
> Sent from my iPhone
>
> > On Sep 25, 2023, at 6:23 PM, Xiangying Meng  wrote:
> >
> > Hi Dave,
> > The uncommitted transactions do not impact actual users' bank accounts.
> > Business Processing System E only reads committed transactional
> > messages and operates users' accounts. It needs Exactly-once semantic.
> > Real-time Monitoring System D reads uncommitted transactional
> > messages. It does not need Exactly-once semantic.
> >
> > They use different subscriptions and choose different isolation
> > levels. One needs transaction, one does not.
> > In general, multiple subscriptions of the same topic do not all
> > require transaction guarantees.
> > Some want low latency without the exact-once semantic guarantee, and
> > some must require the exactly-once guarantee.
> > We just provide a new option for different subscriptions. This should
> > not be a breaking change,right?
>
> Not a breaking change, but it does add to the API.
>
> It should be discussed if this PIP is only for master - 3.2, or if may be 
> cherry picked to current versions.
>
> >
> > Looking forward to your reply.
>
> Thank you,
> Dave
> >
> > Thanks,
> > Xiangying
> >
> >> On Tue, Sep 26, 2023 at 4:09 AM Dave Fisher  wrote:
> >>
> >>
> >>
> >>>> On Sep 20, 2023, at 12:50 AM, Xiangying Meng  
> >>>> wrote:
> >>>
> >>> Hi, all,
> >>>
> >>> Let's consider another example:
> >>>
> >>> **System**: Financial Transaction System
> >>>
> >>> **Operations**: Large volume of deposit and withdrawal operations, a
> >>> small number of transfer operations.
> >>>
> >>> **Roles**:
> >>>
> >>> - **Client A1**
> >>> - **Client A2**
> >>> - **User Account B1**
> >>> - **User Account B2**
> >>> - **Request Topic C**
> >>> - **Real-time Monitoring System D**
> >>> - **Business Processing System E**
> >>>
> >>> **Client Operations**:
> >>>
> >>> - **Withdrawal**: Client A1 decreases the deposit amount from User
> >>> Account B1 or B2.
> >>> - **Deposit**: Client A1 increases the deposit amount in User Account B1 
> >>> or B2.
> >>> - **Transfer**: Client A2 decreases the deposit amount from User
> >>> Account B1 and increases it in User Account B2. Or vice versa.
> >>>
> >>> **Real-time Monitoring System D**: Obtains the latest data from
> >>> Request Topic C as quickly as possible to monitor transaction data and
> >>> changes in bank reserves in real-time. This is necessary for the
> >>> timely detection of anomalies and real-time decision-making.
> >>>
> >>> **Business Processing System E**: Reads data from Request Topic C,
> >>> then actually operates User Accounts B1, B2.
> >>>
> >>> **User Scenario**: Client A1 sends a large number of deposit and
> >>> withdrawal requests to Request Topic C. Client A2 writes a small
> >>> number of transfer requests to Request Topic C.
> >>>
> >>> In this case, Business Processing System E needs a read-committed
> >>> isolation level to ensure operation consistency and Exactly Once
> >>> semantics. The real-time monitoring system does not care if a small
> >>> number of transfer requests are incomplete (dirty data). What it
> >>> cannot tolerate is a situation where a large number of deposit and
> >>> withdrawal requests cannot be presented in real time due to a small
> >>> number of transfer requests (the current situation is that uncommitted
> >>> transaction messages can block the reading of committed transaction
> >>> messages).
> >>
> >> So you are willing to let uncommitted transactions impact actual users 
> >> bank accounts? Are you sure that there is not another way to bypass 
> >> uncommitted records? Letting uncommitted records through is not Exactly 
> >> once.
> >>
> >> Are you ready to rewrite Pulsar’s documentation to explain how normal 
> >&

[VOTE] PIP-302 Introduce refreshAsync API for TableView

[Xiangying Meng]

Hi dev,
   This thread is to start a vote for PIP-302 Add new API
refreshAsync for TableView.
Discuss thread: https://lists.apache.org/thread/o085y2314o0fymvx0x8pojmgjwcwn59q
PIP: https://github.com/apache/pulsar/pull/21166

BR,
Xiangying

Re: [VOTE] PIP-302 Add new API readAllExistingMessages for TableView

[Xiangying Meng]

Close this via https://lists.apache.org/thread/vox93tmj33mms026wt52l92h1wffctbk

On Mon, Sep 25, 2023 at 6:34 PM Xiangying Meng  wrote:
>
> Thank you for your reminder. In our discussion, there were several
> changes to the specific plan and method names, which resulted in the
> PR title not being updated promptly. This was my oversight. The email
> title for the vote was not modified to match the titles of the
> discussed emails.
>
> Regarding my proposal itself, would you happen to have any other questions?
>
> BR,
> Xiangying
>
> On Mon, Sep 25, 2023 at 6:03 PM Zike Yang  wrote:
> >
> > Hi, Xiangying
> >
> > This PIP is a little confusing to me.
> > This mail title states that we are introducing `readAllExistingMessages`.
> > But actually, we only introduced `refreshAsync` in the PIP:
> > https://github.com/apache/pulsar/pull/21166/files#diff-45c655583d6c0c73d87afd3df3fe67f77caadbf1bd691cf8f8211cc89728a1ceR34-R36
> > And the PR title doesn't seem relevant. “PIP-302 Add alwaysRefresh
> > Configuration Option for TableView to Read Latest Values”
> >
> > BR,
> > Zike Yang
> >
> > On Mon, Sep 25, 2023 at 3:25 PM Xiangying Meng  wrote:
> > >
> > > Hi dev,
> > >This thread is to start a vote for PIP-302 Add new API
> > > readAllExistingMessages for TableView.
> > > Discuss thread: 
> > > https://lists.apache.org/thread/o085y2314o0fymvx0x8pojmgjwcwn59q
> > > PIP: https://github.com/apache/pulsar/pull/21166
> > >
> > > BR,
> > > Xiangying

Re: [VOTE] PIP-302 Introduce refreshAsync API for TableView

[Xiangying Meng]

Close this vote with 3 +1(binding)
- Penghui
- Jiwei Guo (Tboy)
- Mattison

On Sun, Oct 8, 2023 at 5:02 PM mattison chao  wrote:
>
> +1(binding)
>
> Best,
> Mattison
> On 8 Oct 2023 at 14:49 +0800, Yunze Xu , wrote:
> > Totally I'm +0 at the moment. I'm still wondering which issue you
> > really want to resolve. I left a comment
> > https://github.com/apache/pulsar/pull/21271#issuecomment-1751899833.
> > Generally you can get latest value unless the producer is far quicker
> > than the reader. However, even with the refreshAsync() method, in the
> > future's callback, the data could still be updated.
> >
> > From my perspective, this proposal only makes sense when you can
> > guarantee there is no more data written to the topic when you call
> > methods of TableView. But if so, a simple `boolean hasReachedLatest()`
> > method could do the trick.
> >
> > Thanks,
> > Yunze
> >
> > On Sun, Oct 8, 2023 at 2:12 PM 太上玄元道君  wrote:
> > >
> > > +1 (no-binding)
> > >
> > >
> > > Xiangying Meng 于2023年9月27日 周三15:05写道：
> > >
> > > > > Hi dev,
> > > > > This thread is to start a vote for PIP-302 Add new API
> > > > > refreshAsync for TableView.
> > > > > Discuss thread:
> > > > > https://lists.apache.org/thread/o085y2314o0fymvx0x8pojmgjwcwn59q
> > > > > PIP: https://github.com/apache/pulsar/pull/21166
> > > > >
> > > > > BR,
> > > > > Xiangying
> > > > >

[DISCUSS] PIP-311 Modify the signature of the `newMultiTransactionMessageAck` method

[Xiangying Meng]

Hi dev,
Currently, the public API `newMultiTransactionMessageAck` in the
`Commands.java` creates the ack commands without request ID and hopes
to wait for the response.

This causes all the requests created by
`newMultiTransactionMessageAck` to be timed out.
I drafted a proposal [0] to modify it.
Looking forward to receiving your feedback and suggestions.

[0]: https://github.com/apache/pulsar/pull/21419

Sincerely,
Xiangying

Re: [ANNOUNCE] Yubiao Feng as new PMC member in Apache Pulsar

[Xiangying Meng]

Congrats! Yubiao.

Thanks,
Xiangying

On Mon, Nov 13, 2023 at 8:15 PM Kai Wang  wrote:

> Congrats!
>
> Thanks,
> Kai
>

[DISCUSS] PIP-321 Split the responsibilities of namespace replication-clusters

[Xiangying Meng]

Hi, Pulsar Community

I drafted a proposal to make the configuration of clusters at the namespace
level clearer. This helps solve the problem of geo-replication not working
correctly at the topic level.

https://github.com/apache/pulsar/pull/21648

I'm looking forward to hearing from you.

BR
Xiangying

Re: [DISCUSS] PIP-321 Split the responsibilities of namespace replication-clusters

[Xiangying Meng]

Hi Rajan and Girish,
Thanks for your reply. About the question you mentioned, there is some
information I want to share with you.
>If anyone wants to setup different replication clusters then either
>those topics can be created under different namespaces or defined at topic
>level policy.

>And users can anyway go and update the namespace's cluster list to add the
>missing cluster.
Because the replication clusters also mean the clusters where the topic can
be created or loaded, the topic-level replication clusters can only be the
subset of namespace-level replication clusters.
Just as Girish mentioned, the users can update the namespace's cluster list
to add the missing cluster.
But there is a problem because the replication clusters as the namespace
level will create a full mesh replication for that namespace across the
clusters defined in
replication-clusters if users want to change the replication policy for
topic-n and do not change the replication policy of other topics, they need
to change all the topic policy under this namespace.

> Pulsar is being used by many legacy systems and changing its
>semantics for specific usecases without considering consequences are
>creating a lot of pain and incompatibility problems for other existing
>systems and let's avoid doing it as we are struggling with such changes and
>breaking compatibility or changing semantics are just not acceptable.

This proposal will not introduce an incompatibility problem, because the
default value of the namespace policy of allowed-clusters and
topic-policy-synchronized-clusters are the replication-clusters.

>Allowed clusters defined at tenant level
>will restrict tenants to create namespaces in regions/clusters where they
>are not allowed.
>As Rajan also mentioned, allowed-clusters field has a different
meaning/purpose.

Allowed clusters defined at the tenant level will restrict tenants from
creating namespaces in regions/clusters where they are not allowed.
Similarly, the allowed clusters defined at the namespace level will
restrict the namespace from creating topics in regions/clusters where they
are not allowed.
What's wrong with this?

Regards,
Xiangying

On Fri, Dec 1, 2023 at 2:35 PM Girish Sharma 
wrote:

> Hi Xiangying,
>
> Shouldn't the solution to the issue mentioned in #21564 [0] mostly be
> around validating that topic level replication clusters are subset of
> namespace level replication clusters?
> It would be a completely compatible change as even today the case where a
> topic has a cluster not defined in namespaces's replication-clusters
> doesn't really work.
> And users can anyway go and update the namespace's cluster list to add the
> missing cluster.
>
> As Rajan also mentioned, allowed-clusters field has a different
> meaning/purpose.
> Regards
>
> On Thu, Nov 30, 2023 at 10:56 AM Xiangying Meng 
> wrote:
>
> > Hi, Pulsar Community
> >
> > I drafted a proposal to make the configuration of clusters at the
> namespace
> > level clearer. This helps solve the problem of geo-replication not
> working
> > correctly at the topic level.
> >
> > https://github.com/apache/pulsar/pull/21648
> >
> > I'm looking forward to hearing from you.
> >
> > BR
> > Xiangying
> >
>
>
> --
> Girish Sharma
>

Re: [DISCUSS] PIP-321 Split the responsibilities of namespace replication-clusters

[Xiangying Meng]

Hi Joe,

You're correct. The initial design of the replication policy leaves room
for improvement. To address this, we aim to refine the cluster settings at
the namespace level in a way that won't impact the existing system. The
replication clusters should solely be used to establish full mesh
replication for that specific namespace, without having any other
definitions or functionalities.

BR,
Xiangying


On Mon, Dec 4, 2023 at 1:52 PM Joe F  wrote:

> >if users want to change the replication policy for
> topic-n and do not change the replication policy of other topics, they need
> to change all the topic policy under this namespace.
>
> This PIP unfortunately  flows from  attempting to solve bad application
> design
>
> A namespace is supposed to represent an application, and the namespace
> policy is an umbrella for a similar set of policies  that applies to all
> topics.  The exceptions would be if a topic had a need for a deficit, The
> case of one topic in the namespace sticking out of the namespace policy
> umbrella is bad  application design in my opinion
>
> -Joe.
>
>
>
> On Sun, Dec 3, 2023 at 6:00 PM Xiangying Meng 
> wrote:
>
> > Hi Rajan and Girish,
> > Thanks for your reply. About the question you mentioned, there is some
> > information I want to share with you.
> > >If anyone wants to setup different replication clusters then either
> > >those topics can be created under different namespaces or defined at
> topic
> > >level policy.
> >
> > >And users can anyway go and update the namespace's cluster list to add
> the
> > >missing cluster.
> > Because the replication clusters also mean the clusters where the topic
> can
> > be created or loaded, the topic-level replication clusters can only be
> the
> > subset of namespace-level replication clusters.
> > Just as Girish mentioned, the users can update the namespace's cluster
> list
> > to add the missing cluster.
> > But there is a problem because the replication clusters as the namespace
> > level will create a full mesh replication for that namespace across the
> > clusters defined in
> > replication-clusters if users want to change the replication policy for
> > topic-n and do not change the replication policy of other topics, they
> need
> > to change all the topic policy under this namespace.
> >
> > > Pulsar is being used by many legacy systems and changing its
> > >semantics for specific usecases without considering consequences are
> > >creating a lot of pain and incompatibility problems for other existing
> > >systems and let's avoid doing it as we are struggling with such changes
> > and
> > >breaking compatibility or changing semantics are just not acceptable.
> >
> > This proposal will not introduce an incompatibility problem, because the
> > default value of the namespace policy of allowed-clusters and
> > topic-policy-synchronized-clusters are the replication-clusters.
> >
> > >Allowed clusters defined at tenant level
> > >will restrict tenants to create namespaces in regions/clusters where
> they
> > >are not allowed.
> > >As Rajan also mentioned, allowed-clusters field has a different
> > meaning/purpose.
> >
> > Allowed clusters defined at the tenant level will restrict tenants from
> > creating namespaces in regions/clusters where they are not allowed.
> > Similarly, the allowed clusters defined at the namespace level will
> > restrict the namespace from creating topics in regions/clusters where
> they
> > are not allowed.
> > What's wrong with this?
> >
> > Regards,
> > Xiangying
> >
> > On Fri, Dec 1, 2023 at 2:35 PM Girish Sharma 
> > wrote:
> >
> > > Hi Xiangying,
> > >
> > > Shouldn't the solution to the issue mentioned in #21564 [0] mostly be
> > > around validating that topic level replication clusters are subset of
> > > namespace level replication clusters?
> > > It would be a completely compatible change as even today the case
> where a
> > > topic has a cluster not defined in namespaces's replication-clusters
> > > doesn't really work.
> > > And users can anyway go and update the namespace's cluster list to add
> > the
> > > missing cluster.
> > >
> > > As Rajan also mentioned, allowed-clusters field has a different
> > > meaning/purpose.
> > > Regards
> > >
> > > On Thu, Nov 30, 2023 at 10:56 AM Xiangying Meng 
> > > wrote:
> > >
> > > > Hi, Pulsar Community
> > > >
> > > > I drafted a proposal to make the configuration of clusters at the
> > > namespace
> > > > level clearer. This helps solve the problem of geo-replication not
> > > working
> > > > correctly at the topic level.
> > > >
> > > > https://github.com/apache/pulsar/pull/21648
> > > >
> > > > I'm looking forward to hearing from you.
> > > >
> > > > BR
> > > > Xiangying
> > > >
> > >
> > >
> > > --
> > > Girish Sharma
> > >
> >
>

Re: [DISCUSS] PIP-321 Split the responsibilities of namespace replication-clusters

[Xiangying Meng]

Hi Girish,

Thank you for your explanation. Because Joe's email referenced the current
implementation of Pulsar, I misunderstood him to be saying that this
current implementation is not good.

A possible use case is where there is one or a small number of topics in
the namespace that store important messages, which need to be replicated to
other clusters. Meanwhile, other topics only need to store data in the
local cluster.

For example, only topic1 needs replication, while topic2 to topic100 do
not. According to the current implementation, we need to set replication
clusters at the namespace level (e.g. cluster1 and cluster2), and then set
the topic-level replication clusters (cluster1) for topic2 to topic100 to
exclude them. It's hard to say that this is a good design.

Best regards.

On Wed, Dec 6, 2023 at 12:49 PM Joe F  wrote:

> Girish,
>
> Thank you for making my point much better than I did ..
>
> -Joe
>
> On Tue, Dec 5, 2023 at 1:45 AM Girish Sharma 
> wrote:
>
> > Hello Xiangying,
> >
> > I believe what Joe here is referring to as "application design" is not
> the
> > design of pulsar or namespace level replication but the design of your
> > application and the dependency that you've put on topic level
> replication.
> >
> > In general, I am aligned with Joe from an application design standpoint.
> A
> > namespace is supposed to represent a single application use case, topic
> > level override of replication clusters helps in cases where there are a
> few
> > exceptional topics which do not need replication in all of the namespace
> > clusters. This helps in saving network bandwidth, storage, CPU, RAM etc
> >
> > But the reason why you've raised this PIP is to bring down the actual
> > replication semantics at a topic level. Yes, namespace level still exists
> > as per your PIP as well, but is basically left only to be a "default in
> > case topic level is missing".
> > This brings me to a very basic question - What's the use case that you
> are
> > trying to solve that needs these changes? Because, then what's stopping
> us
> > from bringing every construct that's at a namespace level (bundling,
> > hardware affinity, etc) down to a topic level?
> >
> > Regards
> >
> > On Tue, Dec 5, 2023 at 2:52 PM Xiangying Meng 
> > wrote:
> >
> > > Hi Joe,
> > >
> > > You're correct. The initial design of the replication policy leaves
> room
> > > for improvement. To address this, we aim to refine the cluster settings
> > at
> > > the namespace level in a way that won't impact the existing system. The
> > > replication clusters should solely be used to establish full mesh
> > > replication for that specific namespace, without having any other
> > > definitions or functionalities.
> > >
> > > BR,
> > > Xiangying
> > >
> > >
> > > On Mon, Dec 4, 2023 at 1:52 PM Joe F  wrote:
> > >
> > > > >if users want to change the replication policy for
> > > > topic-n and do not change the replication policy of other topics,
> they
> > > need
> > > > to change all the topic policy under this namespace.
> > > >
> > > > This PIP unfortunately  flows from  attempting to solve bad
> application
> > > > design
> > > >
> > > > A namespace is supposed to represent an application, and the
> namespace
> > > > policy is an umbrella for a similar set of policies  that applies to
> > all
> > > > topics.  The exceptions would be if a topic had a need for a deficit,
> > The
> > > > case of one topic in the namespace sticking out of the namespace
> policy
> > > > umbrella is bad  application design in my opinion
> > > >
> > > > -Joe.
> > > >
> > > >
> > > >
> > > > On Sun, Dec 3, 2023 at 6:00 PM Xiangying Meng 
> > > > wrote:
> > > >
> > > > > Hi Rajan and Girish,
> > > > > Thanks for your reply. About the question you mentioned, there is
> > some
> > > > > information I want to share with you.
> > > > > >If anyone wants to setup different replication clusters then
> either
> > > > > >those topics can be created under different namespaces or defined
> at
> > > > topic
> > > > > >level policy.
> > > > >
> > > > > >And users can anyway go and update the namespace's cluster list to
> > add
> > &

Re: [DISCUSS] PIP-321 Split the responsibilities of namespace replication-clusters

[Xiangying Meng]

Hi Girish,

>But the reason why you've raised this PIP is to bring down the actual
replication semantics at a topic level. Yes, namespace level still exists
as per your PIP as well, but is basically left only to be a "default in
case topic level is missing".

I'm afraid there's some misunderstanding here. According to the Pulsar
website, replication can actually be enabled at the topic level.

>You can enable geo-replication at namespace or topic level. [1]

So, it's not this proposal that introduces topic-level replication
semantics. Prior to this, topic-level replication was constrained by the
namespace-level replication policy. There are just some problems here. If
replication was not configured at the namespace level, then topic-level
replication would also be ineffective. Moreover, users would not be aware
of this replication failure.

> Yes, namespace level still exists as per your PIP as well, but is
basically left only to be a "default in case topic level is missing".

This behavior is consistent with the current behavior of Pulsar and is not
something introduced by this proposal.
This proposal introduces an `allowed-cluster` configuration at the
namespace level.
As the website states, you can enable replication at either the namespace
or topic level.
But If you only enable replication at the topic level, the replication
configuration would not take effect prior to this proposal.

Before this proposal: even though the topic policy can be updated
successfully, topic1 cannot be created in cluster2.
```
Namespace policy {replication clusters -> local cluster(cluster1)}, topic1
policy {replication clusters {cluster1, cluster2}}
```
After this proposal: you can set allowed clusters at the namespace level,
which specifies the clusters where topics under this namespace are allowed.
Then, the topic-level replication would also be effective, as described on
the Pulsar website.
```
Namespace policy {replication clusters -> local cluster(cluster1), allowed
clusters -> {cluster1, cluster2, cluster3}}, topic1 policy {replication
clusters {cluster1, cluster2}}
```

[1]
https://pulsar.apache.org/docs/3.1.x/administration-geo/#enable-geo-replication


On Wed, Dec 6, 2023 at 7:57 PM Xiangying Meng  wrote:

> Hi Girish,
>
> Thank you for your explanation. Because Joe's email referenced the current
> implementation of Pulsar, I misunderstood him to be saying that this
> current implementation is not good.
>
> A possible use case is where there is one or a small number of topics in
> the namespace that store important messages, which need to be replicated to
> other clusters. Meanwhile, other topics only need to store data in the
> local cluster.
>
> For example, only topic1 needs replication, while topic2 to topic100 do
> not. According to the current implementation, we need to set replication
> clusters at the namespace level (e.g. cluster1 and cluster2), and then set
> the topic-level replication clusters (cluster1) for topic2 to topic100 to
> exclude them. It's hard to say that this is a good design.
>
> Best regards.
>
> On Wed, Dec 6, 2023 at 12:49 PM Joe F  wrote:
>
>> Girish,
>>
>> Thank you for making my point much better than I did ..
>>
>> -Joe
>>
>> On Tue, Dec 5, 2023 at 1:45 AM Girish Sharma 
>> wrote:
>>
>> > Hello Xiangying,
>> >
>> > I believe what Joe here is referring to as "application design" is not
>> the
>> > design of pulsar or namespace level replication but the design of your
>> > application and the dependency that you've put on topic level
>> replication.
>> >
>> > In general, I am aligned with Joe from an application design
>> standpoint. A
>> > namespace is supposed to represent a single application use case, topic
>> > level override of replication clusters helps in cases where there are a
>> few
>> > exceptional topics which do not need replication in all of the namespace
>> > clusters. This helps in saving network bandwidth, storage, CPU, RAM etc
>> >
>> > But the reason why you've raised this PIP is to bring down the actual
>> > replication semantics at a topic level. Yes, namespace level still
>> exists
>> > as per your PIP as well, but is basically left only to be a "default in
>> > case topic level is missing".
>> > This brings me to a very basic question - What's the use case that you
>> are
>> > trying to solve that needs these changes? Because, then what's stopping
>> us
>> > from bringing every construct that's at a namespace level (bundling,
>> > hardware affinity, etc) down to a topic level?
>> >
>> > Regards

Re: [DISCUSS] PIP-321 Split the responsibilities of namespace replication-clusters

[Xiangying Meng]

Hi Girish,

What you are actually opposing is the implementation of true topic-level
geo-replication. You believe that topics should be divided into different
namespaces based on replication. Following this line of thought, what we
should do is restrict the current topic-level replication settings, not
allowing the replication clusters set at the topic level to exceed the
range of replication clusters set in the namespace.

One point that confuses me is that we provide a setting for topic-level
replication clusters, but it can only be used to amend the namespace
settings and cannot work independently. Isn't this also a poor design for
Pulsar?

On Thu, Dec 7, 2023 at 2:28 AM Girish Sharma 
wrote:

> Hello, replies inline.
>
> On Wed, Dec 6, 2023 at 5:28 PM Xiangying Meng 
> wrote:
>
> > Hi Girish,
> >
> > Thank you for your explanation. Because Joe's email referenced the
> current
> > implementation of Pulsar, I misunderstood him to be saying that this
> > current implementation is not good.
> >
> > A possible use case is where there is one or a small number of topics in
> > the namespace that store important messages, which need to be replicated
> to
> > other clusters. Meanwhile, other topics only need to store data in the
> > local cluster.
> >
>
> Is it not possible to simply have the other topics in a namespace which
> allows for that other cluster, and the local topics remain in the namespace
> with local cluster needs. Seems to me like a proper use case of two
> different namespaces as the use case is different in both cases.
>
>
>
> >
> > For example, only topic1 needs replication, while topic2 to topic100 do
> > not. According to the current implementation, we need to set replication
> > clusters at the namespace level (e.g. cluster1 and cluster2), and then
> set
> > the topic-level replication clusters (cluster1) for topic2 to topic100 to
> > exclude them. It's hard to say that this is a good design.
> >
>
> No, all you need is to put topic1 in namespace1 and topic2 to topic100 in
> namespace2 . This is exactly what me and Joe were saying is a bad design
> choice that you are clubbing all 100 topics in same namespace.
>
>
>
> >
> > Best regards.
> >
> > On Wed, Dec 6, 2023 at 12:49 PM Joe F  wrote:
> >
> > > Girish,
> > >
> > > Thank you for making my point much better than I did ..
> > >
> > > -Joe
> > >
> > > On Tue, Dec 5, 2023 at 1:45 AM Girish Sharma 
> > > wrote:
> > >
> > > > Hello Xiangying,
> > > >
> > > > I believe what Joe here is referring to as "application design" is
> not
> > > the
> > > > design of pulsar or namespace level replication but the design of
> your
> > > > application and the dependency that you've put on topic level
> > > replication.
> > > >
> > > > In general, I am aligned with Joe from an application design
> > standpoint.
> > > A
> > > > namespace is supposed to represent a single application use case,
> topic
> > > > level override of replication clusters helps in cases where there
> are a
> > > few
> > > > exceptional topics which do not need replication in all of the
> > namespace
> > > > clusters. This helps in saving network bandwidth, storage, CPU, RAM
> etc
> > > >
> > > > But the reason why you've raised this PIP is to bring down the actual
> > > > replication semantics at a topic level. Yes, namespace level still
> > exists
> > > > as per your PIP as well, but is basically left only to be a "default
> in
> > > > case topic level is missing".
> > > > This brings me to a very basic question - What's the use case that
> you
> > > are
> > > > trying to solve that needs these changes? Because, then what's
> stopping
> > > us
> > > > from bringing every construct that's at a namespace level (bundling,
> > > > hardware affinity, etc) down to a topic level?
> > > >
> > > > Regards
> > > >
> > > > On Tue, Dec 5, 2023 at 2:52 PM Xiangying Meng 
> > > > wrote:
> > > >
> > > > > Hi Joe,
> > > > >
> > > > > You're correct. The initial design of the replication policy leaves
> > > room
> > > > > for improvement. To address this, we aim to refine the cluster
> > settings
> > > > at
> > > > > the namespace level in a way tha

Re: [DISCUSS] PIP-321 Split the responsibilities of namespace replication-clusters

[Xiangying Meng]

Hi Girish,

I'm very pleased that we have reached some consensus now. Pulsar already
supports geo-replication at the topic level, but the existing
implementation of this topic level replication does not match our
expectations.

At the moment, I can think of three directions to solve this problem:

1. Treat this issue as a bug and fix it so that Pulsar can truly support
replication at the topic level.
2. Limit the replication topic policy, so that the replication clusters at
the topic level must be included in the replication clusters configured at
the namespace level. In this case, the topic level replication would serve
as a supplement to the namespace replication, rather than a true topic
level policy.
3. Remove topic level replication.

I lean towards the first option, as it would make Pulsar's replication
configuration more flexible and would not break the previous behavior logic.

>Yes, that's my viewpoint. In case that's not your view point, then in your
>use cases do you ever have more than one namespace inside a tenant?
>With every property coming at topic level, it makes no sense for the
>namespace hierarchy to exist anymore.

I didn't propose this from the perspective of a user, but from the
perspective of a Pulsar maintainer. The replication cluster at the topic
level cannot function independently like other topic policies, and I
attempted to fix it after finding the reason.

>From the user's perspective, I could modify my system to put topics with
the same replication strategy under the same namespace. From the
maintainer's perspective, if a feature can help users use Pulsar more
flexibly and conveniently without introducing risks, then this feature
should be implemented. Perhaps business systems do not want to maintain too
many namespaces, as they would need to configure multiple namespace
policies or it might make their business logic complex. The other
configurations for topics under this namespace might be consistent, with
only a few topics needing to enable replication. In this case, topic level
replication becomes valuable. Therefore, I lean towards the first option,
to solve this problem and make it a truly expected topic policy.

On Thu, Dec 7, 2023 at 12:45 PM Girish Sharma 
wrote:

> Hello Xiangying,
>
>
> On Thu, Dec 7, 2023 at 6:32 AM Xiangying Meng 
> wrote:
>
> > Hi Girish,
> >
> > What you are actually opposing is the implementation of true topic-level
> > geo-replication. You believe that topics should be divided into different
> > namespaces based on replication. Following this line of thought, what we
> > should do is restrict the current topic-level replication settings, not
> > allowing the replication clusters set at the topic level to exceed the
> > range of replication clusters set in the namespace.
> >
>
> Yes, that's my viewpoint. In case that's not your view point, then in your
> use cases do you ever have more than one namespace inside a tenant?
> With every property coming at topic level, it makes no sense for the
> namespace hierarchy to exist anymore.
>
>
> >
> > One point that confuses me is that we provide a setting for topic-level
> > replication clusters, but it can only be used to amend the namespace
> > settings and cannot work independently. Isn't this also a poor design for
> > Pulsar?
> >
>
> This feature was originally added in pulsar without a PIP. And the PR [0]
> also doesn't have much context around why it was needed and why it is being
> added.. So I can't comment on why this was added..
> But my understanding is that even in a situation when the topics are
> divided into proper namespaces based on use cases and suddenly there is an
> exceptional need for one of the existing topics to have lesser replication,
> then instead of following a long exercise of moving that topic to a new
> namespace, you can use this feature.
>
> [0] - https://github.com/apache/pulsar/pull/12136
>
>
>
> >
> > On Thu, Dec 7, 2023 at 2:28 AM Girish Sharma 
> > wrote:
> >
> > > Hello, replies inline.
> > >
> > > On Wed, Dec 6, 2023 at 5:28 PM Xiangying Meng 
> > > wrote:
> > >
> > > > Hi Girish,
> > > >
> > > > Thank you for your explanation. Because Joe's email referenced the
> > > current
> > > > implementation of Pulsar, I misunderstood him to be saying that this
> > > > current implementation is not good.
> > > >
> > > > A possible use case is where there is one or a small number of topics
> > in
> > > > the namespace that store important messages, which need to be
> > replicated
> > > to
> > > > other clusters. Meanwhile

Re: [DISCUSS] PIP-321 Split the responsibilities of namespace replication-clusters

[Xiangying Meng]

Hi Jiwei

Great advice. Thanks for your suggestions and additions.

Thanks,
Xiangying

On Fri, Dec 15, 2023 at 9:41 AM guo jiwei  wrote:

> Hi Xiangying,
>I think  we can rename this PIP to:   *Introduce `allowed-clusters` and
> `topic-policy-synchronized-clusters` to fully support replication on
> message and topic level*
>Currently, we can set replication clusters on the message and topic
> level, but the replication clusters should be a subset of the namespace
> replication clusters. which means :
>If we set namespace replication clusters: cluster1, cluster2, cluster3,
> at most, these three or two clusters can be set on message or topic. If the
> user wanna set cluster4 or others, the replication
>can't work as expected.
>It's easy to reproduce by this test:
>
> >@Test
>
> public void testEnableReplicationInTopicLevel() throws Exception {
>
> // 1. Create namespace and topic
>
> String namespace =
> > BrokerTestUtil.newUniqueName("pulsar/testEnableReplicationInTopicLevel");
>
> String topic1 = "persistent://" + namespace + "/topic-1";
>
> admin1.namespaces().createNamespace(namespace);
>
> admin1.topics().createNonPartitionedTopic(topic1);
>
> // 2. Configure replication clusters for the topic.
>
> admin1.topics().setReplicationClusters(topic1, List.of("r1",
> "r2"));
>
> // 3. Check if the replicator connected successfully.
>
> Awaitility.await().atMost(5, TimeUnit.MINUTES).untilAsserted(() ->
> {
>
> List keys = pulsar1.getBrokerService()
>
> .getTopic(topic1, false).get().get()
>
> .getReplicators().keys();
>
> assertEquals(keys.size(), 1);
>
> assertTrue(pulsar1.getBrokerService()
>
> .getTopic(topic1, false).get().get()
>
> .getReplicators().get(keys.get(0)).isConnected());
>
> });
>
> }
>
>
>   To fully support the replication, we find out an easy way to solve it.
> Introduce `allowed-clusters` on namespace policies, which Xiangying
> explains above.
>   How could this work and solve the issue? The same example :
>   If we set namespace replication clusters: cluster1, cluster2, cluster3,
> and
>set topic1 replication clusters: cluster2, cluster4.
>set topic2 replication clusters: cluster1, cluster4.
>   We must set `allowed-clusters` with cluster1, cluster2, cluster3, and
> cluster4.  The broker side will validate the topic or message replication
> clusters from the `allowed-cluster.`
>   In this way,  we can simplify more codes and logic here.
>   For *`topic-policy-synchronized-clusters` *we also add examples in the
> PIP.
>
>   Hope the explanation could help @Rajan @Girish
>
>
>
>
> Regards
> Jiwei Guo (Tboy)
>
>
> On Thu, Dec 7, 2023 at 10:29 PM Xiangying Meng 
> wrote:
>
> > Hi Girish,
> >
> > I'm very pleased that we have reached some consensus now. Pulsar already
> > supports geo-replication at the topic level, but the existing
> > implementation of this topic level replication does not match our
> > expectations.
> >
> > At the moment, I can think of three directions to solve this problem:
> >
> > 1. Treat this issue as a bug and fix it so that Pulsar can truly support
> > replication at the topic level.
> > 2. Limit the replication topic policy, so that the replication clusters
> at
> > the topic level must be included in the replication clusters configured
> at
> > the namespace level. In this case, the topic level replication would
> serve
> > as a supplement to the namespace replication, rather than a true topic
> > level policy.
> > 3. Remove topic level replication.
> >
> > I lean towards the first option, as it would make Pulsar's replication
> > configuration more flexible and would not break the previous behavior
> > logic.
> >
> > >Yes, that's my viewpoint. In case that's not your view point, then in
> your
> > >use cases do you ever have more than one namespace inside a tenant?
> > >With every property coming at topic level, it makes no sense for the
> > >namespace hierarchy to exist anymore.
> >
> > I didn't propose this from the perspective of a user, but from the
> > perspective of a Pulsar maintainer. The replication cluster at the topic
> > level cannot function independently like other topic policies, and I
> > attempted to fix it after finding the reason.
> >
> &g

Re: [DISCUSS] PIP-321 Split the responsibilities of namespace replication-clusters

[Xiangying Meng]

Hi Penghui

>I'm sorry, I don't fully understand your point here. What is the "support
replication on message and topic level"?

>As I understand, are the `allowed-clusters` and `replication-clusters` more
concise options?

Pulsar support set replication-cluster for per message.
After this proposal, the `replication-clusters` at the topic/namespace
level could be the default value when the message level is not set, and
`allowed-clusters` could be the constraint when setting
`replication-cluster` for the message level.

I am struggling with whether we should introduce message level into the
title of this proposal.
Do you have any suggestions for the title of the proposal?

BR,
Xiangying

Re: [DISSCUSS] PIP-325: Add command to abort transaction

[Xiangying Meng]

Hi, Ruihong

Thanks for your proposal.
I wonder whether we should abort all the transactions one client creates
when the client crushes.
For example, a process builds a Pulsar client and creates a transaction by
this client to do some operations.
If the process crushes, the transaction cannot be committed or aborted.

In this case, aborting the transaction automatically when the client is
crushed may be more helpful than aborting them after restarting.

BR,
Xiangying


On Mon, Dec 18, 2023 at 9:08 AM PengHui Li  wrote:

> Hi, Ruihong
>
> The proposal looks good to me.
> Just left a comment about the security considerations.
> We need to have a clear permission definition for newly added admin API
>
> Regards,
> Penghui
>
> On Sun, Dec 17, 2023 at 1:14 AM |海阔天高 <1373544...@qq.com.invalid> wrote:
>
> > Hi community,
> >
> >
> > PIP-325 introduces a new API for aborting transactions, allowing
> > administrators to proactively abort a transaction when it gets stuck,
> thus
> > preventing consumers from being blocked for an extended period.
> >
> >
> > Hopes for discuss.
> > PIP: https://github.com/apache/pulsar/pull/21731
> > Releted PR: https://github.com/apache/pulsar/pull/21630
> >
> >
> > Thanks,
> >
> >
> > Ruihong
> >
> >
> > |海阔天高
> > 1373544...@qq.com
> >
> >
> >
> >  
>

Re: [DISSCUSS] PIP-325: Add command to abort transaction

[Xiangying Meng]

Hi, Ruihong

This proposal looks good to me.

BR,
Xiangying

On Tue, Dec 19, 2023 at 8:13 PM Xiangying Meng  wrote:

> Hi, Ruihong
>
> Thanks for your proposal.
> I wonder whether we should abort all the transactions one client creates
> when the client crushes.
> For example, a process builds a Pulsar client and creates a transaction by
> this client to do some operations.
> If the process crushes, the transaction cannot be committed or aborted.
>
> In this case, aborting the transaction automatically when the client is
> crushed may be more helpful than aborting them after restarting.
>
> BR,
> Xiangying
>
>
> On Mon, Dec 18, 2023 at 9:08 AM PengHui Li  wrote:
>
>> Hi, Ruihong
>>
>> The proposal looks good to me.
>> Just left a comment about the security considerations.
>> We need to have a clear permission definition for newly added admin API
>>
>> Regards,
>> Penghui
>>
>> On Sun, Dec 17, 2023 at 1:14 AM |海阔天高 <1373544...@qq.com.invalid> wrote:
>>
>> > Hi community,
>> >
>> >
>> > PIP-325 introduces a new API for aborting transactions, allowing
>> > administrators to proactively abort a transaction when it gets stuck,
>> thus
>> > preventing consumers from being blocked for an extended period.
>> >
>> >
>> > Hopes for discuss.
>> > PIP: https://github.com/apache/pulsar/pull/21731
>> > Releted PR: https://github.com/apache/pulsar/pull/21630
>> >
>> >
>> > Thanks,
>> >
>> >
>> > Ruihong
>> >
>> >
>> > |海阔天高
>> > 1373544...@qq.com
>> >
>> >
>> >
>> >  
>>
>

[VOTE]PIP-321: Split the responsibilities of namespace replication-clusters

[Xiangying Meng]

Dear Pulsar community,

I am initiating a voting thread for "PIP-321: Split the responsibilities of
namespace replication-clusters".

Here is the pull request for PIP-321:
https://github.com/apache/pulsar/pull/21648

And the discussion thread:
https://lists.apache.org/thread/87qfp8ht5s0fvw2y4t3j9yzgfmdzmcnz

Highlight:
This proposal introduces a new feature for topics, allowing a topic to be
loaded across different clusters without data replication between these
clusters. It also introduces more granular control over `allowed_clusters`
at the namespace level.

Please review the PIP document and cast your vote!

Thank you,

Xiangying

Re: [VOTE]PIP-321: Split the responsibilities of namespace replication-clusters

[Xiangying Meng]

Close this vote with three bindings.

- Jiwei Guo (Tboy)
- Penghui
- Yubiao Feng

On Fri, Jan 19, 2024 at 6:31 PM Yubiao Feng
 wrote:

> +1 (binding)
>
> Thanks
> Yubiao Feng
>
> On Tue, Jan 16, 2024 at 2:41 PM Xiangying Meng 
> wrote:
>
> > Dear Pulsar community,
> >
> > I am initiating a voting thread for "PIP-321: Split the responsibilities
> of
> > namespace replication-clusters".
> >
> > Here is the pull request for PIP-321:
> > https://github.com/apache/pulsar/pull/21648
> >
> > And the discussion thread:
> > https://lists.apache.org/thread/87qfp8ht5s0fvw2y4t3j9yzgfmdzmcnz
> >
> > Highlight:
> > This proposal introduces a new feature for topics, allowing a topic to be
> > loaded across different clusters without data replication between these
> > clusters. It also introduces more granular control over
> `allowed_clusters`
> > at the namespace level.
> >
> > Please review the PIP document and cast your vote!
> >
> > Thank you,
> >
> > Xiangying
> >
>

Re: [DISCUSS] PIP-335: Oxia metadata support

[Xiangying Meng]

+1

Thanks,
Xiangying

On Thu, Feb 1, 2024 at 11:20 AM Yubiao Feng
 wrote:

> +1
>
> Thanks
> Yubiao Feng
>
> On Thu, Feb 1, 2024 at 7:58 AM Matteo Merli 
> wrote:
>
> > https://github.com/apache/pulsar/pull/22009
> >
> > ===
> >
> > # PIP-335: Supporty Oxia metadata store plugin
> >
> > # Motivation
> >
> > Oxia is a scalable metadata store and coordination system that can be
> used
> > as the core infrastructure
> > to build large scale distributed systems.
> >
> > Oxia was created with the primary goal of providing an alternative Pulsar
> > to replace ZooKeeper as the
> > long term preferred metadata store, overcoming all the current
> limitations
> > in terms of metadata
> > access throughput and data set size.
> >
> > # Goals
> >
> > Add a Pulsar MetadataStore plugin that uses Oxia client SDK.
> >
> > Users will be able to start a Pulsar cluster using just Oxia, without any
> > ZooKeeper involved.
> >
> > ## Not in Scope
> >
> > It's not in the scope of this proposal to change any default behavior or
> > configuration of Pulsar.
> >
> > # Detailed Design
> >
> > ## Design & Implementation Details
> >
> > Oxia semantics and client SDK were already designed with Pulsar and
> > MetadataStore plugin API in mind, so
> > there is not much integration work that needs to be done here.
> >
> > Just a few notes:
> >  1. Oxia client already provides support for transparent batching of read
> > and write operations,
> > so there will be no use of the batching logic in
> > `AbstractBatchedMetadataStore`
> >  2. Oxia does not treat keys as a walkable file-system like interface,
> with
> > directories and files. Instead
> > all the keys are independent. Though Oxia sorting of keys is aware of
> > '/' and provides efficient key
> > range scanning operations to identify the first level children of a
> > given key
> >  3. Oxia, unlike ZooKeeper, doesn't require the parent path of a key to
> > exist. eg: we can create `/a/b/c` key
> > without `/a/b` and `/a` existing.
> > In the Pulsar integration for Oxia we're forcing to create all parent
> > keys when they are not there. This
> > is due to several places in BookKeeper access where it does not
> create
> > the parent keys, though it will
> > later make `getChildren()` operations on the parents.
> >
> > ## Other notes
> >
> > Unlike in the ZooKeeper implementation, the notification of events is
> > guaranteed in Oxia, because the Oxia
> > client SDK will use the transaction offset after server reconnections and
> > session restarted events. This
> > will ensure that brokers cache will always be properly invalidated. We
> will
> > then be able to remove the
> > current 5minutes automatic cache refresh which is in place to prevent the
> > ZooKeeper missed watch issue.
> >
> > # Links
> >
> > Oxia: https://github.com/streamnative/oxia
> > Oxia Java Client SDK: https://github.com/streamnative/oxia-java
> >
> >
> > --
> > Matteo Merli
> > 
> >
>

[DISCUSS] Deletion of Current Ledger upon Rollover

[Xiangying Meng]

Dear Community,

I hope this message finds you well. I am writing to discuss a modification
to the behavior of deleting the current ledger. As you may know, in Pulsar,
the current ledger cannot be deleted because it may still be written to.
However, there is an exception. When the current ledger is rolled over, but
no new messages are written, the current ledger does not change. In this
case, the current ledger will not be written to, but it is also not deleted.

This can be confusing for users, especially when they configure
`managedLedgerMaxLedgerRolloverTimeMinutes` and `retentionTimeInMinutes`.
They expect the current ledger to roll over and then be deleted after
`managedLedgerMaxLedgerRolloverTimeMinutes` and `retentionTimeInMinutes`.
However, in reality, while the current ledger does rollover, it is not
deleted.

The purpose of this discussion is to consider deleting the current ledger
when it is rolled over. The specific implementation can be found at
https://github.com/apache/pulsar/pull/22034.

Looking forward to a productive discussion.

Best Regards,

xiangying

Re: [DISCUSS] Deletion of Current Ledger upon Rollover

[Xiangying Meng]

>The problem is that if the ledger is deleted, the next time a client
produces a message, a new ledger needs to be opened. This is an operation
that may take some time, disrupting latency.

Hi Enrico, I think there may be some misunderstandings. If the state of the
managerLedger is `ClosedLedger`, the current ledger is closed and cannot be
written to. So when a client sends new messages, it must open a new ledger.
This will cause latency regardless of whether the current ledger is
deleted, since the current ledger is already closed.

In the discussion of the PR[1], we found that there are different behaviors
in Pulsar after ledger rollover.
1. If the rollover is triggered in the process of adding an entry, a new
ledger will be created depending on whether there are pending write
operations.
2. In other cases, a new ledger will be created directly regardless of
whether there are pending write operations.

In the latest implementation of the PR, for the first situation, a new
ledger will be created directly even if there are no pending write
operations. Of course, this will increase the overhead of the bookkeeper.
But for normal production and consumption scenarios, there should be no
situation where the new ledger is not written to for a long time. Even for
inactive topics, users can handle this situation according to the
inactive_topic_policies.

[1] - https://github.com/apache/pulsar/pull/22034

On Tue, Feb 6, 2024 at 9:55 PM Enrico Olivelli  wrote:

> Xiangying
>
> Il giorno mar 6 feb 2024 alle ore 13:01 Xiangying Meng
>  ha scritto:
> >
> > Dear Community,
> >
> > I hope this message finds you well. I am writing to discuss a
> modification
> > to the behavior of deleting the current ledger. As you may know, in
> Pulsar,
> > the current ledger cannot be deleted because it may still be written to.
> > However, there is an exception. When the current ledger is rolled over,
> but
> > no new messages are written, the current ledger does not change. In this
> > case, the current ledger will not be written to, but it is also not
> deleted.
>
> I understand the problem and I was surprised about it the first time I saw
> it.
> The problem is that if you delete the ledger the next time a client
> produces a message the broken
> must open a new ledger and this is an operation that may take some
> time, disrupting latency.
>
> It is a trade-off, I know, in production you usually don't need to
> release the disk space.
> Maybe you have a use case in which you write a lot to a topic, then
> you stop writing ?
>
> Maybe you could "unload" the topic, and that will force a ledger
> rollover (with an impact on latency)
>
> Enrico
>
> >
> > This can be confusing for users, especially when they configure
> > `managedLedgerMaxLedgerRolloverTimeMinutes` and `retentionTimeInMinutes`.
> > They expect the current ledger to roll over and then be deleted after
> > `managedLedgerMaxLedgerRolloverTimeMinutes` and `retentionTimeInMinutes`.
> > However, in reality, while the current ledger does rollover, it is not
> > deleted.
> >
> > The purpose of this discussion is to consider deleting the current ledger
> > when it is rolled over. The specific implementation can be found at
> > https://github.com/apache/pulsar/pull/22034.
> >
> > Looking forward to a productive discussion.
> >
> > Best Regards,
> >
> > xiangying
>

[VOTE] Pulsar Release 2.10.6 Candidate 1

[Xiangying Meng]

This is the first release candidate for Apache Pulsar, version 2.10.6.

It fixes the following issues:
https://github.com/apache/pulsar/pulls?q=is:pr+label:cherry-picked/branch-2.10+label:release/2.10.6+is:closed

*** Please download, test and vote on this release. This vote will stay open
for at least 72 hours ***

Note that we are voting upon the source (tag), binaries are provided for
convenience.

Source and binary files:
https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.10.6-candidate-1/

SHA-512 checksums:
09f29265f8173331d4c05b470c4e77a31146172b27ef333f45d8c8a19074ef25061cb1e80872fc45c323c9ce8e2e17989c6df5d991ef84c4d245197303d9e6d7
 apache-pulsar-2.10.6-bin.tar.gz
49c8836882818c6f38748dae26b51c598f163606c16993a3287ab1ce9f853a4aaa43c6729c1b6f6957738b4dead3818cd12026da68b328eb2d4ac0d0214957bb
 apache-pulsar-2.10.6-src.tar.gz

Maven staging repo:
https://repository.apache.org/content/repositories/orgapachepulsar-1270

The tag to be voted upon:
v2.10.6-candidate-1 (9c29b76ff2be865429ad44df8683aec80deacfba)
https://github.com/apache/pulsar/releases/tag/v2.10.6-candidate-1

Pulsar's KEYS file containing PGP keys you use to sign the release:
https://downloads.apache.org/pulsar/KEYS

Docker images:


https://hub.docker.com/layers/xiangyingmeng/pulsar/2.10.6/images/sha256-bf8f36e49ff44ef810ab2c76742121205e51d3a04c79afdb5d288c7d8a06443f?context=repo


https://hub.docker.com/layers/xiangyingmeng/pulsar-all/2.10.6/images/sha256-1b3a10db12f6d5a0acd2d4ed73eb11864b6b598294bb905b6ede34aef1157f23?context=repo

Please download the source package, and follow the README to build
and run the Pulsar standalone service.

Re: [VOTE] Pulsar Release 2.10.6 Candidate 1

[Xiangying Meng]

Dear Zixuan,

Thank you for your email and your ongoing commitment to the Pulsar project.

I wanted to clarify that this release, 2.10.6, is a special case. It
was primarily focused on addressing certain security issues that were
deemed critical. This decision was made following internal discussions
within the PMC.

I completely understand and respect the release policy defined by
Pulsar [0]. Under normal circumstances, we would indeed follow the
policy and consider version 2.10 as EOL, ceasing further maintenance.

However, given the exceptional nature of this release and the
importance of the security issues it addresses, we felt it was
necessary to make an exception in this case.

Thank you for your understanding and for bringing this to our
attention. We appreciate your diligence in adhering to Pulsar's
release policy.

Best regards,

Xiangying

On Wed, Mar 6, 2024 at 4:22 PM Zixuan Liu  wrote:
>
> Thank you for releasing 2.10.6.
>
> According to the release policy defined [0] by Pulsar, this version is EOL and
> does not require further maintenance.
>
> If we need to continue to maintain the 2.10, we must discuss the
> maintenance lifecycle of the 2.10, and update our doc.
>
> - [0] https://pulsar.apache.org/contribute/release-policy/
>
> Thanks,
> Zixuan
>
>
> Xiangying Meng  于2024年3月6日周三 11:15写道：
>
> > This is the first release candidate for Apache Pulsar, version 2.10.6.
> >
> > It fixes the following issues:
> >
> > https://github.com/apache/pulsar/pulls?q=is:pr+label:cherry-picked/branch-2.10+label:release/2.10.6+is:closed
> >
> > *** Please download, test and vote on this release. This vote will stay
> > open
> > for at least 72 hours ***
> >
> > Note that we are voting upon the source (tag), binaries are provided for
> > convenience.
> >
> > Source and binary files:
> > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.10.6-candidate-1/
> >
> > SHA-512 checksums:
> >
> > 09f29265f8173331d4c05b470c4e77a31146172b27ef333f45d8c8a19074ef25061cb1e80872fc45c323c9ce8e2e17989c6df5d991ef84c4d245197303d9e6d7
> >  apache-pulsar-2.10.6-bin.tar.gz
> >
> > 49c8836882818c6f38748dae26b51c598f163606c16993a3287ab1ce9f853a4aaa43c6729c1b6f6957738b4dead3818cd12026da68b328eb2d4ac0d0214957bb
> >  apache-pulsar-2.10.6-src.tar.gz
> >
> > Maven staging repo:
> > https://repository.apache.org/content/repositories/orgapachepulsar-1270
> >
> > The tag to be voted upon:
> > v2.10.6-candidate-1 (9c29b76ff2be865429ad44df8683aec80deacfba)
> > https://github.com/apache/pulsar/releases/tag/v2.10.6-candidate-1
> >
> > Pulsar's KEYS file containing PGP keys you use to sign the release:
> > https://downloads.apache.org/pulsar/KEYS
> >
> > Docker images:
> >
> > 
> >
> > https://hub.docker.com/layers/xiangyingmeng/pulsar/2.10.6/images/sha256-bf8f36e49ff44ef810ab2c76742121205e51d3a04c79afdb5d288c7d8a06443f?context=repo
> >
> > 
> >
> > https://hub.docker.com/layers/xiangyingmeng/pulsar-all/2.10.6/images/sha256-1b3a10db12f6d5a0acd2d4ed73eb11864b6b598294bb905b6ede34aef1157f23?context=repo
> >
> > Please download the source package, and follow the README to build
> > and run the Pulsar standalone service.
> >

Re: [VOTE] Pulsar Release 2.10.6 Candidate 1

[Xiangying Meng]

This right.
Maybe we need some discussion and update our release policy.

Thanks
Xiangying

On Wed, Mar 6, 2024 at 5:09 PM Zixuan Liu  wrote:
>
> It sounds like the version release was triggered due to security issues. If
> so, I think we need to update our release policy.
>
> Only when a fatal security issue occurs can we trigger a release of a new
> version, but we also need to clarify the maintenance cycle, otherwise this
> maintenance is endless.
>
> Thanks,
> Zixuan
>
> Xiangying Meng  于2024年3月6日周三 16:45写道：
>
> > Dear Zixuan,
> >
> > Thank you for your email and your ongoing commitment to the Pulsar project.
> >
> > I wanted to clarify that this release, 2.10.6, is a special case. It
> > was primarily focused on addressing certain security issues that were
> > deemed critical. This decision was made following internal discussions
> > within the PMC.
> >
> > I completely understand and respect the release policy defined by
> > Pulsar [0]. Under normal circumstances, we would indeed follow the
> > policy and consider version 2.10 as EOL, ceasing further maintenance.
> >
> > However, given the exceptional nature of this release and the
> > importance of the security issues it addresses, we felt it was
> > necessary to make an exception in this case.
> >
> > Thank you for your understanding and for bringing this to our
> > attention. We appreciate your diligence in adhering to Pulsar's
> > release policy.
> >
> > Best regards,
> >
> > Xiangying
> >
> > On Wed, Mar 6, 2024 at 4:22 PM Zixuan Liu  wrote:
> > >
> > > Thank you for releasing 2.10.6.
> > >
> > > According to the release policy defined [0] by Pulsar, this version is
> > EOL and
> > > does not require further maintenance.
> > >
> > > If we need to continue to maintain the 2.10, we must discuss the
> > > maintenance lifecycle of the 2.10, and update our doc.
> > >
> > > - [0] https://pulsar.apache.org/contribute/release-policy/
> > >
> > > Thanks,
> > > Zixuan
> > >
> > >
> > > Xiangying Meng  于2024年3月6日周三 11:15写道：
> > >
> > > > This is the first release candidate for Apache Pulsar, version 2.10.6.
> > > >
> > > > It fixes the following issues:
> > > >
> > > >
> > https://github.com/apache/pulsar/pulls?q=is:pr+label:cherry-picked/branch-2.10+label:release/2.10.6+is:closed
> > > >
> > > > *** Please download, test and vote on this release. This vote will stay
> > > > open
> > > > for at least 72 hours ***
> > > >
> > > > Note that we are voting upon the source (tag), binaries are provided
> > for
> > > > convenience.
> > > >
> > > > Source and binary files:
> > > >
> > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.10.6-candidate-1/
> > > >
> > > > SHA-512 checksums:
> > > >
> > > >
> > 09f29265f8173331d4c05b470c4e77a31146172b27ef333f45d8c8a19074ef25061cb1e80872fc45c323c9ce8e2e17989c6df5d991ef84c4d245197303d9e6d7
> > > >  apache-pulsar-2.10.6-bin.tar.gz
> > > >
> > > >
> > 49c8836882818c6f38748dae26b51c598f163606c16993a3287ab1ce9f853a4aaa43c6729c1b6f6957738b4dead3818cd12026da68b328eb2d4ac0d0214957bb
> > > >  apache-pulsar-2.10.6-src.tar.gz
> > > >
> > > > Maven staging repo:
> > > >
> > https://repository.apache.org/content/repositories/orgapachepulsar-1270
> > > >
> > > > The tag to be voted upon:
> > > > v2.10.6-candidate-1 (9c29b76ff2be865429ad44df8683aec80deacfba)
> > > > https://github.com/apache/pulsar/releases/tag/v2.10.6-candidate-1
> > > >
> > > > Pulsar's KEYS file containing PGP keys you use to sign the release:
> > > > https://downloads.apache.org/pulsar/KEYS
> > > >
> > > > Docker images:
> > > >
> > > > 
> > > >
> > > >
> > https://hub.docker.com/layers/xiangyingmeng/pulsar/2.10.6/images/sha256-bf8f36e49ff44ef810ab2c76742121205e51d3a04c79afdb5d288c7d8a06443f?context=repo
> > > >
> > > > 
> > > >
> > > >
> > https://hub.docker.com/layers/xiangyingmeng/pulsar-all/2.10.6/images/sha256-1b3a10db12f6d5a0acd2d4ed73eb11864b6b598294bb905b6ede34aef1157f23?context=repo
> > > >
> > > > Please download the source package, and follow the README to build
> > > > and run the Pulsar standalone service.
> > > >
> >

Re: [VOTE] Pulsar Release 2.10.6 Candidate 1

[Xiangying Meng]

Close this candidate, as we have a new fix, #22202, that needs to be
contained in the release.  I will raise a new candidate soon.


Regards

On Fri, Mar 8, 2024 at 10:06 AM Xiangying Meng  wrote:
>
> This right.
> Maybe we need some discussion and update our release policy.
>
> Thanks
> Xiangying
>
> On Wed, Mar 6, 2024 at 5:09 PM Zixuan Liu  wrote:
> >
> > It sounds like the version release was triggered due to security issues. If
> > so, I think we need to update our release policy.
> >
> > Only when a fatal security issue occurs can we trigger a release of a new
> > version, but we also need to clarify the maintenance cycle, otherwise this
> > maintenance is endless.
> >
> > Thanks,
> > Zixuan
> >
> > Xiangying Meng  于2024年3月6日周三 16:45写道：
> >
> > > Dear Zixuan,
> > >
> > > Thank you for your email and your ongoing commitment to the Pulsar 
> > > project.
> > >
> > > I wanted to clarify that this release, 2.10.6, is a special case. It
> > > was primarily focused on addressing certain security issues that were
> > > deemed critical. This decision was made following internal discussions
> > > within the PMC.
> > >
> > > I completely understand and respect the release policy defined by
> > > Pulsar [0]. Under normal circumstances, we would indeed follow the
> > > policy and consider version 2.10 as EOL, ceasing further maintenance.
> > >
> > > However, given the exceptional nature of this release and the
> > > importance of the security issues it addresses, we felt it was
> > > necessary to make an exception in this case.
> > >
> > > Thank you for your understanding and for bringing this to our
> > > attention. We appreciate your diligence in adhering to Pulsar's
> > > release policy.
> > >
> > > Best regards,
> > >
> > > Xiangying
> > >
> > > On Wed, Mar 6, 2024 at 4:22 PM Zixuan Liu  wrote:
> > > >
> > > > Thank you for releasing 2.10.6.
> > > >
> > > > According to the release policy defined [0] by Pulsar, this version is
> > > EOL and
> > > > does not require further maintenance.
> > > >
> > > > If we need to continue to maintain the 2.10, we must discuss the
> > > > maintenance lifecycle of the 2.10, and update our doc.
> > > >
> > > > - [0] https://pulsar.apache.org/contribute/release-policy/
> > > >
> > > > Thanks,
> > > > Zixuan
> > > >
> > > >
> > > > Xiangying Meng  于2024年3月6日周三 11:15写道：
> > > >
> > > > > This is the first release candidate for Apache Pulsar, version 2.10.6.
> > > > >
> > > > > It fixes the following issues:
> > > > >
> > > > >
> > > https://github.com/apache/pulsar/pulls?q=is:pr+label:cherry-picked/branch-2.10+label:release/2.10.6+is:closed
> > > > >
> > > > > *** Please download, test and vote on this release. This vote will 
> > > > > stay
> > > > > open
> > > > > for at least 72 hours ***
> > > > >
> > > > > Note that we are voting upon the source (tag), binaries are provided
> > > for
> > > > > convenience.
> > > > >
> > > > > Source and binary files:
> > > > >
> > > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.10.6-candidate-1/
> > > > >
> > > > > SHA-512 checksums:
> > > > >
> > > > >
> > > 09f29265f8173331d4c05b470c4e77a31146172b27ef333f45d8c8a19074ef25061cb1e80872fc45c323c9ce8e2e17989c6df5d991ef84c4d245197303d9e6d7
> > > > >  apache-pulsar-2.10.6-bin.tar.gz
> > > > >
> > > > >
> > > 49c8836882818c6f38748dae26b51c598f163606c16993a3287ab1ce9f853a4aaa43c6729c1b6f6957738b4dead3818cd12026da68b328eb2d4ac0d0214957bb
> > > > >  apache-pulsar-2.10.6-src.tar.gz
> > > > >
> > > > > Maven staging repo:
> > > > >
> > > https://repository.apache.org/content/repositories/orgapachepulsar-1270
> > > > >
> > > > > The tag to be voted upon:
> > > > > v2.10.6-candidate-1 (9c29b76ff2be865429ad44df8683aec80deacfba)
> > > > > https://github.com/apache/pulsar/releases/tag/v2.10.6-candidate-1
> > > > >
> > > > > Pulsar's KEYS file containing PGP keys you use to sign the release:
> > > > > https://downloads.apache.org/pulsar/KEYS
> > > > >
> > > > > Docker images:
> > > > >
> > > > > 
> > > > >
> > > > >
> > > https://hub.docker.com/layers/xiangyingmeng/pulsar/2.10.6/images/sha256-bf8f36e49ff44ef810ab2c76742121205e51d3a04c79afdb5d288c7d8a06443f?context=repo
> > > > >
> > > > > 
> > > > >
> > > > >
> > > https://hub.docker.com/layers/xiangyingmeng/pulsar-all/2.10.6/images/sha256-1b3a10db12f6d5a0acd2d4ed73eb11864b6b598294bb905b6ede34aef1157f23?context=repo
> > > > >
> > > > > Please download the source package, and follow the README to build
> > > > > and run the Pulsar standalone service.
> > > > >
> > >

[VOTE] Pulsar Release 2.10.6 Candidate 2

[Xiangying Meng]

This is the first release candidate for Apache Pulsar, version 2.10.6.

It fixes the following issues:
https://github.com/apache/pulsar/pulls?q=is:pr+label:cherry-picked/branch-2.10+label:release/2.10.6+is:closed

*** Please download, test and vote on this release. This vote will stay open
for at least 72 hours ***

Note that we are voting upon the source (tag), binaries are provided for
convenience.

Source and binary files:
https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.10.6-candidate-2/

SHA-512 checksums:
699a18b25828dc1274debb4d20cb22e9700935849243ea1892a15ad7b40c67d42ba46574e96356c39f4963183f080fb154fb42002559469eb1d240ceadf7a76c
 apache-pulsar-2.10.6-bin.tar.gz
fe7230e6c939b4da8da7c80b41611fd05336f83caeea981d16a749988cf4cb10cce64703c2c62aaec9f7ed1a4b353b1f68f8800b5befafe3264a5504750a2b6a
 apache-pulsar-2.10.6-src.tar.gz

Maven staging repo:
https://repository.apache.org/content/repositories/orgapachepulsar-1274

The tag to be voted upon:
v2.10.6-candidate-2 (a76ddbe5af523b4aa541a2272c58f685ef05859f)
https://github.com/apache/pulsar/releases/tag/v2.10.6-candidate-2

Pulsar's KEYS file containing PGP keys you use to sign the release:
https://downloads.apache.org/pulsar/KEYS

Docker images:


https://hub.docker.com/layers/xiangyingmeng/pulsar/2.10.6/images/sha256-743a4b5d79708b7e87d99008882c4d7321433616d7ee3b9a32063c398238decf?context=repo


https://hub.docker.com/layers/xiangyingmeng/pulsar-all/2.10.6/images/sha256-14013a1307c0c44ccb35526866c0775e5978fcb10beae5d5ffbdf4e73a029dc1?context=repo

Please download the source package, and follow the README to build
and run the Pulsar standalone service.

Re: [VOTE] Pulsar Release 2.10.6 Candidate 2

[Xiangying Meng]

Close this vote with 3 bindings.
- Penghui
- Jiwei Guo (Tboy)
- Lari

On Fri, Mar 8, 2024 at 5:04 PM PengHui Li  wrote:
>
> +1 (binding)
>
> - Checked the signatures
> - Build from the source
> - Checked the bookkeeper JNI libs
> org.apache.bookkeeper-circe-checksum-*.jar and
> org.apache.bookkeeper-cpu-affinity-*.jar
> - Verified the Cassandra connector
> - Verified the Stateful Function
> - Verified the Trino connector
> - Tested performance with 100 topics
>
> Regards,
> Penghui
>
> On Fri, Mar 8, 2024 at 4:14 PM guo jiwei  wrote:
>
> > +1 (binding)
> >
> > - Built from source
> > - Checked the signatures
> > - Run standalone
> > - Checked producer and consumer
> > - Verified the Cassandra connector
> > - Verified the Stateful function
> >
> >
> > Regards
> > Jiwei Guo (Tboy)
> >
> >
> > On Fri, Mar 8, 2024 at 3:48 PM Lari Hotari  wrote:
> >
> > > +1 (binding)
> > >
> > > - Built from source
> > > - Checked the signatures of the source and binary release artifacts
> > > - Run standalone
> > > - Checked producer and consumer
> > > - Verified the Cassandra connector
> > > - Verified the Stateful function
> > >
> > > -Lari
> > >
> > > On 2024/03/08 02:08:34 Xiangying Meng wrote:
> > > > This is the first release candidate for Apache Pulsar, version 2.10.6.
> > > >
> > > > It fixes the following issues:
> > > >
> > >
> > https://github.com/apache/pulsar/pulls?q=is:pr+label:cherry-picked/branch-2.10+label:release/2.10.6+is:closed
> > > >
> > > > *** Please download, test and vote on this release. This vote will stay
> > > open
> > > > for at least 72 hours ***
> > > >
> > > > Note that we are voting upon the source (tag), binaries are provided
> > for
> > > > convenience.
> > > >
> > > > Source and binary files:
> > > >
> > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.10.6-candidate-2/
> > > >
> > > > SHA-512 checksums:
> > > >
> > >
> > 699a18b25828dc1274debb4d20cb22e9700935849243ea1892a15ad7b40c67d42ba46574e96356c39f4963183f080fb154fb42002559469eb1d240ceadf7a76c
> > > >  apache-pulsar-2.10.6-bin.tar.gz
> > > >
> > >
> > fe7230e6c939b4da8da7c80b41611fd05336f83caeea981d16a749988cf4cb10cce64703c2c62aaec9f7ed1a4b353b1f68f8800b5befafe3264a5504750a2b6a
> > > >  apache-pulsar-2.10.6-src.tar.gz
> > > >
> > > > Maven staging repo:
> > > >
> > https://repository.apache.org/content/repositories/orgapachepulsar-1274
> > > >
> > > > The tag to be voted upon:
> > > > v2.10.6-candidate-2 (a76ddbe5af523b4aa541a2272c58f685ef05859f)
> > > > https://github.com/apache/pulsar/releases/tag/v2.10.6-candidate-2
> > > >
> > > > Pulsar's KEYS file containing PGP keys you use to sign the release:
> > > > https://downloads.apache.org/pulsar/KEYS
> > > >
> > > > Docker images:
> > > >
> > > > 
> > > >
> > >
> > https://hub.docker.com/layers/xiangyingmeng/pulsar/2.10.6/images/sha256-743a4b5d79708b7e87d99008882c4d7321433616d7ee3b9a32063c398238decf?context=repo
> > > >
> > > > 
> > > >
> > >
> > https://hub.docker.com/layers/xiangyingmeng/pulsar-all/2.10.6/images/sha256-14013a1307c0c44ccb35526866c0775e5978fcb10beae5d5ffbdf4e73a029dc1?context=repo
> > > >
> > > > Please download the source package, and follow the README to build
> > > > and run the Pulsar standalone service.
> > > >
> > >
> >

[ANNOUNCE] Apache Pulsar 2.10.6 released

[Xiangying Meng]

The Apache Pulsar team is proud to announce Apache Pulsar version 2.10.6.

Pulsar is a highly scalable, low latency messaging platform running on
commodity hardware. It provides simple pub-sub semantics over topics,
guaranteed at-least-once delivery of messages, automatic cursor management for
subscribers, and cross-datacenter replication.

For Pulsar release details and downloads, visit:

https://pulsar.apache.org/download

Release Notes are at:
https://pulsar.apache.org/release-notes

We would like to thank the contributors that made the release possible.

Regards,

The Pulsar Team

[DISCUSS] Optimizing the Method of Estimating Message Backlog Size in Pulsar

[Xiangying Meng]

Dear Pulsar Community,

I would like to initiate a discussion regarding the optimization of
the method used for estimating the message backlog size.

In the current implementation, the backlog size is estimated from the
mark delete position to the last confirm position, whereas the backlog
message count is the number of messages from the mark delete position
to the last confirm position, minus the count of individually
acknowledged messages. The inconsistency between these two could
potentially confuse users.

For instance, let's consider there are 3,000 messages in a topic and
all messages except for message 1:0, 1:998, and 3:999 have been
acknowledged by a subscription. When users retrieve the stats of the
subscription, they will find that `msgBacklog` is 3, while
`backlogSize` is 3000 * entry size.

|1:0|...|1:998|...|3:999|

When it comes to the value of `backlogSize`, there seem to be two
different opinions:
1. The backlog size should be consistent with the message backlog, and
it should not include the messages that have been individually
acknowledged.
2. Only the messages before the mark delete position can be deleted,
so we should calculate the backlog size from the mark delete position,
and individual acknowledgments should not affect the calculation of
the backlog size.

I'm interested in hearing how others view this issue. I look forward
to your response.

Best Regards,
Xiangying

[DISCUSS] Optimize the Acktimeout Mechanism in Pulsar Client

[Xiangying Meng]

Dear Pulsar Community,

I would like to initiate a discussion regarding the optimization of
the acktimeout mechanism on the client side. As we all know, the
Pulsar consumer has a configuration for ack timeout that automatically
redelivers unacknowledged messages after a certain period. The
workflow is approximately as follows:

1. Record a message when it is received.
2. Remove the message from the record when the consumer begins to
acknowledge it.
3. A timed task checks whether the messages from the record have
reached the ack timeout and triggers redelivery for those messages.

This workflow has a potential shortcoming: it does not wait for the
ack response before removing the message from the record. If an ack
request is lost during transmission to the broker - due to issues with
proxy processing or buffer overflow after reaching the broker, for
instance - then the ack timeout will not be resent, leading to an ack
hole. While this situation is quite extreme and the likelihood of
occurrence is extremely low, it is nonetheless a possibility. Another,
more common scenario is when the broker fails to process or persist
the ack request.

In such cases, users who are highly sensitive to ack holes may prefer
the message to be removed only after receiving the ack response.
Perhaps we could add a parameter to the acktimeout to determine
whether to wait for the ack response before removing the message.

I am interested in hearing your thoughts on this issue and look
forward to your responses and valuable suggestions.

Best Regards,
Xiangying

Re: [DISCUSS] Optimize the Acktimeout Mechanism in Pulsar Client

[Xiangying Meng]

I have considered this issue, but it may increase the user's consumption delay.
However, it is not appropriate to make modifications in the ack timeout.
Yubiao and I have just discussed this issue, and we should handle it
in the AcknowledgementsGroupingTracker to automatically retry failed
ack requests.

Thanks,

On Wed, Mar 27, 2024 at 5:37 PM ZhangJian He  wrote:
>
> Hi, Xiangying. Have you ever considered the `isAckReceiptEnabled` param?
>
> Thanks
> ZhangJian He
>
>
> On Wed, Mar 27, 2024 at 3:33 PM Xiangying Meng  wrote:
>
> > Dear Pulsar Community,
> >
> > I would like to initiate a discussion regarding the optimization of
> > the acktimeout mechanism on the client side. As we all know, the
> > Pulsar consumer has a configuration for ack timeout that automatically
> > redelivers unacknowledged messages after a certain period. The
> > workflow is approximately as follows:
> >
> > 1. Record a message when it is received.
> > 2. Remove the message from the record when the consumer begins to
> > acknowledge it.
> > 3. A timed task checks whether the messages from the record have
> > reached the ack timeout and triggers redelivery for those messages.
> >
> > This workflow has a potential shortcoming: it does not wait for the
> > ack response before removing the message from the record. If an ack
> > request is lost during transmission to the broker - due to issues with
> > proxy processing or buffer overflow after reaching the broker, for
> > instance - then the ack timeout will not be resent, leading to an ack
> > hole. While this situation is quite extreme and the likelihood of
> > occurrence is extremely low, it is nonetheless a possibility. Another,
> > more common scenario is when the broker fails to process or persist
> > the ack request.
> >
> > In such cases, users who are highly sensitive to ack holes may prefer
> > the message to be removed only after receiving the ack response.
> > Perhaps we could add a parameter to the acktimeout to determine
> > whether to wait for the ack response before removing the message.
> >
> > I am interested in hearing your thoughts on this issue and look
> > forward to your responses and valuable suggestions.
> >
> > Best Regards,
> > Xiangying
> >

Re: [DISCUSS] Optimizing the Method of Estimating Message Backlog Size in Pulsar

[Xiangying Meng]

Agree. While the name might be misleading, it indeed accurately
reflects the actual disk usage situation.


BR

On Wed, Mar 27, 2024 at 3:48 PM Girish Sharma  wrote:
>
> Hi Xiangying,
>
>
> > In the current implementation, the backlog size is estimated from the
> > mark delete position to the last confirm position, whereas the backlog
> > message count is the number of messages from the mark delete position
> > to the last confirm position, minus the count of individually
> > acknowledged messages. The inconsistency between these two could
> > potentially confuse users.
> >
>
> While confusing, it is somewhat accurate. Since the messages can be part of
> the same ledger where some messages are acked, some aren't, we can't delete
> that entire ledger until all messages of the ledger are acked - so it does
> contribute towards size of the backlog from a disk perspective.
> There might be some optimization possible - in a way that we try to figure
> out all completely acked ledgers from markDeletePosition to latest offset
> and remove their size, but what's the ROI there?
>
> So I would say that in your proposal, option 2 (current) is more accurate
> (while not being the best) than option 1.
>
> Regards
> --
> Girish Sharma

Re: [DISCUSS] GEO-replication issues on topic level

[Xiangying Meng]

Hi zixuan,

Thanks for your work in improving this geo-replication issue.
In my opinion, this should be a mistake when implementing pulsar
geo-replication at the topic level.
As we know, after a user configures replication policies at the
namespace level, the topics under the namespace will be created at the
remote clusters when the topic is created in the local cluster. If the
geo-replication policies are enabled at the topic level for a
non-partition topic, the topic will be created automatically in the
remote cluster when building the replicator producer.
However, for a partitioned topic, the topics created automatically are
no-partition topics in the remote cluster.

Therefore, creating topics at the remote clusters when the admin
uploads the replication policies is an acceptable solution. That is
following the current behavior of the topic creating for the
geo-replication enabling in namespace level and no-partition topic at
the topic level.  That is not a break-changing process that does not
require a proposal or a simple proposal to record this improvement.

Thanks,
Xiangying

On Fri, Mar 22, 2024 at 4:44 PM Zixuan Liu  wrote:
>
>  Hi all,
>
> The GEO-replication can be enabled on the Namespace and topic levels. When
> GEO-replication is enabled on the namespace level, it automatically creates
> the topic for the remote cluster, but the topic level misses this feature,
> which can cause unexpected problems.
>
> When two clusters use different global configuration metadatastore, the
> local cluster has a partitioned topic, and then we enable the
> GEO-replication on the topic level, I expect a partitioned topic will be
> created on the remote cluster, not a non-partitioned topic. BTW,
> `allowAutoTopicCreation` was enabled.
>
> There are two options:
>
> 1. When the GEO replication is enabled on the topic level , we can create
> the topic for the remote cluster(usually, we have superuser permissions):
> https://github.com/apache/pulsar/pull/22203
> 2. When the remote cluster has no topic, stop GEO-replication and throw an
> error.
>
> We also need to consider resource/permission issues, so like the remote
> clusters disables topic creation or exceed the maximum number of topics.
>
> Please let me know your thoughts.
>
> More contexts: https://github.com/apache/pulsar/pull/21679 (This is an
> incorrect implement, but there is more context here.)
>
> Thanks,
> Zixuan

[DISCUSS] cherry-pick #22034 Create new ledger after the current ledger is closed

[Xiangying Meng]

Hi all,

I want to start a discussion to cherry-pick #22034 [0] to release branches.
The PR creates a new ledger after the current one is full. This is a
bug fix to resolve the issue where the last ledger could not be
deleted after expiration. Moreover, since there is no need to create a
new ledger only when messages are sent, it can reduce the send
latency.

However, it could be a behavior change and introduce some flaky tests
we have fixed in the master. So I think it may be necessary to have a
discussion before cherry-picking.

The target branches:

- branch-3.0
- branch-3.1
- branch-3.2

[0] https://github.com/apache/pulsar/pull/22034

I will keep the discussion open for at least 48 hours.
If there is no objections, I will perform the cherry-picking.

BR,
Xiangying

RE: Re: [DISCUSS] PIP-359: Support custom message listener executor for specific subscription

[xiangying meng]

On 2024/06/14 03:13:03 Yubiao Feng wrote:
> Same as Lari
> 
> Thanks
> Yubiao Feng
> 
> On Thu, Jun 13, 2024 at 8:05 PM Aurora Twinkle 
> wrote:
> 
> > Hi, Pulsar Community.
> > I open a new PIP for support custom message listener executor for specific
> > subscription to avoid individual subscription listener consuming too much
> > time leading to higher consumption delay in other subscriptions.
> > link: https://github.com/apache/pulsar/pull/22902
> >
> > Thanks,
> > Linlin Duan(AuroraTwinkle)
> >
> 

RE: RE: Re: [DISCUSS] PIP-359: Support custom message listener executor for specific subscription

[xiangying meng]

Good work! Same as Lari.
 
Thanks 
Xiangying meng


On 2024/06/14 03:39:01 xiangying meng wrote:
> 
> 
> On 2024/06/14 03:13:03 Yubiao Feng wrote:
> > Same as Lari
> > 
> > Thanks
> > Yubiao Feng
> > 
> > On Thu, Jun 13, 2024 at 8:05 PM Aurora Twinkle 
> > wrote:
> > 
> > > Hi, Pulsar Community.
> > > I open a new PIP for support custom message listener executor for specific
> > > subscription to avoid individual subscription listener consuming too much
> > > time leading to higher consumption delay in other subscriptions.
> > > link: https://github.com/apache/pulsar/pull/22902
> > >
> > > Thanks,
> > > Linlin Duan(AuroraTwinkle)
> > >
> > 

Re: [VOTE] PIP-359: Support custom message listener executor for specific subscription

[xiangying meng]

+1 (no-binding)

Thanks,
Xiangying

On Wed, Jun 19, 2024 at 2:14 PM Lari Hotari  wrote:

> +1 (binding)
>
> -Lari
>
> On 2024/06/18 16:44:46 Aurora Twinkle wrote:
> > Hi, Pulsar Community: I would like to start the voting thread for
> > PIP-359: Support
> > custom message listener executor for specific subscription.
> > PIP: https://github.com/apache/pulsar/pull/22902
> >
> > Please review and vote on the PIP-359, as follows: [ ] +1, Approve the
> > PIP-359 [ ] -1, Do not approve the PIP-539 (please provide specific
> > comments)
> > Thanks, Linlin Duan(AuroraTwinkle)
> >
>

[DISCUSS] Propose a Contributor Repository for Pulsar

[xiangying meng]

Hello Pulsar Community,

I hope this message finds you well. I'm reaching out to propose the
establishment of a Pulsar Contributor Repository.

This new initiative is designed to provide a dedicated space for
experimental and community-driven features that complement our core
offerings.  This repository would serve as a space for non-core
features and customizations, allowing for greater flexibility and
community involvement.

The proposal can be viewed at this GitHub link:
https://github.com/apache/pulsar/pull/23061/files

I'm looking forward to your thoughts and feedback. Please feel free to
share them on the Pulsar Developer Mailing List.

Thank you in advance for your time and consideration.

Best,

Xiangying

Re: [DISCUSS] Propose a Contributor Repository for Pulsar

[xiangying meng]

>thanks for your proposal, we need more and more energy in the project.

Yes, it requires a lot of participation, but sometimes we also need a start.

>Also we allow only "committers" to commit patches to the repositories that are 
>under the responsibility of the Apache Pulsar PMC, I don't know how we can 
>make this work with a "official fork"

>An alternative is to have a "curated list" of links to personal projects,
but we keep the responsibility over the code on the code owners and
not on the Pulsar PMC

Yes, this is a "curated list" link, but it points to the branch and
commit ID created by the contributor himself. In addition, for some
interface implementation classes, they can be placed in the main
branch of the new contributor's repository and then added to the
"curated list". Only committors or maintainers can add content to the
"curated list". I have detailed these in the proposal.

But no matter which method, each function will be marked in the
document, that is, in the "Featured List" who contributed it. I
understand that this feature is more of a reference feature. Allow
users to choose these customized and risky features to customize their
own versions. This is very useful for some large companies with
development capabilities. At the same time, its responsibility should
belong to the user, although we will also point out who is the
contributor of each feature he chooses. But when using innovative or
customized features, users should also be clear about the risks. I
also wrote about the risks of contributor repositories in the
proposal.

In addition, thank you for your valuable feedback.

Best regards
Xiangying

On Mon, Jul 22, 2024 at 10:55 PM Enrico Olivelli  wrote:
>
> Hello,
> thanks for your proposal, we need more and more energy in the project.
>
> I don't think that this is a good idea.
>
> The main reason is that as an ASF project we must guarantee the quality and
> especially the security of what we provide to the public.
>
> Such kinds of repositories tend to become full of stale code that can
> become a source of security issues.
>
> Also we allow only "committers" to commit patches to the repositories that
> are under the responsibility of the Apache Pulsar PMC, I don't know how we
> can make this work with a "official fork"
>
> Maybe I misunderstood the goal ?
>
> An alternative is to have a "curated list" of links to personal projects,
> but we keep the responsibility over the code on the code owners and not on
> the Pulsar PMC
>
>
> Best regards
> Enrico Olivelli
>
>
>
> Il giorno lun 22 lug 2024 alle ore 15:15 steven lu 
> ha scritto:
>
> > We think this is a very good solution,
> > Our XHS team(@liangyepianzhou <https://github.com/liangyepianzhou>
> > @AuroraTwinkle <https://github.com/AuroraTwinkle> @StevenLuMT
> > <https://github.com/StevenLuMT> @cai152 <https://github.com/cai152>) will
> > submit a few to pulsar-java-contrib
> > <https://github.com/StevenLuMT/pulsar-java-contrib>(
> > https://github.com/StevenLuMT/pulsar-java-contrib) as a primer to let
> > everyone know the role of this library
> >
> > xiangying meng  于2024年7月22日周一 16:33写道：
> >
> > > Hello Pulsar Community,
> > >
> > > I hope this message finds you well. I'm reaching out to propose the
> > > establishment of a Pulsar Contributor Repository.
> > >
> > > This new initiative is designed to provide a dedicated space for
> > > experimental and community-driven features that complement our core
> > > offerings.  This repository would serve as a space for non-core
> > > features and customizations, allowing for greater flexibility and
> > > community involvement.
> > >
> > > The proposal can be viewed at this GitHub link:
> > > https://github.com/apache/pulsar/pull/23061/files
> > >
> > > I'm looking forward to your thoughts and feedback. Please feel free to
> > > share them on the Pulsar Developer Mailing List.
> > >
> > > Thank you in advance for your time and consideration.
> > >
> > > Best,
> > >
> > > Xiangying
> > >
> >

Re: [DISCUSS] Propose a Contributor Repository for Pulsar

[xiangying meng]

Thank you for your feedback, let's further clarify this matter.

The original design of this proposal is a new contributor repository,
and provides two ways to contribute.
The first one:
Contributors contribute through PR and accept code review, and can
only be merged into the main branch after reviewed. The code merged in
this way is various plug-ins based on Pulsar's external interface. It
does not contain any pulsar code, only Pulsar's dependencies.
The second one:
Features that require modifications to the Pulsar main repository code
and may conflict with later versions. My initial idea was for
contributors to create branches and then write the commit id and
features description into the document after the PR is reviewed.

However, after the discussion just now, I agree that the second one
may not be maintained in the branch of the new repository, but in the
contributor's personal repository.

Based on the above content, let's continue to look at the question you raised.

>- How can the Pulsar community maintain  a fork of Pulsar itself  ?

If it is just  plug-in library and a feature list document of the
features in  personal repository, then there is no need to fork Pulsar
itself.

>-  Do you want to "cut releases" out of this fork ? Who is going to validate 
>them ? Who is responsible for security bugs?

We will only need to maintain a documents of feature and plugins list
that have been reviewed by PR. We will not make changes to the Pulsar
code itself. There is no fork of Pulsar. It will have its own version
release process, just like Pulsar.

-> Who is going to use this code ?

IMO, providing various implementation plugins based on Pulsar's
external interface is meaningful in itself. While introducing Pulsar's
own dependencies, introducing plugin libraries to use various already
implemented plugins can reduce development costs. I think some people
will be happy to use them.

As for the other functions in the document, maintained in personal
repositories, they can be used as a reference and provided to
companies that are capable of solving security issues and bugs.

Best regards,
Xiangying

On Tue, Jul 23, 2024 at 12:03 AM Enrico Olivelli  wrote:
>
> Il giorno lun 22 lug 2024 alle ore 17:34 xiangying meng <
> xiangyingme...@gmail.com> ha scritto:
>
> > >thanks for your proposal, we need more and more energy in the project.
> >
> > Yes, it requires a lot of participation, but sometimes we also need a
> > start.
> >
> > >Also we allow only "committers" to commit patches to the repositories
> > that are under the responsibility of the Apache Pulsar PMC, I don't know
> > how we can make this work with a "official fork"
> >
> > >An alternative is to have a "curated list" of links to personal projects,
> > but we keep the responsibility over the code on the code owners and
> > not on the Pulsar PMC
> >
> > Yes, this is a "curated list" link, but it points to the branch and
> > commit ID created by the contributor himself. In addition, for some
> > interface implementation classes, they can be placed in the main
> > branch of the new contributor's repository and then added to the
> > "curated list". Only committors or maintainers can add content to the
> > "curated list". I have detailed these in the proposal.
> >
> > But no matter which method, each function will be marked in the
> > document, that is, in the "Featured List" who contributed it. I
> > understand that this feature is more of a reference feature. Allow
> > users to choose these customized and risky features to customize their
> > own versions. This is very useful for some large companies with
> > development capabilities. At the same time, its responsibility should
> > belong to the user, although we will also point out who is the
> > contributor of each feature he chooses. But when using innovative or
> > customized features, users should also be clear about the risks. I
> > also wrote about the risks of contributor repositories in the
> > proposal.
> >
>
> The main point is that we cannot grant "write" permissions to repositories
> owned by the Pulsar project (the PMC)
> to people who are not "committers". This is a legal thing, no exceptions.
>
> So any contribution must be in the form of a PR, like for any other patches
> contributes to one of the github repositories owned by the Pulsar project
>
> A couple of questions:
> - How can the Pulsar community maintain  a fork of Pulsar itself  ?
> - Do you want to "cut releases" out of this fork ? Who is going to validate
> them ? Who is responsible for security bugs ?

Re: [DISCUSS] Propose a Contributor Repository for Pulsar

[xiangying meng]

Hello Wen Zhi,

This project is a new repository similar to a connector, and it will
not make any modifications to Pulsar itself. [0]

Its code part is simply a plugin library, based on Pulsar's external
interfaces, providing a rich set of implementations. It only retains a
"curated list", which is a series of pointers to personalized features
and experimental features in personal repositories for reference and
inspiration. If these experimental features are recognized by other
users, they can also be gradually merged into the Pulsar main
repository. However, it will not fork the content of the Pulsar
repository for modification in the new repository.
This is the improvement we made after discussion in the last email. I
will synchronize it to the proposal later.
Finally, thank you for your valuable suggestions, and we look forward
to your continued participation.

[0] - https://github.com/apache/pulsar-connectors

Best Regards,
Xiangying

On Tue, Jul 23, 2024 at 10:36 AM WenZhi Feng  wrote:
>
> Hi, xiangying,
>   First of all, thanks for driving the proposal.
>   I have some questions.
>   1. Will this repository maintained by committers/pmc? If the answer is yes, 
> there will be too many new modules introduced into pulsar and hard to 
> maintain. If the answer is no, i think we have better not to include it into 
> pulsar official repository.
>   2. For those plugin-in feature, we may create a new repository like various 
> connectors for pulsar. But for those core code change on pulsar, how can we 
> seperate it with pulsar repository?
>   It is thrilling that users contribute various new features to apache 
> community, enriching the ecosystem. I think we should find out a good way to 
> realize it.
>
> Thanks,
> Wenzhi Feng.
>
> On 2024/07/22 17:40:24 xiangying meng wrote:
> > Thank you for your feedback, let's further clarify this matter.
> >
> > The original design of this proposal is a new contributor repository,
> > and provides two ways to contribute.
> > The first one:
> > Contributors contribute through PR and accept code review, and can
> > only be merged into the main branch after reviewed. The code merged in
> > this way is various plug-ins based on Pulsar's external interface. It
> > does not contain any pulsar code, only Pulsar's dependencies.
> > The second one:
> > Features that require modifications to the Pulsar main repository code
> > and may conflict with later versions. My initial idea was for
> > contributors to create branches and then write the commit id and
> > features description into the document after the PR is reviewed.
> >
> > However, after the discussion just now, I agree that the second one
> > may not be maintained in the branch of the new repository, but in the
> > contributor's personal repository.
> >
> > Based on the above content, let's continue to look at the question you 
> > raised.
> >
> > >- How can the Pulsar community maintain  a fork of Pulsar itself  ?
> >
> > If it is just  plug-in library and a feature list document of the
> > features in  personal repository, then there is no need to fork Pulsar
> > itself.
> >
> > >-  Do you want to "cut releases" out of this fork ? Who is going to 
> > >validate them ? Who is responsible for security bugs?
> >
> > We will only need to maintain a documents of feature and plugins list
> > that have been reviewed by PR. We will not make changes to the Pulsar
> > code itself. There is no fork of Pulsar. It will have its own version
> > release process, just like Pulsar.
> >
> > -> Who is going to use this code ?
> >
> > IMO, providing various implementation plugins based on Pulsar's
> > external interface is meaningful in itself. While introducing Pulsar's
> > own dependencies, introducing plugin libraries to use various already
> > implemented plugins can reduce development costs. I think some people
> > will be happy to use them.
> >
> > As for the other functions in the document, maintained in personal
> > repositories, they can be used as a reference and provided to
> > companies that are capable of solving security issues and bugs.
> >
> > Best regards,
> > Xiangying
> >
> > On Tue, Jul 23, 2024 at 12:03 AM Enrico Olivelli  
> > wrote:
> > >
> > > Il giorno lun 22 lug 2024 alle ore 17:34 xiangying meng <
> > > xiangyingme...@gmail.com> ha scritto:
> > >
> > > > >thanks for your proposal, we need more and more energy in the project.
> > > >
> > > > Yes, it requires a lot of participation, but so

Re: [VOTE] PIP-366: Support to specify different config for Configuration and Local Metadata Store

[xiangying meng]

+1 (no-binding)

Thanks
Xiangying

On Thu, Jul 25, 2024 at 9:50 AM Yubiao Feng
 wrote:
>
> +1 (binding)
>
> Thanks
> Yubiao Feng
>
> On Mon, Jul 22, 2024 at 9:46 AM Kai Wang  wrote:
>
> > Hi all, I want to start a vote on PIP-366: Support to specify different
> > config for Configuration and Local Metadata Store.
> >
> > You can find the proposal at https://github.com/apache/pulsar/pull/23033
> > and
> > the discussion thread at
> > https://lists.apache.org/thread/98ggo1zg1k7dbyx8wr9bc8onm10p16c6
> >
> > The vote will stay open for at least 48 hours.
> > Thanks, Kai
> >

Re: [DISCUSS] Propose a Contributor Repository for Pulsar

[xiangying meng]

Dear Dave and Enrico,

Thank you for your reply and valuable comments. I think the point that
needs to be clarified now is whether this repository needs to be
released. In the original idea, the functions in this repository are
not stable. It will be used to collect various experimental functions
and extended plug-in implementations.
The risk is mentioned in the first version of the proposal. However,
due to lack of experience, I failed to clarify the importance of
release, so there is no clear release description. Thanks to Dave for
helping to further clarify this matter.

- For experimental functions and customized functions that require
modifying the Pulsar source code, I think they must not be released.
- For the plug-in implementation of Pulsar extension, it cannot be
promised to users that it is stable, so it cannot be released, too.

Therefore, we further clarify the positioning of this repository.

- Collect user needs and their various customized plug-in implementations
- Collect and organize various experimental functions and customized functions
- Collect user best practices

The admission standard of the warehouse is that this function can work
normally and meet the needs, but it is not necessarily stable, so it
will not be released.

With the continuous feedback from users, this feature may become
stable, and eventually someone (new contributor or original
contributor) may move this feature to a stable releaseable place. That
is, move it to the `appropriate Pulsar repository and component` you
mentioned.

In addition, regarding the SECURITY.md file, I have added it to the
repository. [0]

Best Regards

xiangying

[0] - https://github.com/StevenLuMT/pulsar-java-contrib/blob/stable/SECURITY.md

On Mon, Jul 29, 2024 at 10:50 PM Dave Fisher  wrote:
>
>
>
> > On Jul 29, 2024, at 6:37 AM, Enrico Olivelli  wrote:
> >
> > Il giorno lun 29 lug 2024 alle ore 11:55 steven lu 
> > ha scritto:
> >
> >> Different from pulsar-adapters, we added some instructions and examples
> >> here https://github.com/StevenLuMT/pulsar-java-contrib
> >
> >
> > This example is clear, thanks !
>
> From your README in the repository.
>
> > Pulsar java contrib is to provide a non-core code maintenance repository to 
> > collect plugin implementations, personalized features, experimental 
> > features, and best practices from users.
>
> These are examples which anyone can use at their own risk and the PMC does 
> not intend to make RELEASES. Is this correct? If so then that should be very 
> clear.
>
> Anything the community decides to support for the future in a release should 
> be moved into the appropriate Pulsar repository and component.
>
> If this were made clear then I would be a +1.
>
> Security vulnerabilities found would still require care and attention from 
> the PMC due to the private nature of this reporting. You will need a 
> SECURITY.md file.
>
> Best,
> Dave
>
> >
> > Let's put it this way: +0
> > I am not against this proposal, but I would support it if we see that the
> > Community and in particular the PMC
> > is willing to maintain it.
> >
> > If the community is not willing to maintain it (and we already have
> > problems with pulsar-adapters for instance) then
> > it is only like adding another dead repository plus the responsibility to
> > fix security issues and keep the examples up to date with the best
> > practices as long as Pulsar evolves
> >
> > If I were you I would start a well curated github repository and encourage
> > people to contribute their code snippets to it.
> >
> > That said, I won't VOTE against the proposal
> >
> > Thanks
> >
> > Enrico
> >
> >
> >
> >>
> >>
> >> Enrico Olivelli  于2024年7月23日周二 17:45写道：
> >>
> >>> Probably this repository is already what you want to do:
> >>> https://github.com/apache/pulsar-adapters
> >>>
> >>> It contains a lot of different stuff that we don't keep in the main
> >> pulsar
> >>> repository.
> >>>
> >>> Enrico
> >>>
> >>>
> >>> Il giorno mar 23 lug 2024 alle ore 09:51 Jia Zhai 
> >> ha
> >>> scritto:
> >>>
> >>>> Thanks for the proposal. It is a good idea. It is also not a bad idea
> >> to
> >>>> incubate it outside of Pulsar official repository at first.
> >>>>
> >>>> On Tue, Jul 23, 2024 at 2:59 PM Aurora Twinkle <
> >> foreverlove...@gmail.com
> >>>>
> >>>> wrote:
> >>>>
> >>>>> Hi:
> >>>>> 

[VOTE] PIP-367 Propose a Contributor Repository for Pulsar

[xiangying meng]

Hi all, I want to start a vote on PIP-367: Propose a Contributor
Repository for Pulsar

You can find the proposal at https://github.com/apache/pulsar/pull/23061 and
the discussion thread at
https://lists.apache.org/thread/75y70j6pqw15271b0tq8j637tb02mrwo

The vote will stay open for at least 48 hours.

BR,
Xiangying

Re: [DISCUSS] Propose a Contributor Repository for Pulsar

[xiangying meng]

Dear all,

I would like to express my appreciation to all of you who have shown
support for my recent proposal. Your feedback and encouragement are
greatly appreciated and have been instrumental in moving the
discussion forward.

I am pleased to announce that I have initiated a vote on this
proposal. You can find the details and participate in the voting
through the following link:

https://lists.apache.org/thread/td0j8l1c3l93nny0m5smnsdmb91j1n2y

Your active participation is crucial for the success of this
initiative, and I look forward to a positive outcome.

Once again, thank you for your valuable input and support.

Best regards,
Xiangying

On Sat, Aug 3, 2024 at 3:38 AM Dave Fisher  wrote:
>
> Hi - I renamed the PIP PR to include the PIP number - PIP-367. - 
> https://github.com/apache/pulsar/pull/23061
>
> I think that you have covered concerns and it is time to start a separate 
> VOTE thread.
>
> Assuming that the PIP passes then we can create the new repository by 
> following your example and adjust the files to best fit policies.
>
> Best Regards,
> Dave
>
> > On Jul 29, 2024, at 7:45 PM, xiangying meng  wrote:
> >
> > Dear Dave and Enrico,
> >
> > Thank you for your reply and valuable comments. I think the point that
> > needs to be clarified now is whether this repository needs to be
> > released. In the original idea, the functions in this repository are
> > not stable. It will be used to collect various experimental functions
> > and extended plug-in implementations.
> > The risk is mentioned in the first version of the proposal. However,
> > due to lack of experience, I failed to clarify the importance of
> > release, so there is no clear release description. Thanks to Dave for
> > helping to further clarify this matter.
> >
> > - For experimental functions and customized functions that require
> > modifying the Pulsar source code, I think they must not be released.
> > - For the plug-in implementation of Pulsar extension, it cannot be
> > promised to users that it is stable, so it cannot be released, too.
> >
> > Therefore, we further clarify the positioning of this repository.
> >
> > - Collect user needs and their various customized plug-in implementations
> > - Collect and organize various experimental functions and customized 
> > functions
> > - Collect user best practices
> >
> > The admission standard of the warehouse is that this function can work
> > normally and meet the needs, but it is not necessarily stable, so it
> > will not be released.
> >
> > With the continuous feedback from users, this feature may become
> > stable, and eventually someone (new contributor or original
> > contributor) may move this feature to a stable releaseable place. That
> > is, move it to the `appropriate Pulsar repository and component` you
> > mentioned.
> >
> > In addition, regarding the SECURITY.md file, I have added it to the
> > repository. [0]
> >
> > Best Regards
> >
> > xiangying
> >
> > [0] - 
> > https://github.com/StevenLuMT/pulsar-java-contrib/blob/stable/SECURITY.md
> >
> > On Mon, Jul 29, 2024 at 10:50 PM Dave Fisher  wrote:
> >>
> >>
> >>
> >>> On Jul 29, 2024, at 6:37 AM, Enrico Olivelli  wrote:
> >>>
> >>> Il giorno lun 29 lug 2024 alle ore 11:55 steven lu 
> >>> ha scritto:
> >>>
> >>>> Different from pulsar-adapters, we added some instructions and examples
> >>>> here https://github.com/StevenLuMT/pulsar-java-contrib
> >>>
> >>>
> >>> This example is clear, thanks !
> >>
> >> From your README in the repository.
> >>
> >>> Pulsar java contrib is to provide a non-core code maintenance repository 
> >>> to collect plugin implementations, personalized features, experimental 
> >>> features, and best practices from users.
> >>
> >> These are examples which anyone can use at their own risk and the PMC does 
> >> not intend to make RELEASES. Is this correct? If so then that should be 
> >> very clear.
> >>
> >> Anything the community decides to support for the future in a release 
> >> should be moved into the appropriate Pulsar repository and component.
> >>
> >> If this were made clear then I would be a +1.
> >>
> >> Security vulnerabilities found would still require care and attention from 
> >> the PMC due to the private nature of this reporting. You will need a 
> >> SECURITY.md file.
> >>
> >> Best,
> 

Re: [VOTE] PIP-368: Support lookup based on the lookup properties

[xiangying meng]

+1 nonbinding

BR,
Xiangying

On Mon, Aug 5, 2024 at 10:02 AM Yunze Xu  wrote:
>
> +1 (binding)
>
> Thanks,
> Yunze
>
> On Thu, Aug 1, 2024 at 9:39 PM Jie crossover  wrote:
> >
> > +1 nonbinding
> > --
> > Best Regards！
> > crossoverJie
> >
> >
> > Tao Jiuming  于2024年8月1日周四 19:38写道：
> >
> > > +1 nonbinding
> > >
> > > Zike Yang 于2024年8月1日 周四19:08写道：
> > >
> > > > Hi, everyone,
> > > >
> > > > I want to start a vote on PIP-368: Support lookup based on the lookup
> > > > properties
> > > >
> > > > You can find the proposal at
> > > > https://github.com/apache/pulsar/pull/23075 and the discussion thread
> > > > at https://lists.apache.org/thread/7n2gncxk3c5q8dxj8fw9y5gcwg6jjg6z
> > > >
> > > > The vote will stay open for at least 48 hours.
> > > >
> > > > Thanks,
> > > > Zike Yang
> > > >
> > >

Re: [VOTE] PIP-367 Propose a Contributor Repository for Pulsar

[xiangying meng]

Thanks, everyone.

Close this vote with 3 binding +1s:
- Dave Fisher
- Enrico Olivelli
- Yunze Xu

Thanks,
Xiangying


On Mon, Aug 19, 2024 at 4:01 PM Yunze Xu  wrote:
>
> Well, let me change my binding -1 to binding +1.
>
> Thanks,
> Yunze
>
> On Mon, Aug 19, 2024 at 3:22 PM steven lu  wrote:
> >
> > In the past period of time, we have initiated several discussions with
> > other companies and Pulsar community developers, and improved the project
> > as follows:
> >
> > As you can see, there are one license [1], one SECURITY file [2], CI
> > workflows[3], and more commits in the last two weeks [4].
> > there are one best practice part[5] and execution plan [6].
> >
> > Secondly, maintaining this repo requires Pulsar PMC's care. now there are
> > two PMC members who support to maintain it. [7]
> >
> >
> > [1] license file:
> > https://github.com/StevenLuMT/pulsar-java-contrib?tab=License-1-ov-file
> > [2] SECURITY file:
> > https://github.com/StevenLuMT/pulsar-java-contrib?tab=security-ov-file
> > [3] CI workflows:
> > https://github.com/StevenLuMT/pulsar-java-contrib/pull/8/checks
> > [4] commits in the last two weeks:
> > https://github.com/StevenLuMT/pulsar-java-contrib/commits/stable/
> > [5] best practice:
> > https://github.com/StevenLuMT/pulsar-java-contrib/tree/stable/pulsar-client-common-contrib/src/main/java/org/apache/pulsar/client/api/impl
> > [6] execution plan:
> > https://github.com/StevenLuMT/pulsar-java-contrib/issues/7
> > [7] VOTE for PIP-367:
> > https://lists.apache.org/thread/td0j8l1c3l93nny0m5smnsdmb91j1n2y
> >
> > On Sat, Aug 3, 2024 at 5:25 PM xiangying meng  wrote:
> >
> > > Hi all, I want to start a vote on PIP-367: Propose a Contributor
> > > Repository for Pulsar
> > >
> > > You can find the proposal at https://github.com/apache/pulsar/pull/23061
> > > and
> > > the discussion thread at
> > > https://lists.apache.org/thread/75y70j6pqw15271b0tq8j637tb02mrwo
> > >
> > > The vote will stay open for at least 48 hours.
> > >
> > > BR,
> > > Xiangying
> > >

Re: [VOTE] PIP-367 Propose a Contributor Repository for Pulsar

[xiangying meng]

Thank you for your attention. I believe that PIP-371 is very
meaningful for this contributor repository. The contributor repository
is not just a plugin library. It also includes some extensible and
experimental features. We welcome all contributors in the community to
contribute their own ideas. After the new repository is created, we
can find a way to promote your feature into the contributor
repository. Of course, this may involve some code changes to avoid
unnecessary coupling with the Pulsar main repository code.

Thanks,
Xiangying

On Mon, Aug 19, 2024 at 4:31 PM SiNan Liu  wrote:
>
> Sorry, I would like to know more about it.
> PIP-371(https://github.com/apache/pulsar/pull/23194)
>
> Can this be contributed to this repo? I think the repo is just a plug-in
> implementation of the interface?
>
>
> Thanks,
> sinan
>
>
> xiangying meng 于2024年8月19日 周一16:09写道：
>
> > Thanks, everyone.
> >
> > Close this vote with 3 binding +1s:
> > - Dave Fisher
> > - Enrico Olivelli
> > - Yunze Xu
> >
> > Thanks,
> > Xiangying
> >
> >
> > On Mon, Aug 19, 2024 at 4:01 PM Yunze Xu  wrote:
> > >
> > > Well, let me change my binding -1 to binding +1.
> > >
> > > Thanks,
> > > Yunze
> > >
> > > On Mon, Aug 19, 2024 at 3:22 PM steven lu  wrote:
> > > >
> > > > In the past period of time, we have initiated several discussions with
> > > > other companies and Pulsar community developers, and improved the
> > project
> > > > as follows:
> > > >
> > > > As you can see, there are one license [1], one SECURITY file [2], CI
> > > > workflows[3], and more commits in the last two weeks [4].
> > > > there are one best practice part[5] and execution plan [6].
> > > >
> > > > Secondly, maintaining this repo requires Pulsar PMC's care. now there
> > are
> > > > two PMC members who support to maintain it. [7]
> > > >
> > > >
> > > > [1] license file:
> > > >
> > https://github.com/StevenLuMT/pulsar-java-contrib?tab=License-1-ov-file
> > > > [2] SECURITY file:
> > > > https://github.com/StevenLuMT/pulsar-java-contrib?tab=security-ov-file
> > > > [3] CI workflows:
> > > > https://github.com/StevenLuMT/pulsar-java-contrib/pull/8/checks
> > > > [4] commits in the last two weeks:
> > > > https://github.com/StevenLuMT/pulsar-java-contrib/commits/stable/
> > > > [5] best practice:
> > > >
> > https://github.com/StevenLuMT/pulsar-java-contrib/tree/stable/pulsar-client-common-contrib/src/main/java/org/apache/pulsar/client/api/impl
> > > > [6] execution plan:
> > > > https://github.com/StevenLuMT/pulsar-java-contrib/issues/7
> > > > [7] VOTE for PIP-367:
> > > > https://lists.apache.org/thread/td0j8l1c3l93nny0m5smnsdmb91j1n2y
> > > >
> > > > On Sat, Aug 3, 2024 at 5:25 PM xiangying meng 
> > wrote:
> > > >
> > > > > Hi all, I want to start a vote on PIP-367: Propose a Contributor
> > > > > Repository for Pulsar
> > > > >
> > > > > You can find the proposal at
> > https://github.com/apache/pulsar/pull/23061
> > > > > and
> > > > > the discussion thread at
> > > > > https://lists.apache.org/thread/75y70j6pqw15271b0tq8j637tb02mrwo
> > > > >
> > > > > The vote will stay open for at least 48 hours.
> > > > >
> > > > > BR,
> > > > > Xiangying
> > > > >
> >

Re: [DISCUSS] PIP-373: Add a topic's system prop that indicates whether users have published TXN messages in before

[xiangying meng]

Good idea. I still have some questions about this.

Since we can add data to the topic attribute, why not directly put the
snapshot or snapshot segment position into the topic position?
This method has two advantages:
1. System properties of topics that do not use transactions are not
added. It will not affect common topics.
2. Topics using transactions can be recovered faster.

Of course, the disadvantage is that topics using transactions will
need to update these properties frequently.

Thanks,

Xiangying

On Thu, Aug 22, 2024 at 10:53 AM Ran Gao  wrote:
>
> LGTM
>
> Not all topics use the transaction feature, it can save a lot of resources to 
> help reduce broker recovery time, and it's convenient to calculate topics 
> count that use transactions.
>
> Thanks,
> Ran Gao
>
> On 2024/08/21 14:27:25 Yubiao Feng wrote:
> > Hi all
> >
> > I drafted a proposal to add a topic's system prop that indicates whether
> > users have published TXN messages in before.
> >
> > https://github.com/apache/pulsar/pull/23210
> >
> > I'm looking forward to hearing from you.
> >
> > Thanks
> > Yubiao Feng
> >

Re: [DISCUSS] PIP-373: Add a topic's system prop that indicates whether users have published TXN messages in before

[xiangying meng]

Hi yubiao
Then return to your plan, must we set the system property to false for
topics that do not use transactions? I understand that it is enough to
only set the system properties of the topics that used the
transaction.

Thanks,
Xiangying Meng

On Fri, Aug 23, 2024 at 4:16 PM Yubiao Feng
 wrote:
>
> Hi Xiangying
>
> > Since we can add data to the topic attribute,
> > why not directly put the
> > snapshot or snapshot segment position into
> > the topic position?
>
> Reason 1:
> - The rate of loading the topic is once many days
> - The rate of taking Transaction Buffer snapshot is per 5 seconds
>
> The changes you suggested are to use write ZK per 5 seconds instead of
> reading data from BK once many days, it is not better.
>
> Reason 2:
> Once you move snapshots from BK to ZK, we can difficultly guarantee the
> compatibility of upgrading and downgrading.
>
> Thanks
> Yubioa Feng
>
> On Thu, Aug 22, 2024 at 4:52 PM xiangying meng  wrote:
>
> > Good idea. I still have some questions about this.
> >
> > Since we can add data to the topic attribute, why not directly put the
> > snapshot or snapshot segment position into the topic position?
> > This method has two advantages:
> > 1. System properties of topics that do not use transactions are not
> > added. It will not affect common topics.
> > 2. Topics using transactions can be recovered faster.
> >
> > Of course, the disadvantage is that topics using transactions will
> > need to update these properties frequently.
> >
> > Thanks,
> >
> > Xiangying
> >
> > On Thu, Aug 22, 2024 at 10:53 AM Ran Gao  wrote:
> > >
> > > LGTM
> > >
> > > Not all topics use the transaction feature, it can save a lot of
> > resources to help reduce broker recovery time, and it's convenient to
> > calculate topics count that use transactions.
> > >
> > > Thanks,
> > > Ran Gao
> > >
> > > On 2024/08/21 14:27:25 Yubiao Feng wrote:
> > > > Hi all
> > > >
> > > > I drafted a proposal to add a topic's system prop that indicates
> > whether
> > > > users have published TXN messages in before.
> > > >
> > > > https://github.com/apache/pulsar/pull/23210
> > > >
> > > > I'm looking forward to hearing from you.
> > > >
> > > > Thanks
> > > > Yubiao Feng
> > > >
> >

Re: [DISCUSS] PIP-373: Add a topic's system prop that indicates whether users have published TXN messages in before

[xiangying meng]

Hi Yubiao
If it's just for compatibility, wouldn't it be better to add a global
property? There is no need to add it for every topic.

Thanks

On Fri, Aug 23, 2024 at 5:23 PM Yubiao Feng
 wrote:
>
> Hi Xiangying
>
> > Must we set the system property to
> > false for topics that do not use
> > transactions?
>
> It is needed because the `false` can be used to determine whether the topic
> was created before upgrading to the version that contains the current PIP
> or not, which is useful for compatibility support. see detail at the
> section Compatibility in the PIP. For example:
> - Before upgrading: the topic's property is empty, and it contains TXN
> messages. Brokers should start a TB recovery task for this topic because
> the property is empty.
> - After upgrading: the topics property is `false`, and it does not contain
> TXN messages.
>
> Thanks
> Yubiao Feng
>
> On Fri, Aug 23, 2024 at 4:47 PM xiangying meng  wrote:
>
> > Hi yubiao
> > Then return to your plan, must we set the system property to false for
> > topics that do not use transactions? I understand that it is enough to
> > only set the system properties of the topics that used the
> > transaction.
> >
> > Thanks,
> > Xiangying Meng
> >
> > On Fri, Aug 23, 2024 at 4:16 PM Yubiao Feng
> >  wrote:
> > >
> > > Hi Xiangying
> > >
> > > > Since we can add data to the topic attribute,
> > > > why not directly put the
> > > > snapshot or snapshot segment position into
> > > > the topic position?
> > >
> > > Reason 1:
> > > - The rate of loading the topic is once many days
> > > - The rate of taking Transaction Buffer snapshot is per 5 seconds
> > >
> > > The changes you suggested are to use write ZK per 5 seconds instead of
> > > reading data from BK once many days, it is not better.
> > >
> > > Reason 2:
> > > Once you move snapshots from BK to ZK, we can difficultly guarantee the
> > > compatibility of upgrading and downgrading.
> > >
> > > Thanks
> > > Yubioa Feng
> > >
> > > On Thu, Aug 22, 2024 at 4:52 PM xiangying meng 
> > wrote:
> > >
> > > > Good idea. I still have some questions about this.
> > > >
> > > > Since we can add data to the topic attribute, why not directly put the
> > > > snapshot or snapshot segment position into the topic position?
> > > > This method has two advantages:
> > > > 1. System properties of topics that do not use transactions are not
> > > > added. It will not affect common topics.
> > > > 2. Topics using transactions can be recovered faster.
> > > >
> > > > Of course, the disadvantage is that topics using transactions will
> > > > need to update these properties frequently.
> > > >
> > > > Thanks,
> > > >
> > > > Xiangying
> > > >
> > > > On Thu, Aug 22, 2024 at 10:53 AM Ran Gao  wrote:
> > > > >
> > > > > LGTM
> > > > >
> > > > > Not all topics use the transaction feature, it can save a lot of
> > > > resources to help reduce broker recovery time, and it's convenient to
> > > > calculate topics count that use transactions.
> > > > >
> > > > > Thanks,
> > > > > Ran Gao
> > > > >
> > > > > On 2024/08/21 14:27:25 Yubiao Feng wrote:
> > > > > > Hi all
> > > > > >
> > > > > > I drafted a proposal to add a topic's system prop that indicates
> > > > whether
> > > > > > users have published TXN messages in before.
> > > > > >
> > > > > > https://github.com/apache/pulsar/pull/23210
> > > > > >
> > > > > > I'm looking forward to hearing from you.
> > > > > >
> > > > > > Thanks
> > > > > > Yubiao Feng
> > > > > >
> > > >
> >

Re: [DISCUSS] PIP-373: Add a topic's system prop that indicates whether users have published TXN messages in before

[xiangying meng]

Let me explain this solution further. We can determine whether this
namespace has used transactions through the data of the system topic.
If it has not been used, setting a global property is enough. If the
newly created topics in this namespace use transactions, add a
property with a value of true for the topics that use transactions.

For namespaces that have mixed transactions, your solution is correct.

However, for namespaces that do not use transactions at all, topics
that do not use transactions do not need a property with a value of
false.

Of course, my solution may be ill-considered, and you are welcome to
point it out.

Thanks,
Xiangying Meng

On Fri, Aug 23, 2024 at 6:02 PM xiangying meng  wrote:
>
> Hi Yubiao
> If it's just for compatibility, wouldn't it be better to add a global
> property? There is no need to add it for every topic.
>
> Thanks
>
> On Fri, Aug 23, 2024 at 5:23 PM Yubiao Feng
>  wrote:
> >
> > Hi Xiangying
> >
> > > Must we set the system property to
> > > false for topics that do not use
> > > transactions?
> >
> > It is needed because the `false` can be used to determine whether the topic
> > was created before upgrading to the version that contains the current PIP
> > or not, which is useful for compatibility support. see detail at the
> > section Compatibility in the PIP. For example:
> > - Before upgrading: the topic's property is empty, and it contains TXN
> > messages. Brokers should start a TB recovery task for this topic because
> > the property is empty.
> > - After upgrading: the topics property is `false`, and it does not contain
> > TXN messages.
> >
> > Thanks
> > Yubiao Feng
> >
> > On Fri, Aug 23, 2024 at 4:47 PM xiangying meng  wrote:
> >
> > > Hi yubiao
> > > Then return to your plan, must we set the system property to false for
> > > topics that do not use transactions? I understand that it is enough to
> > > only set the system properties of the topics that used the
> > > transaction.
> > >
> > > Thanks,
> > > Xiangying Meng
> > >
> > > On Fri, Aug 23, 2024 at 4:16 PM Yubiao Feng
> > >  wrote:
> > > >
> > > > Hi Xiangying
> > > >
> > > > > Since we can add data to the topic attribute,
> > > > > why not directly put the
> > > > > snapshot or snapshot segment position into
> > > > > the topic position?
> > > >
> > > > Reason 1:
> > > > - The rate of loading the topic is once many days
> > > > - The rate of taking Transaction Buffer snapshot is per 5 seconds
> > > >
> > > > The changes you suggested are to use write ZK per 5 seconds instead of
> > > > reading data from BK once many days, it is not better.
> > > >
> > > > Reason 2:
> > > > Once you move snapshots from BK to ZK, we can difficultly guarantee the
> > > > compatibility of upgrading and downgrading.
> > > >
> > > > Thanks
> > > > Yubioa Feng
> > > >
> > > > On Thu, Aug 22, 2024 at 4:52 PM xiangying meng 
> > > wrote:
> > > >
> > > > > Good idea. I still have some questions about this.
> > > > >
> > > > > Since we can add data to the topic attribute, why not directly put the
> > > > > snapshot or snapshot segment position into the topic position?
> > > > > This method has two advantages:
> > > > > 1. System properties of topics that do not use transactions are not
> > > > > added. It will not affect common topics.
> > > > > 2. Topics using transactions can be recovered faster.
> > > > >
> > > > > Of course, the disadvantage is that topics using transactions will
> > > > > need to update these properties frequently.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Xiangying
> > > > >
> > > > > On Thu, Aug 22, 2024 at 10:53 AM Ran Gao  wrote:
> > > > > >
> > > > > > LGTM
> > > > > >
> > > > > > Not all topics use the transaction feature, it can save a lot of
> > > > > resources to help reduce broker recovery time, and it's convenient to
> > > > > calculate topics count that use transactions.
> > > > > >
> > > > > > Thanks,
> > > > > > Ran Gao
> > > > > >
> > > > > > On 2024/08/21 14:27:25 Yubiao Feng wrote:
> > > > > > > Hi all
> > > > > > >
> > > > > > > I drafted a proposal to add a topic's system prop that indicates
> > > > > whether
> > > > > > > users have published TXN messages in before.
> > > > > > >
> > > > > > > https://github.com/apache/pulsar/pull/23210
> > > > > > >
> > > > > > > I'm looking forward to hearing from you.
> > > > > > >
> > > > > > > Thanks
> > > > > > > Yubiao Feng
> > > > > > >
> > > > >
> > >

[DISCUSS] Cherry-pick PIP-359 Support custom message listener executor for specific subscription

[xiangying meng]

Hi all

I would like to start a discussion to cherry-pick PIP-359 into `branch-3.0`

This PIP is very useful for scenarios where the same Client starts
multiple subscriptions. It provides a useful custom message listener
to avoid multiple subscriptions from competing for listeners and
blocking each other. There is no change in the default behavior, so we
can cherry-pick it gracefully.

The PIP-370 includes the following PRs
- https://github.com/apache/pulsar/pull/22861

The thread will remain in the open state for 48 hours.
If there is no objection, I will perform the cherry-picking.

Thanks

Xiangying Meng

Re: [DISCUSS] Cherry-pick PIP-359 Support custom message listener executor for specific subscription

[xiangying meng]

Hi yubiao
Oh, sorry. I used your cherry-pick email template. It looks really
professional and is very informative.
Unfortunately, I forgot to change the PIP number somewhere. Thanks for
your reminder and supplement.

Thanks

On Mon, Aug 26, 2024 at 10:47 AM Yubiao Feng
 wrote:
>
> +1
>
>  BTW. there is a mistake in your email content: The "PIP-xxx" includes the
> following PRs
>
> On Mon, Aug 26, 2024 at 10:39 AM xiangying meng 
> wrote:
>
> > Hi all
> >
> > I would like to start a discussion to cherry-pick PIP-359 into `branch-3.0`
> >
> > This PIP is very useful for scenarios where the same Client starts
> > multiple subscriptions. It provides a useful custom message listener
> > to avoid multiple subscriptions from competing for listeners and
> > blocking each other. There is no change in the default behavior, so we
> > can cherry-pick it gracefully.
> >
> > The PIP-370 includes the following PRs
> > - https://github.com/apache/pulsar/pull/22861
> >
> > The thread will remain in the open state for 48 hours.
> > If there is no objection, I will perform the cherry-picking.
> >
> > Thanks
> >
> > Xiangying Meng
> >

Re: [DISCUSS] PIP-375 Expose the Admin client configs: readTimeout, requestTimeout, and connectionTimeout

[xiangying meng]

+1, this would be very useful.

Thanks,
Xiangying Meng

On Mon, Aug 26, 2024 at 6:12 PM Yubiao Feng
 wrote:
>
> Hi all
>
> I drafted a proposal to expose the Admin client configs: `readTimeout`,
> `requestTimeout`, and `connectionTimeout`
>
> https://github.com/apache/pulsar/pull/23222
>
> I'm looking forward to hearing from you.
>
> Thanks
> Yubiao Feng

Re: {DISCUSS] PIP-374: Visibility of messages in receiverQueue for the consumers

[xiangying meng]

Good Work! This interceptor can be useful in many ways.

Thanks

On Mon, Sep 2, 2024 at 10:14 AM Cong Zhao  wrote:
>
> Good work
>
> Thanks
> Cong
>
> On 2024/08/30 04:57:01 "Bhat, Vinay" wrote:
> > Hi All,
> >
> > I drafted a proposal for Visibility of messages in receiverQueue for the 
> > consumers
> >
> > https://github.com/apache/pulsar/pull/23235
> > I'm looking forward to hearing from you.
> >
> > Thanks
> > Vinay
> >

Re: [VOTE] PIP-375 Expose the Admin client configs: readTimeout, requestTimeout, and connectionTimeout

[xiangying meng]

+1 (non-binding)

On Fri, Aug 30, 2024 at 1:14 AM Apurva Telang  wrote:
>
> +1 (non-binding)
>
> On Thu, Aug 29, 2024 at 4:38 AM Lari Hotari  wrote:
>
> > +1 (binding)
> >
> > -Lari
> >
> > On 2024/08/28 04:45:53 Yubiao Feng wrote:
> > > Hi all
> > >
> > > I want to start a vote on PIP-375: expose the Admin client configs:
> > > readTimeout, requestTimeout, and connectionTimeout
> > >
> > > Proposal link: https://github.com/apache/pulsar/pull/23222
> > > Discussion link:
> > > https://lists.apache.org/thread/nwvc1jc29k0hnvtn6qtpdyb8b5xp3ch7
> > >
> > > The vote will stay open for at least 48 hours.
> > >
> > > Thanks
> > > Yubiao Feng
> > >
> >
>
>
> --
> Best regards,
> Apurva Telang.

Re: [DISCUSS] Cherry-pick PIP-359 Support custom message listener executor for specific subscription

[xiangying meng]

Hi all

Cherry-picked.

Thanks

On Mon, Aug 26, 2024 at 10:52 AM xiangying meng  wrote:
>
> Hi yubiao
> Oh, sorry. I used your cherry-pick email template. It looks really
> professional and is very informative.
> Unfortunately, I forgot to change the PIP number somewhere. Thanks for
> your reminder and supplement.
>
> Thanks
>
> On Mon, Aug 26, 2024 at 10:47 AM Yubiao Feng
>  wrote:
> >
> > +1
> >
> >  BTW. there is a mistake in your email content: The "PIP-xxx" includes the
> > following PRs
> >
> > On Mon, Aug 26, 2024 at 10:39 AM xiangying meng 
> > wrote:
> >
> > > Hi all
> > >
> > > I would like to start a discussion to cherry-pick PIP-359 into 
> > > `branch-3.0`
> > >
> > > This PIP is very useful for scenarios where the same Client starts
> > > multiple subscriptions. It provides a useful custom message listener
> > > to avoid multiple subscriptions from competing for listeners and
> > > blocking each other. There is no change in the default behavior, so we
> > > can cherry-pick it gracefully.
> > >
> > > The PIP-370 includes the following PRs
> > > - https://github.com/apache/pulsar/pull/22861
> > >
> > > The thread will remain in the open state for 48 hours.
> > > If there is no objection, I will perform the cherry-picking.
> > >
> > > Thanks
> > >
> > > Xiangying Meng
> > >

Re: [VOTE] PIP-196 Segmented transaction buffer snapshot

[Xiangying Meng]

Hi, community
This proposal has some updates. The latest version of the proposal can be
found here
<https://docs.google.com/document/d/1hBk2nGcj0Os-ULi2q404gCoxIsPleGY8p5H1hqWD5kI/edit#>
.
Feel free to comment on this doc.
Sincerely,
Xiangying

On Wed, Sep 7, 2022 at 4:55 PM Xiangying Meng  wrote:

> Hi, community
> I,d like to start a vote for the PIP-196
> <https://github.com/apache/pulsar/issues/16913> Segmented transaction
> buffer snapshot.
> And the discussion can be found here
> <https://lists.apache.org/thread/bqoy3oz8flvxy7xpmnw81cr4c9sz5vy0>.
>
> Sincerely,
> Xiangying
>
>