Hi,
The vote is open in this email thread.
https://lists.apache.org/thread/6y8407pw4fv21n2n0cbrvsspg5tok0h7
Regards,
Heesung
On Fri, Nov 4, 2022 at 3:07 PM Heesung Sohn
wrote:
> Hi,
>
> > I would like to understand what the performance tradeoffs are for the
> changes that you (as an individua
Hi,
> I would like to understand what the performance tradeoffs are for the
changes that you (as an individual) and others are proposing.
The problem that this PIP is trying to solve is how to support custom topic
compaction logic. (currently, Pulsar only takes the latest message in the
topic comp
> On Nov 4, 2022, at 10:28 AM, Heesung Sohn
> wrote:
>
> Hi,
>
> I think `bool shouldKeepLeft(T prev, T cur)` is clearer. I updated the PIP.
>
> Hopefully, we provided enough context about this PIP and the design
> trade-off as well.
I would like to understand what the performance tradeoff
Hi,
I think `bool shouldKeepLeft(T prev, T cur)` is clearer. I updated the PIP.
Hopefully, we provided enough context about this PIP and the design
trade-off as well.
I will send out a vote email soon.
Thank you,
Heesung
On Thu, Nov 3, 2022 at 9:59 PM Michael Marshall
wrote:
> Thank you
Thank you for your detailed responses, Heesung.
> We are not planning to expose this feature to users
> soon unless demanded and proven to be stable.
In that case, I think we should move forward with this PIP. I have a
different opinion about the trade offs for the two designs, but none
of my con
Hi,
I have a different thought about my previous comment.
- Agreed with your point that we should merge CompactionStrategy APIs. I
updated the interface proposal in the PIP. I replaced `"isValid",
"isMergeEnabled", and "merge"` apis with "compact" api.
boolean isValid(T prev, T cur)
boolean isMe
Oops! Michael, I apologize for the typo in your name.
On Thu, Nov 3, 2022 at 9:47 AM Heesung Sohn
wrote:
> Hi Machel,
>
> Here are my additional comments regarding your earlier email.
>
> - I updated the PIP title to show that this will impact table view as well.
>
> - PIP-192 : https://github.c
Hi Machel,
Here are my additional comments regarding your earlier email.
- I updated the PIP title to show that this will impact table view as well.
- PIP-192 : https://github.com/apache/pulsar/issues/16691 shows the general
idea of the states and their actions, and I defined the actual states h
Hi,
Thank you for the great comments.
Please find my comments inline too.
Regards,
Heesung
On Mon, Oct 31, 2022 at 10:59 PM Michael Marshall
wrote:
> > I think we lose a single linearized view.
>
> Which linearized view are we losing, and what is the role of that
> linearized view? I think I mi
> I think we lose a single linearized view.
Which linearized view are we losing, and what is the role of that
linearized view? I think I might be missing why it is important. I
agree that consumers won't know about each unsuccessful attempted
acquisition of a bundle, but that seems like unnecessar
Hi,
Also, I thought about some concurrent assignment scenarios between
pre-filter vs post-filter.
Example 1: When the topic broadcast is slower than the concurrent
assignment requests
With pre-filter + two-topics (non-compacted and compacted topics)
t1: A -> non-compacted topic // broker A publi
Hi Michael,
For the pre-prefilter(pre-compaction) option,
I think the leader requires a write-through cache to compact messages based
on the latest states. Otherwise, the leader needs to wait for the last msg
from the (compacted) topic before compacting the next msg for the same
bundle.
Pulsar gu
Sharing some more thoughts. We could alternatively use two topics
instead of one. In this design, the first topic is the unfiltered
write ahead log that represents many writers (brokers) trying to
acquire ownership of bundles. The second topic is the distilled log
that represents the "winners" or t
Thanks for your proposal, Heesung.
Fundamentally, we have the problems listed in this PIP because we have
multiple writers instead of just one writer. Can we solve this problem
by changing our write pattern? What if we use the leader broker as the
single writer? That broker would intercept attempt
+1(non-binding)
thanks,
bo
Heesung Sohn 于2022年10月19日周三 07:54写道：
>
> Hi pulsar-dev community,
>
> I raised a pip to discuss : PIP-215: Configurable Topic Compaction Strategy
>
> PIP link: https://github.com/apache/pulsar/issues/18099
>
> Regards,
> Heesung
Hi Heesung,
Thanks for the explanation
I got your point now.
We don't know when the topic compaction task will be triggered.
If we are not able to write the final decision message to the topic.
We will lose the first message of the key.
It makes sense to me.
+1
Penghui
On Tue, Oct 25, 2022 at 1
Hi,
Regarding the release plan of this strategic compaction, I think we can
take a conservative approach.
First, since this is required for the system topic introduced in PIP-192,
we can make this strategic compaction internal-only(the PIP-192, new broker
load balancer will be the only use-case i
Hi, please find my answers inline.
On Sun, Oct 23, 2022 at 7:11 PM PengHui Li wrote:
> Sorry, heesung,
>
> I think I used a confusing name "leader election".
> Actually, I meant to say "topic owner".
>
> From my understanding, the issue is if we are using the table view for a
> compacted topic.
Sorry, heesung,
I think I used a confusing name "leader election".
Actually, I meant to say "topic owner".
>From my understanding, the issue is if we are using the table view for a
compacted topic.
We will always get the last value of a key. But it will not work for the
"broker ownership conflict
Hi Penghui,
I put my answers inline.
On Thu, Oct 20, 2022 at 5:11 PM PengHui Li wrote:
> Hi Heesung.
>
> Is it possible to send the promoted value to the topic again to achieve
> eventual consistency?
>
Yes, as long as the state change is valid, BSC will accept it and broadcast
it to all broke
Hi Heesung.
Is it possible to send the promoted value to the topic again to achieve
eventual consistency?
For example:
We have 3 brokers, broker-a, broker-b, and broker-c
The message for leader election could be "own: broker-b", "own: broker-c",
"own: broker-a"
The broker-b will win in the end.
Oops.
I forgot to mention another important item. I added it below(in bold).
Pros:
- It supports more distributed load balance operations(bundle assignment)
in a sequentially consistent manner
- For really large clusters, by a partitioned system topic, BSC can be more
scalable than the current sin
Hi,
On Wed, Oct 19, 2022 at 2:06 AM 丛搏 wrote:
> Hi, Heesung:
> I have some doubts.
> I review your PIP-192: New Pulsar Broker Load Balancer. I found that
> unload topic uses the leader broker to do, (Assigning, Return) uses
> the lookup request broker. why (Assigning, Return) not use a leader
>
Hi, Heesung:
I have some doubts.
I review your PIP-192: New Pulsar Broker Load Balancer. I found that
unload topic uses the leader broker to do, (Assigning, Return) uses
the lookup request broker. why (Assigning, Return) not use a leader
broker?
I can think of a few reasons:
1. reduce leader broker
24 matches
Mail list logo
