Hello, Jason
I'd like to attend this project and do some development, could you tell where
shall I start ?
Thanks
Jianjie
On 2020/04/16 23:44:59, Jason Gustafson wrote:
> Hi All,
>
> I'd like to start a discussion on KIP-595:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-595%
Hi, Jason
I'd like to attend to this project and do some development, could you tell
me where shall I started?
Thanks
Jianjie
On 2020/04/16 23:44:59, Jason Gustafson wrote:
> Hi All,
>
> I'd like to start a discussion on KIP-595:
> https://cwiki.apache.org/confluence/display/KAFKA/
Hey Jun,
> Thanks for the response. Should we make clusterId a nullable field
consistently in all new requests?
Yes, makes sense. I updated the proposal.
-Jason
On Fri, Jul 31, 2020 at 4:17 PM Jun Rao wrote:
> Hi, Jason,
>
> Thanks for the response. Should we make clusterId a nullable field
>
Hi, Jason,
Thanks for the response. Should we make clusterId a nullable field
consistently in all new requests?
Jun
On Wed, Jul 29, 2020 at 12:20 PM Jason Gustafson wrote:
> Hey Jun,
>
> I added a section on "Cluster Bootstrapping" which discusses clusterId
> generation and the process through
Hey Jun,
I added a section on "Cluster Bootstrapping" which discusses clusterId
generation and the process through which brokers find the current leader.
The quick summary is that the first controller will be responsible for
generating the clusterId and persisting it in the metadata log. Before th
On Mon, Jul 27, 2020 at 4:58 AM Unmesh Joshi wrote:
> Just checked etcd and zookeeper code, and both support leader to step down
> as a follower to make sure there are no two leaders if the leader has been
> disconnected from the majority of the followers
> For etcd this is https://github.com/etc
Just checked etcd and zookeeper code, and both support leader to step down
as a follower to make sure there are no two leaders if the leader has been
disconnected from the majority of the followers
For etcd this is https://github.com/etcd-io/etcd/issues/3866
For Zookeeper its https://issues.apache.
>>Could you clarify on this question? Which part of the raft group doesn't
>>know about leader dis-connection?
The leader of the controller quorum is partitioned from the controller
cluster, and a different leader is elected for the remaining controller
cluster.
I think there are two things here,
1
Thanks for the questions Unmesh!
On Sun, Jul 26, 2020 at 6:18 AM Unmesh Joshi wrote:
> Hi,
>
> In the FetchRequest Handling, how to make sure we handle scenarios where
> the leader might have been disconnected from the cluster, but doesn't know
> yet?
>
Could you clarify on this question? Which
Hi,
In the FetchRequest Handling, how to make sure we handle scenarios where
the leader might have been disconnected from the cluster, but doesn't know
yet?
As discussed in the Raft Thesis section 6.4, the linearizable semantics of
read requests is implemented in LogCabin by sending heartbeat to f
Hi, Jason,
Thanks for the reply.
101. Sounds good. Regarding clusterId, I am not sure storing it in the
metadata log is enough. For example, the vote request includes clusterId.
So, no one can vote until they know the clusterId. Also, it would be useful
to support the case when a voter completely
Hello Tom,
Thanks for reviewing the doc! Some replies inlined.
On Tue, Jul 14, 2020 at 4:11 AM Tom Bentley wrote:
> Hi Jason, Ghouzang and Boyang,
>
> First, thanks for the KIP. I've not a few minor suggestions and nits, but
> on the whole it was pretty clear and understandable.
>
> 1. § Motiva
Hello Jun,
Thanks for your comments, answering some of them inlined below.
On Thu, Jul 16, 2020 at 3:44 PM Jun Rao wrote:
> Hi, Jason,
>
> Thanks for the updated KIP. Looks good overall. A few more comments below.
>
> 101. I still don't see a section on bootstrapping related issues. It would
>
Hi Jun,
Thanks for the questions.
101. I am treating some of the bootstrapping problems as out of the scope
of this KIP. I am working on a separate proposal which addresses
bootstrapping security credentials specifically. Here is a rough sketch of
how I am seeing it:
1. Dynamic broker configurat
Hi, Jason,
Thanks for the updated KIP. Looks good overall. A few more comments below.
101. I still don't see a section on bootstrapping related issues. It would
be useful to document if/how the following is supported.
101.1 Currently, we support auto broker id generation. Is this supported
for bo
Hi Jason, Ghouzang and Boyang,
First, thanks for the KIP. I've not a few minor suggestions and nits, but
on the whole it was pretty clear and understandable.
1. § Motivation
"We have intentionally avoided any assumption about the representation
of the log and its semantics."
I found this a
Hi All,
Just a quick update on the proposal. We have decided to move quorum
reassignment to a separate KIP:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-642%3A+Dynamic+quorum+reassignment.
The way this ties into cluster bootstrapping is complicated, so we felt we
needed a bit more time fo
@Jun Rao
Regarding your comment about log compaction. After some deep-diving into
this we've decided to propose a new snapshot-based log cleaning mechanism
which would be used to replace the current compaction mechanism for this
meta log. A new KIP will be proposed specifically for this idea.
Al
Thanks for the suggestions Guozhang.
On Thu, Jun 11, 2020 at 2:51 PM Guozhang Wang wrote:
> Hello Boyang,
>
> Thanks for the updated information. A few questions here:
>
> 1) Should the quorum-file also update to support multi-raft?
>
> I'm neutral about this, as we don't know yet how the multi-
Hello Boyang,
Thanks for the updated information. A few questions here:
1) Should the quorum-file also update to support multi-raft?
2) In the previous proposal, there's fields in the FetchQuorumRecords like
latestDirtyOffset, is that dropped intentionally?
3) I think we also need to elaborate
Hey all,
follow-up on the previous email, we made some more updates:
1. The Alter/DescribeQuorum RPCs are also re-structured to use multi-raft.
2. We add observer status into the DescribeQuorumResponse as we see it is a
low hanging fruit which is very useful for user debugging and reassignment.
Hey all,
Thanks for the great discussions so far. I'm posting some KIP updates from
our working group discussion:
1. We will be changing the core RPCs from single-raft API to multi-raft.
This means all protocols will be "batch" in the first version, but the KIP
itself only illustrates the design
Hi, Guozhang and Jason,
Thanks for the reply. A couple of more replies.
102. Still not sure about this. How is the tombstone issue addressed in the
non-voter and the observer. They can die at any point and restart at an
arbitrary later time, and the advancing of the firstDirty offset and the
rem
Hello folks,
I've also updated the KIP wiki page adding a section of the proposed
metrics for this Raft protocol:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-595%3A+A+Raft+Protocol+for+the+Metadata+Quorum#KIP-595:ARaftProtocolfortheMetadataQuorum-Metrics
Please let us know if you have
Thanks Jun. Further replies are in-lined.
On Mon, May 4, 2020 at 11:58 AM Jun Rao wrote:
> Hi, Guozhang,
>
> Thanks for the reply. A few more replies inlined below.
>
> On Sun, May 3, 2020 at 6:33 PM Guozhang Wang wrote:
>
> > Hello Jun,
> >
> > Thanks for your comments! I'm replying inline bel
Hi Jun,
Thanks for the comments. Here are a few quick responses.
> My question was on the auto generated broker id. Currently, the broker can
choose to have its broker Id auto generated. The generation is done through
ZK to guarantee uniqueness. Without ZK, it's not clear how the broker id is
aut
Hi, Guozhang,
Thanks for the reply. A few more replies inlined below.
On Sun, May 3, 2020 at 6:33 PM Guozhang Wang wrote:
> Hello Jun,
>
> Thanks for your comments! I'm replying inline below:
>
> On Fri, May 1, 2020 at 12:36 PM Jun Rao wrote:
>
> > 101. Bootstrapping related issues.
> > 101.1
Hello Jun,
Thanks for your comments! I'm replying inline below:
On Fri, May 1, 2020 at 12:36 PM Jun Rao wrote:
> Hi, Jason,
>
> Thanks for the KIP. Great writeup. A few comments below.
>
> 100. Do we need AlterQuorum in the first version? Quorum changes are rare
> and the implementation is invo
Hi, Jason,
Thanks for the KIP. Great writeup. A few comments below.
100. Do we need AlterQuorum in the first version? Quorum changes are rare
and the implementation is involved. ZK doesn't have that until 4 years
after the initial version. Dropping that in the first version could speed
up this KI
Hello Leonard,
Thanks for your comments, I'm relying in line below:
On Wed, Apr 29, 2020 at 1:58 AM Wang (Leonard) Ge wrote:
> Hi Kafka developers,
>
> It's great to see this proposal and it took me some time to finish reading
> it.
>
> And I have the following questions about the Proposal:
>
>
Hi Kafka developers,
It's great to see this proposal and it took me some time to finish reading
it.
And I have the following questions about the Proposal:
- How do we plan to test this design to ensure its correctness? Or more
broadly, how do we ensure that our new ‘pull’ based model is fu
Hi Jason,
It's amazing to see this coming together :)
I haven't had a chance to read in detail, but I read the outline and a few
things jumped out at me.
First, for every epoch that is 32 bits rather than 64, I sort of wonder if
that's a good long-term choice. I keep reading about stuff like
As promised, here is a link to the current prototype:
https://github.com/confluentinc/kafka/tree/kafka-raft.
Thanks,
Jason
On Mon, Apr 20, 2020 at 10:53 AM Jason Gustafson wrote:
> Hi Deng,
>
> Thanks for the question. I mentioned this in the rejected alternatives
> section. The current proposa
Hi Deng,
Thanks for the question. I mentioned this in the rejected alternatives
section. The current proposal is only for metadata, but I am definitely in
favor of using Raft for partition replication in the long term as well.
There are some interesting tradeoffs in terms of fault tolerance, laten
Big +1 for your initiative and I have a question, we implement the
Raft protocol
just to be used in the management of metadata in Zookeeper or we will also
use it to replace the current logical of managing log-replica since the
algorithm we used to manage log-replica is analogous to Raft.
On Fri,
Hi All,
I'd like to start a discussion on KIP-595:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-595%3A+A+Raft+Protocol+for+the+Metadata+Quorum.
This proposal specifies a Raft protocol to ultimately replace Zookeeper as
documented in KIP-500. Please take a look and share your thoughts.
A
36 matches
Mail list logo
