> >
> > > > > Ismael
> > > > >
> > > > > On Tue, Apr 12, 2016 at 2:19 AM, Jun Rao wrote:
> > > > >
> > > > > > Grant,
> > > > > >
> > > > > > Thanks for the upd
m me.
> > > > >
> > > > > Jun
> > > > >
> > > > > On Mon, Apr 11, 2016 at 10:42 AM, Grant Henke >
> > > > wrote:
> > > > >
> > > > > > Based on the discussion in the previous vote thread
&
gt; > On Mon, Apr 11, 2016 at 10:42 AM, Grant Henke
> > > wrote:
> > > >
> > > > > Based on the discussion in the previous vote thread
> > > > > <
> > > > >
> > > >
> > >
> >
> http://search-h
Jun
> > >
> > > On Mon, Apr 11, 2016 at 10:42 AM, Grant Henke
> > wrote:
> > >
> > > > Based on the discussion in the previous vote thread
> > > > <
> > > >
> > >
> >
> http://search-hadoo
> On Mon, Apr 11, 2016 at 10:42 AM, Grant Henke
> wrote:
> >
> > > Based on the discussion in the previous vote thread
> > > <
> > >
> >
> http://search-hadoop.com/m/uyzND1xlaiU10QlYX&subj=+VOTE+KIP+4+Metadata+Schema
> > > >
> >
+1
On Apr 11, 2016 21:55, "Gwen Shapira" wrote:
> +1
>
> On Mon, Apr 11, 2016 at 10:42 AM, Grant Henke wrote:
> > Based on the discussion in the previous vote thread
> > <
> http://search-hadoop.com/m/uyzND1xlaiU10QlYX&subj=+VOTE+KIP+4+Metadata+Sche
+1
On Mon, Apr 11, 2016 at 10:42 AM, Grant Henke wrote:
> Based on the discussion in the previous vote thread
> <http://search-hadoop.com/m/uyzND1xlaiU10QlYX&subj=+VOTE+KIP+4+Metadata+Schema>
> I also would like to include a behavior change to the MetadataResponse. I
>
; <
> >
> http://search-hadoop.com/m/uyzND1xlaiU10QlYX&subj=+VOTE+KIP+4+Metadata+Schema
> > >
> > I also would like to include a behavior change to the MetadataResponse. I
> > have update the wiki
> > <
> >
> https://cwiki.apache.org/co
Grant,
Thanks for the updated version. +1 from me.
Jun
On Mon, Apr 11, 2016 at 10:42 AM, Grant Henke wrote:
> Based on the discussion in the previous vote thread
> <
> http://search-hadoop.com/m/uyzND1xlaiU10QlYX&subj=+VOTE+KIP+4+Metadata+Schema
> >
> I also would l
+1.
On Mon, Apr 11, 2016 at 10:42 AM, Grant Henke wrote:
> Based on the discussion in the previous vote thread
> <
> http://search-hadoop.com/m/uyzND1xlaiU10QlYX&subj=+VOTE+KIP+4+Metadata+Schema
> >
> I also would like to include a behavior change to the MetadataResp
Based on the discussion in the previous vote thread
<http://search-hadoop.com/m/uyzND1xlaiU10QlYX&subj=+VOTE+KIP+4+Metadata+Schema>
I also would like to include a behavior change to the MetadataResponse. I
have update the wiki
<https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-
Jun,
That makes sense. It is important that we have an accurate view of the
metadata in the cluster. Especially if it will be used for the admin tools.
I will make the changes to support this and be backwards compatible with v0
requests and see if it can get into this initial patch. I will post a
Grant,
The limitation with the current MetadataResponse is that if a broker is
down, all replicas on that broker will be missing in the assigned replica
list in the response. Now, imagine that you want to use MetadataRequest to
do a describe of a topic, it's weird that you don't see the full assig
Sounds good.
On Fri, Apr 8, 2016 at 11:37 AM, Grant Henke wrote:
> Guozhang,
>
> I agree there is a gap. Thats what I was trying to say in the last email.
> But I also, don't see a great/safe way to fix it by changing what topics
> are included in the metadata.
>
> Perhaps instead, I can add a s
; > A pull request is available implementing the proposed changes here:
> > https://github.com/apache/kafka/pull/1095
> >
> > Here are some links to past discussions on the mailing list:
> > http://search-hadoop.com/m/uyzND1pd4T52H1m0u1&subj=Re+KIP+4+Wiki+Update
> >
> >
> http://search-hadoop.com/m/uyzND1J2IXeSNXAT&subj=Metadata+and+ACLs+wire+protocol+review+KIP+4+
> >
> > Here is the previous vote discussion (please take a look and discuss
> > there):
> >
> >
> http://search-hadoop.com/m/uyzND1xlaiU10QlYX&subj=+VOTE+KIP+4+Metadata+Schema
> >
> > Thank you,
> > Grant
> > --
> > Grant Henke
> > Software Engineer | Cloudera
> > gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
> >
>
--
-- Guozhang
e:
> https://github.com/apache/kafka/pull/1095
>
> Here are some links to past discussions on the mailing list:
> http://search-hadoop.com/m/uyzND1pd4T52H1m0u1&subj=Re+KIP+4+Wiki+Update
>
> http://search-hadoop.com/m/uyzND1J2IXeSNXAT&subj=Metadata+and+ACLs+wire+protocol+review+KIP+4+
>
> Here is the previous vote discussion (please take a look and discuss
> there):
>
> http://search-hadoop.com/m/uyzND1xlaiU10QlYX&subj=+VOTE+KIP+4+Metadata+Schema
>
> Thank you,
> Grant
> --
> Grant Henke
> Software Engineer | Cloudera
> gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
>
sed changes here:
> https://github.com/apache/kafka/pull/1095
>
> Here are some links to past discussions on the mailing list:
> http://search-hadoop.com/m/uyzND1pd4T52H1m0u1&subj=Re+KIP+4+Wiki+Update
>
> http://search-hadoop.com/m/uyzND1J2IXeSNXAT&subj=Metadata+and+ACLs+wire+protocol+review+KIP+4+
>
> Here is the previous vote discussion (please take a look and discuss
> there):
>
> http://search-hadoop.com/m/uyzND1xlaiU10QlYX&subj=+VOTE+KIP+4+Metadata+Schema
>
> Thank you,
> Grant
> --
> Grant Henke
> Software Engineer | Cloudera
> gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
>
://github.com/apache/kafka/pull/1095
Here are some links to past discussions on the mailing list:
http://search-hadoop.com/m/uyzND1pd4T52H1m0u1&subj=Re+KIP+4+Wiki+Update
http://search-hadoop.com/m/uyzND1J2IXeSNXAT&subj=Metadata+and+ACLs+wire+protocol+review+KIP+4+
Here is the previous vote di
Hi Jun,
I am looking at the changes required for the below request:
5. You will return no error and 4,5,6 as replicas. The response also
> includes a list of live brokers. So the client can figure out 5 is not live
> directly w/o relying on the error code.
The challenge here is that I need to s
Guozhang,
I agree there is a gap. Thats what I was trying to say in the last email.
But I also, don't see a great/safe way to fix it by changing what topics
are included in the metadata.
Perhaps instead, I can add a special error code to the CreateTopic request
to tell the user the topic they are
I feel that "a delete and then create action may fail with a topic exists
exception...which the user could retry until succeeded" has some flaw,
since we cannot distinguish the case 1) the topic not marked for deleted,
but deletion is not complete, from 2) the topic is being created, but
creation i
Thanks for the feedback Guozhang and Gwen.
Gwen, I agree with you on this. I am not sure its something we can/should
tackle here. Especially before the release. I can leave the delete flag off
of the changes.
What that means for KIP-4, is that a client won't be able to differentiate
between a top
Given that we are very close to the release, if we are changing the
Metadata cache + topic deletion logic, I'd like a good number of system
tests to appear with the patch.
On Thu, Apr 7, 2016 at 10:53 AM, Gwen Shapira wrote:
> This will change some logic though, right?
>
> IIRC, right now produc
This will change some logic though, right?
IIRC, right now produce/fetch requests to marked-for-deletion topics fail
because the topics are simple not around. You get a generic "doesn't exist"
error. If we keep these topics and add a flag, we'll need to find all the
places with this implicit logic
Hmm, I think since in the original protocol, metadata response do not have
information for "marked for deleted topics" and hence we want to remove
that topic from returning in response by cleaning the metadata cache once
it is marked to deletion.
With the new format, I think it is OK to delay the
I am testing the marked for deletion flag in the metadata and ran into some
challenges.
It turns out that as soon as a topic is marked for deletion it may be
purged from the metadata cache. This means that Metadata responses
can't/don't return the topic. Though the topic may still exist if its not
5. You will return no error and 4,5,6 as replicas. The response also
includes a list of live brokers. So the client can figure out 5 is not live
directly w/o relying on the error code.
Thanks,
Jun
On Tue, Apr 5, 2016 at 5:05 PM, Grant Henke wrote:
> Hi Jun,
>
> See my responses below:
>
> 2. T
After the discussion today about the clarity and flexibility of the
flag/lists for internal topics and deleted topics, I think I will switch
back to using booleans inside the topic metadata. This is a clearer
representation of the intent, should not have too much overhead (especially
because users
Hi Jun,
See my responses below:
2. The issues that I was thinking are the following. (a) Say the controller
> has topic deletion disabled and a topic deletion request is submitted to
> ZK. In this case, the controller will ignore this request. However, the
> broker may pick up the topic deletion
Grant,
2. The issues that I was thinking are the following. (a) Say the controller
has topic deletion disabled and a topic deletion request is submitted to
ZK. In this case, the controller will ignore this request. However, the
broker may pick up the topic deletion marker in a transient window. (b
Hi Jun,
Please See my responses below:
Hmm, I am not sure about the listener approach. It ignores configs like
> enable.topic.deletion and also opens the door for potential ordering issues
> since now there are two separate paths for propagating the metadata to the
> brokers.
This mechanism is
Grant,
2. Hmm, I am not sure about the listener approach. It ignores configs like
enable.topic.deletion and also opens the door for potential ordering issues
since now there are two separate paths for propagating the metadata to the
brokers. Could we just leave out markedForDeletion for now? In th
Responding to a few of the other comments:
it seems that you propagated
> the topic deletion marker by having the replicaManager read from ZK
> directly. It seems that it would be simpler/consistent if the controller
> propagates that information directly through UpdateMetaRequest.
I was told th
Hi Jun and Ismael,
Initially I had 2 booleans used to indicate if a topic was internal and if
a topic was marked for deletion. To save space on large deployments, Ismael
suggested I break out the internal topics and deleted topics into their own
lists. The idea was that instead of 2 bytes added pe
Hi Jun,
A couple of comments inline.
On Sun, Apr 3, 2016 at 5:55 PM, Jun Rao wrote:
> 1. It seems a bit weird to return just a list of internal topics w/o the
> corresponding metadata. It also seems a bit weird to return the internal
> topics even if the client doesn't ask for it.
Good point.
Grant,
Thanks for the writeup. A few comments.
1. It seems a bit weird to return just a list of internal topics w/o the
corresponding metadata. It also seems a bit weird to return the internal
topics even if the client doesn't ask for it. Would it be better to just
add a flag in topic_metadata to
+1 (non-binding)
On Fri, Apr 1, 2016 at 6:38 PM, Ashish Singh wrote:
> +1 (non-binding)
>
> On Fri, Apr 1, 2016 at 10:16 AM, Gwen Shapira wrote:
>
> > +1
> >
> > On Fri, Apr 1, 2016 at 10:12 AM, Jason Gustafson
> > wrote:
> >
> > > +1 (non-binding)
> > >
> > > On Fri, Apr 1, 2016 at 8:19 AM, G
+1 (non-binding)
On Fri, Apr 1, 2016 at 10:16 AM, Gwen Shapira wrote:
> +1
>
> On Fri, Apr 1, 2016 at 10:12 AM, Jason Gustafson
> wrote:
>
> > +1 (non-binding)
> >
> > On Fri, Apr 1, 2016 at 8:19 AM, Grant Henke wrote:
> >
> > > I would like to start the voting process for the "KIP-4 Metadata
+1
On Fri, Apr 1, 2016 at 10:12 AM, Jason Gustafson wrote:
> +1 (non-binding)
>
> On Fri, Apr 1, 2016 at 8:19 AM, Grant Henke wrote:
>
> > I would like to start the voting process for the "KIP-4 Metadata Schema
> > changes". This is not a vote for all of KIP-4, but specifically for the
> > meta
+1 (non-binding)
On Fri, Apr 1, 2016 at 8:19 AM, Grant Henke wrote:
> I would like to start the voting process for the "KIP-4 Metadata Schema
> changes". This is not a vote for all of KIP-4, but specifically for the
> metadata changes. I have included the exact changes below for clarity:
>
> > M
I would like to start the voting process for the "KIP-4 Metadata Schema
changes". This is not a vote for all of KIP-4, but specifically for the
metadata changes. I have included the exact changes below for clarity:
> Metadata Request (version 1)
>
>
>
> MetadataRequest => [topics]
>
> Stays the sa
41 matches
Mail list logo
