Hi Grant, One question that occurred to me is whether we want to take the chance to make it possible to pass a regex pattern for the desired topics in the metadata request. This would potentially improve the efficiency of `KafkaConsumer.subscribe` significantly for cases where there are large number of topics and partitions (we currently get the metadata for _all_ topics and apply the regular expression on the client).
Ismael On Fri, Apr 1, 2016 at 3:29 AM, Grant Henke <ghe...@cloudera.com> wrote: > I have update the wiki and patch based on the feedback on the metadata > changes. Please take a look and let me know if there are any concerns or > issues. > > > - Wiki: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-MetadataSchema > - PR: https://github.com/apache/kafka/pull/1095 > > I will hold a vote on the metadata change soon if no major issues are > raised. > > On Thu, Mar 31, 2016 at 6:33 PM, Jason Gustafson <ja...@confluent.io> > wrote: > > > I also prefer B. In addition to being intuitive, it seems less > error-prone > > in the long term, though it might be a little annoying for clients > > maintaining backwards compatibility. > > > > -Jason > > > > On Thu, Mar 31, 2016 at 1:24 PM, Ismael Juma <ism...@juma.me.uk> wrote: > > > > > I prefer B, the fact that we version the protocol means that we can fix > > > mistakes instead of living with them forever. We should take advantage > of > > > that. > > > > > > Ismael > > > > > > On Thu, Mar 31, 2016 at 9:15 PM, Grant Henke <ghe...@cloudera.com> > > wrote: > > > > > > > Looking for some resolution on the "No Topics" change. > > > > > > > > I am thinking that using null in the protocol isn't that complex, and > > it > > > > avoids various edge cases with having multiple fields. That leaves us > > > with > > > > 2 options: > > > > > > > > - A: null = no topics, empty = all topics > > > > - B: null = all topics, empty = no topics > > > > > > > > A is nice because it just adds new functionality, existing logic > > doesn't > > > > change > > > > B is nice because its more "intuitive", but has the drawback of > > changing > > > > what empty means from request v0 > > > > > > > > I do not have a strong opinion on the approach taken, which makes me > > lean > > > > towards option A. Keep in mind at the user level, the apis in the > > various > > > > clients can map this however they like. > > > > > > > > Does anyone feel strongly about the choice? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Mar 31, 2016 at 9:21 AM, Grant Henke <ghe...@cloudera.com> > > > wrote: > > > > > > > > > I had a second look at the proposed changes to Metadata Request and > > > > >> Response and it seems to me that having a `controller_id` field > > would > > > be > > > > >> more efficient for non-trivial cases than having a `is_controller` > > > field > > > > >> for each broker (which would be false for all but 1 case). > > > > > > > > > > > > > > > I agree this is better. I will update it. > > > > > > > > > > Similar, but less clear is the best way to encode > > `marked_for_deletion` > > > > and > > > > >> `is_internal`. These will also be false for most topics (there is > > only > > > > one > > > > >> internal topic at the moment, for example), so it may make sense > to > > > > have a > > > > >> topics_marked_for_deletion and internal_topics in the response. > > > Because > > > > >> topics are identified by strings, it is not as clear-cut as the > > > > >> controller_id case, but it still seems like it would be a win for > > when > > > > it > > > > >> matters most (when the number of topics is large). > > > > >> > > > > > > > > > > Thats an interesting idea. I can try making this change to see what > > it > > > > > would look like. > > > > > > > > > > Thanks, > > > > > Grant > > > > > > > > > > On Thu, Mar 31, 2016 at 8:59 AM, Ismael Juma <ism...@juma.me.uk> > > > wrote: > > > > > > > > > >> Hi Grant, > > > > >> > > > > >> I had a second look at the proposed changes to Metadata Request > and > > > > >> Response and it seems to me that having a `controller_id` field > > would > > > be > > > > >> more efficient for non-trivial cases than having a `is_controller` > > > field > > > > >> for each broker (which would be false for all but 1 case). > > > > >> > > > > >> Similar, but less clear is the best way to encode > > > `marked_for_deletion` > > > > >> and > > > > >> `is_internal`. These will also be false for most topics (there is > > only > > > > one > > > > >> internal topic at the moment, for example), so it may make sense > to > > > > have a > > > > >> topics_marked_for_deletion and internal_topics in the response. > > > Because > > > > >> topics are identified by strings, it is not as clear-cut as the > > > > >> controller_id case, but it still seems like it would be a win for > > when > > > > it > > > > >> matters most (when the number of topics is large). > > > > > > > > > > > > > > >> Ismael > > > > >> > > > > >> On Mon, Mar 14, 2016 at 10:07 PM, Grant Henke < > ghe...@cloudera.com> > > > > >> wrote: > > > > >> > > > > >> > I have been updating the KIP-4 wiki page based on the last KIP > > call > > > > and > > > > >> > wanted to get some review and discussion around the server side > > > > >> > implementation for admin requests. Both the "ideal" > functionality > > > and > > > > >> the > > > > >> > "intermediated" functionality. The updates are still in > progress, > > > but > > > > >> this > > > > >> > section is the most critical and will likely have the most > > > discussion. > > > > >> This > > > > >> > topic has had a few shifts in perspective and various > discussions > > on > > > > >> > synchronous vs asynchronous server support. The wiki contains my > > > > current > > > > >> > perspective on the challenges and approach. > > > > >> > > > > > >> > If you have any thoughts or feedback on the "Server-side Admin > > > Request > > > > >> > handlers" section here > > > > >> > < > > > > >> > > > > > >> > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-2.Server-sideAdminRequesthandlers > > > > >> > >. > > > > >> > Lets discuss them in this thread. > > > > >> > > > > > >> > For reference the last KIP discussion can be viewed here: > > > > >> > https://youtu.be/rFW0-zJqg5I?t=12m30s > > > > >> > > > > > >> > Thank you, > > > > >> > Grant > > > > >> > -- > > > > >> > Grant Henke > > > > >> > Software Engineer | Cloudera > > > > >> > gr...@cloudera.com | twitter.com/gchenke | > > > linkedin.com/in/granthenke > > > > >> > > > > > >> > > > > > > > > > > > > > > > > > > > > -- > > > > > Grant Henke > > > > > Software Engineer | Cloudera > > > > > gr...@cloudera.com | twitter.com/gchenke | > > linkedin.com/in/granthenke > > > > > > > > > > > > > > > > > > > > > -- > > > > Grant Henke > > > > Software Engineer | Cloudera > > > > gr...@cloudera.com | twitter.com/gchenke | > linkedin.com/in/granthenke > > > > > > > > > > > > > -- > Grant Henke > Software Engineer | Cloudera > gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke >
