Hi Hu Xi, On Mon, Feb 6, 2017, at 19:37, Hu Xi wrote: > Two things I want to confirm. Please advise. > > > 1. Seems the KIP only cares about topic management things. Is there any > plan for this KIP to merge the feature of what `GetOffsetShell` script > offers? Since a lot of people really want to know/monitor how many > committed records have been created for a topic.
Yes, I think we should provide the ability to get offsets for a topic in a follow-on KIP. We are also planning on adding ACL management and AlterTopic functionality, in follow-on work. > > > 2. Since deleting topic is a totally async process, is there any way for > me to make sure the topic is deleted successfully after invoking > deleteTopic once the KIP is implemented? One way would be to periodically list the topics and wait for it to go away. best, Colin > > > Regards, > > > -- huxi > > > ________________________________ > 发件人: radai <radai.rosenbl...@gmail.com> > 发送时间: 2017年2月7日 10:46 > 收件人: dev@kafka.apache.org > 主题: Re: [DISCUSS] KIP-117: Add a public AdministrativeClient API for > Kafka admin operations > > even assuming all consumers use kafka for offset storage, would it be > possible to get this information from a single broker without "reaching > out" to all brokers in a cluster? > > On Mon, Feb 6, 2017 at 2:05 PM, Jianbin Wei <jianbin....@netskope.com> > wrote: > > > In the specify group information, can we also return information like > > partition assignment for each member, the lag/offset of each > > member/partition? It would be useful for Ops/Admin regarding the health of > > the consumer group. > > > > Regards, > > > > -- Jianbin > > > > > On Feb 6, 2017, at 13:54, Guozhang Wang <wangg...@gmail.com> wrote: > > > > > > Some follow-up on 2) / 3) below. > > > > > > On Mon, Feb 6, 2017 at 11:21 AM, Colin McCabe <cmcc...@apache.org > > <mailto:cmcc...@apache.org>> wrote: > > > > > >> On Fri, Feb 3, 2017, at 16:25, Guozhang Wang wrote: > > >>> Thanks for the proposal Colin. A few comments below: > > >> > > >> Thanks for taking a look, Guozhang. > > >> > > >>> > > >>> 1. There are a couple of classes that looks new to me but not defined > > >>> anywhere. For example: NewTopic (topic name and configs?), TopicInfo > > (is > > >>> this a wrapper of MetadataResponse.TopicMetadata?), NodeApiVersions, > > >>> GroupOverview. > > >>> Could you provide their class definitions? > > >> > > >> Good point. I will add them in the KIP. > > >> > > >> NodeApiVersions is at > > >> ./clients/src/main/java/org/apache/kafka/clients/NodeApiVersions.java > > >> > > >>> > > >>> 2. In Streams, we would like to replace its own ` > > >>> org.apache.kafka.streams.processor.internals.StreamsKafkaClient` class > > >>> with > > >>> this new admin client. One additional request though, is that for > > create > > >>> / > > >>> delete topics, we'd like to use a different "flag" as BLOCKING, which > > >>> means > > >>> the response will not be sent back until the controller has updated its > > >>> own > > >>> metadata cache for the topic, and even STRICT_BLOCKING, which means the > > >>> response will not be sent back until the metadata has been propagated > > to > > >>> the whole cluster. > > >> > > >> Hmm. It seems like this would require additional RPCs or changes to > > >> existing RPCs on the server. So we should handle this in a follow-on > > >> KIP, I think. > > >> > > >> > > > I agree for STRICT_BLOCKING, while for BLOCKING, it is already supported > > as > > > of today I think, and Streams' KafkaClient is using that mechanism as > > well. > > > > > > > > >>> > > >>> 3. I'm wondering what's the usage of "public Map<Node, > > >>> Try<List<GroupOverview>>> getAllGroups()", or rather, would it be more > > >>> useful to get a specific group information given the group id? > > Otherwise > > >>> we > > >>> need to query each one of the coordinator. > > >> > > >> That's a good point. We should have an API that gets information about > > >> a specific group, by querying only the coordinator for that group. By > > >> the way, what specific group information should we expose, besides name > > >> and protocolType? > > >> > > >> > > > I think these can all be returned? > > > > > > (groupID, protocolType, generationID, state, members: [memberID, > > > clientHost], leaderID (nullable) ) > > > > > > > > >>> > > >>> 4. I'm +1 with Ismael's suggestion for having the AdminClient interface > > >>> with a KafkaAdminClient impl, this at least allows easier mocks for > > unit > > >>> testing. > > >> > > >> Yeah, I agree. Hopefully that will also make it a little clearer what > > >> the boundary is between the internal functions and classes and the > > >> public API. I'll update the KIP accordingly. > > >> > > >> thanks, > > >> Colin > > >> > > >>> > > >>> Guozhang > > >>> > > >>> > > >>> > > >>> On Fri, Feb 3, 2017 at 10:40 AM, Colin McCabe <cmcc...@apache.org> > > >> wrote: > > >>> > > >>>> On Thu, Feb 2, 2017, at 15:02, Ismael Juma wrote: > > >>>>> Hi Colin, > > >>>>> > > >>>>> Thanks for the KIP, great to make progress on this. I have some > > >> initial > > >>>>> comments, will post more later: > > >>>>> > > >>>>> 1. We have KafkaProducer that implements the Producer interface and > > >>>>> KafkaConsumer that implements the Consumer interface. Maybe we could > > >> have > > >>>>> KafkaAdminClient that implements the AdminClient interface? Or maybe > > >> just > > >>>>> KafkaAdmin. Not sure, some ideas for consideration. Also, I don't > > >> think > > >>>>> we > > >>>>> should worry about a name clash with the internal AdminClient > > >> written in > > >>>>> Scala. That will go away soon enough and choosing a good name for the > > >>>>> public class is what matters. > > >>>> > > >>>> Hi Ismael, > > >>>> > > >>>> Thanks for taking a look. > > >>>> > > >>>> I guess my thought process was that users might find it confusing if > > >> the > > >>>> public API and the old private API had the same name. "What do you > > >>>> mean, I have to upgrade to release X to get AdminClient, I have it > > >> right > > >>>> here?" I do have a slight preference for the shorter name, though, so > > >>>> if this isn't a worry, we can change it to AdminClient. > > >>>> > > >>>>> > > >>>>> 2. We should include the proposed package name in the KIP > > >>>>> (probably org.apache.kafka.clients.admin?). > > >>>> > > >>>> Good idea. I will add the package name to the KIP. > > >>>> > > >>>>> > > >>>>> 3. It would be good to list the supported configs. > > >>>> > > >>>> OK > > >>>> > > >>>>> > > >>>>> 4. KIP-107, which passed the vote, specifies the introduction of a > > >> method > > >>>>> to AdminClient with the following signature. We should figure out > > >> how it > > >>>>> would look given this proposal. > > >>>>> > > >>>>> Future<Map<TopicPartition, PurgeDataResult>> > > >>>>> purgeDataBefore(Map<TopicPartition, Long> offsetForPartition) > > >>>>> > > >>>>> 5. I am not sure about rejecting the Futures-based API. I think I > > >> would > > >>>>> prefer that, personally. Grant had an interesting idea of not > > >> exposing > > >>>>> the > > >>>>> batch methods in the AdminClient to start with to keep it simple and > > >>>>> relying on a Future based API to make it easy for users to run things > > >>>>> concurrently. This is consistent with the producer... > > >>>> > > >>>> So, there are two ways that an operation can be "async" here which are > > >>>> very separate. > > >>>> > > >>>> There is "async on the server." This basically means that we tell the > > >>>> server to do something and don't wait for a confirmation that it > > >>>> succeeded. For example, in the current proposal, users can call > > >>>> createTopic(new Topic(...), CreateTopicFlags.NONBLOCKING). The call > > >>>> will wait for the server to get the request, which will go into > > >>>> purgatory. Later on, the request may succeed or fail, but the client > > >>>> won't know either way. In RPC terms, this means we set the timeout > > >>>> value to 0. > > >>>> > > >>>> Then there is "async on the client." This just means that the client > > >>>> thread doesn't block-- instead, it gets back a Future (or similar > > >>>> object). What this boils down to in terms of implementation is that a > > >>>> message gets put on some queue somewhere and the client thread > > >> continues > > >>>> running. > > >>>> > > >>>> "async on the client" tends to be good when you want to churn out a > > ton > > >>>> of requests without using lots of threads. However, it is more > > >>>> confusing mental model for most programmers. > > >>>> > > >>>> You can easily translate a Futures-based API into a blocking API by > > >>>> having blocking shims that just call create the Future and call get(). > > >>>> Similarly, you can transform a blocking API into a Futures-based API > > by > > >>>> using a thread pool. Thread pools use resources, though, whereas > > >> having > > >>>> function shims does not. > > >>>> > > >>>> I haven't seen any discussion here about what we gain here by using a > > >>>> Futures-based API. It makes sense to use Futures in the Producer, > > >> since > > >>>> they're more flexible, and users are potentially creating lots and > > lots > > >>>> of messages. I'm not sure if users would want to do lots and lots of > > >>>> admin operations with a single thread. I'd be curious to hear a > > little > > >>>> more from potential end-users about what API would be most flexible > > and > > >>>> usable for them. I'm open to ideas. > > >>>> > > >>>> best, > > >>>> Colin > > >>>> > > >>>>> > > >>>>> Ismael > > >>>>> > > >>>>> On Thu, Feb 2, 2017 at 6:54 PM, Colin McCabe <cmcc...@apache.org> > > >> wrote: > > >>>>> > > >>>>>> Hi all, > > >>>>>> > > >>>>>> I wrote up a Kafka improvement proposal for adding an > > >>>>>> AdministrativeClient interface. This is a continuation of the > > >> work on > > >>>>>> KIP-4 towards centralized administrative operations. Please check > > >> out > > >>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > >>>> 117%3A+Add+a+public+ > > >>>>>> AdministrativeClient+API+for+Kafka+admin+operations > > >>>>>> > > >>>>>> regards, > > >>>>>> Colin > > >>>>>> > > >>>> > > >>> > > >>> > > >>> > > >>> -- > > >>> -- Guozhang > > >> > > > > > > > > > > > > -- > > > -- Guozhang > > > >
