Looks good overall. Some comments below. 1. The using of ellipsis: This may make passing a list of items from a collection to the api a bit harder. Suppose that you have a list of topics stored in
ArrayList<String> topics; If you want subscribe to all topics in one call, you will have to do: String[] topicArray = new String[topics.size()]; consumer.subscribe(topics.toArray(topicArray)); 2. It would be good to document that the following apis are mutually exclusive. Also, if the partition level subscription is specified, there is no group management. Finally, unsubscribe() can only be used to cancel subscriptions with the same pattern. For example, you can't unsubscribe at the partition level if the subscription is done at the topic level. *subscribe*(java.lang.String... topics) *subscribe*(java.lang.String topic, int... partitions) 3.commit(): The following comment in the doc should probably say "commit offsets for partitions assigned to this consumer". If no partitions are specified, commits offsets for the subscribed list of topics and partitions to Kafka. 4. There is inconsistency in specifying partitions. Sometimes we use TopicPartition and some other times we use String and int (see examples below). void onPartitionsAssigned(Consumer consumer, TopicPartition...partitions) public void *subscribe*(java.lang.String topic, int... partitions) Thanks, Jun On Mon, Feb 10, 2014 at 10:54 AM, Neha Narkhede <neha.narkh...@gmail.com>wrote: > As mentioned in previous emails, we are also working on a re-implementation > of the consumer. I would like to use this email thread to discuss the > details of the public API. I would also like us to be picky about this > public api now so it is as good as possible and we don't need to break it > in the future. > > The best way to get a feel for the API is actually to take a look at the > javadoc< > http://people.apache.org/~nehanarkhede/kafka-0.9-consumer-javadoc/doc/kafka/clients/consumer/KafkaConsumer.html > >, > the hope is to get the api docs good enough so that it is self-explanatory. > You can also take a look at the configs > here< > http://people.apache.org/~nehanarkhede/kafka-0.9-consumer-javadoc/doc/kafka/clients/consumer/ConsumerConfig.html > > > > Some background info on implementation: > > At a high level the primary difference in this consumer is that it removes > the distinction between the "high-level" and "low-level" consumer. The new > consumer API is non blocking and instead of returning a blocking iterator, > the consumer provides a poll() API that returns a list of records. We think > this is better compared to the blocking iterators since it effectively > decouples the threading strategy used for processing messages from the > consumer. It is worth noting that the consumer is entirely single threaded > and runs in the user thread. The advantage is that it can be easily > rewritten in less multi-threading-friendly languages. The consumer batches > data and multiplexes I/O over TCP connections to each of the brokers it > communicates with, for high throughput. The consumer also allows long poll > to reduce the end-to-end message latency for low throughput data. > > The consumer provides a group management facility that supports the concept > of a group with multiple consumer instances (just like the current > consumer). This is done through a custom heartbeat and group management > protocol transparent to the user. At the same time, it allows users the > option to subscribe to a fixed set of partitions and not use group > management at all. The offset management strategy defaults to Kafka based > offset management and the API provides a way for the user to use a > customized offset store to manage the consumer's offsets. > > A key difference in this consumer also is the fact that it does not depend > on zookeeper at all. > > More details about the new consumer design are > here< > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+0.9+Consumer+Rewrite+Design > > > > Please take a look at the new > API< > http://people.apache.org/~nehanarkhede/kafka-0.9-consumer-javadoc/doc/kafka/clients/consumer/KafkaConsumer.html > >and > give us any thoughts you may have. > > Thanks, > Neha >
