Hi Jeff, The java doc is very nice, thank you and thanks to whoever wrote it.
I do have one question about the API. For what we're doing, it's important for us to calculate the "lag" or pending message count. Today we do that using the simple consumer to ask kafka for the committed offset (because we're using kafka to store offsets) and another similar call to kafka for the last offset in the log. It appears that I can get the last committed offset from the KafkaConsumer by calling committed(TopicPartition) which returns the OffsetAndMetadata for the partition. How would I get the offset for the last record in the log? Do I still need to use the simple consumer for this? Thanks, Cliff On Thu, Nov 5, 2015 at 11:58 AM, Jeff Holoman <jholo...@cloudera.com> wrote: > The best thing that I know is the latest javadoc that's committed to trunk: > > > https://github.com/apache/kafka/blob/ef5d168cc8f10ad4f0efe9df4cbe849a4b35496e/clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java > > Thanks > > Jeff > > > > On Thu, Nov 5, 2015 at 12:51 PM, Cliff Rhyne <crh...@signal.co> wrote: > > > Hi Jeff, > > > > Is there a writeup of how to use the new consumer API (either in general > or > > for Java)? I've seen various proposals but I don't see a recent one on > the > > actual implementation. My team wants to start the development work to > > migrate to 0.9. > > > > Thanks, > > Cliff > > > > On Thu, Nov 5, 2015 at 11:18 AM, Jeff Holoman <jholo...@cloudera.com> > > wrote: > > > > > Prabhjot, > > > > > > The answer changes slightly for the Producer and Consumer and depends > on > > > your timeline and comfort with using new APIs > > > > > > Today and in the future, for the Producer, you should be using the > "new" > > > producer, which isn't all that new anymore: > > > org.apache.kafka.clients.producer.KafkaProducer; > > > > > > > > > Today with 0.9 yet to be released you'd likely want to use the > High-Level > > > Consumer. This is covered in the official docs here: > > > > https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example > > > and > > > in this blog post > > > > > > > > > http://ingest.tips/2014/10/12/kafka-high-level-consumer-frequently-missing-pieces/ > > > along > > > with most of the other examples that you'll find. > > > > > > After .9 is released, I'd encourage you to take a look at the new > > Consumer > > > API. This has a lot of advantages in terms of offset management and > will > > be > > > the only consumer client that fully supports security features like SSL > > > that are slated to be released into the platform. > > > > > > Your choice of development language is entirely up to you. Note that > the > > > only version of clients that will be maintained in the project going > > > forward are being implemented in Java, so Scala or Java shouldn't > matter > > > too much for you. > > > > > > Hope this helps > > > > > > Jeff > > > > > > > > > On Thu, Nov 5, 2015 at 12:14 PM, Prabhjot Bharaj < > prabhbha...@gmail.com> > > > wrote: > > > > > > > Hello Folks, > > > > > > > > Requesting your expertise on this. > > > > I see that under core/src/main/scala/kafka/producer/, there are many > > > > implementations - Producer.scala and SyncProducer.scala > > > > > > > > Also, going via the producerPerformance.scala, there are 2 > > > implementations > > > > - NewShinyProducer (which points to KafkaProducer.java) and the > > > OldProducer > > > > > > > > Similar might be the case with Consumers, but I have not seen that > yet. > > > > > > > > Please let me know which producer and consumer is supposed to be used > > and > > > > which ones will be phased out in future releases, so I can focus on > > only > > > 1 > > > > type of producer and consumer (high level as well as simple) > > > > > > > > Thanks, > > > > Prabhjot > > > > > > > > Thanks, > > > > Prabhjot > > > > > > > > On Thu, Nov 5, 2015 at 3:55 PM, Prabhjot Bharaj < > prabhbha...@gmail.com > > > > > > > wrote: > > > > > > > > > Adding users as well > > > > > > > > > > On Thu, Nov 5, 2015 at 3:37 PM, Prabhjot Bharaj < > > prabhbha...@gmail.com > > > > > > > > > wrote: > > > > > > > > > >> Hi, > > > > >> > > > > >> I'm using the latest update: 0.8.2.2 > > > > >> I would like to use the latest producer and consumer apis > > > > >> over the past few weeks, I have tried to do some performance > > > > benchmarking > > > > >> using the producer and consumer scripts provided in the bin > > directory. > > > > It > > > > >> was a fun activity and I have learnt a lot about kafka. > > > > >> > > > > >> But, I have also experienced that sometimes the implementation of > > the > > > > >> performance scripts was not up-to-date or some items were > different > > > than > > > > >> the documentation > > > > >> > > > > >> Now, I would like to develop my application with kafka. I'm > > > comfortable > > > > >> using scala/java > > > > >> > > > > >> Please let me know which producer and consumer (both high level > and > > > > >> simple) class/object should I be using > > > > >> > > > > >> Thanks a lot, > > > > >> Prabhjot > > > > >> > > > > > > > > > > > > > > > > > > > > -- > > > > > --------------------------------------------------------- > > > > > "There are only 10 types of people in the world: Those who > understand > > > > > binary, and those who don't" > > > > > > > > > > > > > > > > > > > > > -- > > > > --------------------------------------------------------- > > > > "There are only 10 types of people in the world: Those who understand > > > > binary, and those who don't" > > > > > > > > > > > > > > > > -- > > > Jeff Holoman > > > Systems Engineer > > > > > > > > > > > -- > > Cliff Rhyne > > Software Engineering Lead > > e: crh...@signal.co > > signal.co > > ________________________ > > > > Cut Through the Noise > > > > This e-mail and any files transmitted with it are for the sole use of the > > intended recipient(s) and may contain confidential and privileged > > information. Any unauthorized use of this email is strictly prohibited. > > ©2015 Signal. All rights reserved. > > > > > > -- > Jeff Holoman > Systems Engineer > -- Cliff Rhyne Software Engineering Lead e: crh...@signal.co signal.co ________________________ Cut Through the Noise This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized use of this email is strictly prohibited. ©2015 Signal. All rights reserved.
