Also, a newbie, using 0.9.0.1, I think you meant
auto.offset.reset=earliest, did the OP have an intention to use his own
commit strategy/management by setting enable.auto.commit=false?
With the auto.offset.reset=earliest, a "new" consumer will get the earliest
partition offsets and commit them and start consuming, going forward, they
will use whatever was committed to continue where they left off.
You could also do this programatically as:
consumer.subscribe(Collections.singletonList(KafkaProperties.topic));
consumer.poll(0); // << this will update its current
subscriptions first
// 2. Get these partitions
consumer.seekToBeginning(); // << then seek to the beginning
consumer.poll(0);
Feel free to correct me.
On Tue, Apr 12, 2016 at 10:10 PM, Pradeep Bhattiprolu <pbhatt...@gmail.com>
wrote:
> I tried both the approaches stated above, with no luck :(.
> Let me give concrete examples of what i am trying to achieve here :
>
> 1) Kafka Producer adds multiple JSON messages to a particular topic in the
> message broker (done, this part works)
> 2) I want to have multiple consumers identified under a single consumer
> group to read these messages and perform an index into a search engine. To
> have multiple consumers, I have created a process which spawns multiple
> java threads.
> 3) While creating these threads and adding them to the pool, I select one
> thread as a "leader" thread. The only difference between a leader thread
> and other threads is that the "leader" thread sets the offset to beginning
> by calling Consumer.seekToBeginning() (with no parameters).
> 4) Other threads pick out messages based on the last committed offsets of
> the "leader" thread and other consumer threads.
>
> The idea is to ensure that every thread group of consumers reading a single
> topic should always read from the start of the topic. This is a fair
> assumption to make within my application that each consumer group should
> read from the start.
>
> PS: I am using the new KafkaConsumer class and the latest 0.9.0.0 version
> of the kafka-clients dependency in my project.
>
> Any help / code samples to help me move forward are highly appreciated.
>
> Thanks
> Pradeep
>
> On Sun, Apr 10, 2016 at 2:43 AM, Liquan Pei <liquan...@gmail.com> wrote:
>
> > Hi Predeep,
> >
> > I think I misinterpreted your question. Are you trying to consume a topic
> > multiple times for each consumer instance or consume one topic with
> > multiple consumer instances?
> >
> > In case that you want to consume a topic multiple times with one consumer
> > instance, `seekToBeginning`` will reset the offset to the beginning in
> the
> > next ``poll``.
> >
> > In case that you want each thread to consume the same topic multiple
> times,
> > you need to use multiple consumer groups. Otherwise, only one consumer
> > instance will be will be consuming the topic.
> >
> > Thanks,
> > Liquan
> >
> >
> >
> > On Sun, Apr 10, 2016 at 12:21 AM, Harsha <ka...@harsha.io> wrote:
> >
> > > Pradeep,
> > > How about
> > >
> > >
> >
> https://kafka.apache.org/090/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#seekToBeginning%28org.apache.kafka.common.TopicPartition...%29
> > >
> > > -Harsha
> > >
> > > On Sat, Apr 9, 2016, at 09:48 PM, Pradeep Bhattiprolu wrote:
> > > > Liquan , thanks for the response.
> > > > By setting the auto commit to false do i have to manage queue offset
> > > > manually ?
> > > > I am running a multiple threads with each thread being a consumer, it
> > > > would
> > > > be complicated to manage offsets across threads, if i dont use
> kafka's
> > > > automatic consumer group abstraction.
> > > >
> > > > Thanks
> > > > Pradeep
> > > >
> > > > On Sat, Apr 9, 2016 at 3:12 AM, Liquan Pei <liquan...@gmail.com>
> > wrote:
> > > >
> > > > > Hi Pradeep,
> > > > >
> > > > > Can you try to set enable.auto.commit = false if you want to read
> to
> > > the
> > > > > earliest offset? According to the documentation, auto.offset.reset
> > > controls
> > > > > what to do when there is no initial offset in Kafka or if the
> current
> > > > > offset does not exist any more on the server (e.g. because that
> data
> > > has
> > > > > been deleted). In case that auto commit is enabled, the committed
> > > offset is
> > > > > available in some servers.
> > > > >
> > > > > Thanks,
> > > > > Liquan
> > > > >
> > > > > On Fri, Apr 8, 2016 at 10:44 PM, Pradeep Bhattiprolu <
> > > pbhatt...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi All
> > > > > >
> > > > > > I am a newbie to kafka. I am using the new Consumer API in a
> thread
> > > > > acting
> > > > > > as a consumer for a topic in Kafka.
> > > > > > For my testing and other purposes I have read the queue multiple
> > > times
> > > > > > using console-consumer.sh script of kafka.
> > > > > >
> > > > > > To start reading the message from the beginning in my java code
> , I
> > > have
> > > > > > set the value of the auto.offset.reset to "earliest".
> > > > > >
> > > > > > However that property does not guarantee that i begin reading the
> > > > > messages
> > > > > > from start, it goes by the most recent smallest offset for the
> > > consumer
> > > > > > group.
> > > > > >
> > > > > > Here is my question,
> > > > > > Is there a assured way of starting to read the messages from
> > > beginning
> > > > > from
> > > > > > Java based Kafka Consumer ?
> > > > > > Once I reset one of my consumers to zero, do i have to do offset
> > > > > management
> > > > > > myself for other consumer threads or does kafka automatically
> lower
> > > the
> > > > > > offset to the first threads read offset ?
> > > > > >
> > > > > > Any information / material pointing to the solution are highly
> > > > > appreciated.
> > > > > >
> > > > > > Thanks
> > > > > > Pradeep
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Liquan Pei
> > > > > Software Engineer, Confluent Inc
> > > > >
> > >
> >
> >
> >
> > --
> > Liquan Pei
> > Software Engineer, Confluent Inc
> >
>
--
Radha Krishna, Proddaturi
253-234-5657
