Any idea why I started receiving mails from this list as of 2:43 am today?
I didn't make any changes or subscribe to anything. I even clicked
unsubscribe earlier today and am still receiving mails.
Maybe there's a misconfiguration in your email list?
Kelly
On Wed, Oct 25, 2017 at 1:44 PM, Hans Jespersen <h...@confluent.io> wrote:
> I think you are just missing the —execute flag.
>
> -hans
>
> > On Oct 25, 2017, at 1:24 PM, Ted Yu <yuzhih...@gmail.com> wrote:
> >
> > I wonder if you have hit KAFKA-5600.
> >
> > Is it possible that you try out 0.11.0.1 ?
> >
> > Thanks
> >
> >> On Wed, Oct 25, 2017 at 1:15 PM, Dan Markhasin <minimi...@gmail.com>
> wrote:
> >>
> >> I am using 0.11.0.0.
> >>
> >> There is no difference configuration-wise - both have 10 partitions and
> 2
> >> replicas. There are no errors in the logs, but looking in the data
> folder
> >> it seems like Kafka is not updating the timeindex file for data1_log -
> >> notice how the timeindex file for the current log segment is not being
> >> updated.
> >>
> >> bash-4.2$ pwd
> >> /kafka/data/data1_log-1
> >> bash-4.2$ ls -ltr | tail
> >> -rw-rw-r-- 1 ibiuser it 1073731573 Oct 25 01:21 00000000000337554984.log
> >> -rw-rw-r-- 1 ibiuser it 943616 Oct 25 01:21
> 00000000000337554984.index
> >> -rw-rw-r-- 1 ibiuser it 1073734199 Oct 25 13:38 00000000000339816017.log
> >> -rw-rw-r-- 1 ibiuser it 10485756 Oct 25 13:38
> >> 00000000000341934289.timeindex
> >> -rw-rw-r-- 1 ibiuser it 10 Oct 25 13:38
> >> 00000000000341934289.snapshot
> >> -rw-rw-r-- 1 ibiuser it 0 Oct 25 13:38
> >> 00000000000339816017.timeindex
> >> -rw-rw-r-- 1 ibiuser it 566712 Oct 25 13:38
> 00000000000339816017.index
> >> -rw-rw-r-- 1 ibiuser it 17 Oct 25 20:23 leader-epoch-checkpoint
> >> -rw-rw-r-- 1 ibiuser it 10485760 Oct 25 23:03
> 00000000000341934289.index
> >> -rw-rw-r-- 1 ibiuser it 461590419 Oct 25 23:04 00000000000341934289.log
> >>
> >> For comparison, the beats topic:
> >>
> >> bash-4.2$ cd ../beats-1
> >> bash-4.2$ ls -ltr
> >> total 3212088
> >> -rw-rw-r-- 1 ibiuser it 17 Oct 25 00:23 leader-epoch-checkpoint
> >> -rw-rw-r-- 1 ibiuser it 10 Oct 25 20:04
> >> 00000000000188672034.snapshot
> >> -rw-rw-r-- 1 ibiuser it 2773008 Oct 25 20:04
> >> 00000000000185224087.timeindex
> >> -rw-rw-r-- 1 ibiuser it 1073741779 Oct 25 20:04 00000000000185224087.log
> >> -rw-rw-r-- 1 ibiuser it 1967440 Oct 25 20:04
> 00000000000185224087.index
> >> -rw-rw-r-- 1 ibiuser it 10485760 Oct 25 23:03
> 00000000000188672034.index
> >> -rw-rw-r-- 1 ibiuser it 10485756 Oct 25 23:04
> >> 00000000000188672034.timeindex
> >> -rw-rw-r-- 1 ibiuser it 50166645 Oct 25 23:04 00000000000188672034.log
> >>
> >>
> >> To give some context to why I'm even trying to reset the offsets, we had
> >> encountered a strange situation earlier today:
> >>
> >> 1) One of the brokers had a hardware failure, and had to be rebuilt from
> >> scratch (data partition was gone)
> >> 2) When it went down, we noticed a spike in lag in one particular
> consumer
> >> group - it seems to have reset its offset to an earlier point in time
> (but
> >> not the earliest offset of the topic); I have read other messages on
> this
> >> mailing list of users who experienced the same behavior with 0.11.0.0
> >> 3) The broker was reinstalled and rejoined the cluster with the same
> >> broker.id (but with no data on it) - it rebalanced and eventually all
> >> replicas became synced and the cluster was functioning normally.
> >> 4) I then decided to bounce the same broker again to see if I can
> reproduce
> >> the issue I saw in #2 - and as soon as the broker was restarted, the
> exact
> >> same consumer group had its offset reset again and was lagging with
> >> millions of records behind the current offset.
> >> 5) I then tried to manually reset the consumer group's offset to a few
> >> minutes before I restarted the broker, only to discover this strange
> >> behavior where no matter which datetime value I provided, it kept
> resetting
> >> to the latest offset.
> >>
> >>
> >>> On 25 October 2017 at 22:48, Ted Yu <yuzhih...@gmail.com> wrote:
> >>>
> >>> Do you mind providing a bit more information ?
> >>>
> >>> Release of Kafka you use
> >>>
> >>> Any difference between data1_log and the other, normal topic ?
> >>>
> >>> Probably check the broker log where data1_log is hosted - see if there
> is
> >>> some clue.
> >>>
> >>> Thanks
> >>>
> >>> On Wed, Oct 25, 2017 at 12:11 PM, Dan Markhasin <minimi...@gmail.com>
> >>> wrote:
> >>>
> >>>> I'm trying to use the kafka-consumer-groups.sh tool in order to rewind
> >> a
> >>>> consumer group's offset, however it seems to be returning the latest
> >>> offset
> >>>> regarding of the requested offset.
> >>>>
> >>>> You can see in the below example that two consecutive commands to
> reset
> >>> the
> >>>> offset to a specific point in time return different (increasing)
> >> offsets,
> >>>> which are actually the latest offsets for the topic.
> >>>>
> >>>> - The consumer group ("test_consumer") is a console consumer that was
> >>>> started with --from-beginning and terminated after a few seconds, just
> >>>> enough for it to commit its offsets.
> >>>> - The topic data1_log is very busy with thousands of incoming messages
> >>> per
> >>>> second
> >>>> - The datetime value provided is approx. 5 hours earlier than the
> >> current
> >>>> UTC time
> >>>>
> >>>> [admin@broker01] ~> /kafka/latest/bin/kafka-consumer-groups.sh
> >>>> --bootstrap-server localhost:9092 --reset-offsets --group
> test_consumer
> >>>> --topic data1_log --to-datetime '2017-10-25T13:40:00.000'
> >>>> Note: This will only show information about consumers that use the
> Java
> >>>> consumer API (non-ZooKeeper-based consumers).
> >>>>
> >>>>
> >>>> TOPIC PARTITION NEW-OFFSET
> >>>> data1_log 8 301485420
> >>>> data1_log 1 342788637
> >>>> data1_log 7 287621428
> >>>> data1_log 3 268612266
> >>>> data1_log 0 201860717
> >>>> data1_log 9 202749553
> >>>> data1_log 4 188974032
> >>>> data1_log 6 234308481
> >>>> data1_log 2 263507741
> >>>> data1_log 5 232707238
> >>>>
> >>>> [admin@broker01] ~> /kafka/latest/bin/kafka-consumer-groups.sh
> >>>> --bootstrap-server localhost:9092 --reset-offsets --group
> test_consumer
> >>>> --topic data1_log --to-datetime '2017-10-25T13:40:00.000'
> >>>> Note: This will only show information about consumers that use the
> Java
> >>>> consumer API (non-ZooKeeper-based consumers).
> >>>>
> >>>>
> >>>> TOPIC PARTITION NEW-OFFSET
> >>>> data1_log 8 301485491
> >>>> data1_log 1 342788779
> >>>> data1_log 7 287621534
> >>>> data1_log 3 268612364
> >>>> data1_log 0 201860796
> >>>> data1_log 9 202749620
> >>>> data1_log 4 188974068
> >>>> data1_log 6 234308564
> >>>> data1_log 2 263507823
> >>>> data1_log 5 232707293
> >>>>
> >>>> This issue seems to be topic-specific - there is a different topic
> >> (also
> >>>> very active) where the same command consistently returns the correct
> >>>> offsets fixed in the time for the requested datetime.
> >>>>
> >>>> What could be the issue here?
> >>>>
> >>>> Thanks,
> >>>> Dan
> >>>>
> >>>
> >>
>
