[
https://issues.apache.org/jira/browse/KAFKA-6572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16372986#comment-16372986
]
Sharon Lucas edited comment on KAFKA-6572 at 2/22/18 4:03 PM:
--------------------------------------------------------------
[~huxi_2b] No. There were tons of messages whose timestamp was greater than
the timestamp we specified.
was (Author: slucas):
No. There were tons of messages whose timestamp was greater than the timestamp
we specified.
> kafka-consumer-groups does not reset offsets to specified datetime correctly
> ----------------------------------------------------------------------------
>
> Key: KAFKA-6572
> URL: https://issues.apache.org/jira/browse/KAFKA-6572
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 1.0.0
> Reporter: Sharon Lucas
> Priority: Major
>
> With Kafka 1.0.0 we are seeing a problem using the kafka-consumer-groups.sh
> --reset-offsets option to reset offsets to a specific date/time in our
> production environment.
> We first tried to use the kafka-consumer-groups.sh command with the
> --reset-offsets option and with option --to-datetime 2018-02-10T00:00:00.000
> in our staging environment and it worked correctly. Running the following
> command changed it to start processing logs from February 12, 2018 (4 days
> ago) for a topic that had a large lag. We did a dry run to verify before
> running with the --execute option.
> {code:java}
> root@mlpstagemon0101a:/# /opt/kafka/bin/kafka-consumer-groups.sh
> --bootstrap-server NN.NNN.NN.NN:9092 --group logstash-elasticsearch-latest
> --to-datetime 2018-02-12T00:00:00.000-06:00 --reset-offsets --topic
> staging-mon01-rg-elasticsearch --execute{code}
> We stopped the kafka mirrors that process this topic before resetting the
> offsets and started the kafka mirrors after rsetting the offsets. We
> verified that it correctly started processing logs from February 12, 2018.
> Then we tried resetting offsets in a production environment for a topic that
> had a very large lag using option --to-datetime 2018-02-10T00:00:00.000 and
> it did not work as expected. We stopped the kafka mirrors that process this
> topic before resetting the offsets and did a dry run to see what the new
> offsets would be:
> {code:java}
> root@mlplon0401e:# /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server
> NN.N.NNN.NNN:9092 --group mirror-consumer-ams03-geo-earliest --to-datetime
> 2018-02-10T00:00:00.000 --reset-offsets --topic prod_in-ams03-geo-ca_access
> Note: This will not show information about old Zookeeper-based consumers.
> ^@^@^@^@
> TOPIC PARTITION NEW-OFFSET
> prod_in-ams03-geo-ca_access 52 52084147
> prod_in-ams03-geo-ca_access 106 52154199
> prod_in-ams03-geo-ca_access 75 52148673
> prod_in-ams03-geo-ca_access 61 52130753
> prod_in-ams03-geo-ca_access 49 52151667
> prod_in-ams03-geo-ca_access 48 52145233
> prod_in-ams03-geo-ca_access 27 52092805
> prod_in-ams03-geo-ca_access 26 52139644
> prod_in-ams03-geo-ca_access 65 52157504
> prod_in-ams03-geo-ca_access 105 52166289
> prod_in-ams03-geo-ca_access 38 52160464
> prod_in-ams03-geo-ca_access 22 52093451
> prod_in-ams03-geo-ca_access 4 52151660
> prod_in-ams03-geo-ca_access 90 52160296
> prod_in-ams03-geo-ca_access 25 52161691
> prod_in-ams03-geo-ca_access 13 52145828
> prod_in-ams03-geo-ca_access 56 52162867
> prod_in-ams03-geo-ca_access 42 52072094
> prod_in-ams03-geo-ca_access 7 52069496
> prod_in-ams03-geo-ca_access 117 52087078
> prod_in-ams03-geo-ca_access 32 52073732
> prod_in-ams03-geo-ca_access 102 52082022
> prod_in-ams03-geo-ca_access 76 52141018
> prod_in-ams03-geo-ca_access 83 52154542
> prod_in-ams03-geo-ca_access 72 52095051
> prod_in-ams03-geo-ca_access 85 52149907
> prod_in-ams03-geo-ca_access 119 52134435
> prod_in-ams03-geo-ca_access 113 52159340
> prod_in-ams03-geo-ca_access 55 52146597
> prod_in-ams03-geo-ca_access 18 52149079
> prod_in-ams03-geo-ca_access 35 52149058
> prod_in-ams03-geo-ca_access 99 52143277
> prod_in-ams03-geo-ca_access 41 52158872
> prod_in-ams03-geo-ca_access 112 52083901
> prod_in-ams03-geo-ca_access 34 52137932
> prod_in-ams03-geo-ca_access 89 52158135
> prod_in-ams03-geo-ca_access 40 52144442
> prod_in-ams03-geo-ca_access 53 52138400
> prod_in-ams03-geo-ca_access 19 52144966
> prod_in-ams03-geo-ca_access 44 52166404
> prod_in-ams03-geo-ca_access 31 52155685
> prod_in-ams03-geo-ca_access 10 52152151
> prod_in-ams03-geo-ca_access 98 52145378
> prod_in-ams03-geo-ca_access 69 52153436
> prod_in-ams03-geo-ca_access 92 52093455
> prod_in-ams03-geo-ca_access 5 52156448
> prod_in-ams03-geo-ca_access 17 52063697
> prod_in-ams03-geo-ca_access 24 52146410
> prod_in-ams03-geo-ca_access 28 52156955
> prod_in-ams03-geo-ca_access 82 52081194
> prod_in-ams03-geo-ca_access 101 52153592
> prod_in-ams03-geo-ca_access 58 52153765
> prod_in-ams03-geo-ca_access 94 52162827
> prod_in-ams03-geo-ca_access 116 52155307
> prod_in-ams03-geo-ca_access 77 52072906
> prod_in-ams03-geo-ca_access 68 52150283
> prod_in-ams03-geo-ca_access 64 52155580
> prod_in-ams03-geo-ca_access 109 52159915
> prod_in-ams03-geo-ca_access 111 52153547
> prod_in-ams03-geo-ca_access 2 52080434
> prod_in-ams03-geo-ca_access 74 52148060
> prod_in-ams03-geo-ca_access 47 52090187
> prod_in-ams03-geo-ca_access 59 52145161
> prod_in-ams03-geo-ca_access 1 52151228
> prod_in-ams03-geo-ca_access 104 52159932
> prod_in-ams03-geo-ca_access 14 52158366
> prod_in-ams03-geo-ca_access 86 52153030
> prod_in-ams03-geo-ca_access 81 52155507
> prod_in-ams03-geo-ca_access 11 52158931
> prod_in-ams03-geo-ca_access 100 52154646
> prod_in-ams03-geo-ca_access 23 52169713
> prod_in-ams03-geo-ca_access 45 52151320
> prod_in-ams03-geo-ca_access 6 52157025
> prod_in-ams03-geo-ca_access 87 52088913
> prod_in-ams03-geo-ca_access 107 52086878
> prod_in-ams03-geo-ca_access 16 52146481
> prod_in-ams03-geo-ca_access 20 52148093
> prod_in-ams03-geo-ca_access 103 52152437
> prod_in-ams03-geo-ca_access 9 52158998
> prod_in-ams03-geo-ca_access 29 52151112
> prod_in-ams03-geo-ca_access 54 52149226
> prod_in-ams03-geo-ca_access 36 52139520
> prod_in-ams03-geo-ca_access 0 52152902
> prod_in-ams03-geo-ca_access 84 52156057
> prod_in-ams03-geo-ca_access 78 52155766
> prod_in-ams03-geo-ca_access 118 52147043
> prod_in-ams03-geo-ca_access 62 52087864
> prod_in-ams03-geo-ca_access 108 52154615
> prod_in-ams03-geo-ca_access 88 52147651
> prod_in-ams03-geo-ca_access 39 52145255
> prod_in-ams03-geo-ca_access 115 52166466
> prod_in-ams03-geo-ca_access 51 52150136
> prod_in-ams03-geo-ca_access 3 52147544
> prod_in-ams03-geo-ca_access 73 52157150
> prod_in-ams03-geo-ca_access 66 52158808
> prod_in-ams03-geo-ca_access 67 52075616
> prod_in-ams03-geo-ca_access 97 52086900
> prod_in-ams03-geo-ca_access 15 52153225
> prod_in-ams03-geo-ca_access 57 52097899
> prod_in-ams03-geo-ca_access 8 52159097
> prod_in-ams03-geo-ca_access 114 52149412
> prod_in-ams03-geo-ca_access 37 52088584
> prod_in-ams03-geo-ca_access 30 52153155
> prod_in-ams03-geo-ca_access 79 52143773
> prod_in-ams03-geo-ca_access 12 52089423
> prod_in-ams03-geo-ca_access 21 52148187
> prod_in-ams03-geo-ca_access 50 52150414
> prod_in-ams03-geo-ca_access 60 52144624
> prod_in-ams03-geo-ca_access 33 52149807
> prod_in-ams03-geo-ca_access 46 52165008
> prod_in-ams03-geo-ca_access 110 52153643
> prod_in-ams03-geo-ca_access 95 52154736
> prod_in-ams03-geo-ca_access 71 52150981
> prod_in-ams03-geo-ca_access 96 52145091
> prod_in-ams03-geo-ca_access 93 52149375
> prod_in-ams03-geo-ca_access 91 52160207
> prod_in-ams03-geo-ca_access 43 52151689
> prod_in-ams03-geo-ca_access 63 52164909
> prod_in-ams03-geo-ca_access 80 52151810
> prod_in-ams03-geo-ca_access 70 52154522{code}
> Then we ran the same command with the --execute option to the command to
> reset the offsets:
> {code:java}
> root@mlplon0401e:# /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server
> NN.N.NNN.NNN:9092 --group mirror-consumer-ams03-geo-earliest --to-datetime
> 2018-02-10T00:00:00.000 --reset-offsets --topic prod_in-ams03-geo-ca_access
> --execute{code}
> And started the kafka mirrors for this topic.
> However, it didn't work as expected for this topic in our production
> environment.
> The total log lag before resetting the offsets was 1,691,782,349:
> {code:java}
> root@mlbams0301a:/var/log/kafka# while true; do echo -n `date +'%Y-%m-%d
> %H:%M:%S'`; echo -n " "; /opt/kafka/bin/kafka-consumer-groups.sh
> --bootstrap-server NN.N.NNN.NNN:9092 --describe --group
> mirror-consumer-ams03-geo-earliest 2>/dev/null | awk '{if ($1 ~
> "prod_in-ams03-geo-ca_access") { current_offset+=$3; end_offset+=$4; }}
> END{print "lag:", end_offset-current_offset;}'; sleep 15; done
> 2018-02-14 17:31:01 lag: 1691782349{code}
> After resetting the offsets, the log lag dropped to 24,330:
> {code:java}
> 2018-02-14 17:31:17 lag: 24330{code}
> This was too much of a drop in the log lag. This shows it did not really
> reset the offsets to start processing logs starting at Feb 10th (4 days ago).
> We should be seeing a lag of around 37,170,000 if it worked correctly.
> Instead, it seemed to reset the offsets to start processing logs starting
> today (Feb 14th).
> We also tried resetting the offsets to start processing logs starting at
> February 8th (after stopping the mirrors) and it also did not work correctly.
> Again, it seemed like it did a seek to the current date rather than
> resetting offsets to the date we specified.
> {code:java}
> root@mlplon0401e:# /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server
> NN.N.NNN.NNN:9092 --group mirror-consumer-ams03-geo-earliest --to-datetime
> 2018-02-08T00:00:00.000 --reset-offsets --topic prod_in-ams03-geo-ca_access
> Note: This will not show information about old Zookeeper-based consumers.
> ^@^@^@
> TOPIC PARTITION NEW-OFFSET
> prod_in-ams03-geo-ca_access 52 52087106
> prod_in-ams03-geo-ca_access 106 52156813
> prod_in-ams03-geo-ca_access 75 52151755
> prod_in-ams03-geo-ca_access 61 52133608
> prod_in-ams03-geo-ca_access 49 52154385
> prod_in-ams03-geo-ca_access 48 52148247
> prod_in-ams03-geo-ca_access 27 52095874
> prod_in-ams03-geo-ca_access 26 52142331
> prod_in-ams03-geo-ca_access 65 52160181
> prod_in-ams03-geo-ca_access 105 52169166
> prod_in-ams03-geo-ca_access 38 52163198
> prod_in-ams03-geo-ca_access 22 52096231
> prod_in-ams03-geo-ca_access 4 52154512
> prod_in-ams03-geo-ca_access 90 52163215
> prod_in-ams03-geo-ca_access 25 52164527
> prod_in-ams03-geo-ca_access 13 52148553
> prod_in-ams03-geo-ca_access 56 52165558
> prod_in-ams03-geo-ca_access 42 52075104
> prod_in-ams03-geo-ca_access 7 52072333
> prod_in-ams03-geo-ca_access 117 52090029
> prod_in-ams03-geo-ca_access 32 52076469
> prod_in-ams03-geo-ca_access 102 52085066
> prod_in-ams03-geo-ca_access 76 52143820
> prod_in-ams03-geo-ca_access 83 52157217
> prod_in-ams03-geo-ca_access 72 52098060
> prod_in-ams03-geo-ca_access 85 52152612
> prod_in-ams03-geo-ca_access 119 52137225
> prod_in-ams03-geo-ca_access 113 52162012
> prod_in-ams03-geo-ca_access 55 52149399
> prod_in-ams03-geo-ca_access 18 52151998
> prod_in-ams03-geo-ca_access 35 52151756
> prod_in-ams03-geo-ca_access 99 52146244
> prod_in-ams03-geo-ca_access 41 52161567
> prod_in-ams03-geo-ca_access 112 52086712
> prod_in-ams03-geo-ca_access 34 52140841
> prod_in-ams03-geo-ca_access 89 52160809
> prod_in-ams03-geo-ca_access 40 52147177
> prod_in-ams03-geo-ca_access 53 52141117
> prod_in-ams03-geo-ca_access 19 52147752
> prod_in-ams03-geo-ca_access 44 52169047
> prod_in-ams03-geo-ca_access 31 52158455
> prod_in-ams03-geo-ca_access 10 52155007
> prod_in-ams03-geo-ca_access 98 52148061
> prod_in-ams03-geo-ca_access 69 52156414
> prod_in-ams03-geo-ca_access 92 52096106
> prod_in-ams03-geo-ca_access 5 52159142
> prod_in-ams03-geo-ca_access 17 52066397
> prod_in-ams03-geo-ca_access 24 52149433
> prod_in-ams03-geo-ca_access 28 52159909
> prod_in-ams03-geo-ca_access 82 52083925
> prod_in-ams03-geo-ca_access 101 52156350
> prod_in-ams03-geo-ca_access 58 52156500
> prod_in-ams03-geo-ca_access 94 52165527
> prod_in-ams03-geo-ca_access 116 52158005
> prod_in-ams03-geo-ca_access 77 52075690
> prod_in-ams03-geo-ca_access 68 52152975
> prod_in-ams03-geo-ca_access 64 52158277
> prod_in-ams03-geo-ca_access 109 52162696
> prod_in-ams03-geo-ca_access 111 52156535
> prod_in-ams03-geo-ca_access 2 52083131
> prod_in-ams03-geo-ca_access 74 52150817
> prod_in-ams03-geo-ca_access 47 52092867
> prod_in-ams03-geo-ca_access 59 52147851
> prod_in-ams03-geo-ca_access 1 52154047
> prod_in-ams03-geo-ca_access 104 52162658
> prod_in-ams03-geo-ca_access 14 52161087
> prod_in-ams03-geo-ca_access 86 52155828
> prod_in-ams03-geo-ca_access 81 52158537
> prod_in-ams03-geo-ca_access 11 52161597
> prod_in-ams03-geo-ca_access 100 52157311
> prod_in-ams03-geo-ca_access 23 52172470
> prod_in-ams03-geo-ca_access 45 52154257
> prod_in-ams03-geo-ca_access 6 52160103
> prod_in-ams03-geo-ca_access 87 52091756
> prod_in-ams03-geo-ca_access 107 52089608
> prod_in-ams03-geo-ca_access 16 52149265
> prod_in-ams03-geo-ca_access 20 52150732
> prod_in-ams03-geo-ca_access 103 52155276
> prod_in-ams03-geo-ca_access 9 52161987
> prod_in-ams03-geo-ca_access 29 52153796
> prod_in-ams03-geo-ca_access 54 52152245
> prod_in-ams03-geo-ca_access 36 52142445
> prod_in-ams03-geo-ca_access 0 52155910
> prod_in-ams03-geo-ca_access 84 52159052
> prod_in-ams03-geo-ca_access 78 52158632
> prod_in-ams03-geo-ca_access 118 52149746
> prod_in-ams03-geo-ca_access 62 52090560
> prod_in-ams03-geo-ca_access 108 52157511
> prod_in-ams03-geo-ca_access 88 52150632
> prod_in-ams03-geo-ca_access 39 52148150
> prod_in-ams03-geo-ca_access 115 52169203
> prod_in-ams03-geo-ca_access 51 52153200
> prod_in-ams03-geo-ca_access 3 52150514
> prod_in-ams03-geo-ca_access 73 52159766
> prod_in-ams03-geo-ca_access 66 52161783
> prod_in-ams03-geo-ca_access 67 52078371
> prod_in-ams03-geo-ca_access 97 52089805
> prod_in-ams03-geo-ca_access 15 52156319
> prod_in-ams03-geo-ca_access 57 52100913
> prod_in-ams03-geo-ca_access 8 52161732
> prod_in-ams03-geo-ca_access 114 52152390
> prod_in-ams03-geo-ca_access 37 52091579
> prod_in-ams03-geo-ca_access 30 52156230
> prod_in-ams03-geo-ca_access 79 52146601
> prod_in-ams03-geo-ca_access 12 52092444
> prod_in-ams03-geo-ca_access 21 52151103
> prod_in-ams03-geo-ca_access 50 52153126
> prod_in-ams03-geo-ca_access 60 52147546
> prod_in-ams03-geo-ca_access 33 52152862
> prod_in-ams03-geo-ca_access 46 52167625
> prod_in-ams03-geo-ca_access 110 52156369
> prod_in-ams03-geo-ca_access 95 52157493
> prod_in-ams03-geo-ca_access 71 52153659
> prod_in-ams03-geo-ca_access 96 52148024
> prod_in-ams03-geo-ca_access 93 52152429
> prod_in-ams03-geo-ca_access 91 52162944
> prod_in-ams03-geo-ca_access 43 52154651
> prod_in-ams03-geo-ca_access 63 52167783
> prod_in-ams03-geo-ca_access 80 52154510
> prod_in-ams03-geo-ca_access 70 52157348{code}
> If there is an invalid committed offset, could that maybe have caused it to
> not work correctly? Even so, this should not be the result.
> We really want to be able to use this tool to reset offsets to a specified
> date.
> Let me know if you need more information.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
