[
https://issues.apache.org/jira/browse/KAFKA-513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13590216#comment-13590216
]
Swapnil Ghike edited comment on KAFKA-513 at 3/1/13 3:53 AM:
-------------------------------------------------------------
There are many changes, the non-tool part of the patch is not complete yet. I
will first make some comments on the tool that I have written up and give some
examples below. Let me know if you want to make any changes to the tool's
input/output:
8.1 Yes
8.2 Using dash now.
8.3 If we provide a directory as input, then the question would be whether 1.
the user needs to create this directory 2. the tool looks for certain file names
I think we could avoid these issues by providing two options: 1. comma
separated list of log files 2. log file name regex (the user would be able to
specify full path of the file name regex). Examples below.
8.4 Ok. The merge tool will look in a time interval closed at both the start
time and end time specified.
8.5 Cool. The merge tool accepts exactly one topic in arguments.
i. If only the topic is specified, the tool will look for log entries of all
its partitions.
ii. The tool will also accept a comma separated list of partitions. When this
list is specified, the topic must be specified, and the tool will look for only
those entries for topic and partitions specified.
iii. If no topic is specified in arguments, the merge tool will merge log
entries of all topics.
8.6
i. Implemented n-way merge with output buffer of size 1MB.
ii. The tool merges logs while maintaining chronological order. So if the tool
is used to merge logs of multiple partitions or multiple topics, the output
will only be ordered wrt time, but will be jumbled up wrt partitions or topics.
As discussed offline, in this case, the user is expected to perform additional
screening of the output.
iii. Of course, if the user specifies only one topic and one partition, the
output will not appear jumbled.
Exmaples:
~/kafka/kafka/bin$ ./kafka-run-class.sh kafka.tools.stateChangeLogMerger
--logs-regex ../state-change.log* --topic testfoo --partitions 0 --start-time
"2013-02-28 13:41:35,891" --end-time "2013-02-28 13:41:35,959"
[2013-02-28 13:41:35,891] INFO [Partition state machine on Controller 0]:
Elected leader 0 for Offline partition [testfoo, 0] (partitionStateChangeLogger)
[2013-02-28 13:41:35,892] INFO [Partition state machine on Controller 0]:
Partition [testfoo, 0] state changed from OnlinePartition to OnlinePartition
with leader 0 (partitionStateChangeLogger)
[2013-02-28 13:41:35,900] DEBUG Controller 0, epoch 9 sending LeaderAndIsr
request with correlationId 1to broker 0 for partition [testfoo,0]
(partitionStateChangeLogger)
[2013-02-28 13:41:35,923] INFO [Replica state machine on Controller 0]: Replica
0 for partition [testfoo, 0] state changed to OnlineReplica
(partitionStateChangeLogger)
[2013-02-28 13:41:35,924] DEBUG Controller 0, epoch 9 sending LeaderAndIsr
request with correlationId 2to broker 0 for partition [testfoo,0]
(partitionStateChangeLogger)
[2013-02-28 13:41:35,958] INFO [Replica Manager on Broker 0]: Received
LeaderAndIsr request from controller 0, epoch 2, starting leader state
transition for partition [testfoo, 0] (partitionStateChangeLogger)
[2013-02-28 13:41:35,959] INFO [Replica Manager on Broker 0]: Completed leader
state transition for partition [testfoo, 0] (partitionStateChangeLogger)
~/kafka/kafka/bin$ ./kafka-run-class.sh kafka.tools.stateChangeLogMerger
Provide at least one of the two arguments "[logs]" or "[logs-regex]"
~/kafka/kafka/bin$ ./kafka-run-class.sh kafka.tools.stateChangeLogMerger --logs
../state-change.log --partitions 0,1
"[topic]" required with partition ids
was (Author: swapnilghike):
There are many changes, the non-tool part of the patch is not complete yet.
I will first make some comments on the tool that I have written up and give
some examples below. Let me know if you want to make any changes to the tool's
input/output:
8.1 Yes
8.2 Using dash now.
8.3 If we provide a directory as input, then the question would be whether 1.
the user needs to create this directory 2. the tool looks for certain file names
I think we could avoid these issues by providing two options: 1. comma
separated list of log files 2. log file name regex (the user would be able to
specify full path of the file name regex). Examples below.
8.4 Ok. The merge tool will look in a time interval closed at both the start
time and end time specified.
8.5 Cool. The merge tool accepts exactly one topic in arguments.
i. If only the topic is specified, the tool will look for log entries of all
its partitions.
ii. The tool will also accept a comma separated list of partitions. When this
list is specified, the topic must be specified, and the tool will look for only
those entries for topic and partitions specified.
iii. If no topic is specified in arguments, the merge tool will merge log
entries of all topics.
8.6
i. Implemented n-way merge with output buffer of size 1MB.
ii. The tool merges logs while maintaining chronological order. So if the tool
is used to merge logs of multiple partitions or multiple topics, the output
will only be ordered wrt time, but will be jumbled up wrt partitions or topics.
As discussed offline, in this case, the user is expected to perform additional
screening of the output.
iii. Of course, if the user specifies only one topic and one partition, the
output will not appear jumbled.
Exmaples:
sghike@sghike-ld:~/kafka/kafka/bin$ ./kafka-run-class.sh
kafka.tools.stateChangeLogMerger --logs-regex ../state-change.log* --topic
testfoo --partitions 0 --start-time "2013-02-28 13:41:35,891" --end-time
"2013-02-28 13:41:35,959"
[2013-02-28 13:41:35,891] INFO [Partition state machine on Controller 0]:
Elected leader 0 for Offline partition [testfoo, 0] (partitionStateChangeLogger)
[2013-02-28 13:41:35,892] INFO [Partition state machine on Controller 0]:
Partition [testfoo, 0] state changed from OnlinePartition to OnlinePartition
with leader 0 (partitionStateChangeLogger)
[2013-02-28 13:41:35,900] DEBUG Controller 0, epoch 9 sending LeaderAndIsr
request with correlationId 1to broker 0 for partition [testfoo,0]
(partitionStateChangeLogger)
[2013-02-28 13:41:35,923] INFO [Replica state machine on Controller 0]: Replica
0 for partition [testfoo, 0] state changed to OnlineReplica
(partitionStateChangeLogger)
[2013-02-28 13:41:35,924] DEBUG Controller 0, epoch 9 sending LeaderAndIsr
request with correlationId 2to broker 0 for partition [testfoo,0]
(partitionStateChangeLogger)
[2013-02-28 13:41:35,958] INFO [Replica Manager on Broker 0]: Received
LeaderAndIsr request from controller 0, epoch 2, starting leader state
transition for partition [testfoo, 0] (partitionStateChangeLogger)
[2013-02-28 13:41:35,959] INFO [Replica Manager on Broker 0]: Completed leader
state transition for partition [testfoo, 0] (partitionStateChangeLogger)
sghike@sghike-ld:~/kafka/kafka/bin$ ./kafka-run-class.sh
kafka.tools.stateChangeLogMerger
Provide at least one of the two arguments "[logs]" or "[logs-regex]"
sghike@sghike-ld:~/kafka/kafka/bin$ ./kafka-run-class.sh
kafka.tools.stateChangeLogMerger --logs ../state-change.log --partitions 0,1
"[topic]" required with partition ids
> Add state change log to Kafka brokers
> -------------------------------------
>
> Key: KAFKA-513
> URL: https://issues.apache.org/jira/browse/KAFKA-513
> Project: Kafka
> Issue Type: Sub-task
> Affects Versions: 0.8
> Reporter: Neha Narkhede
> Assignee: Swapnil Ghike
> Priority: Blocker
> Labels: p1, replication, tools
> Fix For: 0.8
>
> Attachments: kafka-513-v1.patch, kafka-513-v2.patch,
> kafka-513-v3.patch
>
> Original Estimate: 96h
> Remaining Estimate: 96h
>
> Once KAFKA-499 is checked in, every controller to broker communication can be
> modelled as a state change for one or more partitions. Every state change
> request will carry the controller epoch. If there is a problem with the state
> of some partitions, it will be good to have a tool that can create a timeline
> of requested and completed state changes. This will require each broker to
> output a state change log that has entries like
> [2012-09-10 10:06:17,280] broker 1 received request LeaderAndIsr() for
> partition [foo, 0] from controller 2, epoch 1
> [2012-09-10 10:06:17,350] broker 1 completed request LeaderAndIsr() for
> partition [foo, 0] from controller 2, epoch 1
> On controller, this will look like -
> [2012-09-10 10:06:17,198] controller 2, epoch 1, initiated state change
> request LeaderAndIsr() for partition [foo, 0]
> We need a tool that can collect the state change log from all brokers and
> create a per-partition timeline of state changes -
> [foo, 0]
> [2012-09-10 10:06:17,198] controller 2, epoch 1 initiated state change
> request LeaderAndIsr()
> [2012-09-10 10:06:17,280] broker 1 received request LeaderAndIsr() from
> controller 2, epoch 1
> [2012-09-10 10:06:17,350] broker 1 completed request LeaderAndIsr() from
> controller 2, epoch 1
> This JIRA involves adding the state change log to each broker and adding the
> tool to create the timeline
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
