[
https://issues.apache.org/jira/browse/KAFKA-513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Swapnil Ghike updated KAFKA-513:
--------------------------------
Attachment: kafka-513-v1.patch
Patch v1:
1. Created a tool PartitionStateChangeLogMerger
- It looks for a topicAndPartition in the format [%s, %d] and a data in the
format "yyyy-MM-dd HH:mm:ss,SSS".
- Uses a TreeMap to maintain a log for each topicAndPartition sorted according
to date.
- Run it as ./kafka-run-class.sh kafka.tools.PartitionStateChangeLogMerger
--file testfile --file testfile1
2. Added a class PartitionStateChangeLogger.
- It uses a separate logger in log4j properties file and a fileAppender.
- It uses [%d] %p %m%n format.
eg -
[2012-12-12 18:27:46,913] DEBUG preRegister called.
Server=com.sun.jmx.mbeanserver.JmxMBeanServer@70922804,
name=log4j:logger=partitionStateChangeLogger
[2012-12-12 18:27:49,995] INFO [Replica state machine on Controller 0]: Replica
0 for partition [foo, 0] state changed to OnlineReplica
[2012-12-12 18:27:49,995] INFO [Replica state machine on Controller 0]: Replica
0 for partition [too, 0] state changed to OnlineReplica
[2012-12-12 18:27:50,004] DEBUG Controller 0, epoch 4 sending LeaderAndIsr
request to broker 0 for partition [foo, 0]
[2012-12-12 18:27:50,004] DEBUG Controller 0, epoch 4 sending LeaderAndIsr
request to broker 0 for partition [too, 0]
[2012-12-12 18:27:50,048] INFO [Replica Manager on Broker 0]: Received
LeaderAndIsr request from controller 0, epoch 0, becoming Leader for partition
[foo, 0]
[2012-12-12 18:27:50,051] TRACE Partition [foo, 0] on broker 0: Started to
become leader at the request { "ISR":"0","leader":"0","leaderEpoch":"0" }
[2012-12-12 18:27:50,062] INFO [Replica Manager on Broker 0]: Completed the
leader state transition for partition [foo, 0]
[2012-12-12 18:27:50,062] INFO [Replica Manager on Broker 0]: Received
LeaderAndIsr request from controller 0, epoch 0, becoming Leader for partition
[too, 0]
[2012-12-12 18:27:50,062] TRACE Partition [too, 0] on broker 0: Started to
become leader at the request { "ISR":"0","leader":"0","leaderEpoch":"0" }
[2012-12-12 18:27:50,063] INFO [Replica Manager on Broker 0]: Completed the
leader state transition for partition [too, 0]
I am not sure how to prevent it from printing the first statement.
3. A bunch of logging statements use PartitionStateChangeLogger in Partition,
PartitionStateMachine, ReplicaStateMachine, ReplicaManager,
ControllerChannelManager,
4. Included controllerId in LeaderAndIsrRequest via
ControllerBrokerRequestBatch.
5. Modification to two unit tests because of the change in LeaderAndIsrRequest
constructor.
Testing done -
Unit tests, Sanity test, merged two test log files.
What should be the location of the partition state change log file?
> Add state change log to Kafka brokers
> -------------------------------------
>
> Key: KAFKA-513
> URL: https://issues.apache.org/jira/browse/KAFKA-513
> Project: Kafka
> Issue Type: Sub-task
> Affects Versions: 0.8
> Reporter: Neha Narkhede
> Assignee: Swapnil Ghike
> Priority: Blocker
> Labels: replication, tools
> Fix For: 0.8
>
> Attachments: kafka-513-v1.patch
>
> Original Estimate: 96h
> Remaining Estimate: 96h
>
> Once KAFKA-499 is checked in, every controller to broker communication can be
> modelled as a state change for one or more partitions. Every state change
> request will carry the controller epoch. If there is a problem with the state
> of some partitions, it will be good to have a tool that can create a timeline
> of requested and completed state changes. This will require each broker to
> output a state change log that has entries like
> [2012-09-10 10:06:17,280] broker 1 received request LeaderAndIsr() for
> partition [foo, 0] from controller 2, epoch 1
> [2012-09-10 10:06:17,350] broker 1 completed request LeaderAndIsr() for
> partition [foo, 0] from controller 2, epoch 1
> On controller, this will look like -
> [2012-09-10 10:06:17,198] controller 2, epoch 1, initiated state change
> request LeaderAndIsr() for partition [foo, 0]
> We need a tool that can collect the state change log from all brokers and
> create a per-partition timeline of state changes -
> [foo, 0]
> [2012-09-10 10:06:17,198] controller 2, epoch 1 initiated state change
> request LeaderAndIsr()
> [2012-09-10 10:06:17,280] broker 1 received request LeaderAndIsr() from
> controller 2, epoch 1
> [2012-09-10 10:06:17,350] broker 1 completed request LeaderAndIsr() from
> controller 2, epoch 1
> This JIRA involves adding the state change log to each broker and adding the
> tool to create the timeline
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
