Re: Get all the partion details from zookeeper for a topic
Hi Neha,
Thanks for your reply.
ZkClient zk = new ZkClient(serverstring );
List partitions = zk.getChildren("/brokers/topics/test-topic/1");
// my topic = test-topic, broker id = 1
This broker is associated with partition 0. So I am expecting output should
be [0] but I am getting empty response [].
Is there anything wrong in the way I am calling it?
Please help.
Thanks
Sourabh
On Wed, Feb 27, 2013 at 11:22 PM, Neha Narkhede wrote:
> Sourabh,
>
> In 0.7, you can use zkclient or the zookeeper client library to query the
> following zookeeper path to get a number of partitions for a topic -
>
> get /brokers/topics/[topic]/[broker-id]
>
> This will give you # of partitions/broker, you can then add up this value
> across all brokers.
>
> Thanks,
> Neha
>
>
> On Wed, Feb 27, 2013 at 4:32 AM, sourabh chaki >wrote:
>
> > Hi All,
> >
> > I am using zookeeper based connector to get data from Kafka Server. Is
> > there any way to get all the partitions from zookeeper for a given topic?
> >
> > I need do do that through java code.
> >
> > Please help!!!
> >
> > Thanks.
> >
>
[jira] [Commented] (KAFKA-739) Handle null values in Message payload
[ https://issues.apache.org/jira/browse/KAFKA-739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13593523#comment-13593523 ] Jay Kreps commented on KAFKA-739: - No, actually it is just git diff against the base revision. > Handle null values in Message payload > - > > Key: KAFKA-739 > URL: https://issues.apache.org/jira/browse/KAFKA-739 > Project: Kafka > Issue Type: Bug >Reporter: Jay Kreps >Assignee: Jay Kreps > Fix For: 0.8.1 > > Attachments: KAFKA-739-v1.patch > > > Add tests for null message payloads in producer, server, and consumer. > Ensure log cleaner treats these as deletes. > Test that null keys are rejected on dedupe logs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-772) System Test Transient Failure on testcase_0122
[ https://issues.apache.org/jira/browse/KAFKA-772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13593540#comment-13593540 ] Neha Narkhede commented on KAFKA-772: - Yeah, probably ok to skip the message > System Test Transient Failure on testcase_0122 > -- > > Key: KAFKA-772 > URL: https://issues.apache.org/jira/browse/KAFKA-772 > Project: Kafka > Issue Type: Bug >Affects Versions: 0.8 >Reporter: John Fung >Assignee: Sriram Subramanian > Labels: kafka-0.8, p1 > Fix For: 0.8 > > Attachments: KAFKA-772.patch, testcase_0122.tar.gz, > testcase_0125.tar.gz > > > * This test case is failing randomly in the past few weeks. Please note there > is a small % data loss allowance for the test case with Ack = 1. But the > failure in this case is the mismatch of log segment checksum across the > replicas. > * Test description: > 3 brokers cluster > Replication factor = 3 > No. topic = 2 > No. partitions = 3 > Controlled failure (kill -15) > Ack = 1 > * Test case output > _test_case_name : testcase_0122 > _test_class_name : ReplicaBasicTest > arg : auto_create_topic : true > arg : bounce_broker : true > arg : broker_type : leader > arg : message_producing_free_time_sec : 15 > arg : num_iteration : 3 > arg : num_partition : 3 > arg : replica_factor : 3 > arg : sleep_seconds_between_producer_calls : 1 > validation_status : > Leader Election Latency - iter 1 brokerid 3 : 377.00 ms > Leader Election Latency - iter 2 brokerid 1 : 374.00 ms > Leader Election Latency - iter 3 brokerid 2 : 384.00 ms > Leader Election Latency MAX : 384.00 > Leader Election Latency MIN : 374.00 > Unique messages from consumer on [test_1] at > simple_consumer_test_1-0_r1.log : 1750 > Unique messages from consumer on [test_1] at > simple_consumer_test_1-0_r2.log : 1750 > Unique messages from consumer on [test_1] at > simple_consumer_test_1-0_r3.log : 1750 > Unique messages from consumer on [test_1] at > simple_consumer_test_1-1_r1.log : 1750 > Unique messages from consumer on [test_1] at > simple_consumer_test_1-1_r2.log : 1750 > Unique messages from consumer on [test_1] at > simple_consumer_test_1-1_r3.log : 1750 > Unique messages from consumer on [test_1] at > simple_consumer_test_1-2_r1.log : 1500 > Unique messages from consumer on [test_1] at > simple_consumer_test_1-2_r2.log : 1500 > Unique messages from consumer on [test_1] at > simple_consumer_test_1-2_r3.log : 1500 > Unique messages from consumer on [test_2] : 5000 > Unique messages from consumer on [test_2] at > simple_consumer_test_2-0_r1.log : 1714 > Unique messages from consumer on [test_2] at > simple_consumer_test_2-0_r2.log : 1714 > Unique messages from consumer on [test_2] at > simple_consumer_test_2-0_r3.log : 1680 > Unique messages from consumer on [test_2] at > simple_consumer_test_2-1_r1.log : 1708 > Unique messages from consumer on [test_2] at > simple_consumer_test_2-1_r2.log : 1708 > Unique messages from consumer on [test_2] at > simple_consumer_test_2-1_r3.log : 1708 > Unique messages from consumer on [test_2] at > simple_consumer_test_2-2_r1.log : 1469 > Unique messages from consumer on [test_2] at > simple_consumer_test_2-2_r2.log : 1469 > Unique messages from consumer on [test_2] at > simple_consumer_test_2-2_r3.log : 1469 > Unique messages from producer on [test_2] : 4900 > Validate for data matched on topic [test_1] across replicas : PASSED > Validate for data matched on topic [test_2] : FAILED > Validate for data matched on topic [test_2] across replicas : FAILED > Validate for merged log segment checksum in cluster [source] : FAILED > Validate leader election successful : PASSED -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Get all the partion details from zookeeper for a topic
The problem is that you need to invoke getData, not getChildren on this
path - /brokers/topics/test-topic/1. That will return the # of partitions
on this broker, which is 1.
Thanks,
Neha
On Tue, Mar 5, 2013 at 6:48 AM, sourabh chaki wrote:
> Hi Neha,
>
> Thanks for your reply.
>
> ZkClient zk = new ZkClient(serverstring );
> List partitions = zk.getChildren("/brokers/topics/test-topic/1");
> // my topic = test-topic, broker id = 1
>
> This broker is associated with partition 0. So I am expecting output should
> be [0] but I am getting empty response [].
>
> Is there anything wrong in the way I am calling it?
> Please help.
>
> Thanks
> Sourabh
>
>
> On Wed, Feb 27, 2013 at 11:22 PM, Neha Narkhede >wrote:
>
> > Sourabh,
> >
> > In 0.7, you can use zkclient or the zookeeper client library to query the
> > following zookeeper path to get a number of partitions for a topic -
> >
> > get /brokers/topics/[topic]/[broker-id]
> >
> > This will give you # of partitions/broker, you can then add up this value
> > across all brokers.
> >
> > Thanks,
> > Neha
> >
> >
> > On Wed, Feb 27, 2013 at 4:32 AM, sourabh chaki > >wrote:
> >
> > > Hi All,
> > >
> > > I am using zookeeper based connector to get data from Kafka Server. Is
> > > there any way to get all the partitions from zookeeper for a given
> topic?
> > >
> > > I need do do that through java code.
> > >
> > > Please help!!!
> > >
> > > Thanks.
> > >
> >
>
[jira] [Resolved] (KAFKA-105) switch to using slf4j
[ https://issues.apache.org/jira/browse/KAFKA-105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao resolved KAFKA-105. --- Resolution: Won't Fix Since we now have the Logger class, which gives us most of the benefits of slf4j api, we can resolve this jira for now. > switch to using slf4j > - > > Key: KAFKA-105 > URL: https://issues.apache.org/jira/browse/KAFKA-105 > Project: Kafka > Issue Type: Improvement > Components: core >Reporter: Chris Burroughs >Assignee: Chris Burroughs > > Breaking out discussion from KAFKA-96. > slf4j has a cleaner (and faster) way of not logging: > http://www.slf4j.org/faq.html#logging_performance that avoid both the > redundant if's and string con-catting. slf4j is just an api, we would still > use log4j for the actual logging in the standalone server. This also makes > it easier on downstream projects who are not using log4j. The downside is > that java logging is notoriously complicated, and now we would have two > logging frameworks that can break. > Discussion from other projects; ZOOKEEPER-850, SOLR-560 HBASE-2608, > CASSANDRA-625, THRIFT-558 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (KAFKA-787) Simple Consumer connecting to Broker that is not the Leader generates wrong error
Chris Curtin created KAFKA-787: -- Summary: Simple Consumer connecting to Broker that is not the Leader generates wrong error Key: KAFKA-787 URL: https://issues.apache.org/jira/browse/KAFKA-787 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8 Reporter: Chris Curtin 0.8.0 HEAD from 3/4/2013 Using a SimpleConsumer, connect to a Broker that is NOT the leader for the topic/partition you want to fetch, but IS a replica. On fetch the error returned is '5' 'ErrorMapping.LeaderNotAvailableCode' Per email thread with Neha: "Ideally, you should get back ErrorMapping.NotLeaderForPartitionCode" Key part of the test is the Broker must be a replica. If the topic/partition is not on the Broker you get a different (correct) error. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (KAFKA-787) Simple Consumer connecting to Broker that is not the Leader generates wrong error
[ https://issues.apache.org/jira/browse/KAFKA-787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede reassigned KAFKA-787: --- Assignee: Neha Narkhede > Simple Consumer connecting to Broker that is not the Leader generates wrong > error > - > > Key: KAFKA-787 > URL: https://issues.apache.org/jira/browse/KAFKA-787 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 0.8 >Reporter: Chris Curtin >Assignee: Neha Narkhede > > 0.8.0 HEAD from 3/4/2013 > Using a SimpleConsumer, connect to a Broker that is NOT the leader for the > topic/partition you want to fetch, but IS a replica. > On fetch the error returned is '5' 'ErrorMapping.LeaderNotAvailableCode' > Per email thread with Neha: > "Ideally, you should get back ErrorMapping.NotLeaderForPartitionCode" > Key part of the test is the Broker must be a replica. If the topic/partition > is not on the Broker you get a different (correct) error. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-787) Simple Consumer connecting to Broker that is not the Leader generates wrong error
[ https://issues.apache.org/jira/browse/KAFKA-787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-787: Labels: kafka-0.8 p2 (was: ) > Simple Consumer connecting to Broker that is not the Leader generates wrong > error > - > > Key: KAFKA-787 > URL: https://issues.apache.org/jira/browse/KAFKA-787 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 0.8 >Reporter: Chris Curtin >Assignee: Neha Narkhede > Labels: kafka-0.8, p2 > > 0.8.0 HEAD from 3/4/2013 > Using a SimpleConsumer, connect to a Broker that is NOT the leader for the > topic/partition you want to fetch, but IS a replica. > On fetch the error returned is '5' 'ErrorMapping.LeaderNotAvailableCode' > Per email thread with Neha: > "Ideally, you should get back ErrorMapping.NotLeaderForPartitionCode" > Key part of the test is the Broker must be a replica. If the topic/partition > is not on the Broker you get a different (correct) error. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-785) Resolve bugs in PreferredReplicaLeaderElection admin tool
[ https://issues.apache.org/jira/browse/KAFKA-785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13593643#comment-13593643 ] Neha Narkhede commented on KAFKA-785: - +1 > Resolve bugs in PreferredReplicaLeaderElection admin tool > - > > Key: KAFKA-785 > URL: https://issues.apache.org/jira/browse/KAFKA-785 > Project: Kafka > Issue Type: Bug >Affects Versions: 0.8 >Reporter: Swapnil Ghike >Assignee: Sriram Subramanian >Priority: Blocker > Labels: kafka-0.8, p2 > Fix For: 0.8 > > Attachments: KAFKA-785.patch > > > 1. Since we run the preferred replica election on all partitions if the > jsonFile is empty, the jsonFileOpt should not be checked in the code as a > required option. > 2. We should not pass "" to Utils.readFileAsString > 3. KAFKA-780 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (KAFKA-785) Resolve bugs in PreferredReplicaLeaderElection admin tool
[ https://issues.apache.org/jira/browse/KAFKA-785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede closed KAFKA-785. --- > Resolve bugs in PreferredReplicaLeaderElection admin tool > - > > Key: KAFKA-785 > URL: https://issues.apache.org/jira/browse/KAFKA-785 > Project: Kafka > Issue Type: Bug >Affects Versions: 0.8 >Reporter: Swapnil Ghike >Assignee: Sriram Subramanian >Priority: Blocker > Labels: kafka-0.8, p2 > Fix For: 0.8 > > Attachments: KAFKA-785.patch > > > 1. Since we run the preferred replica election on all partitions if the > jsonFile is empty, the jsonFileOpt should not be checked in the code as a > required option. > 2. We should not pass "" to Utils.readFileAsString > 3. KAFKA-780 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (KAFKA-785) Resolve bugs in PreferredReplicaLeaderElection admin tool
[ https://issues.apache.org/jira/browse/KAFKA-785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede resolved KAFKA-785. - Resolution: Fixed Checked in. > Resolve bugs in PreferredReplicaLeaderElection admin tool > - > > Key: KAFKA-785 > URL: https://issues.apache.org/jira/browse/KAFKA-785 > Project: Kafka > Issue Type: Bug >Affects Versions: 0.8 >Reporter: Swapnil Ghike >Assignee: Sriram Subramanian >Priority: Blocker > Labels: kafka-0.8, p2 > Fix For: 0.8 > > Attachments: KAFKA-785.patch > > > 1. Since we run the preferred replica election on all partitions if the > jsonFile is empty, the jsonFileOpt should not be checked in the code as a > required option. > 2. We should not pass "" to Utils.readFileAsString > 3. KAFKA-780 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (KAFKA-780) Reassign partitions tool produces NPE in shutdown handler
[ https://issues.apache.org/jira/browse/KAFKA-780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sriram Subramanian closed KAFKA-780. > Reassign partitions tool produces NPE in shutdown handler > - > > Key: KAFKA-780 > URL: https://issues.apache.org/jira/browse/KAFKA-780 > Project: Kafka > Issue Type: Sub-task >Affects Versions: 0.8 >Reporter: Swapnil Ghike >Assignee: Sriram Subramanian >Priority: Blocker > Labels: kafka-0.8, p2 > Fix For: 0.8 > > > sghike@sghike-mn:~/kafka-local/kafka/bin$ ./kafka-reassign-partitions.sh > --zookeeper localhost:2181 --path-to-json-file test1.json > [2013-03-03 03:43:54,513] INFO Starting ZkClient event thread. > (org.I0Itec.zkclient.ZkEventThread) > [2013-03-03 03:43:54,534] INFO Client > environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011 05:47 GMT > (org.apache.zookeeper.ZooKeeper) > [2013-03-03 03:43:54,535] INFO Client environment:host.name=192.168.1.107 > (org.apache.zookeeper.ZooKeeper) > [2013-03-03 03:43:54,535] INFO Client environment:java.version=1.6.0_41 > (org.apache.zookeeper.ZooKeeper) > [2013-03-03 03:43:54,535] INFO Client environment:java.vendor=Apple Inc. > (org.apache.zookeeper.ZooKeeper) > [2013-03-03 03:43:54,535] INFO Client > environment:java.home=/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home > (org.apache.zookeeper.ZooKeeper) > [2013-03-03 03:43:54,535] INFO Client > environment:java.class.path=:/Users/sghike/.ivy2/cache/org.xerial.snappy/snappy-java/bundles/snappy-java-1.0.4.1.jar:/Users/sghike/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.8.0.jar:~/Users/sghike/.ivy2/cache/org.scala-lang/scala-compiler/jars/scala-compiler-2.8.0.jar:/Users/sghike/.ivy2/cache/log4j/log4j/jars/log4j-1.2.15.jar:/Users/sghike/.ivy2/cache/org.slf4j/slf4j-api/jars/slf4j-api-1.6.4.jar:/Users/sghike/.ivy2/cache/org.apache.zookeeper/zookeeper/jars/zookeeper-3.3.4.jar:/Users/sghike/.ivy2/cache/net.sf.jopt-simple/jopt-simple/jars/jopt-simple-3.2.jar:./../core/target/scala-2.8.0/kafka_2.8.0-0.8-SNAPSHOT.jar:./../core/lib/metrics-annotation-3.0.0-c0c8be71.jar:./../core/lib/metrics-core-3.0.0-c0c8be71.jar:./../core/lib/zkclient-20120522.jar:./../perf/target/scala-2.8.0/kafka-perf_2.8.0-0.8-SNAPSHOT.jar > (org.apache.zookeeper.ZooKeeper) > [2013-03-03 03:43:54,535] INFO Client > environment:java.library.path=.:/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java > (org.apache.zookeeper.ZooKeeper) > [2013-03-03 03:43:54,535] INFO Client > environment:java.io.tmpdir=/var/folders/py/4syf5w4d1lg9h4byyrr4pm1mgp/T/ > (org.apache.zookeeper.ZooKeeper) > [2013-03-03 03:43:54,535] INFO Client environment:java.compiler= > (org.apache.zookeeper.ZooKeeper) > [2013-03-03 03:43:54,535] INFO Client environment:os.name=Mac OS X > (org.apache.zookeeper.ZooKeeper) > [2013-03-03 03:43:54,535] INFO Client environment:os.arch=x86_64 > (org.apache.zookeeper.ZooKeeper) > [2013-03-03 03:43:54,535] INFO Client environment:os.version=10.7.5 > (org.apache.zookeeper.ZooKeeper) > [2013-03-03 03:43:54,535] INFO Client environment:user.name=sghike > (org.apache.zookeeper.ZooKeeper) > [2013-03-03 03:43:54,535] INFO Client environment:user.home=/Users/sghike > (org.apache.zookeeper.ZooKeeper) > [2013-03-03 03:43:54,536] INFO Client > environment:user.dir=/Users/sghike/kafka-local/kafka/bin > (org.apache.zookeeper.ZooKeeper) > [2013-03-03 03:43:54,536] INFO Initiating client connection, > connectString=localhost:2181 sessionTimeout=3 > watcher=org.I0Itec.zkclient.ZkClient@67da (org.apache.zookeeper.ZooKeeper) > [2013-03-03 03:43:54,550] INFO Opening socket connection to server > localhost/0:0:0:0:0:0:0:1:2181 (org.apache.zookeeper.ClientCnxn) > [2013-03-03 03:43:54,558] INFO Socket connection established to > localhost/0:0:0:0:0:0:0:1:2181, initiating session > (org.apache.zookeeper.ClientCnxn) > [2013-03-03 03:43:54,766] INFO Session establishment complete on server > localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x13d3009f743, negotiated > timeout = 3 (org.apache.zookeeper.ClientCnxn) > [2013-03-03 03:43:54,768] INFO zookeeper state changed (SyncConnected) > (org.I0Itec.zkclient.ZkClient) > Successfully started reassignment of partitions Map([foo,0] -> > WrappedArray(0)) > [2013-03-03 03:43:54,847] INFO Terminate ZkClient event thread. > (org.I0Itec.zkclient.ZkEventThread) > [2013-03-03 03:43:54,848] INFO Session: 0x13d3009f743 closed > (org.apache.zookeeper.ZooKeeper) > [2013-03-03 03:43:54,848] INFO EventThread shut down > (org.apache.zookeeper.ClientCnxn) > Exception in thread "Thread-3" java.lang.NullPointerException > at org.I0Itec.zkclient.ZkClient$2.call(ZkClient.java:416) > at org.I0Itec.zkclient.ZkClient$2.call(ZkClient.java:413
[jira] [Resolved] (KAFKA-780) Reassign partitions tool produces NPE in shutdown handler
[ https://issues.apache.org/jira/browse/KAFKA-780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sriram Subramanian resolved KAFKA-780. -- Resolution: Fixed Resolved by KAFKA-785 > Reassign partitions tool produces NPE in shutdown handler > - > > Key: KAFKA-780 > URL: https://issues.apache.org/jira/browse/KAFKA-780 > Project: Kafka > Issue Type: Sub-task >Affects Versions: 0.8 >Reporter: Swapnil Ghike >Assignee: Sriram Subramanian >Priority: Blocker > Labels: kafka-0.8, p2 > Fix For: 0.8 > > > sghike@sghike-mn:~/kafka-local/kafka/bin$ ./kafka-reassign-partitions.sh > --zookeeper localhost:2181 --path-to-json-file test1.json > [2013-03-03 03:43:54,513] INFO Starting ZkClient event thread. > (org.I0Itec.zkclient.ZkEventThread) > [2013-03-03 03:43:54,534] INFO Client > environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011 05:47 GMT > (org.apache.zookeeper.ZooKeeper) > [2013-03-03 03:43:54,535] INFO Client environment:host.name=192.168.1.107 > (org.apache.zookeeper.ZooKeeper) > [2013-03-03 03:43:54,535] INFO Client environment:java.version=1.6.0_41 > (org.apache.zookeeper.ZooKeeper) > [2013-03-03 03:43:54,535] INFO Client environment:java.vendor=Apple Inc. > (org.apache.zookeeper.ZooKeeper) > [2013-03-03 03:43:54,535] INFO Client > environment:java.home=/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home > (org.apache.zookeeper.ZooKeeper) > [2013-03-03 03:43:54,535] INFO Client > environment:java.class.path=:/Users/sghike/.ivy2/cache/org.xerial.snappy/snappy-java/bundles/snappy-java-1.0.4.1.jar:/Users/sghike/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.8.0.jar:~/Users/sghike/.ivy2/cache/org.scala-lang/scala-compiler/jars/scala-compiler-2.8.0.jar:/Users/sghike/.ivy2/cache/log4j/log4j/jars/log4j-1.2.15.jar:/Users/sghike/.ivy2/cache/org.slf4j/slf4j-api/jars/slf4j-api-1.6.4.jar:/Users/sghike/.ivy2/cache/org.apache.zookeeper/zookeeper/jars/zookeeper-3.3.4.jar:/Users/sghike/.ivy2/cache/net.sf.jopt-simple/jopt-simple/jars/jopt-simple-3.2.jar:./../core/target/scala-2.8.0/kafka_2.8.0-0.8-SNAPSHOT.jar:./../core/lib/metrics-annotation-3.0.0-c0c8be71.jar:./../core/lib/metrics-core-3.0.0-c0c8be71.jar:./../core/lib/zkclient-20120522.jar:./../perf/target/scala-2.8.0/kafka-perf_2.8.0-0.8-SNAPSHOT.jar > (org.apache.zookeeper.ZooKeeper) > [2013-03-03 03:43:54,535] INFO Client > environment:java.library.path=.:/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java > (org.apache.zookeeper.ZooKeeper) > [2013-03-03 03:43:54,535] INFO Client > environment:java.io.tmpdir=/var/folders/py/4syf5w4d1lg9h4byyrr4pm1mgp/T/ > (org.apache.zookeeper.ZooKeeper) > [2013-03-03 03:43:54,535] INFO Client environment:java.compiler= > (org.apache.zookeeper.ZooKeeper) > [2013-03-03 03:43:54,535] INFO Client environment:os.name=Mac OS X > (org.apache.zookeeper.ZooKeeper) > [2013-03-03 03:43:54,535] INFO Client environment:os.arch=x86_64 > (org.apache.zookeeper.ZooKeeper) > [2013-03-03 03:43:54,535] INFO Client environment:os.version=10.7.5 > (org.apache.zookeeper.ZooKeeper) > [2013-03-03 03:43:54,535] INFO Client environment:user.name=sghike > (org.apache.zookeeper.ZooKeeper) > [2013-03-03 03:43:54,535] INFO Client environment:user.home=/Users/sghike > (org.apache.zookeeper.ZooKeeper) > [2013-03-03 03:43:54,536] INFO Client > environment:user.dir=/Users/sghike/kafka-local/kafka/bin > (org.apache.zookeeper.ZooKeeper) > [2013-03-03 03:43:54,536] INFO Initiating client connection, > connectString=localhost:2181 sessionTimeout=3 > watcher=org.I0Itec.zkclient.ZkClient@67da (org.apache.zookeeper.ZooKeeper) > [2013-03-03 03:43:54,550] INFO Opening socket connection to server > localhost/0:0:0:0:0:0:0:1:2181 (org.apache.zookeeper.ClientCnxn) > [2013-03-03 03:43:54,558] INFO Socket connection established to > localhost/0:0:0:0:0:0:0:1:2181, initiating session > (org.apache.zookeeper.ClientCnxn) > [2013-03-03 03:43:54,766] INFO Session establishment complete on server > localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x13d3009f743, negotiated > timeout = 3 (org.apache.zookeeper.ClientCnxn) > [2013-03-03 03:43:54,768] INFO zookeeper state changed (SyncConnected) > (org.I0Itec.zkclient.ZkClient) > Successfully started reassignment of partitions Map([foo,0] -> > WrappedArray(0)) > [2013-03-03 03:43:54,847] INFO Terminate ZkClient event thread. > (org.I0Itec.zkclient.ZkEventThread) > [2013-03-03 03:43:54,848] INFO Session: 0x13d3009f743 closed > (org.apache.zookeeper.ZooKeeper) > [2013-03-03 03:43:54,848] INFO EventThread shut down > (org.apache.zookeeper.ClientCnxn) > Exception in thread "Thread-3" java.lang.NullPointerException > at org.I0Itec.zkclient.ZkClient$2.call(ZkClient.java:416) >
[jira] [Commented] (KAFKA-496) high level producer send should return a response
[ https://issues.apache.org/jira/browse/KAFKA-496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13593775#comment-13593775 ] Matan Safriel commented on KAFKA-496: - I spent some time trying to first-iteration-refactor the producer's sending activity, as a basis for playing around with adding some persistence to it I'll probably wait for this item here (KAFKA-496) to be implemented first. I'm writing here just to add that it would potentially be nice if the producer's wrappers around sending messages would become simplified in the code through some behavior-maintaining refactoring. The producer internals around managing the queue and around sync/async flows can probably be made much simpler in terms of class and method relationships (or the relationships between producer and producer.async), as part of modifications implied on the previous posts above. I'm writing this as it may seem that KAFKA-496 here may take care of refactoring in this area anyway. It seems this can help future modifications... > high level producer send should return a response > - > > Key: KAFKA-496 > URL: https://issues.apache.org/jira/browse/KAFKA-496 > Project: Kafka > Issue Type: Bug > Components: core >Reporter: Jun Rao >Assignee: Jay Kreps >Priority: Blocker > Labels: features > Fix For: 0.8.1 > > Original Estimate: 72h > Remaining Estimate: 72h > > Currently, Producer.send() doesn't return any value. In 0.8, since each > produce request will be acked, we should pass the response back. What we can > do is that if the producer is in sync mode, we can return a map of > (topic,partitionId) -> (errorcode, offset). If the producer is in async mode, > we can just return a null. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (KAFKA-788) Periodic refresh of topic metadata on the producer doesn't include all topics
Neha Narkhede created KAFKA-788: --- Summary: Periodic refresh of topic metadata on the producer doesn't include all topics Key: KAFKA-788 URL: https://issues.apache.org/jira/browse/KAFKA-788 Project: Kafka Issue Type: Bug Components: producer Affects Versions: 0.8 Reporter: Neha Narkhede Assignee: Jun Rao We added a patch to the producer to refresh the metadata for all topics periodically. However, the producer only does this for the topics in the last batch. But some topics sent by the producer could be low throughput and might not be present in every batch. If we bounce the cluster or if brokers fail and leaders change, the metadata for those low throughput topic is not refreshed by this periodic topic metadata request. The next produce request for those topics have to fail and then a separate metadata request needs to be reissued to handle the produce request. This is especially a problem for the migration tool. So even if the producer had a chance to refresh the metadata when the leader changed, it throws LeaderNotAvailableExceptions much later when it sends a request for that topic. I propose we just fetch data for all topics sent by the producer in the periodic refresh of topic metadata -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (KAFKA-788) Periodic refresh of topic metadata on the producer doesn't include all topics
[ https://issues.apache.org/jira/browse/KAFKA-788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede reassigned KAFKA-788: --- Assignee: Neha Narkhede (was: Jun Rao) > Periodic refresh of topic metadata on the producer doesn't include all topics > - > > Key: KAFKA-788 > URL: https://issues.apache.org/jira/browse/KAFKA-788 > Project: Kafka > Issue Type: Bug > Components: producer >Affects Versions: 0.8 >Reporter: Neha Narkhede >Assignee: Neha Narkhede > Labels: kafka-0.8, p2 > Original Estimate: 24h > Remaining Estimate: 24h > > We added a patch to the producer to refresh the metadata for all topics > periodically. However, the producer only does this for the topics in the last > batch. But some topics sent by the producer could be low throughput and might > not be present in every batch. If we bounce the cluster or if brokers fail > and leaders change, the metadata for those low throughput topic is not > refreshed by this periodic topic metadata request. The next produce request > for those topics have to fail and then a separate metadata request needs to > be reissued to handle the produce request. This is especially a problem for > the migration tool. So even if the producer had a chance to refresh the > metadata when the leader changed, it throws LeaderNotAvailableExceptions much > later when it sends a request for that topic. > I propose we just fetch data for all topics sent by the producer in the > periodic refresh of topic metadata -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (KAFKA-789) Producer-side persistence for delivery guarantee
Matan Safriel created KAFKA-789: --- Summary: Producer-side persistence for delivery guarantee Key: KAFKA-789 URL: https://issues.apache.org/jira/browse/KAFKA-789 Project: Kafka Issue Type: Improvement Components: producer Reporter: Matan Safriel Assignee: Jun Rao Priority: Minor Fix For: 0.9 A suggestion for higher guarantee for the part of entering messages into Kafka through it's producer. It aims to address the case that the entire set of broker replicas for a topic and partition is not available. Currently, in that case, data is lost. When a message set exhausts the send retry counter, the message set will be simply dropped. It would be nice being able to provide higher guarantee that a message passed to the producer would eventually be received by the broker. In an environment with some disk space to spare for this on the producer side, persisting to disk would seem to enable keeping messages for later retry (until defined space limits are exhausted). Thus somewhat elevating the level of guarantee. One way to facilitate this would be capitalizing on https://issues.apache.org/jira/browse/KAFKA-496, as the feedback it will add will enable knowing what needs to be retried again later. Changes to the producer or a wrapper around it (that may require access to the partitioning functions) would be able to persist failed message sets and manage delivery with a nice level of guarantee. As it would affect performance and use disks, should probably be a non-default option. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (KAFKA-790) Kafka server throws UnknownTopicOrPartitionException right after becoming leader for a new partition
Neha Narkhede created KAFKA-790:
---
Summary: Kafka server throws UnknownTopicOrPartitionException
right after becoming leader for a new partition
Key: KAFKA-790
URL: https://issues.apache.org/jira/browse/KAFKA-790
Project: Kafka
Issue Type: Bug
Components: replication
Affects Versions: 0.8
Reporter: Neha Narkhede
Assignee: Neha Narkhede
Priority: Blocker
2013/03/05 01:24:01.207 INFO [ReplicaManager] [kafka-request-handler-5] [kafka]
[] Replica Manager on Broker 273: Completed leader and isr request Name:
LeaderAndIsrRequest; Version: 0; CorrelationId: 15
38; ClientId: ; AckTimeoutMs: 1000 ms; ControllerEpoch: 3; PartitionStateInfo:
(foo,7) -> PartitionStateInfo(LeaderIsrAndControllerEpoch({ "ISR":"273,271",
"leader":"273", "leaderEpoch":"0" },3),2),(foo,4) ->
PartitionStateInfo(LeaderIsrAndControllerEpoch({ "ISR":"270,273",
"leader":"270", "leaderEpoch":"0" },3),2),(foo,1) ->
PartitionStateInfo(LeaderIsrAndControllerEpoch({ "ISR":"273,270",
"leader":"273", "leaderEpoch":"0" },3),2); Leaders:
id:270,host:host270,port:10251,id:273,host:host273
,port:10251
2013/03/05 01:24:01.473 INFO [Processor] [kafka-processor-10251-4] [kafka] []
Closing socket connection to /172.20.72.66.
2013/03/05 01:24:01.505 INFO [Processor] [kafka-processor-10251-4] [kafka] []
Closing socket connection to /172.20.72.244.
2013/03/05 01:24:01.521 INFO [Processor] [kafka-processor-10251-5] [kafka] []
Closing socket connection to /172.20.72.243.
2013/03/05 01:24:01.555 INFO [Processor] [kafka-processor-10251-3] [kafka] []
Closing socket connection to /172.20.72.11.
2013/03/05 01:24:01.568 INFO [Processor] [kafka-processor-10251-1] [kafka] []
Closing socket connection to /172.20.72.66.
2013/03/05 01:24:01.606 INFO [Processor] [kafka-processor-10251-0] [kafka] []
Closing socket connection to /172.20.72.66.
2013/03/05 01:24:01.758 INFO [Processor] [kafka-processor-10251-0] [kafka] []
Closing socket connection to /172.20.72.66.
2013/03/05 01:24:01.776 ERROR [KafkaApis] [kafka-request-handler-7] [kafka] []
[KafkaApi-273] error when processing request (foo,1,0,1048576)
kafka.common.UnknownTopicOrPartitionException: Topic foo partition 1 doesn't
exist on 273
at
kafka.server.ReplicaManager.getLeaderReplicaIfLocal(ReplicaManager.scala:170)
at
kafka.server.KafkaApis.kafka$server$KafkaApis$$readMessageSet(KafkaApis.scala:301)
at
kafka.server.KafkaApis$$anonfun$kafka$server$KafkaApis$$readMessageSets$1.apply(KafkaApis.scala:268)
at
kafka.server.KafkaApis$$anonfun$kafka$server$KafkaApis$$readMessageSets$1.apply(KafkaApis.scala:264)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
at scala.collection.immutable.Map$Map2.foreach(Map.scala:127)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:206)
at scala.collection.immutable.Map$Map2.map(Map.scala:110)
at
kafka.server.KafkaApis.kafka$server$KafkaApis$$readMessageSets(KafkaApis.scala:264)
at kafka.server.KafkaApis.handleFetchRequest(KafkaApis.scala:231)
at kafka.server.KafkaApis.handle(KafkaApis.scala:61)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:41)
at java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-790) Kafka server throws UnknownTopicOrPartitionException right after becoming leader for a new partition
[
https://issues.apache.org/jira/browse/KAFKA-790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Neha Narkhede updated KAFKA-790:
Labels: kafka-0.8 p1 (was: )
> Kafka server throws UnknownTopicOrPartitionException right after becoming
> leader for a new partition
>
>
> Key: KAFKA-790
> URL: https://issues.apache.org/jira/browse/KAFKA-790
> Project: Kafka
> Issue Type: Bug
> Components: replication
>Affects Versions: 0.8
>Reporter: Neha Narkhede
>Assignee: Neha Narkhede
>Priority: Blocker
> Labels: kafka-0.8, p1
>
> 2013/03/05 01:24:01.207 INFO [ReplicaManager] [kafka-request-handler-5]
> [kafka] [] Replica Manager on Broker 273: Completed leader and isr request
> Name: LeaderAndIsrRequest; Version: 0; CorrelationId: 15
> 38; ClientId: ; AckTimeoutMs: 1000 ms; ControllerEpoch: 3;
> PartitionStateInfo: (foo,7) ->
> PartitionStateInfo(LeaderIsrAndControllerEpoch({ "ISR":"273,271",
> "leader":"273", "leaderEpoch":"0" },3),2),(foo,4) ->
> PartitionStateInfo(LeaderIsrAndControllerEpoch({ "ISR":"270,273",
> "leader":"270", "leaderEpoch":"0" },3),2),(foo,1) ->
> PartitionStateInfo(LeaderIsrAndControllerEpoch({ "ISR":"273,270",
> "leader":"273", "leaderEpoch":"0" },3),2); Leaders:
> id:270,host:host270,port:10251,id:273,host:host273
> ,port:10251
> 2013/03/05 01:24:01.473 INFO [Processor] [kafka-processor-10251-4] [kafka] []
> Closing socket connection to /172.20.72.66.
> 2013/03/05 01:24:01.505 INFO [Processor] [kafka-processor-10251-4] [kafka] []
> Closing socket connection to /172.20.72.244.
> 2013/03/05 01:24:01.521 INFO [Processor] [kafka-processor-10251-5] [kafka] []
> Closing socket connection to /172.20.72.243.
> 2013/03/05 01:24:01.555 INFO [Processor] [kafka-processor-10251-3] [kafka] []
> Closing socket connection to /172.20.72.11.
> 2013/03/05 01:24:01.568 INFO [Processor] [kafka-processor-10251-1] [kafka] []
> Closing socket connection to /172.20.72.66.
> 2013/03/05 01:24:01.606 INFO [Processor] [kafka-processor-10251-0] [kafka] []
> Closing socket connection to /172.20.72.66.
> 2013/03/05 01:24:01.758 INFO [Processor] [kafka-processor-10251-0] [kafka] []
> Closing socket connection to /172.20.72.66.
> 2013/03/05 01:24:01.776 ERROR [KafkaApis] [kafka-request-handler-7] [kafka]
> [] [KafkaApi-273] error when processing request (foo,1,0,1048576)
> kafka.common.UnknownTopicOrPartitionException: Topic foo partition 1 doesn't
> exist on 273
> at
> kafka.server.ReplicaManager.getLeaderReplicaIfLocal(ReplicaManager.scala:170)
> at
> kafka.server.KafkaApis.kafka$server$KafkaApis$$readMessageSet(KafkaApis.scala:301)
> at
> kafka.server.KafkaApis$$anonfun$kafka$server$KafkaApis$$readMessageSets$1.apply(KafkaApis.scala:268)
> at
> kafka.server.KafkaApis$$anonfun$kafka$server$KafkaApis$$readMessageSets$1.apply(KafkaApis.scala:264)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
> at scala.collection.immutable.Map$Map2.foreach(Map.scala:127)
> at
> scala.collection.TraversableLike$class.map(TraversableLike.scala:206)
> at scala.collection.immutable.Map$Map2.map(Map.scala:110)
> at
> kafka.server.KafkaApis.kafka$server$KafkaApis$$readMessageSets(KafkaApis.scala:264)
> at kafka.server.KafkaApis.handleFetchRequest(KafkaApis.scala:231)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:61)
> at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:41)
> at java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (KAFKA-783) Preferred replica assignment on leader failure may not be correct
[ https://issues.apache.org/jira/browse/KAFKA-783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede reassigned KAFKA-783: --- Assignee: Sriram Subramanian (was: Neha Narkhede) Related to the preferred replica tool tests that you are doing > Preferred replica assignment on leader failure may not be correct > - > > Key: KAFKA-783 > URL: https://issues.apache.org/jira/browse/KAFKA-783 > Project: Kafka > Issue Type: Bug > Components: replication >Affects Versions: 0.8 > Environment: $ uname -a > Linux vrd01.atlnp1 2.6.32-279.el6.x86_64 #1 SMP Fri Jun 22 12:19:21 UTC 2012 > x86_64 x86_64 x86_64 GNU/Linux > $ java -version > java version "1.6.0_25" > Java(TM) SE Runtime Environment (build 1.6.0_25-b06) > Java HotSpot(TM) 64-Bit Server VM (build 20.0-b11, mixed mode) > Kafka 0.8.0 loaded from HEAD on 1/29/2013 >Reporter: Chris Curtin >Assignee: Sriram Subramanian > > Based on an email thread in the user group, Neha asked me to submit this. > Original question: "> I ran another test, again starting with a full cluster > and all partitions > > had a full set of copies. When I stop the broker which was leader for 9 of > > the 10 partitions, the leaders were all elected on one machine instead of > > the set of 3. Should the leaders have been better spread out? Also the > > copies weren’t fully populated either." > Neha: "For problem 2, we always try to make the preferred replica (1st replica > in the list of all replicas for a partition) the leader, if it is > available. We intended to spread the preferred replica for all partitions > for a topic evenly across the brokers. If this is not happening, we need to > look into it. Please can you file a bug and describe your test case there ?" > Configuration: > 4 node cluster > 1 topic with 3 replicas > 10 partitions: 0-9 below > Current status: > Partition: 0:vrd01.atlnp1 R:[ vrd03.atlnp1 vrd04.atlnp1 vrd01.atlnp1] I:[ > vrd01.atlnp1 vrd03.atlnp1 vrd04.atlnp1] > Partition: 1:vrd01.atlnp1 R:[ vrd04.atlnp1 vrd01.atlnp1 vrd02.atlnp1] I:[ > vrd01.atlnp1 vrd04.atlnp1 vrd02.atlnp1] > Partition: 2:vrd01.atlnp1 R:[ vrd01.atlnp1 vrd02.atlnp1 vrd03.atlnp1] I:[ > vrd01.atlnp1 vrd03.atlnp1 vrd02.atlnp1] > Partition: 3:vrd03.atlnp1 R:[ vrd02.atlnp1 vrd03.atlnp1 vrd04.atlnp1] I:[ > vrd03.atlnp1 vrd04.atlnp1 vrd02.atlnp1] > Partition: 4:vrd01.atlnp1 R:[ vrd03.atlnp1 vrd01.atlnp1 vrd02.atlnp1] I:[ > vrd01.atlnp1 vrd03.atlnp1 vrd02.atlnp1] > Partition: 5:vrd03.atlnp1 R:[ vrd04.atlnp1 vrd02.atlnp1 vrd03.atlnp1] I:[ > vrd03.atlnp1 vrd04.atlnp1 vrd02.atlnp1] > Partition: 6:vrd01.atlnp1 R:[ vrd01.atlnp1 vrd03.atlnp1 vrd04.atlnp1] I:[ > vrd01.atlnp1 vrd03.atlnp1 vrd04.atlnp1] > Partition: 7:vrd01.atlnp1 R:[ vrd02.atlnp1 vrd04.atlnp1 vrd01.atlnp1] I:[ > vrd01.atlnp1 vrd04.atlnp1 vrd02.atlnp1] > Partition: 8:vrd03.atlnp1 R:[ vrd03.atlnp1 vrd02.atlnp1 vrd04.atlnp1] I:[ > vrd03.atlnp1 vrd04.atlnp1 vrd02.atlnp1] > Partition: 9:vrd01.atlnp1 R:[ vrd04.atlnp1 vrd03.atlnp1 vrd01.atlnp1] I:[ > vrd01.atlnp1 vrd03.atlnp1 vrd04.atlnp1] > Shutdown vrd03: > Partition: 0:vrd01.atlnp1 R:[ ] I:[] > Partition: 1:vrd01.atlnp1 R:[ vrd04.atlnp1 vrd01.atlnp1 vrd02.atlnp1] I:[ > vrd01.atlnp1 vrd04.atlnp1 vrd02.atlnp1] > Partition: 2:vrd01.atlnp1 R:[ ] I:[] > *Partition: 3:vrd04.atlnp1 R:[ ] I:[] > Partition: 4:vrd01.atlnp1 R:[ ] I:[] > *Partition: 5:vrd04.atlnp1 R:[ ] I:[] > Partition: 6:vrd01.atlnp1 R:[ ] I:[] > Partition: 7:vrd01.atlnp1 R:[ vrd02.atlnp1 vrd04.atlnp1 vrd01.atlnp1] I:[ > vrd01.atlnp1 vrd04.atlnp1 vrd02.atlnp1] > *Partition: 8:vrd04.atlnp1 R:[ ] I:[] > Partition: 9:vrd01.atlnp1 R:[ ] I:[] > (* means leader changed) > Note that partitions 3, 5 and 8 were assigned new leaders. > Per an email group thread with Neha, the new leader should be assigned from > the preferred replica. So 3 should have gotten vrd02, 5, vrd04 and 8 vrd02 > (since 03 was shutdown). Instead 3 got vrd04, 5 got vrd04 and 8 got vrd04. > Restarting vrd03 led to: > Partition: 0:vrd01.atlnp1 R:[ vrd03.atlnp1 vrd04.atlnp1 vrd01.atlnp1] I:[ > vrd01.atlnp1 vrd04.atlnp1 vrd03.atlnp1] > Partition: 1:vrd01.atlnp1 R:[ vrd04.atlnp1 vrd01.atlnp1 vrd02.atlnp1] I:[ > vrd01.atlnp1 vrd04.atlnp1 vrd02.atlnp1] > Partition: 2:vrd01.atlnp1 R:[ vrd01.atlnp1 vrd02.atlnp1 vrd03.atlnp1] I:[ > vrd01.atlnp1 vrd02.atlnp1 vrd03.atlnp1] > Partition: 3:vrd04.atlnp1 R:[ vrd02.atlnp1 vrd03.atlnp1 vrd04.atlnp1] I:[ > vrd04.atlnp1 vrd02.atlnp1 vrd03.atlnp1] > Partition: 4:vrd01.atlnp1 R:[ vrd03.atlnp1 vrd01.atlnp1 vrd02.atlnp1] I:[ > vrd01.atlnp1 vrd02.atlnp1 vrd03.atlnp1] > Partition: 5:vrd04.atlnp1 R:[ vrd04.atlnp1 vrd02.atlnp1 vrd03.atlnp1] I:[ > vrd04.atlnp1 vrd02.atlnp1 vrd03.atlnp1] > Partition: 6:vrd01.atlnp1 R:[ vrd01.atlnp1 vrd03.atlnp1 vrd04.atlnp1] I:[ > vrd01.
[jira] [Updated] (KAFKA-783) Preferred replica assignment on leader failure may not be correct
[ https://issues.apache.org/jira/browse/KAFKA-783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-783: Labels: kafka-0.8 p2 (was: ) > Preferred replica assignment on leader failure may not be correct > - > > Key: KAFKA-783 > URL: https://issues.apache.org/jira/browse/KAFKA-783 > Project: Kafka > Issue Type: Bug > Components: replication >Affects Versions: 0.8 > Environment: $ uname -a > Linux vrd01.atlnp1 2.6.32-279.el6.x86_64 #1 SMP Fri Jun 22 12:19:21 UTC 2012 > x86_64 x86_64 x86_64 GNU/Linux > $ java -version > java version "1.6.0_25" > Java(TM) SE Runtime Environment (build 1.6.0_25-b06) > Java HotSpot(TM) 64-Bit Server VM (build 20.0-b11, mixed mode) > Kafka 0.8.0 loaded from HEAD on 1/29/2013 >Reporter: Chris Curtin >Assignee: Sriram Subramanian > Labels: kafka-0.8, p2 > > Based on an email thread in the user group, Neha asked me to submit this. > Original question: "> I ran another test, again starting with a full cluster > and all partitions > > had a full set of copies. When I stop the broker which was leader for 9 of > > the 10 partitions, the leaders were all elected on one machine instead of > > the set of 3. Should the leaders have been better spread out? Also the > > copies weren’t fully populated either." > Neha: "For problem 2, we always try to make the preferred replica (1st replica > in the list of all replicas for a partition) the leader, if it is > available. We intended to spread the preferred replica for all partitions > for a topic evenly across the brokers. If this is not happening, we need to > look into it. Please can you file a bug and describe your test case there ?" > Configuration: > 4 node cluster > 1 topic with 3 replicas > 10 partitions: 0-9 below > Current status: > Partition: 0:vrd01.atlnp1 R:[ vrd03.atlnp1 vrd04.atlnp1 vrd01.atlnp1] I:[ > vrd01.atlnp1 vrd03.atlnp1 vrd04.atlnp1] > Partition: 1:vrd01.atlnp1 R:[ vrd04.atlnp1 vrd01.atlnp1 vrd02.atlnp1] I:[ > vrd01.atlnp1 vrd04.atlnp1 vrd02.atlnp1] > Partition: 2:vrd01.atlnp1 R:[ vrd01.atlnp1 vrd02.atlnp1 vrd03.atlnp1] I:[ > vrd01.atlnp1 vrd03.atlnp1 vrd02.atlnp1] > Partition: 3:vrd03.atlnp1 R:[ vrd02.atlnp1 vrd03.atlnp1 vrd04.atlnp1] I:[ > vrd03.atlnp1 vrd04.atlnp1 vrd02.atlnp1] > Partition: 4:vrd01.atlnp1 R:[ vrd03.atlnp1 vrd01.atlnp1 vrd02.atlnp1] I:[ > vrd01.atlnp1 vrd03.atlnp1 vrd02.atlnp1] > Partition: 5:vrd03.atlnp1 R:[ vrd04.atlnp1 vrd02.atlnp1 vrd03.atlnp1] I:[ > vrd03.atlnp1 vrd04.atlnp1 vrd02.atlnp1] > Partition: 6:vrd01.atlnp1 R:[ vrd01.atlnp1 vrd03.atlnp1 vrd04.atlnp1] I:[ > vrd01.atlnp1 vrd03.atlnp1 vrd04.atlnp1] > Partition: 7:vrd01.atlnp1 R:[ vrd02.atlnp1 vrd04.atlnp1 vrd01.atlnp1] I:[ > vrd01.atlnp1 vrd04.atlnp1 vrd02.atlnp1] > Partition: 8:vrd03.atlnp1 R:[ vrd03.atlnp1 vrd02.atlnp1 vrd04.atlnp1] I:[ > vrd03.atlnp1 vrd04.atlnp1 vrd02.atlnp1] > Partition: 9:vrd01.atlnp1 R:[ vrd04.atlnp1 vrd03.atlnp1 vrd01.atlnp1] I:[ > vrd01.atlnp1 vrd03.atlnp1 vrd04.atlnp1] > Shutdown vrd03: > Partition: 0:vrd01.atlnp1 R:[ ] I:[] > Partition: 1:vrd01.atlnp1 R:[ vrd04.atlnp1 vrd01.atlnp1 vrd02.atlnp1] I:[ > vrd01.atlnp1 vrd04.atlnp1 vrd02.atlnp1] > Partition: 2:vrd01.atlnp1 R:[ ] I:[] > *Partition: 3:vrd04.atlnp1 R:[ ] I:[] > Partition: 4:vrd01.atlnp1 R:[ ] I:[] > *Partition: 5:vrd04.atlnp1 R:[ ] I:[] > Partition: 6:vrd01.atlnp1 R:[ ] I:[] > Partition: 7:vrd01.atlnp1 R:[ vrd02.atlnp1 vrd04.atlnp1 vrd01.atlnp1] I:[ > vrd01.atlnp1 vrd04.atlnp1 vrd02.atlnp1] > *Partition: 8:vrd04.atlnp1 R:[ ] I:[] > Partition: 9:vrd01.atlnp1 R:[ ] I:[] > (* means leader changed) > Note that partitions 3, 5 and 8 were assigned new leaders. > Per an email group thread with Neha, the new leader should be assigned from > the preferred replica. So 3 should have gotten vrd02, 5, vrd04 and 8 vrd02 > (since 03 was shutdown). Instead 3 got vrd04, 5 got vrd04 and 8 got vrd04. > Restarting vrd03 led to: > Partition: 0:vrd01.atlnp1 R:[ vrd03.atlnp1 vrd04.atlnp1 vrd01.atlnp1] I:[ > vrd01.atlnp1 vrd04.atlnp1 vrd03.atlnp1] > Partition: 1:vrd01.atlnp1 R:[ vrd04.atlnp1 vrd01.atlnp1 vrd02.atlnp1] I:[ > vrd01.atlnp1 vrd04.atlnp1 vrd02.atlnp1] > Partition: 2:vrd01.atlnp1 R:[ vrd01.atlnp1 vrd02.atlnp1 vrd03.atlnp1] I:[ > vrd01.atlnp1 vrd02.atlnp1 vrd03.atlnp1] > Partition: 3:vrd04.atlnp1 R:[ vrd02.atlnp1 vrd03.atlnp1 vrd04.atlnp1] I:[ > vrd04.atlnp1 vrd02.atlnp1 vrd03.atlnp1] > Partition: 4:vrd01.atlnp1 R:[ vrd03.atlnp1 vrd01.atlnp1 vrd02.atlnp1] I:[ > vrd01.atlnp1 vrd02.atlnp1 vrd03.atlnp1] > Partition: 5:vrd04.atlnp1 R:[ vrd04.atlnp1 vrd02.atlnp1 vrd03.atlnp1] I:[ > vrd04.atlnp1 vrd02.atlnp1 vrd03.atlnp1] > Partition: 6:vrd01.atlnp1 R:[ vrd01.atlnp1 vrd03.atlnp1 vrd04.atlnp1] I:[ > vrd01.atlnp1 vrd04.atlnp1 vrd03.atlnp1] > Partition: 7:vrd01.atlnp1 R:[
[jira] [Updated] (KAFKA-782) replicas not being displayed in TopicMetadataResponse when replica's Broker is shutdown
[
https://issues.apache.org/jira/browse/KAFKA-782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Neha Narkhede updated KAFKA-782:
Labels: kafka-0.8 p2 (was: )
> replicas not being displayed in TopicMetadataResponse when replica's Broker
> is shutdown
> ---
>
> Key: KAFKA-782
> URL: https://issues.apache.org/jira/browse/KAFKA-782
> Project: Kafka
> Issue Type: Bug
> Components: replication
>Affects Versions: 0.8
> Environment: $ uname -a
> Linux vrd01.atlnp1 2.6.32-279.el6.x86_64 #1 SMP Fri Jun 22 12:19:21 UTC 2012
> x86_64 x86_64 x86_64 GNU/Linux
> $ java -version
> java version "1.6.0_25"
> Java(TM) SE Runtime Environment (build 1.6.0_25-b06)
> Java HotSpot(TM) 64-Bit Server VM (build 20.0-b11, mixed mode)
> Kafka 0.8.0 loaded from HEAD on 1/29/2013
>Reporter: Chris Curtin
>Assignee: Neha Narkhede
> Labels: kafka-0.8, p2
>
> Setup
> - 4 nodes in the cluster
> - topic has 10 partitions, numbered 0-9 in the output below
> - configure the cluster so one of the nodes doesn't hold any leader copies of
> the partition. I did this by shutting down a node, waiting for reassignment
> of leaders and starting again
> Output of the TopicMetadataResponse call:
> Partition: 0:vrd01.atlnp1 R:[ vrd03.atlnp1 vrd04.atlnp1 vrd01.atlnp1] I:[
> vrd01.atlnp1 vrd03.atlnp1 vrd04.atlnp1]
> Partition: 1:vrd01.atlnp1 R:[ vrd04.atlnp1 vrd01.atlnp1 vrd02.atlnp1] I:[
> vrd01.atlnp1 vrd04.atlnp1 vrd02.atlnp1]
> Partition: 2:vrd01.atlnp1 R:[ vrd01.atlnp1 vrd02.atlnp1 vrd03.atlnp1] I:[
> vrd01.atlnp1 vrd03.atlnp1 vrd02.atlnp1]
> Partition: 3:vrd03.atlnp1 R:[ vrd02.atlnp1 vrd03.atlnp1 vrd04.atlnp1] I:[
> vrd03.atlnp1 vrd04.atlnp1 vrd02.atlnp1]
> Partition: 4:vrd01.atlnp1 R:[ vrd03.atlnp1 vrd01.atlnp1 vrd02.atlnp1] I:[
> vrd01.atlnp1 vrd03.atlnp1 vrd02.atlnp1]
> Partition: 5:vrd03.atlnp1 R:[ vrd04.atlnp1 vrd02.atlnp1 vrd03.atlnp1] I:[
> vrd03.atlnp1 vrd04.atlnp1 vrd02.atlnp1]
> Partition: 6:vrd01.atlnp1 R:[ vrd01.atlnp1 vrd03.atlnp1 vrd04.atlnp1] I:[
> vrd01.atlnp1 vrd03.atlnp1 vrd04.atlnp1]
> Partition: 7:vrd01.atlnp1 R:[ vrd02.atlnp1 vrd04.atlnp1 vrd01.atlnp1] I:[
> vrd01.atlnp1 vrd04.atlnp1 vrd02.atlnp1]
> Partition: 8:vrd03.atlnp1 R:[ vrd03.atlnp1 vrd02.atlnp1 vrd04.atlnp1] I:[
> vrd03.atlnp1 vrd04.atlnp1 vrd02.atlnp1]
> Partition: 9:vrd01.atlnp1 R:[ vrd04.atlnp1 vrd03.atlnp1 vrd01.atlnp1] I:[
> vrd01.atlnp1 vrd03.atlnp1 vrd04.atlnp1]
> Note that vrd02.atlnp1 does not contain any leader replicas.
> Shutdown vrd02 normally. Run TopicMetadataResponse again:
> Partition: 0:vrd01.atlnp1 R:[ vrd03.atlnp1 vrd04.atlnp1 vrd01.atlnp1] I:[
> vrd01.atlnp1 vrd03.atlnp1 vrd04.atlnp1]
> Partition: 1:vrd01.atlnp1 R:[ ] I:[]
> Partition: 2:vrd01.atlnp1 R:[ ] I:[]
> Partition: 3:vrd03.atlnp1 R:[ ] I:[]
> Partition: 4:vrd01.atlnp1 R:[ ] I:[]
> Partition: 5:vrd03.atlnp1 R:[ ] I:[]
> Partition: 6:vrd01.atlnp1 R:[ vrd01.atlnp1 vrd03.atlnp1 vrd04.atlnp1] I:[
> vrd01.atlnp1 vrd03.atlnp1 vrd04.atlnp1]
> Partition: 7:vrd01.atlnp1 R:[ ] I:[]
> Partition: 8:vrd03.atlnp1 R:[ ] I:[]
> Partition: 9:vrd01.atlnp1 R:[ vrd04.atlnp1 vrd03.atlnp1 vrd01.atlnp1] I:[
> vrd01.atlnp1 vrd03.atlnp1 vrd04.atlnp1]
> Note that the partitions where vrd02 was in the replica set no longer show
> any replicas.
> Not clear if the list of replicas isn't being set correctly or the replicas
> aren't associated with the partition any longer.
> Java code:
> kafka.javaapi.consumer.SimpleConsumer consumer = new
> SimpleConsumer("vrd01.atlnp1",
> 9092,
> 10,
> 64 * 1024, "test");
> List topics2 = new ArrayList();
> topics2.add("storm-anon");
> TopicMetadataRequest req = new TopicMetadataRequest(topics2);
> kafka.javaapi.TopicMetadataResponse resp = consumer.send(req);
> List data3 = resp.topicsMetadata();
> for (kafka.javaapi.TopicMetadata item : data3) {
>for (kafka.javaapi.PartitionMetadata part:
> item.partitionsMetadata() ) {
>String replicas = "";
>String isr = "";
>for (kafka.cluster.Broker replica: part.replicas() ) {
>replicas += " " + replica.host();
>}
>for (kafka.cluster.Broker replica: part.isr() ) {
>isr += " " + replica.host();
>}
> System.out.println( "Partition: " + part.partitionId() + ":"
> + part.leader().host() + " R:[ " + replicas + "] I:[" + isr + "]");
>}
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-784) creating topic without partitions, deleting then creating with partition causes errors in 'kafka-list-topic'
[ https://issues.apache.org/jira/browse/KAFKA-784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13594069#comment-13594069 ] Neha Narkhede commented on KAFKA-784: - The delete topic functionality is not yet implemented in Kafka. Somehow a half baked version made it to the 0.8 branch. We plan to do this before release, we can revisit this at that time. > creating topic without partitions, deleting then creating with partition > causes errors in 'kafka-list-topic' > > > Key: KAFKA-784 > URL: https://issues.apache.org/jira/browse/KAFKA-784 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 0.8 > Environment: 0.8.0 head as of 3/4/2013 >Reporter: Chris Curtin >Priority: Minor > > Create a new topic using the command line: > ./kafka-create-topic.sh --topic trash-1 --replica 3 --zookeeper localhost > Realize you forgot to add the partition command, so remove it: > ./kafka-delete-topic.sh --topic trash-1 --zookeeper localhost > Recreate it with partitions: > ./kafka-create-topic.sh --topic trash-1 --replica 3 --zookeeper localhost > --partition 5 > Try to get a listing: > ./kafka-list-topic.sh --topic trash-1 --zookeeper localhost > Errors: > [2013-03-04 14:15:23,876] ERROR Error while fetching metadata for partition > [trash-1,0] (kafka.admin.AdminUtils$) > kafka.common.LeaderNotAvailableException: Leader not available for topic > trash-1 partition 0 > at kafka.admin.AdminUtils$$anonfun$3.apply(AdminUtils.scala:120) > at kafka.admin.AdminUtils$$anonfun$3.apply(AdminUtils.scala:103) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206) > at > scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:61) > at scala.collection.immutable.List.foreach(List.scala:45) > at > scala.collection.TraversableLike$class.map(TraversableLike.scala:206) > at scala.collection.immutable.List.map(List.scala:45) > at > kafka.admin.AdminUtils$.kafka$admin$AdminUtils$$fetchTopicMetadataFromZk(AdminUtils.scala:103) > at > kafka.admin.AdminUtils$.fetchTopicMetadataFromZk(AdminUtils.scala:92) > at kafka.admin.ListTopicCommand$.showTopic(ListTopicCommand.scala:80) > at > kafka.admin.ListTopicCommand$$anonfun$main$2.apply(ListTopicCommand.scala:66) > at > kafka.admin.ListTopicCommand$$anonfun$main$2.apply(ListTopicCommand.scala:65) > at > scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:61) > at scala.collection.immutable.List.foreach(List.scala:45) > at kafka.admin.ListTopicCommand$.main(ListTopicCommand.scala:65) > at kafka.admin.ListTopicCommand.main(ListTopicCommand.scala) > Caused by: kafka.common.LeaderNotAvailableException: No leader exists for > partition 0 > at kafka.admin.AdminUtils$$anonfun$3.apply(AdminUtils.scala:117) > ... 16 more > topic: trash-1 > PartitionMetadata(0,None,List(),List(),5) > Can't recover until you restart all the Brokers in the cluster. Then the list > command works correctly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-787) Simple Consumer connecting to Broker that is not the Leader generates wrong error
[ https://issues.apache.org/jira/browse/KAFKA-787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-787: Attachment: kafka-787.patch Minor fix for the correct error code > Simple Consumer connecting to Broker that is not the Leader generates wrong > error > - > > Key: KAFKA-787 > URL: https://issues.apache.org/jira/browse/KAFKA-787 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 0.8 >Reporter: Chris Curtin >Assignee: Neha Narkhede > Labels: kafka-0.8, p2 > Attachments: kafka-787.patch > > > 0.8.0 HEAD from 3/4/2013 > Using a SimpleConsumer, connect to a Broker that is NOT the leader for the > topic/partition you want to fetch, but IS a replica. > On fetch the error returned is '5' 'ErrorMapping.LeaderNotAvailableCode' > Per email thread with Neha: > "Ideally, you should get back ErrorMapping.NotLeaderForPartitionCode" > Key part of the test is the Broker must be a replica. If the topic/partition > is not on the Broker you get a different (correct) error. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-787) Simple Consumer connecting to Broker that is not the Leader generates wrong error
[ https://issues.apache.org/jira/browse/KAFKA-787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-787: Status: Patch Available (was: Open) > Simple Consumer connecting to Broker that is not the Leader generates wrong > error > - > > Key: KAFKA-787 > URL: https://issues.apache.org/jira/browse/KAFKA-787 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 0.8 >Reporter: Chris Curtin >Assignee: Neha Narkhede > Labels: kafka-0.8, p2 > Attachments: kafka-787.patch > > > 0.8.0 HEAD from 3/4/2013 > Using a SimpleConsumer, connect to a Broker that is NOT the leader for the > topic/partition you want to fetch, but IS a replica. > On fetch the error returned is '5' 'ErrorMapping.LeaderNotAvailableCode' > Per email thread with Neha: > "Ideally, you should get back ErrorMapping.NotLeaderForPartitionCode" > Key part of the test is the Broker must be a replica. If the topic/partition > is not on the Broker you get a different (correct) error. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-786) Use "withRequiredArg" while parsing jopt options in all tools
[
https://issues.apache.org/jira/browse/KAFKA-786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Swapnil Ghike updated KAFKA-786:
Attachment: kafka-786-v2.patch
Jun raised a good question: What do we do for options like "print" or "verify"
or "enable" etc? These options typically don't expect a boolean argument, it
should be enough to specify these options on the command line to sort of enable
them in the program.
The answer is to not use any of withRequiredArg() or withOptionalArg(). We can
simply write something like the following:
val printOpt = parser.accepts("print-data-log", "if set, printing the messages
content when dumping data logs")
Later we should check in the program like the following:
if(options.has(printOpt) print() else doNothing()
> Use "withRequiredArg" while parsing jopt options in all tools
> -
>
> Key: KAFKA-786
> URL: https://issues.apache.org/jira/browse/KAFKA-786
> Project: Kafka
> Issue Type: Bug
>Reporter: Swapnil Ghike
>Assignee: Swapnil Ghike
>Priority: Blocker
> Labels: kafka-0.8, p2
> Attachments: kafka-786.patch, kafka-786-v2.patch
>
>
> While parsing jopt Options in our tools, we sometimes use withRequiredArg()
> and sometimes use withOptionalArg(). I think this confusing and we should
> always use withRequiredArg().
> withOptionalArg() allows you to provide an option without an argument. For
> instance, the following commands will yield the same result if xyz was a
> parser option that accepted an optional argument and was provided a default
> in the tool's code:
> kafka-tool.sh --xyz
> kafka-tool.sh
> I don't quite see the need to allow the 1st command, think that writing code
> will be less confusing if we allowed only the second command. To do that, we
> can make all options require arguments. These arguments will need to be given
> via command line or via a default in the code. So if xyz was an option that
> required an argument then you will see the following:
> kafka-tool.sh --xyz
> Option ['xyz'] requires an argument //printed by jOpt
> kafka-tool.sh --xyz argumentVal
> // Kafka tool proceeds
> If you want to use a default value specified for xyz in the code, then simply
> run ./kafka-tool.sh.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-104) Invalid topics prevent broker start-up
[ https://issues.apache.org/jira/browse/KAFKA-104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13594131#comment-13594131 ] Colin B. commented on KAFKA-104: I am unable to reproduce this issue. It is likely this was fixed in another issue such as: https://issues.apache.org/jira/browse/KAFKA-495 > Invalid topics prevent broker start-up > -- > > Key: KAFKA-104 > URL: https://issues.apache.org/jira/browse/KAFKA-104 > Project: Kafka > Issue Type: Bug >Reporter: Joel Koshy >Priority: Minor > > Excerpt from a trace we saw today. If there is a topic directory with an > invalid name, then the registerMBean call throws an exception which > eventually results in a start failure. Also, if you try sending messages with > an invalid topic to a broker, the broker's logs get flooded with these > exceptions. Easy fix is to wrap the registerMBean with Utils.swallow; and > probably prevent ProducerRequest from accepting invalid topics. > 2011/08/15 20:43:50.834 FATAL [KafkaServer] [main] [kafka] > javax.management.MalformedObjectNameException: Invalid character '"' in value > part of property > at javax.management.ObjectName.construct(ObjectName.java:602) > at javax.management.ObjectName.(ObjectName.java:1403) > at kafka.utils.Utils$.registerMBean(Utils.scala:372) > at kafka.log.Log.(Log.scala:120) > at kafka.log.LogManager$$anonfun$2.apply(LogManager.scala:70) > at kafka.log.LogManager$$anonfun$2.apply(LogManager.scala:65) > at > scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:34) > at scala.collection.mutable.ArrayOps.foreach(ArrayOps.scala:34) > at kafka.log.LogManager.(LogManager.scala:65) > at kafka.server.KafkaServer.startup(KafkaServer.scala:60) > at > kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:46) > at com.linkedin.kafka.KafkaStartable.start(KafkaStartable.java:54) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (KAFKA-791) Fix validation bugs in System Test
John Fung created KAFKA-791: --- Summary: Fix validation bugs in System Test Key: KAFKA-791 URL: https://issues.apache.org/jira/browse/KAFKA-791 Project: Kafka Issue Type: Bug Reporter: John Fung Assignee: John Fung The following issues are found in data / log checksum match in System Test: 1. kafka_system_test_utils.validate_simple_consumer_data_matched It reports PASSED even some log segments don't match 2. kafka_system_test_utils.validate_data_matched It reports PASSED in the Ack=1 cases even data loss is greater than the tolerance (1%). 3. kafka_system_test_utils.validate_simple_consumer_data_matched It gets a unique set of MessageID to validate. It should leave all MessageID as is and the test case should fail if duplicates are detected. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-791) Fix validation bugs in System Test
[ https://issues.apache.org/jira/browse/KAFKA-791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-791: Issue Type: Task (was: Bug) > Fix validation bugs in System Test > -- > > Key: KAFKA-791 > URL: https://issues.apache.org/jira/browse/KAFKA-791 > Project: Kafka > Issue Type: Task >Reporter: John Fung >Assignee: John Fung > Labels: replication-testing > > The following issues are found in data / log checksum match in System Test: > 1. kafka_system_test_utils.validate_simple_consumer_data_matched > It reports PASSED even some log segments don't match > 2. kafka_system_test_utils.validate_data_matched > It reports PASSED in the Ack=1 cases even data loss is greater than the > tolerance (1%). > 3. kafka_system_test_utils.validate_simple_consumer_data_matched > It gets a unique set of MessageID to validate. It should leave all MessageID > as is and the test case should fail if duplicates are detected. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (KAFKA-792) Update multiple attributes in testcase_xxxx_properties.json
John Fung created KAFKA-792: --- Summary: Update multiple attributes in testcase__properties.json Key: KAFKA-792 URL: https://issues.apache.org/jira/browse/KAFKA-792 Project: Kafka Issue Type: Task Reporter: John Fung Assignee: John Fung The following are some of the properties need to be updated in some testcase__properties.json: log.segment.bytes default.replication.factor num.partitions -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-792) Update multiple attributes in testcase_xxxx_properties.json
[ https://issues.apache.org/jira/browse/KAFKA-792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-792: Description: The following are some of the properties need to be updated in some testcase__properties.json. These changes have been patched in local Hudson for a while. Create this new JIRA to check in these changes. log.segment.bytes default.replication.factor num.partitions was: The following are some of the properties need to be updated in some testcase__properties.json: log.segment.bytes default.replication.factor num.partitions > Update multiple attributes in testcase__properties.json > --- > > Key: KAFKA-792 > URL: https://issues.apache.org/jira/browse/KAFKA-792 > Project: Kafka > Issue Type: Task >Reporter: John Fung >Assignee: John Fung > Labels: replication-testing > > The following are some of the properties need to be updated in some > testcase__properties.json. These changes have been patched in local > Hudson for a while. Create this new JIRA to check in these changes. > log.segment.bytes > default.replication.factor > num.partitions -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (KAFKA-104) Invalid topics prevent broker start-up
[ https://issues.apache.org/jira/browse/KAFKA-104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede resolved KAFKA-104. - Resolution: Cannot Reproduce > Invalid topics prevent broker start-up > -- > > Key: KAFKA-104 > URL: https://issues.apache.org/jira/browse/KAFKA-104 > Project: Kafka > Issue Type: Bug >Reporter: Joel Koshy >Priority: Minor > > Excerpt from a trace we saw today. If there is a topic directory with an > invalid name, then the registerMBean call throws an exception which > eventually results in a start failure. Also, if you try sending messages with > an invalid topic to a broker, the broker's logs get flooded with these > exceptions. Easy fix is to wrap the registerMBean with Utils.swallow; and > probably prevent ProducerRequest from accepting invalid topics. > 2011/08/15 20:43:50.834 FATAL [KafkaServer] [main] [kafka] > javax.management.MalformedObjectNameException: Invalid character '"' in value > part of property > at javax.management.ObjectName.construct(ObjectName.java:602) > at javax.management.ObjectName.(ObjectName.java:1403) > at kafka.utils.Utils$.registerMBean(Utils.scala:372) > at kafka.log.Log.(Log.scala:120) > at kafka.log.LogManager$$anonfun$2.apply(LogManager.scala:70) > at kafka.log.LogManager$$anonfun$2.apply(LogManager.scala:65) > at > scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:34) > at scala.collection.mutable.ArrayOps.foreach(ArrayOps.scala:34) > at kafka.log.LogManager.(LogManager.scala:65) > at kafka.server.KafkaServer.startup(KafkaServer.scala:60) > at > kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:46) > at com.linkedin.kafka.KafkaStartable.start(KafkaStartable.java:54) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (KAFKA-104) Invalid topics prevent broker start-up
[ https://issues.apache.org/jira/browse/KAFKA-104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede closed KAFKA-104. --- > Invalid topics prevent broker start-up > -- > > Key: KAFKA-104 > URL: https://issues.apache.org/jira/browse/KAFKA-104 > Project: Kafka > Issue Type: Bug >Reporter: Joel Koshy >Priority: Minor > > Excerpt from a trace we saw today. If there is a topic directory with an > invalid name, then the registerMBean call throws an exception which > eventually results in a start failure. Also, if you try sending messages with > an invalid topic to a broker, the broker's logs get flooded with these > exceptions. Easy fix is to wrap the registerMBean with Utils.swallow; and > probably prevent ProducerRequest from accepting invalid topics. > 2011/08/15 20:43:50.834 FATAL [KafkaServer] [main] [kafka] > javax.management.MalformedObjectNameException: Invalid character '"' in value > part of property > at javax.management.ObjectName.construct(ObjectName.java:602) > at javax.management.ObjectName.(ObjectName.java:1403) > at kafka.utils.Utils$.registerMBean(Utils.scala:372) > at kafka.log.Log.(Log.scala:120) > at kafka.log.LogManager$$anonfun$2.apply(LogManager.scala:70) > at kafka.log.LogManager$$anonfun$2.apply(LogManager.scala:65) > at > scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:34) > at scala.collection.mutable.ArrayOps.foreach(ArrayOps.scala:34) > at kafka.log.LogManager.(LogManager.scala:65) > at kafka.server.KafkaServer.startup(KafkaServer.scala:60) > at > kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:46) > at com.linkedin.kafka.KafkaStartable.start(KafkaStartable.java:54) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-791) Fix validation bugs in System Test
[ https://issues.apache.org/jira/browse/KAFKA-791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-791: Description: The following issues are found in data / log checksum match in System Test: 1. kafka_system_test_utils.validate_simple_consumer_data_matched It reports PASSED even some log segments don't match 2. kafka_system_test_utils.validate_data_matched (this is fixed and patched in local Hudson for some time) It reports PASSED in the Ack=1 cases even data loss is greater than the tolerance (1%). 3. kafka_system_test_utils.validate_simple_consumer_data_matched It gets a unique set of MessageID to validate. It should leave all MessageID as is and the test case should fail if duplicates are detected. was: The following issues are found in data / log checksum match in System Test: 1. kafka_system_test_utils.validate_simple_consumer_data_matched It reports PASSED even some log segments don't match 2. kafka_system_test_utils.validate_data_matched It reports PASSED in the Ack=1 cases even data loss is greater than the tolerance (1%). 3. kafka_system_test_utils.validate_simple_consumer_data_matched It gets a unique set of MessageID to validate. It should leave all MessageID as is and the test case should fail if duplicates are detected. > Fix validation bugs in System Test > -- > > Key: KAFKA-791 > URL: https://issues.apache.org/jira/browse/KAFKA-791 > Project: Kafka > Issue Type: Task >Reporter: John Fung >Assignee: John Fung > Labels: replication-testing > > The following issues are found in data / log checksum match in System Test: > 1. kafka_system_test_utils.validate_simple_consumer_data_matched > It reports PASSED even some log segments don't match > 2. kafka_system_test_utils.validate_data_matched (this is fixed and patched > in local Hudson for some time) > It reports PASSED in the Ack=1 cases even data loss is greater than the > tolerance (1%). > 3. kafka_system_test_utils.validate_simple_consumer_data_matched > It gets a unique set of MessageID to validate. It should leave all MessageID > as is and the test case should fail if duplicates are detected. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-513) Add state change log to Kafka brokers
[ https://issues.apache.org/jira/browse/KAFKA-513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Swapnil Ghike updated KAFKA-513: Attachment: kafka-513-v5.patch Thank you for the detailed suggestions, they were incredibly helpful. Made changes according to last review. Additional comments: 1.1 For this, the constructors of ControllerChannelManagaer, RequestToSendThread have to accept a ControllerContext. 4, 5. Yes, it makes sense to get rid of the logIdent for state change logs. I have removed the wrapper around the Logging trait as well. 7.1 Earlier I was thinking that the list of files should be appended if the user specified both options together. But you are right, it may be confusing and anyways the user can include more files using the regex option. > Add state change log to Kafka brokers > - > > Key: KAFKA-513 > URL: https://issues.apache.org/jira/browse/KAFKA-513 > Project: Kafka > Issue Type: Sub-task >Affects Versions: 0.8 >Reporter: Neha Narkhede >Assignee: Swapnil Ghike >Priority: Blocker > Labels: p1, replication, tools > Fix For: 0.8 > > Attachments: kafka-513-v1.patch, kafka-513-v2.patch, > kafka-513-v3.patch, kafka-513-v4.patch, kafka-513-v5.patch > > Original Estimate: 96h > Remaining Estimate: 96h > > Once KAFKA-499 is checked in, every controller to broker communication can be > modelled as a state change for one or more partitions. Every state change > request will carry the controller epoch. If there is a problem with the state > of some partitions, it will be good to have a tool that can create a timeline > of requested and completed state changes. This will require each broker to > output a state change log that has entries like > [2012-09-10 10:06:17,280] broker 1 received request LeaderAndIsr() for > partition [foo, 0] from controller 2, epoch 1 > [2012-09-10 10:06:17,350] broker 1 completed request LeaderAndIsr() for > partition [foo, 0] from controller 2, epoch 1 > On controller, this will look like - > [2012-09-10 10:06:17,198] controller 2, epoch 1, initiated state change > request LeaderAndIsr() for partition [foo, 0] > We need a tool that can collect the state change log from all brokers and > create a per-partition timeline of state changes - > [foo, 0] > [2012-09-10 10:06:17,198] controller 2, epoch 1 initiated state change > request LeaderAndIsr() > [2012-09-10 10:06:17,280] broker 1 received request LeaderAndIsr() from > controller 2, epoch 1 > [2012-09-10 10:06:17,350] broker 1 completed request LeaderAndIsr() from > controller 2, epoch 1 > This JIRA involves adding the state change log to each broker and adding the > tool to create the timeline -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-513) Add state change log to Kafka brokers
[ https://issues.apache.org/jira/browse/KAFKA-513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13594379#comment-13594379 ] Neha Narkhede commented on KAFKA-513: - I think there is some issue with patch v5. I get the following error while trying to apply the patch - nnarkhed-mn:kafka-git nnarkhed$ patch -p1 -i ~/Projects/kafka-patches/kafka-513-v5.patch patching file bin/kafka-run-class.sh patching file config/log4j.properties patch: malformed patch at line 39: diff --git a/core/src/main/scala/kafka/api/FetchRequest.scala b/core/src/main/scala/kafka/api/FetchRequest.scala > Add state change log to Kafka brokers > - > > Key: KAFKA-513 > URL: https://issues.apache.org/jira/browse/KAFKA-513 > Project: Kafka > Issue Type: Sub-task >Affects Versions: 0.8 >Reporter: Neha Narkhede >Assignee: Swapnil Ghike >Priority: Blocker > Labels: p1, replication, tools > Fix For: 0.8 > > Attachments: kafka-513-v1.patch, kafka-513-v2.patch, > kafka-513-v3.patch, kafka-513-v4.patch, kafka-513-v5.patch > > Original Estimate: 96h > Remaining Estimate: 96h > > Once KAFKA-499 is checked in, every controller to broker communication can be > modelled as a state change for one or more partitions. Every state change > request will carry the controller epoch. If there is a problem with the state > of some partitions, it will be good to have a tool that can create a timeline > of requested and completed state changes. This will require each broker to > output a state change log that has entries like > [2012-09-10 10:06:17,280] broker 1 received request LeaderAndIsr() for > partition [foo, 0] from controller 2, epoch 1 > [2012-09-10 10:06:17,350] broker 1 completed request LeaderAndIsr() for > partition [foo, 0] from controller 2, epoch 1 > On controller, this will look like - > [2012-09-10 10:06:17,198] controller 2, epoch 1, initiated state change > request LeaderAndIsr() for partition [foo, 0] > We need a tool that can collect the state change log from all brokers and > create a per-partition timeline of state changes - > [foo, 0] > [2012-09-10 10:06:17,198] controller 2, epoch 1 initiated state change > request LeaderAndIsr() > [2012-09-10 10:06:17,280] broker 1 received request LeaderAndIsr() from > controller 2, epoch 1 > [2012-09-10 10:06:17,350] broker 1 completed request LeaderAndIsr() from > controller 2, epoch 1 > This JIRA involves adding the state change log to each broker and adding the > tool to create the timeline -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Get all the partion details from zookeeper for a topic
Hi Neha,
I am using zkclient version 0.1. I dont see any method "getData" in
zkclient. Please let me know which version should I use.
Thanks
Sourabh
On Tue, Mar 5, 2013 at 10:01 PM, Neha Narkhede wrote:
> The problem is that you need to invoke getData, not getChildren on this
> path - /brokers/topics/test-topic/1. That will return the # of partitions
> on this broker, which is 1.
>
> Thanks,
> Neha
>
>
> On Tue, Mar 5, 2013 at 6:48 AM, sourabh chaki >wrote:
>
> > Hi Neha,
> >
> > Thanks for your reply.
> >
> > ZkClient zk = new ZkClient(serverstring );
> > List partitions = zk.getChildren("/brokers/topics/test-topic/1");
> > // my topic = test-topic, broker id = 1
> >
> > This broker is associated with partition 0. So I am expecting output
> should
> > be [0] but I am getting empty response [].
> >
> > Is there anything wrong in the way I am calling it?
> > Please help.
> >
> > Thanks
> > Sourabh
> >
> >
> > On Wed, Feb 27, 2013 at 11:22 PM, Neha Narkhede > >wrote:
> >
> > > Sourabh,
> > >
> > > In 0.7, you can use zkclient or the zookeeper client library to query
> the
> > > following zookeeper path to get a number of partitions for a topic -
> > >
> > > get /brokers/topics/[topic]/[broker-id]
> > >
> > > This will give you # of partitions/broker, you can then add up this
> value
> > > across all brokers.
> > >
> > > Thanks,
> > > Neha
> > >
> > >
> > > On Wed, Feb 27, 2013 at 4:32 AM, sourabh chaki <
> chaki.sour...@gmail.com
> > > >wrote:
> > >
> > > > Hi All,
> > > >
> > > > I am using zookeeper based connector to get data from Kafka Server.
> Is
> > > > there any way to get all the partitions from zookeeper for a given
> > topic?
> > > >
> > > > I need do do that through java code.
> > > >
> > > > Please help!!!
> > > >
> > > > Thanks.
> > > >
> > >
> >
>
