[jira] [Created] (KAFKA-1077) OutOfMemoryError when consume large messaages
Xiejing created KAFKA-1077: -- Summary: OutOfMemoryError when consume large messaages Key: KAFKA-1077 URL: https://issues.apache.org/jira/browse/KAFKA-1077 Project: Kafka Issue Type: Bug Components: config, network Reporter: Xiejing Assignee: Jun Rao We set 'socket.request.max.bytes' to 100 * 1024 * 1024, but still see OutOfMemoryError when consuming messages(size 1M). e.g. [08/10/13 05:44:47:047 AM EDT] 102 ERROR network.BoundedByteBufferReceive: OOME with size 858861616 java.lang.OutOfMemoryError: Java heap space 858861616 is much larger than 100 * 1024 * 1024 but no InvalidRequestException is thrown in BoundedByteBufferReceive -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (KAFKA-1069) MBean kafka.cluster.Partition report wrong UnderReplicated status
[ https://issues.apache.org/jira/browse/KAFKA-1069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladislav Pernin updated KAFKA-1069: Affects Version/s: (was: 0.8.1) 0.8 > MBean kafka.cluster.Partition report wrong UnderReplicated status > - > > Key: KAFKA-1069 > URL: https://issues.apache.org/jira/browse/KAFKA-1069 > Project: Kafka > Issue Type: Bug >Affects Versions: 0.8 >Reporter: Vladislav Pernin > > Happens on kafka-0.8.0-beta1. > The MBean kafka.cluster.Partition shows an UnderReplicated status which is > wrong. > Let's take a simple example : > - one topic named topictest > - replication factor = 2 > - 3 nodes > Output of kafka-list-topic.sh command : > topic: topictestpartition: 0leader: 2 replicas: 3,2 isr: > 2,3 > topic: topictestpartition: 1leader: 1 replicas: 1,3 isr: > 1,3 > topic: topictestpartition: 2leader: 1 replicas: 2,1 isr: > 1,2 > topic: topictestpartition: 3leader: 1 replicas: 3,1 isr: > 1,3 > topic: topictestpartition: 4leader: 1 replicas: 1,2 isr: > 1,2 > topic: topictestpartition: 5leader: 2 replicas: 2,3 isr: > 2,3 > topic: topictestpartition: 6leader: 2 replicas: 3,2 isr: > 2,3 > topic: topictestpartition: 7leader: 1 replicas: 1,3 isr: > 1,3 > So everything is ok, each partition have one follower and the ISR size is 2. > Node 1 which is leader of partition 1 : > MBean "kafka.cluster":type="Partition",name="acorreler-1-UnderReplicated" = 0 > Node 3 which if a follower of partition 1 : > "kafka.cluster":type="Partition",name="acorreler-1-UnderReplicated" = 1 > On each node, the MBean > "kafka.server":type="ReplicaManager",name="UnderReplicatedPartitions" reports > 0 which is correct. > So it seems that the followers of a partition believe that their partitions > replicates are under replicated. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (KAFKA-1077) OutOfMemoryError when consume large messaages
[ https://issues.apache.org/jira/browse/KAFKA-1077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13789290#comment-13789290 ] Jun Rao commented on KAFKA-1077: Is this in 0.7 or 0.8? I assume the error is in the consumer side. It seems that you are fetching many topic/partitions, whose total response size is 850M. You may need to increase jvm heap size or using a smaller fetch size. > OutOfMemoryError when consume large messaages > - > > Key: KAFKA-1077 > URL: https://issues.apache.org/jira/browse/KAFKA-1077 > Project: Kafka > Issue Type: Bug > Components: config, network >Reporter: Xiejing >Assignee: Jun Rao > > We set 'socket.request.max.bytes' to 100 * 1024 * 1024, but still see > OutOfMemoryError when consuming messages(size 1M). > e.g. > [08/10/13 05:44:47:047 AM EDT] 102 ERROR network.BoundedByteBufferReceive: > OOME with size 858861616 > java.lang.OutOfMemoryError: Java heap space > 858861616 is much larger than 100 * 1024 * 1024 but no > InvalidRequestException is thrown in BoundedByteBufferReceive -- This message was sent by Atlassian JIRA (v6.1#6144)
Review Request 14534: Patch for KAFKA-1069
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/14534/ --- Review request for kafka. Bugs: KAFKA-1069 https://issues.apache.org/jira/browse/KAFKA-1069 Repository: kafka Description --- kafka-1069 Diffs - core/src/main/scala/kafka/cluster/Partition.scala a9bb3c8c450f99d8e110ae6709c5fe1fe09a8f77 Diff: https://reviews.apache.org/r/14534/diff/ Testing --- Thanks, Jun Rao
[jira] [Updated] (KAFKA-1069) MBean kafka.cluster.Partition report wrong UnderReplicated status
[ https://issues.apache.org/jira/browse/KAFKA-1069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-1069: --- Attachment: KAFKA-1069.patch > MBean kafka.cluster.Partition report wrong UnderReplicated status > - > > Key: KAFKA-1069 > URL: https://issues.apache.org/jira/browse/KAFKA-1069 > Project: Kafka > Issue Type: Bug >Affects Versions: 0.8 >Reporter: Vladislav Pernin > Attachments: KAFKA-1069.patch > > > Happens on kafka-0.8.0-beta1. > The MBean kafka.cluster.Partition shows an UnderReplicated status which is > wrong. > Let's take a simple example : > - one topic named topictest > - replication factor = 2 > - 3 nodes > Output of kafka-list-topic.sh command : > topic: topictestpartition: 0leader: 2 replicas: 3,2 isr: > 2,3 > topic: topictestpartition: 1leader: 1 replicas: 1,3 isr: > 1,3 > topic: topictestpartition: 2leader: 1 replicas: 2,1 isr: > 1,2 > topic: topictestpartition: 3leader: 1 replicas: 3,1 isr: > 1,3 > topic: topictestpartition: 4leader: 1 replicas: 1,2 isr: > 1,2 > topic: topictestpartition: 5leader: 2 replicas: 2,3 isr: > 2,3 > topic: topictestpartition: 6leader: 2 replicas: 3,2 isr: > 2,3 > topic: topictestpartition: 7leader: 1 replicas: 1,3 isr: > 1,3 > So everything is ok, each partition have one follower and the ISR size is 2. > Node 1 which is leader of partition 1 : > MBean "kafka.cluster":type="Partition",name="acorreler-1-UnderReplicated" = 0 > Node 3 which if a follower of partition 1 : > "kafka.cluster":type="Partition",name="acorreler-1-UnderReplicated" = 1 > On each node, the MBean > "kafka.server":type="ReplicaManager",name="UnderReplicatedPartitions" reports > 0 which is correct. > So it seems that the followers of a partition believe that their partitions > replicates are under replicated. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (KAFKA-1069) MBean kafka.cluster.Partition report wrong UnderReplicated status
[ https://issues.apache.org/jira/browse/KAFKA-1069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13789292#comment-13789292 ] Jun Rao commented on KAFKA-1069: Created reviewboard https://reviews.apache.org/r/14534/ > MBean kafka.cluster.Partition report wrong UnderReplicated status > - > > Key: KAFKA-1069 > URL: https://issues.apache.org/jira/browse/KAFKA-1069 > Project: Kafka > Issue Type: Bug >Affects Versions: 0.8 >Reporter: Vladislav Pernin > Attachments: KAFKA-1069.patch > > > Happens on kafka-0.8.0-beta1. > The MBean kafka.cluster.Partition shows an UnderReplicated status which is > wrong. > Let's take a simple example : > - one topic named topictest > - replication factor = 2 > - 3 nodes > Output of kafka-list-topic.sh command : > topic: topictestpartition: 0leader: 2 replicas: 3,2 isr: > 2,3 > topic: topictestpartition: 1leader: 1 replicas: 1,3 isr: > 1,3 > topic: topictestpartition: 2leader: 1 replicas: 2,1 isr: > 1,2 > topic: topictestpartition: 3leader: 1 replicas: 3,1 isr: > 1,3 > topic: topictestpartition: 4leader: 1 replicas: 1,2 isr: > 1,2 > topic: topictestpartition: 5leader: 2 replicas: 2,3 isr: > 2,3 > topic: topictestpartition: 6leader: 2 replicas: 3,2 isr: > 2,3 > topic: topictestpartition: 7leader: 1 replicas: 1,3 isr: > 1,3 > So everything is ok, each partition have one follower and the ISR size is 2. > Node 1 which is leader of partition 1 : > MBean "kafka.cluster":type="Partition",name="acorreler-1-UnderReplicated" = 0 > Node 3 which if a follower of partition 1 : > "kafka.cluster":type="Partition",name="acorreler-1-UnderReplicated" = 1 > On each node, the MBean > "kafka.server":type="ReplicaManager",name="UnderReplicatedPartitions" reports > 0 which is correct. > So it seems that the followers of a partition believe that their partitions > replicates are under replicated. -- This message was sent by Atlassian JIRA (v6.1#6144)
0.8 or 0.7.2 .Net client
Dear Kafka Dev Team, Could you please let me know if there is a 0.8 or 0.7.2 .Net client I could use for my project? Thank you very much. Best regards, Janos
Re: Review Request 14534: Patch for KAFKA-1069
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/14534/#review26773 --- Ship it! Ship It! - Neha Narkhede On Oct. 8, 2013, 3:25 p.m., Jun Rao wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/14534/ > --- > > (Updated Oct. 8, 2013, 3:25 p.m.) > > > Review request for kafka. > > > Bugs: KAFKA-1069 > https://issues.apache.org/jira/browse/KAFKA-1069 > > > Repository: kafka > > > Description > --- > > kafka-1069 > > > Diffs > - > > core/src/main/scala/kafka/cluster/Partition.scala > a9bb3c8c450f99d8e110ae6709c5fe1fe09a8f77 > > Diff: https://reviews.apache.org/r/14534/diff/ > > > Testing > --- > > > Thanks, > > Jun Rao > >
[jira] [Commented] (KAFKA-1069) MBean kafka.cluster.Partition report wrong UnderReplicated status
[ https://issues.apache.org/jira/browse/KAFKA-1069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13789310#comment-13789310 ] Neha Narkhede commented on KAFKA-1069: -- Thanks for the patch, +1 > MBean kafka.cluster.Partition report wrong UnderReplicated status > - > > Key: KAFKA-1069 > URL: https://issues.apache.org/jira/browse/KAFKA-1069 > Project: Kafka > Issue Type: Bug >Affects Versions: 0.8 >Reporter: Vladislav Pernin > Attachments: KAFKA-1069.patch > > > Happens on kafka-0.8.0-beta1. > The MBean kafka.cluster.Partition shows an UnderReplicated status which is > wrong. > Let's take a simple example : > - one topic named topictest > - replication factor = 2 > - 3 nodes > Output of kafka-list-topic.sh command : > topic: topictestpartition: 0leader: 2 replicas: 3,2 isr: > 2,3 > topic: topictestpartition: 1leader: 1 replicas: 1,3 isr: > 1,3 > topic: topictestpartition: 2leader: 1 replicas: 2,1 isr: > 1,2 > topic: topictestpartition: 3leader: 1 replicas: 3,1 isr: > 1,3 > topic: topictestpartition: 4leader: 1 replicas: 1,2 isr: > 1,2 > topic: topictestpartition: 5leader: 2 replicas: 2,3 isr: > 2,3 > topic: topictestpartition: 6leader: 2 replicas: 3,2 isr: > 2,3 > topic: topictestpartition: 7leader: 1 replicas: 1,3 isr: > 1,3 > So everything is ok, each partition have one follower and the ISR size is 2. > Node 1 which is leader of partition 1 : > MBean "kafka.cluster":type="Partition",name="acorreler-1-UnderReplicated" = 0 > Node 3 which if a follower of partition 1 : > "kafka.cluster":type="Partition",name="acorreler-1-UnderReplicated" = 1 > On each node, the MBean > "kafka.server":type="ReplicaManager",name="UnderReplicatedPartitions" reports > 0 which is correct. > So it seems that the followers of a partition believe that their partitions > replicates are under replicated. -- This message was sent by Atlassian JIRA (v6.1#6144)
Re: Review Request 14534: Patch for KAFKA-1069
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/14534/#review26774 --- Ship it! Ship It! - Guozhang Wang On Oct. 8, 2013, 3:25 p.m., Jun Rao wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/14534/ > --- > > (Updated Oct. 8, 2013, 3:25 p.m.) > > > Review request for kafka. > > > Bugs: KAFKA-1069 > https://issues.apache.org/jira/browse/KAFKA-1069 > > > Repository: kafka > > > Description > --- > > kafka-1069 > > > Diffs > - > > core/src/main/scala/kafka/cluster/Partition.scala > a9bb3c8c450f99d8e110ae6709c5fe1fe09a8f77 > > Diff: https://reviews.apache.org/r/14534/diff/ > > > Testing > --- > > > Thanks, > > Jun Rao > >
[jira] [Created] (KAFKA-1078) Update System Test to handle controller data returned by ZK
John Fung created KAFKA-1078:
Summary: Update System Test to handle controller data returned by
ZK
Key: KAFKA-1078
URL: https://issues.apache.org/jira/browse/KAFKA-1078
Project: Kafka
Issue Type: Bug
Reporter: John Fung
There is a change in the result returned by ZK when querying for controller:
2013-10-07 09:39:15,599 - DEBUG - executing command [ssh localhost
"JAVA_HOME=/export/apps/jdk/JDK-1_6_0_27
/home/user/kafka/bin/kafka-run-class.sh org.apache.zookeeper.ZooKeeperMain
-server localhost:2188 get /controller 2> /dev/null | tail -1"]
(kafka_system_test_utils)
1. Previously it returned : 1
2. It is currently returning : { "brokerid":1, "timestamp":"1381163914835",
"version":1 }
The impact is that System Test doesn't have the correct entity ID for bouncing
controller. There are a few test cases that are impacted by this change when
they are bouncing the controller.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Closed] (KAFKA-1069) MBean kafka.cluster.Partition report wrong UnderReplicated status
[ https://issues.apache.org/jira/browse/KAFKA-1069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao closed KAFKA-1069. -- > MBean kafka.cluster.Partition report wrong UnderReplicated status > - > > Key: KAFKA-1069 > URL: https://issues.apache.org/jira/browse/KAFKA-1069 > Project: Kafka > Issue Type: Bug >Affects Versions: 0.8 >Reporter: Vladislav Pernin >Assignee: Jun Rao > Fix For: 0.8 > > Attachments: KAFKA-1069.patch > > > Happens on kafka-0.8.0-beta1. > The MBean kafka.cluster.Partition shows an UnderReplicated status which is > wrong. > Let's take a simple example : > - one topic named topictest > - replication factor = 2 > - 3 nodes > Output of kafka-list-topic.sh command : > topic: topictestpartition: 0leader: 2 replicas: 3,2 isr: > 2,3 > topic: topictestpartition: 1leader: 1 replicas: 1,3 isr: > 1,3 > topic: topictestpartition: 2leader: 1 replicas: 2,1 isr: > 1,2 > topic: topictestpartition: 3leader: 1 replicas: 3,1 isr: > 1,3 > topic: topictestpartition: 4leader: 1 replicas: 1,2 isr: > 1,2 > topic: topictestpartition: 5leader: 2 replicas: 2,3 isr: > 2,3 > topic: topictestpartition: 6leader: 2 replicas: 3,2 isr: > 2,3 > topic: topictestpartition: 7leader: 1 replicas: 1,3 isr: > 1,3 > So everything is ok, each partition have one follower and the ISR size is 2. > Node 1 which is leader of partition 1 : > MBean "kafka.cluster":type="Partition",name="acorreler-1-UnderReplicated" = 0 > Node 3 which if a follower of partition 1 : > "kafka.cluster":type="Partition",name="acorreler-1-UnderReplicated" = 1 > On each node, the MBean > "kafka.server":type="ReplicaManager",name="UnderReplicatedPartitions" reports > 0 which is correct. > So it seems that the followers of a partition believe that their partitions > replicates are under replicated. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Resolved] (KAFKA-1069) MBean kafka.cluster.Partition report wrong UnderReplicated status
[ https://issues.apache.org/jira/browse/KAFKA-1069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao resolved KAFKA-1069. Resolution: Fixed Fix Version/s: 0.8 Assignee: Jun Rao Thanks for the review. Committed to 0.8. > MBean kafka.cluster.Partition report wrong UnderReplicated status > - > > Key: KAFKA-1069 > URL: https://issues.apache.org/jira/browse/KAFKA-1069 > Project: Kafka > Issue Type: Bug >Affects Versions: 0.8 >Reporter: Vladislav Pernin >Assignee: Jun Rao > Fix For: 0.8 > > Attachments: KAFKA-1069.patch > > > Happens on kafka-0.8.0-beta1. > The MBean kafka.cluster.Partition shows an UnderReplicated status which is > wrong. > Let's take a simple example : > - one topic named topictest > - replication factor = 2 > - 3 nodes > Output of kafka-list-topic.sh command : > topic: topictestpartition: 0leader: 2 replicas: 3,2 isr: > 2,3 > topic: topictestpartition: 1leader: 1 replicas: 1,3 isr: > 1,3 > topic: topictestpartition: 2leader: 1 replicas: 2,1 isr: > 1,2 > topic: topictestpartition: 3leader: 1 replicas: 3,1 isr: > 1,3 > topic: topictestpartition: 4leader: 1 replicas: 1,2 isr: > 1,2 > topic: topictestpartition: 5leader: 2 replicas: 2,3 isr: > 2,3 > topic: topictestpartition: 6leader: 2 replicas: 3,2 isr: > 2,3 > topic: topictestpartition: 7leader: 1 replicas: 1,3 isr: > 1,3 > So everything is ok, each partition have one follower and the ISR size is 2. > Node 1 which is leader of partition 1 : > MBean "kafka.cluster":type="Partition",name="acorreler-1-UnderReplicated" = 0 > Node 3 which if a follower of partition 1 : > "kafka.cluster":type="Partition",name="acorreler-1-UnderReplicated" = 1 > On each node, the MBean > "kafka.server":type="ReplicaManager",name="UnderReplicatedPartitions" reports > 0 which is correct. > So it seems that the followers of a partition believe that their partitions > replicates are under replicated. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (KAFKA-1078) Update System Test to handle controller data returned by ZK
[
https://issues.apache.org/jira/browse/KAFKA-1078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Fung updated KAFKA-1078:
-
Attachment: kafka-1078.patch
Updated kafka_system_test_utils.get_controller_attributes to read the json data
of controller.
> Update System Test to handle controller data returned by ZK
> ---
>
> Key: KAFKA-1078
> URL: https://issues.apache.org/jira/browse/KAFKA-1078
> Project: Kafka
> Issue Type: Bug
>Reporter: John Fung
> Attachments: kafka-1078.patch
>
>
> There is a change in the result returned by ZK when querying for controller:
> 2013-10-07 09:39:15,599 - DEBUG - executing command [ssh localhost
> "JAVA_HOME=/export/apps/jdk/JDK-1_6_0_27
> /home/user/kafka/bin/kafka-run-class.sh org.apache.zookeeper.ZooKeeperMain
> -server localhost:2188 get /controller 2> /dev/null | tail -1"]
> (kafka_system_test_utils)
> 1. Previously it returned : 1
> 2. It is currently returning : { "brokerid":1, "timestamp":"1381163914835",
> "version":1 }
> The impact is that System Test doesn't have the correct entity ID for
> bouncing controller. There are a few test cases that are impacted by this
> change when they are bouncing the controller.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (KAFKA-1078) Update System Test to handle controller data returned by ZK
[
https://issues.apache.org/jira/browse/KAFKA-1078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Fung updated KAFKA-1078:
-
Status: Patch Available (was: Open)
> Update System Test to handle controller data returned by ZK
> ---
>
> Key: KAFKA-1078
> URL: https://issues.apache.org/jira/browse/KAFKA-1078
> Project: Kafka
> Issue Type: Bug
>Reporter: John Fung
> Attachments: kafka-1078.patch
>
>
> There is a change in the result returned by ZK when querying for controller:
> 2013-10-07 09:39:15,599 - DEBUG - executing command [ssh localhost
> "JAVA_HOME=/export/apps/jdk/JDK-1_6_0_27
> /home/user/kafka/bin/kafka-run-class.sh org.apache.zookeeper.ZooKeeperMain
> -server localhost:2188 get /controller 2> /dev/null | tail -1"]
> (kafka_system_test_utils)
> 1. Previously it returned : 1
> 2. It is currently returning : { "brokerid":1, "timestamp":"1381163914835",
> "version":1 }
> The impact is that System Test doesn't have the correct entity ID for
> bouncing controller. There are a few test cases that are impacted by this
> change when they are bouncing the controller.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Closed] (KAFKA-1078) Update System Test to handle controller data returned by ZK
[
https://issues.apache.org/jira/browse/KAFKA-1078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jun Rao closed KAFKA-1078.
--
> Update System Test to handle controller data returned by ZK
> ---
>
> Key: KAFKA-1078
> URL: https://issues.apache.org/jira/browse/KAFKA-1078
> Project: Kafka
> Issue Type: Bug
>Reporter: John Fung
>Assignee: John Fung
> Fix For: 0.8
>
> Attachments: kafka-1078.patch
>
>
> There is a change in the result returned by ZK when querying for controller:
> 2013-10-07 09:39:15,599 - DEBUG - executing command [ssh localhost
> "JAVA_HOME=/export/apps/jdk/JDK-1_6_0_27
> /home/user/kafka/bin/kafka-run-class.sh org.apache.zookeeper.ZooKeeperMain
> -server localhost:2188 get /controller 2> /dev/null | tail -1"]
> (kafka_system_test_utils)
> 1. Previously it returned : 1
> 2. It is currently returning : { "brokerid":1, "timestamp":"1381163914835",
> "version":1 }
> The impact is that System Test doesn't have the correct entity ID for
> bouncing controller. There are a few test cases that are impacted by this
> change when they are bouncing the controller.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (KAFKA-1078) Update System Test to handle controller data returned by ZK
[
https://issues.apache.org/jira/browse/KAFKA-1078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jun Rao updated KAFKA-1078:
---
Resolution: Fixed
Fix Version/s: 0.8
Assignee: John Fung
Status: Resolved (was: Patch Available)
Thanks for the patch. Committed to 0.8.
> Update System Test to handle controller data returned by ZK
> ---
>
> Key: KAFKA-1078
> URL: https://issues.apache.org/jira/browse/KAFKA-1078
> Project: Kafka
> Issue Type: Bug
>Reporter: John Fung
>Assignee: John Fung
> Fix For: 0.8
>
> Attachments: kafka-1078.patch
>
>
> There is a change in the result returned by ZK when querying for controller:
> 2013-10-07 09:39:15,599 - DEBUG - executing command [ssh localhost
> "JAVA_HOME=/export/apps/jdk/JDK-1_6_0_27
> /home/user/kafka/bin/kafka-run-class.sh org.apache.zookeeper.ZooKeeperMain
> -server localhost:2188 get /controller 2> /dev/null | tail -1"]
> (kafka_system_test_utils)
> 1. Previously it returned : 1
> 2. It is currently returning : { "brokerid":1, "timestamp":"1381163914835",
> "version":1 }
> The impact is that System Test doesn't have the correct entity ID for
> bouncing controller. There are a few test cases that are impacted by this
> change when they are bouncing the controller.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Comment Edited] (KAFKA-1078) Update System Test to handle controller data returned by ZK
[
https://issues.apache.org/jira/browse/KAFKA-1078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13789350#comment-13789350
]
John Fung edited comment on KAFKA-1078 at 10/8/13 4:27 PM:
---
1. Before the patch (no controller entity id found and system test throws
exception)
2013-10-04 01:04:38,070 - INFO -
==
2013-10-04 01:04:38,070 - INFO - Found controller with entity id:
2013-10-04 01:04:38,070 - INFO -
==
2013-10-04 01:04:38,070 - INFO -
==
2013-10-04 01:04:38,070 - INFO - stopping controller : { "brokerid":1,
"timestamp":"1380873836165", "version":1 }
2013-10-04 01:04:38,070 - INFO -
==
2013-10-04 01:04:38,071 - INFO -
==
2013-10-04 01:04:38,071 - INFO - Exception while running test ''
2013-10-04 01:04:38,071 - INFO -
==
Traceback (most recent call last):
File
"/home/user/kafka/system_test/replication_testsuite/replica_basic_test.py",
line 301, in runTest
kafka_system_test_utils.stop_remote_entity(self.systemTestEnv,
controllerDict["entity_id"],
self.testcaseEnv.entityBrokerParentPidDict[controllerDict["entity_id"]])
KeyError: ''
2. After the patch (controller entity id found)
2013-10-07 16:30:48,122 - INFO -
==
2013-10-07 16:30:48,122 - INFO - Found controller with entity id: 1
2013-10-07 16:30:48,122 - INFO -
==
2013-10-07 16:30:48,123 - INFO -
==
2013-10-07 16:30:48,123 - INFO - stopping controller : 1
2013-10-07 16:30:48,123 - INFO -
==
2013-10-07 16:30:48,123 - DEBUG - executing command [ssh localhost 'pid=26805;
prev_pid=""; echo $pid; while [[ "x$pid" != "x" ]]; do prev_pid=$pid; for
child in $(ps -o pid,ppid a
x | awk "{ if ( \$2 == $pid ) { print \$1 }}"); do echo $child; pid=$child;
done; if [ $prev_pid == $pid ]; then break; fi; done' 2> /dev/null
(system_test_utils)
2013-10-07 16:30:48,477 - DEBUG - terminating (SIGTERM) process id: 26805 in
host: localhost (kafka_system_test_utils)
was (Author: jfung):
1. Before the patch (no controller entity id found)
2013-10-04 01:04:38,070 - INFO -
==
2013-10-04 01:04:38,070 - INFO - Found controller with entity id:
2013-10-04 01:04:38,070 - INFO -
==
2013-10-04 01:04:38,070 - INFO -
==
2013-10-04 01:04:38,070 - INFO - stopping controller : { "brokerid":1,
"timestamp":"1380873836165", "version":1 }
2013-10-04 01:04:38,070 - INFO -
==
2013-10-04 01:04:38,071 - INFO -
==
2013-10-04 01:04:38,071 - INFO - Exception while running test ''
2013-10-04 01:04:38,071 - INFO -
==
Traceback (most recent call last):
File
"/home/user/kafka/system_test/replication_testsuite/replica_basic_test.py",
line 301, in runTest
kafka_system_test_utils.stop_remote_entity(self.systemTestEnv,
controllerDict["entity_id"],
self.testcaseEnv.entityBrokerParentPidDict[controllerDict["entity_id"]])
KeyError: ''
2. After the patch (controller entity id found)
2013-10-07 16:30:48,122 - INFO -
==
2013-10-07 16:30:48,122 - INFO - Found controller with entity id: 1
2013-10-07 16:30:48,122 - INFO -
==
2013-10-07 16:30:48,123 - INFO -
==
2013-10-07 16:30:48,123 - INFO - stopping controller : 1
2013-10-07 16:30:48,123 - INFO -
==
2013-10-07 16:30:48,123 - DEBUG - executing command [ssh localhost 'pid=26805;
prev_pid=""; echo $pid; while [[ "x$pid" != "x" ]]; do prev_pid=$pid; for
child in $(ps -o pid,ppid a
x | awk "{ if ( \$2 == $pid ) { print \$1 }}"); do echo $child; pid=$child;
done; if [ $prev_pid == $pid ]; then break; fi; done' 2> /dev/null
(system_test_utils)
2013-10-07 16:30:48,477 - DEBUG - terminating (SIGTERM) process id: 26805 in
host: localhost (kafka_system_test_utils)
> Update System Test to handle controller data returned by ZK
> ---
>
> Key: KAFKA-1078
> URL: https://issues.apache.org/jira/browse/KAFKA-1078
> Project: Kafka
> Issue Type: Bug
[jira] [Commented] (KAFKA-1078) Update System Test to handle controller data returned by ZK
[
https://issues.apache.org/jira/browse/KAFKA-1078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13789350#comment-13789350
]
John Fung commented on KAFKA-1078:
--
1. Before the patch (no controller entity id found)
2013-10-04 01:04:38,070 - INFO -
==
2013-10-04 01:04:38,070 - INFO - Found controller with entity id:
2013-10-04 01:04:38,070 - INFO -
==
2013-10-04 01:04:38,070 - INFO -
==
2013-10-04 01:04:38,070 - INFO - stopping controller : { "brokerid":1,
"timestamp":"1380873836165", "version":1 }
2013-10-04 01:04:38,070 - INFO -
==
2013-10-04 01:04:38,071 - INFO -
==
2013-10-04 01:04:38,071 - INFO - Exception while running test ''
2013-10-04 01:04:38,071 - INFO -
==
Traceback (most recent call last):
File
"/home/user/kafka/system_test/replication_testsuite/replica_basic_test.py",
line 301, in runTest
kafka_system_test_utils.stop_remote_entity(self.systemTestEnv,
controllerDict["entity_id"],
self.testcaseEnv.entityBrokerParentPidDict[controllerDict["entity_id"]])
KeyError: ''
2. After the patch (controller entity id found)
2013-10-07 16:30:48,122 - INFO -
==
2013-10-07 16:30:48,122 - INFO - Found controller with entity id: 1
2013-10-07 16:30:48,122 - INFO -
==
2013-10-07 16:30:48,123 - INFO -
==
2013-10-07 16:30:48,123 - INFO - stopping controller : 1
2013-10-07 16:30:48,123 - INFO -
==
2013-10-07 16:30:48,123 - DEBUG - executing command [ssh localhost 'pid=26805;
prev_pid=""; echo $pid; while [[ "x$pid" != "x" ]]; do prev_pid=$pid; for
child in $(ps -o pid,ppid a
x | awk "{ if ( \$2 == $pid ) { print \$1 }}"); do echo $child; pid=$child;
done; if [ $prev_pid == $pid ]; then break; fi; done' 2> /dev/null
(system_test_utils)
2013-10-07 16:30:48,477 - DEBUG - terminating (SIGTERM) process id: 26805 in
host: localhost (kafka_system_test_utils)
> Update System Test to handle controller data returned by ZK
> ---
>
> Key: KAFKA-1078
> URL: https://issues.apache.org/jira/browse/KAFKA-1078
> Project: Kafka
> Issue Type: Bug
>Reporter: John Fung
>Assignee: John Fung
> Fix For: 0.8
>
> Attachments: kafka-1078.patch
>
>
> There is a change in the result returned by ZK when querying for controller:
> 2013-10-07 09:39:15,599 - DEBUG - executing command [ssh localhost
> "JAVA_HOME=/export/apps/jdk/JDK-1_6_0_27
> /home/user/kafka/bin/kafka-run-class.sh org.apache.zookeeper.ZooKeeperMain
> -server localhost:2188 get /controller 2> /dev/null | tail -1"]
> (kafka_system_test_utils)
> 1. Previously it returned : 1
> 2. It is currently returning : { "brokerid":1, "timestamp":"1381163914835",
> "version":1 }
> The impact is that System Test doesn't have the correct entity ID for
> bouncing controller. There are a few test cases that are impacted by this
> change when they are bouncing the controller.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
Review Request 14538: Patch for KAFKA-954
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/14538/ --- Review request for kafka. Bugs: KAFKA-954 https://issues.apache.org/jira/browse/KAFKA-954 Repository: kafka Description --- kafka-954 Diffs - README.md 9879427d4103354c7c3872c0b6caf454c1204a76 Diff: https://reviews.apache.org/r/14538/diff/ Testing --- Thanks, Jun Rao
[jira] [Updated] (KAFKA-954) tidy up README file for better general availability
[ https://issues.apache.org/jira/browse/KAFKA-954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-954: -- Attachment: KAFKA-954.patch > tidy up README file for better general availability > --- > > Key: KAFKA-954 > URL: https://issues.apache.org/jira/browse/KAFKA-954 > Project: Kafka > Issue Type: Bug >Affects Versions: 0.8 >Reporter: Joe Stein >Assignee: Jun Rao >Priority: Blocker > Labels: 0.8.0-beta1 > Fix For: 0.8 > > Attachments: KAFKA-954.patch > > > e.g. how to start server after building and all would be good too -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (KAFKA-954) tidy up README file for better general availability
[ https://issues.apache.org/jira/browse/KAFKA-954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13789379#comment-13789379 ] Jun Rao commented on KAFKA-954: --- Created reviewboard https://reviews.apache.org/r/14538/ > tidy up README file for better general availability > --- > > Key: KAFKA-954 > URL: https://issues.apache.org/jira/browse/KAFKA-954 > Project: Kafka > Issue Type: Bug >Affects Versions: 0.8 >Reporter: Joe Stein >Assignee: Jun Rao >Priority: Blocker > Labels: 0.8.0-beta1 > Fix For: 0.8 > > Attachments: KAFKA-954.patch > > > e.g. how to start server after building and all would be good too -- This message was sent by Atlassian JIRA (v6.1#6144)
Why JUnit3?
Hi, kafka team and all! Is there anything that stops kafka from moving to scalatest completely? It is supported by major IDE players (IDEA, Eclipse plugin) and pretty mature atm. Alternatively we could just bump to junit4, if we're not ready to be completely scalish. If you concerned about change in dependencies, we could target this move for 0.8.1 or even 0.9.
Re: Review Request 14538: Patch for KAFKA-954
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/14538/#review26776 --- Ship it! Ship It! - Joe Stein On Oct. 8, 2013, 4:50 p.m., Jun Rao wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/14538/ > --- > > (Updated Oct. 8, 2013, 4:50 p.m.) > > > Review request for kafka. > > > Bugs: KAFKA-954 > https://issues.apache.org/jira/browse/KAFKA-954 > > > Repository: kafka > > > Description > --- > > kafka-954 > > > Diffs > - > > README.md 9879427d4103354c7c3872c0b6caf454c1204a76 > > Diff: https://reviews.apache.org/r/14538/diff/ > > > Testing > --- > > > Thanks, > > Jun Rao > >
Re: Why JUnit3?
Sounds like a good contrib for 0.9 would you create a JIRA and contrib a patch for it? I would be happy to help review and commit it if the changes make sense. I can't say I know off the top of my head what changes there would be maybe highlight those in the ticket , some other pros for doing this are always good imho, keeping current in larger communities makes the project better supported is the benefit for making changes to "things not broken" that have larger impact so I am almost always a +1 on these On Tue, Oct 8, 2013 at 1:06 PM, Kostya Golikov wrote: > Hi, kafka team and all! > > Is there anything that stops kafka from moving to scalatest completely? It > is supported by major IDE players (IDEA, Eclipse plugin) and pretty mature > atm. Alternatively we could just bump to junit4, if we're not ready to be > completely scalish. If you concerned about change in dependencies, we could > target this move for 0.8.1 or even 0.9. >
Re: Review Request 14538: Patch for KAFKA-954
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/14538/#review26777 --- Ship it! Ship It! - Neha Narkhede On Oct. 8, 2013, 4:50 p.m., Jun Rao wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/14538/ > --- > > (Updated Oct. 8, 2013, 4:50 p.m.) > > > Review request for kafka. > > > Bugs: KAFKA-954 > https://issues.apache.org/jira/browse/KAFKA-954 > > > Repository: kafka > > > Description > --- > > kafka-954 > > > Diffs > - > > README.md 9879427d4103354c7c3872c0b6caf454c1204a76 > > Diff: https://reviews.apache.org/r/14538/diff/ > > > Testing > --- > > > Thanks, > > Jun Rao > >
[jira] [Created] (KAFKA-1079) Liars in PrimitiveApiTest that promise to test api in compression mode, but don't do this actually
Kostya Golikov created KAFKA-1079: - Summary: Liars in PrimitiveApiTest that promise to test api in compression mode, but don't do this actually Key: KAFKA-1079 URL: https://issues.apache.org/jira/browse/KAFKA-1079 Project: Kafka Issue Type: Test Components: core Affects Versions: 0.8 Reporter: Kostya Golikov Priority: Minor Long time ago (0.7) we had ByteBufferMessageSet as a part of api and BBM allowed us to control compression. Times goes on and now PrimitiveApiTest have methods that promise to test api with compression enabled, but in fact they don't. Moreover this methods almost entirely copy their counterparts without compression. In particular I'm talking about `testProduceAndMultiFetch` / `testProduceAndMultiFetchWithCompression` and `testMultiProduce`/`testMultiProduceWithCompression` pairs. The fix could be super-easy and soundness -- just parameterize methods with producer of each type (with/without compression). Sadly but it isn't feasible for junit3, so straightforward solution is to do the same ugly thing as `testDefaultEncoderProducerAndFetchWithCompression` method does -- forget about class-wide producer and roll-out it's own. I will attach path if that is a problem indeed. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (KAFKA-1079) Liars in PrimitiveApiTest that promise to test api in compression mode, but don't do this actually
[ https://issues.apache.org/jira/browse/KAFKA-1079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kostya Golikov updated KAFKA-1079: -- Description: Long time ago (0.7) we had ByteBufferMessageSet as a part of api and it's allowed us to control compression. Times goes on and now PrimitiveApiTest have methods that promise to test api with compression enabled, but in fact they don't. Moreover this methods almost entirely copy their counterparts without compression. In particular I'm talking about `testProduceAndMultiFetch` / `testProduceAndMultiFetchWithCompression` and `testMultiProduce`/`testMultiProduceWithCompression` pairs. The fix could be super-easy and soundness -- just parameterize methods with producer of each type (with/without compression). Sadly but it isn't feasible for junit3, so straightforward solution is to do the same ugly thing as `testDefaultEncoderProducerAndFetchWithCompression` method does -- forget about class-wide producer and roll-out it's own. I will attach path if that is a problem indeed. was: Long time ago (0.7) we had ByteBufferMessageSet as a part of api and BBM allowed us to control compression. Times goes on and now PrimitiveApiTest have methods that promise to test api with compression enabled, but in fact they don't. Moreover this methods almost entirely copy their counterparts without compression. In particular I'm talking about `testProduceAndMultiFetch` / `testProduceAndMultiFetchWithCompression` and `testMultiProduce`/`testMultiProduceWithCompression` pairs. The fix could be super-easy and soundness -- just parameterize methods with producer of each type (with/without compression). Sadly but it isn't feasible for junit3, so straightforward solution is to do the same ugly thing as `testDefaultEncoderProducerAndFetchWithCompression` method does -- forget about class-wide producer and roll-out it's own. I will attach path if that is a problem indeed. > Liars in PrimitiveApiTest that promise to test api in compression mode, but > don't do this actually > -- > > Key: KAFKA-1079 > URL: https://issues.apache.org/jira/browse/KAFKA-1079 > Project: Kafka > Issue Type: Test > Components: core >Affects Versions: 0.8 >Reporter: Kostya Golikov >Priority: Minor > Labels: newbie, test > > Long time ago (0.7) we had ByteBufferMessageSet as a part of api and it's > allowed us to control compression. Times goes on and now PrimitiveApiTest > have methods that promise to test api with compression enabled, but in fact > they don't. Moreover this methods almost entirely copy their counterparts > without compression. In particular I'm talking about > `testProduceAndMultiFetch` / `testProduceAndMultiFetchWithCompression` and > `testMultiProduce`/`testMultiProduceWithCompression` pairs. > The fix could be super-easy and soundness -- just parameterize methods with > producer of each type (with/without compression). Sadly but it isn't feasible > for junit3, so straightforward solution is to do the same ugly thing as > `testDefaultEncoderProducerAndFetchWithCompression` method does -- forget > about class-wide producer and roll-out it's own. I will attach path if that > is a problem indeed. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Work started] (KAFKA-954) tidy up README file for better general availability
[ https://issues.apache.org/jira/browse/KAFKA-954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KAFKA-954 started by Jun Rao. > tidy up README file for better general availability > --- > > Key: KAFKA-954 > URL: https://issues.apache.org/jira/browse/KAFKA-954 > Project: Kafka > Issue Type: Bug >Affects Versions: 0.8 >Reporter: Joe Stein >Assignee: Jun Rao >Priority: Blocker > Labels: 0.8.0-beta1 > Fix For: 0.8 > > Attachments: KAFKA-954.patch > > > e.g. how to start server after building and all would be good too -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (KAFKA-954) tidy up README file for better general availability
[ https://issues.apache.org/jira/browse/KAFKA-954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-954: -- Status: Patch Available (was: In Progress) > tidy up README file for better general availability > --- > > Key: KAFKA-954 > URL: https://issues.apache.org/jira/browse/KAFKA-954 > Project: Kafka > Issue Type: Bug >Affects Versions: 0.8 >Reporter: Joe Stein >Assignee: Jun Rao >Priority: Blocker > Labels: 0.8.0-beta1 > Fix For: 0.8 > > Attachments: KAFKA-954.patch > > > e.g. how to start server after building and all would be good too -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (KAFKA-954) tidy up README file for better general availability
[ https://issues.apache.org/jira/browse/KAFKA-954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-954: -- Resolution: Fixed Status: Resolved (was: Patch Available) Thanks for the review. Committed to 0.8. > tidy up README file for better general availability > --- > > Key: KAFKA-954 > URL: https://issues.apache.org/jira/browse/KAFKA-954 > Project: Kafka > Issue Type: Bug >Affects Versions: 0.8 >Reporter: Joe Stein >Assignee: Jun Rao >Priority: Blocker > Labels: 0.8.0-beta1 > Fix For: 0.8 > > Attachments: KAFKA-954.patch > > > e.g. how to start server after building and all would be good too -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Closed] (KAFKA-954) tidy up README file for better general availability
[ https://issues.apache.org/jira/browse/KAFKA-954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao closed KAFKA-954. - > tidy up README file for better general availability > --- > > Key: KAFKA-954 > URL: https://issues.apache.org/jira/browse/KAFKA-954 > Project: Kafka > Issue Type: Bug >Affects Versions: 0.8 >Reporter: Joe Stein >Assignee: Jun Rao >Priority: Blocker > Labels: 0.8.0-beta1 > Fix For: 0.8 > > Attachments: KAFKA-954.patch > > > e.g. how to start server after building and all would be good too -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (KAFKA-369) remove ZK dependency on producer
[ https://issues.apache.org/jira/browse/KAFKA-369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13789689#comment-13789689 ] Otis Gospodnetic commented on KAFKA-369: I'm completely late on this and don't have the full context, so my apologies if this is a naive question, but why would you want to have addresses of brokers in producer's configs and not have producer get that from Zookeeper? I thought using ZK for cluster-wide config sharing was a good thing, but there must be some reason why you chose not to do that here. Can anyone shed some light on this 14 months after the commit? :) > remove ZK dependency on producer > > > Key: KAFKA-369 > URL: https://issues.apache.org/jira/browse/KAFKA-369 > Project: Kafka > Issue Type: Sub-task > Components: core >Affects Versions: 0.8 >Reporter: Jun Rao >Assignee: Yang Ye > Fix For: 0.8 > > Attachments: kafka_369_v1.diff, kafka_369_v2.diff, kafka_369_v3.diff, > kafka_369_v4.diff, kafka_369_v5.diff, kafka_369_v6.diff, kafka_369_v7.diff, > kafka_369_v8.diff, kafka_369_v9.diff > > Original Estimate: 252h > Remaining Estimate: 252h > > Currently, the only place that ZK is actually used is in BrokerPartitionInfo. > We use ZK to get a list of brokers for making TopicMetadataRequest requests. > Instead, we can provide a list of brokers in the producer config directly. > That way, the producer client is no longer dependant on ZK. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (KAFKA-369) remove ZK dependency on producer
[ https://issues.apache.org/jira/browse/KAFKA-369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13789801#comment-13789801 ] Jay Kreps commented on KAFKA-369: - I guess all distributed systems have a problem of bootstrapping knowledge about the cluster. In 0.7 the client had a list of hard-coded zk brokers which you connect to, and from this you can discover the state of the kafka cluster. There are a number of problems with embedding zookeeper in clients at large scale: 1. Makes language support hard 2. GC in clients causes havoc 3. Massive zookeeper load Instead we use the brokers for cluster discovery. As before you need a couple hardcoded urls to bootstrap metadata from. These can be any three random kafka brokers (or else use DNS or a vip) this is by no means intended to be a complete list of your brokers and the producer is not limited in any way to producing to the brokers it uses to get cluster metadata. These hosts are used only to find out the state of the cluster the first time. This isn't worse then the zookeeper situation (you still hardcode a couple urls) but gets rid of all the other zk issues. > remove ZK dependency on producer > > > Key: KAFKA-369 > URL: https://issues.apache.org/jira/browse/KAFKA-369 > Project: Kafka > Issue Type: Sub-task > Components: core >Affects Versions: 0.8 >Reporter: Jun Rao >Assignee: Yang Ye > Fix For: 0.8 > > Attachments: kafka_369_v1.diff, kafka_369_v2.diff, kafka_369_v3.diff, > kafka_369_v4.diff, kafka_369_v5.diff, kafka_369_v6.diff, kafka_369_v7.diff, > kafka_369_v8.diff, kafka_369_v9.diff > > Original Estimate: 252h > Remaining Estimate: 252h > > Currently, the only place that ZK is actually used is in BrokerPartitionInfo. > We use ZK to get a list of brokers for making TopicMetadataRequest requests. > Instead, we can provide a list of brokers in the producer config directly. > That way, the producer client is no longer dependant on ZK. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (KAFKA-369) remove ZK dependency on producer
[ https://issues.apache.org/jira/browse/KAFKA-369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13789851#comment-13789851 ] Neha Narkhede commented on KAFKA-369: - In addition to the above, the cluster metadata approach is more scalable since you get all metadata for several topics in one RPC roundtrip vs reading several paths per topic from zookeeper. > remove ZK dependency on producer > > > Key: KAFKA-369 > URL: https://issues.apache.org/jira/browse/KAFKA-369 > Project: Kafka > Issue Type: Sub-task > Components: core >Affects Versions: 0.8 >Reporter: Jun Rao >Assignee: Yang Ye > Fix For: 0.8 > > Attachments: kafka_369_v1.diff, kafka_369_v2.diff, kafka_369_v3.diff, > kafka_369_v4.diff, kafka_369_v5.diff, kafka_369_v6.diff, kafka_369_v7.diff, > kafka_369_v8.diff, kafka_369_v9.diff > > Original Estimate: 252h > Remaining Estimate: 252h > > Currently, the only place that ZK is actually used is in BrokerPartitionInfo. > We use ZK to get a list of brokers for making TopicMetadataRequest requests. > Instead, we can provide a list of brokers in the producer config directly. > That way, the producer client is no longer dependant on ZK. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (KAFKA-369) remove ZK dependency on producer
[ https://issues.apache.org/jira/browse/KAFKA-369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13789893#comment-13789893 ] Otis Gospodnetic commented on KAFKA-369: Interesting. Sounds like brokers act as "portals into the whole Kafka cluster", so to speak. I wasn't aware of these ZK issues. Just for my edification - are those issues specific to the ZK client that was used or how it was used in Kafka producer? > remove ZK dependency on producer > > > Key: KAFKA-369 > URL: https://issues.apache.org/jira/browse/KAFKA-369 > Project: Kafka > Issue Type: Sub-task > Components: core >Affects Versions: 0.8 >Reporter: Jun Rao >Assignee: Yang Ye > Fix For: 0.8 > > Attachments: kafka_369_v1.diff, kafka_369_v2.diff, kafka_369_v3.diff, > kafka_369_v4.diff, kafka_369_v5.diff, kafka_369_v6.diff, kafka_369_v7.diff, > kafka_369_v8.diff, kafka_369_v9.diff > > Original Estimate: 252h > Remaining Estimate: 252h > > Currently, the only place that ZK is actually used is in BrokerPartitionInfo. > We use ZK to get a list of brokers for making TopicMetadataRequest requests. > Instead, we can provide a list of brokers in the producer config directly. > That way, the producer client is no longer dependant on ZK. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (KAFKA-1077) OutOfMemoryError when consume large messaages
[
https://issues.apache.org/jira/browse/KAFKA-1077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13789938#comment-13789938
]
Xiejing commented on KAFKA-1077:
Thanks Rao. The error was thrown in BoundedByteBufferReceive.scala, which seems
has protection mechanism
if(size > maxSize)
throw new InvalidRequestException("Request of length %d is not valid,
it is larger than the maximum size of %d bytes.".format(size, maxSize))
contentBuffer = byteBufferAllocate(size)
So what makes me confused is, in my case, 'size' is 850M and 'maxSize' is 100M,
why InvalidRequestException hasn't been thrown?
> OutOfMemoryError when consume large messaages
> -
>
> Key: KAFKA-1077
> URL: https://issues.apache.org/jira/browse/KAFKA-1077
> Project: Kafka
> Issue Type: Bug
> Components: config, network
>Reporter: Xiejing
>Assignee: Jun Rao
>
> We set 'socket.request.max.bytes' to 100 * 1024 * 1024, but still see
> OutOfMemoryError when consuming messages(size 1M).
> e.g.
> [08/10/13 05:44:47:047 AM EDT] 102 ERROR network.BoundedByteBufferReceive:
> OOME with size 858861616
> java.lang.OutOfMemoryError: Java heap space
> 858861616 is much larger than 100 * 1024 * 1024 but no
> InvalidRequestException is thrown in BoundedByteBufferReceive
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (KAFKA-1079) Liars in PrimitiveApiTest that promise to test api in compression mode, but don't do this actually
[ https://issues.apache.org/jira/browse/KAFKA-1079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13789985#comment-13789985 ] Jun Rao commented on KAFKA-1079: Thanks for filing the jira. Yes, these two tests are supposed to enable compression, but they are not. I agree that using parameterized methods for unit tests will be ideal. I can't remember why we used junit3. So, it may not be necessary. Look forward to your patch. > Liars in PrimitiveApiTest that promise to test api in compression mode, but > don't do this actually > -- > > Key: KAFKA-1079 > URL: https://issues.apache.org/jira/browse/KAFKA-1079 > Project: Kafka > Issue Type: Test > Components: core >Affects Versions: 0.8 >Reporter: Kostya Golikov >Priority: Minor > Labels: newbie, test > > Long time ago (0.7) we had ByteBufferMessageSet as a part of api and it's > allowed us to control compression. Times goes on and now PrimitiveApiTest > have methods that promise to test api with compression enabled, but in fact > they don't. Moreover this methods almost entirely copy their counterparts > without compression. In particular I'm talking about > `testProduceAndMultiFetch` / `testProduceAndMultiFetchWithCompression` and > `testMultiProduce`/`testMultiProduceWithCompression` pairs. > The fix could be super-easy and soundness -- just parameterize methods with > producer of each type (with/without compression). Sadly but it isn't feasible > for junit3, so straightforward solution is to do the same ugly thing as > `testDefaultEncoderProducerAndFetchWithCompression` method does -- forget > about class-wide producer and roll-out it's own. I will attach path if that > is a problem indeed. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (KAFKA-1077) OutOfMemoryError when consume large messaages
[ https://issues.apache.org/jira/browse/KAFKA-1077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13789992#comment-13789992 ] Jun Rao commented on KAFKA-1077: That check is on the broker side. So if you send a request larger than 100MB, the broker will throw an exception. In your case, I thought you get the OOME on the consumer. So, the fetch request is not big, but fetch response is. > OutOfMemoryError when consume large messaages > - > > Key: KAFKA-1077 > URL: https://issues.apache.org/jira/browse/KAFKA-1077 > Project: Kafka > Issue Type: Bug > Components: config, network >Reporter: Xiejing >Assignee: Jun Rao > > We set 'socket.request.max.bytes' to 100 * 1024 * 1024, but still see > OutOfMemoryError when consuming messages(size 1M). > e.g. > [08/10/13 05:44:47:047 AM EDT] 102 ERROR network.BoundedByteBufferReceive: > OOME with size 858861616 > java.lang.OutOfMemoryError: Java heap space > 858861616 is much larger than 100 * 1024 * 1024 but no > InvalidRequestException is thrown in BoundedByteBufferReceive -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (KAFKA-369) remove ZK dependency on producer
[ https://issues.apache.org/jira/browse/KAFKA-369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13790018#comment-13790018 ] Neha Narkhede commented on KAFKA-369: - [~otis] Those zk issues exist with any zookeeper client. > remove ZK dependency on producer > > > Key: KAFKA-369 > URL: https://issues.apache.org/jira/browse/KAFKA-369 > Project: Kafka > Issue Type: Sub-task > Components: core >Affects Versions: 0.8 >Reporter: Jun Rao >Assignee: Yang Ye > Fix For: 0.8 > > Attachments: kafka_369_v1.diff, kafka_369_v2.diff, kafka_369_v3.diff, > kafka_369_v4.diff, kafka_369_v5.diff, kafka_369_v6.diff, kafka_369_v7.diff, > kafka_369_v8.diff, kafka_369_v9.diff > > Original Estimate: 252h > Remaining Estimate: 252h > > Currently, the only place that ZK is actually used is in BrokerPartitionInfo. > We use ZK to get a list of brokers for making TopicMetadataRequest requests. > Instead, we can provide a list of brokers in the producer config directly. > That way, the producer client is no longer dependant on ZK. -- This message was sent by Atlassian JIRA (v6.1#6144)
Review Request 14553: Move AddPartitions into TopicCommand in trunk
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/14553/ --- Review request for kafka. Bugs: KAFKA-1052 https://issues.apache.org/jira/browse/KAFKA-1052 Repository: kafka Description --- Move AddPartitions into TopicCommand Diffs - core/src/main/scala/kafka/admin/AddPartitionsCommand.scala c74d9c224565d475d4104467ba36c0a287b68239 core/src/main/scala/kafka/admin/AdminUtils.scala 6560fc668e8b8feedc1e3f63c11aac2ff370c8d0 core/src/main/scala/kafka/admin/TopicCommand.scala 06bbd37a5af1ea5bd5f6c42cb9063cc7d6fd381b core/src/test/scala/unit/kafka/admin/AddPartitionsTest.scala 09254ccac2a4d5885a372f1f21b0a5813a930ded Diff: https://reviews.apache.org/r/14553/diff/ Testing --- Thanks, Sriram Subramanian
[jira] [Updated] (KAFKA-1052) integrate add-partitions command into topicCommand
[ https://issues.apache.org/jira/browse/KAFKA-1052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sriram Subramanian updated KAFKA-1052: -- Attachment: KAFKA-1052.patch > integrate add-partitions command into topicCommand > -- > > Key: KAFKA-1052 > URL: https://issues.apache.org/jira/browse/KAFKA-1052 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 0.8.1 >Reporter: Jun Rao >Assignee: Sriram Subramanian > Attachments: KAFKA-1052.patch > > > After merging from 0.8 (kafka-1051), we dragged in a new admin command > add-partitions. This needs to be integrated with the general topicCommand. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (KAFKA-1052) integrate add-partitions command into topicCommand
[ https://issues.apache.org/jira/browse/KAFKA-1052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13790090#comment-13790090 ] Sriram Subramanian commented on KAFKA-1052: --- Created reviewboard https://reviews.apache.org/r/14554/ > integrate add-partitions command into topicCommand > -- > > Key: KAFKA-1052 > URL: https://issues.apache.org/jira/browse/KAFKA-1052 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 0.8.1 >Reporter: Jun Rao >Assignee: Sriram Subramanian > Attachments: KAFKA-1052.patch > > > After merging from 0.8 (kafka-1051), we dragged in a new admin command > add-partitions. This needs to be integrated with the general topicCommand. -- This message was sent by Atlassian JIRA (v6.1#6144)
Review Request 14554: Move AddPartitions into TopicCommand in trunk
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/14554/ --- Review request for kafka. Bugs: KAFKA-1052 https://issues.apache.org/jira/browse/KAFKA-1052 Repository: kafka Description --- Move AddPartitions into TopicCommand Diffs - core/src/main/scala/kafka/admin/AddPartitionsCommand.scala c74d9c224565d475d4104467ba36c0a287b68239 core/src/main/scala/kafka/admin/AdminUtils.scala 6560fc668e8b8feedc1e3f63c11aac2ff370c8d0 core/src/main/scala/kafka/admin/TopicCommand.scala 06bbd37a5af1ea5bd5f6c42cb9063cc7d6fd381b core/src/test/scala/unit/kafka/admin/AddPartitionsTest.scala 09254ccac2a4d5885a372f1f21b0a5813a930ded Diff: https://reviews.apache.org/r/14554/diff/ Testing --- Thanks, Sriram Subramanian
