[jira] [Updated] (KAFKA-1913) App hungs when calls producer.send to wrong IP of Kafka broker

[Igor Khomenko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Igor Khomenko updated KAFKA-1913:
-
Environment: OS X 10.10.1, Java 7, AWS Linux  (was: OS X 10.10.1, Java 7)

> App hungs when calls producer.send to wrong IP of Kafka broker
> --
>
> Key: KAFKA-1913
> URL: https://issues.apache.org/jira/browse/KAFKA-1913
> Project: Kafka
>  Issue Type: Bug
>  Components: producer 
>Affects Versions: 0.8.1.1
> Environment: OS X 10.10.1, Java 7, AWS Linux
>Reporter: Igor Khomenko
>Assignee: Jun Rao
> Fix For: 0.8.1.2
>
>
> I have next test code to check the Kafka functionality:
> {code}
> package com.company;
> import kafka.common.FailedToSendMessageException;
> import kafka.javaapi.producer.Producer;
> import kafka.producer.KeyedMessage;
> import kafka.producer.ProducerConfig;
> import java.util.Date;
> import java.util.Properties;
> public class Main {
> public static void main(String[] args) {
> Properties props = new Properties();
> props.put("metadata.broker.list", "192.168.9.3:9092");
> props.put("serializer.class", "com.company.KafkaMessageSerializer");
> props.put("request.required.acks", "1");
> ProducerConfig config = new ProducerConfig(props);
> // The first is the type of the Partition key, the second the type of 
> the message.
> Producer messagesProducer = new Producer String>(config);
> // Send
> String topicName = "my_messages";
> String message = "hello world";
> KeyedMessage data = new KeyedMessage String>(topicName, message);
> try {
> System.out.println(new Date() + ": sending...");
> messagesProducer.send(data);
> System.out.println(new Date() +  ": sent");
> }catch (FailedToSendMessageException e){
> System.out.println("e: " + e);
> e.printStackTrace();
> }catch (Exception exc){
> System.out.println("e: " + exc);
> exc.printStackTrace();
> }
> }
> }
> {code}
> {code}
> package com.company;
> import kafka.serializer.Encoder;
> import kafka.utils.VerifiableProperties;
> /**
>  * Created by igorkhomenko on 2/2/15.
>  */
> public class KafkaMessageSerializer implements Encoder {
> public KafkaMessageSerializer(VerifiableProperties verifiableProperties) {
> /* This constructor must be present for successful compile. */
> }
> @Override
> public byte[] toBytes(String entity) {
> byte [] serializedMessage = doCustomSerialization(entity);
> return serializedMessage;
> }
> private byte[] doCustomSerialization(String entity) {
> return entity.getBytes();
> }
> }
> {code}
> Here is also GitHub version https://github.com/soulfly/Kafka-java-producer
> So it just hungs on next line:
> {code}
> messagesProducer.send(data)
> {code}
> When I replaced the brokerlist to
> {code}
> props.put("metadata.broker.list", "localhost:9092");
> {code}
> then I got an exception:
> {code}
> kafka.common.FailedToSendMessageException: Failed to send messages after 3 
> tries.
> {code}
> so it's okay
> Why it hungs with wrong brokerlist? Any ideas?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1913) App hungs when calls producer.send to wrong IP of Kafka broker

[Igor Khomenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14304798#comment-14304798
 ] 

Igor Khomenko commented on KAFKA-1913:
--

Any ideas so far? 

> App hungs when calls producer.send to wrong IP of Kafka broker
> --
>
> Key: KAFKA-1913
> URL: https://issues.apache.org/jira/browse/KAFKA-1913
> Project: Kafka
>  Issue Type: Bug
>  Components: producer 
>Affects Versions: 0.8.1.1
> Environment: OS X 10.10.1, Java 7, AWS Linux
>Reporter: Igor Khomenko
>Assignee: Jun Rao
> Fix For: 0.8.1.2
>
>
> I have next test code to check the Kafka functionality:
> {code}
> package com.company;
> import kafka.common.FailedToSendMessageException;
> import kafka.javaapi.producer.Producer;
> import kafka.producer.KeyedMessage;
> import kafka.producer.ProducerConfig;
> import java.util.Date;
> import java.util.Properties;
> public class Main {
> public static void main(String[] args) {
> Properties props = new Properties();
> props.put("metadata.broker.list", "192.168.9.3:9092");
> props.put("serializer.class", "com.company.KafkaMessageSerializer");
> props.put("request.required.acks", "1");
> ProducerConfig config = new ProducerConfig(props);
> // The first is the type of the Partition key, the second the type of 
> the message.
> Producer messagesProducer = new Producer String>(config);
> // Send
> String topicName = "my_messages";
> String message = "hello world";
> KeyedMessage data = new KeyedMessage String>(topicName, message);
> try {
> System.out.println(new Date() + ": sending...");
> messagesProducer.send(data);
> System.out.println(new Date() +  ": sent");
> }catch (FailedToSendMessageException e){
> System.out.println("e: " + e);
> e.printStackTrace();
> }catch (Exception exc){
> System.out.println("e: " + exc);
> exc.printStackTrace();
> }
> }
> }
> {code}
> {code}
> package com.company;
> import kafka.serializer.Encoder;
> import kafka.utils.VerifiableProperties;
> /**
>  * Created by igorkhomenko on 2/2/15.
>  */
> public class KafkaMessageSerializer implements Encoder {
> public KafkaMessageSerializer(VerifiableProperties verifiableProperties) {
> /* This constructor must be present for successful compile. */
> }
> @Override
> public byte[] toBytes(String entity) {
> byte [] serializedMessage = doCustomSerialization(entity);
> return serializedMessage;
> }
> private byte[] doCustomSerialization(String entity) {
> return entity.getBytes();
> }
> }
> {code}
> Here is also GitHub version https://github.com/soulfly/Kafka-java-producer
> So it just hungs on next line:
> {code}
> messagesProducer.send(data)
> {code}
> When I replaced the brokerlist to
> {code}
> props.put("metadata.broker.list", "localhost:9092");
> {code}
> then I got an exception:
> {code}
> kafka.common.FailedToSendMessageException: Failed to send messages after 3 
> tries.
> {code}
> so it's okay
> Why it hungs with wrong brokerlist? Any ideas?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 30126: Patch for KAFKA-1845

[Andrii Biletskyi]

> On Feb. 3, 2015, 2:06 p.m., Jeff Holoman wrote:
> > core/src/main/scala/kafka/server/KafkaConfig.scala, line 834
> > 
> >
> > Is this is the right way to do this rather than having these in 
> > Validators?

Some of the broker settings validations are quite intricate - they involve 
other settings to be compared (e.g. replicaFetchWaitMaxMs <= 
replicaSocketTimeoutMs), which we can't do inside ConfigDef.define.


- Andrii


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30126/#review70748
---


On Jan. 21, 2015, 5:49 p.m., Andrii Biletskyi wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30126/
> ---
> 
> (Updated Jan. 21, 2015, 5:49 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1845
> https://issues.apache.org/jira/browse/KAFKA-1845
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-1845 - Fixed merge conflicts, ported added configs to KafkaConfig
> 
> 
> KAFKA-1845 - KafkaConfig to ConfigDef: moved validateValues so it's called on 
> instantiating KafkaConfig
> 
> 
> KAFKA-1845 - KafkaConfig to ConfigDef: MaxConnectionsPerIpOverrides refactored
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/common/config/ConfigDef.java 
> 98cb79b701918eca3f6ca9823b6c7b7c97b3ecec 
>   core/src/main/scala/kafka/Kafka.scala 
> 77a49e12af6f869e63230162e9f87a7b0b12b610 
>   core/src/main/scala/kafka/controller/KafkaController.scala 
> 66df6d2fbdbdd556da6bea0df84f93e0472c8fbf 
>   core/src/main/scala/kafka/controller/PartitionLeaderSelector.scala 
> 4a31c7271c2d0a4b9e8b28be729340ecfa0696e5 
>   core/src/main/scala/kafka/server/KafkaConfig.scala 
> 6d74983472249eac808d361344c58cc2858ec971 
>   core/src/main/scala/kafka/server/KafkaServer.scala 
> 89200da30a04943f0b9befe84ab17e62b747c8c4 
>   core/src/main/scala/kafka/server/ReplicaFetcherThread.scala 
> 6879e730282185bda3d6bc3659cb15af0672cecf 
>   core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala 
> e63558889272bc76551accdfd554bdafde2e0dd6 
>   core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala 
> 90c0b7a19c7af8e5416e4bdba62b9824f1abd5ab 
>   core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 
> b15237b76def3b234924280fa3fdb25dbb0cc0dc 
>   core/src/test/scala/unit/kafka/admin/AddPartitionsTest.scala 
> 1bf2667f47853585bc33ffb3e28256ec5f24ae84 
>   core/src/test/scala/unit/kafka/admin/AdminTest.scala 
> e28979827110dfbbb92fe5b152e7f1cc973de400 
>   core/src/test/scala/unit/kafka/admin/DeleteTopicTest.scala 
> 33c27678bf8ae8feebcbcdaa4b90a1963157b4a5 
>   core/src/test/scala/unit/kafka/consumer/ConsumerIteratorTest.scala 
> c0355cc0135c6af2e346b4715659353a31723b86 
>   
> core/src/test/scala/unit/kafka/consumer/ZookeeperConsumerConnectorTest.scala 
> a17e8532c44aadf84b8da3a57bcc797a848b5020 
>   core/src/test/scala/unit/kafka/integration/AutoOffsetResetTest.scala 
> 95303e098d40cd790fb370e9b5a47d20860a6da3 
>   core/src/test/scala/unit/kafka/integration/FetcherTest.scala 
> 25845abbcad2e79f56f729e59239b738d3ddbc9d 
>   core/src/test/scala/unit/kafka/integration/PrimitiveApiTest.scala 
> a5386a03b62956bc440b40783247c8cdf7432315 
>   core/src/test/scala/unit/kafka/integration/RollingBounceTest.scala 
> eab4b5f619015af42e4554660eafb5208e72ea33 
>   core/src/test/scala/unit/kafka/integration/TopicMetadataTest.scala 
> 35dc071b1056e775326981573c9618d8046e601d 
>   core/src/test/scala/unit/kafka/integration/UncleanLeaderElectionTest.scala 
> ba3bcdcd1de9843e75e5395dff2fc31b39a5a9d5 
>   
> core/src/test/scala/unit/kafka/javaapi/consumer/ZookeeperConsumerConnectorTest.scala
>  d6248b09bb0f86ee7d3bd0ebce5b99135491453b 
>   core/src/test/scala/unit/kafka/log/LogTest.scala 
> c2dd8eb69da8c0982a0dd20231c6f8bd58eb623e 
>   core/src/test/scala/unit/kafka/log4j/KafkaLog4jAppenderTest.scala 
> 4ea0489c9fd36983fe190491a086b39413f3a9cd 
>   core/src/test/scala/unit/kafka/metrics/MetricsTest.scala 
> 3cf23b3d6d4460535b90cfb36281714788fc681c 
>   core/src/test/scala/unit/kafka/producer/AsyncProducerTest.scala 
> 1db6ac329f7b54e600802c8a623f80d159d4e69b 
>   core/src/test/scala/unit/kafka/producer/ProducerTest.scala 
> ce65dab4910d9182e6774f6ef1a7f45561ec0c23 
>   core/src/test/scala/unit/kafka/producer/SyncProducerTest.scala 
> d60d8e0f49443f4dc8bc2cad6e2f951eda28f5cb 
>   core/src/test/scala/unit/kafka/server/AdvertiseBrokerTest.scala 
> f0c4a56b61b4f081cf4bee799c6e9c523ff45e19 
>   core/src/test/scala/unit/kafka/server/DynamicConfigChangeTest.scala 
> ad121169a5e80ebe1d311b95b219841ed69388e2 
>   core/src/test/scala/unit/kafka/server/HighwatermarkPersistenceTest.scala 
> 8913fc

Re: Review Request 30126: Patch for KAFKA-1845

[Andrii Biletskyi]

> On Jan. 22, 2015, 7:57 p.m., Gwen Shapira wrote:
> > core/src/main/scala/kafka/server/KafkaConfig.scala, lines 743-758
> > 
> >
> > This looks fairly generic. Shouldn't it be somewhere where it can be 
> > reused by other Config classes? Perhaps in ConfigDef or something similar?

I've fixed the function slightly - "valid" predicate is needless here (it was 
ported from VerifiableProperties), so it's just really Try[] arround already 
utility function Utils.parseCsvMap(?str) . I doubt it worth moving simple 
try-catch to separate function now... Hope this resolves.


- Andrii


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30126/#review69224
---


On Jan. 21, 2015, 5:49 p.m., Andrii Biletskyi wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30126/
> ---
> 
> (Updated Jan. 21, 2015, 5:49 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1845
> https://issues.apache.org/jira/browse/KAFKA-1845
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-1845 - Fixed merge conflicts, ported added configs to KafkaConfig
> 
> 
> KAFKA-1845 - KafkaConfig to ConfigDef: moved validateValues so it's called on 
> instantiating KafkaConfig
> 
> 
> KAFKA-1845 - KafkaConfig to ConfigDef: MaxConnectionsPerIpOverrides refactored
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/common/config/ConfigDef.java 
> 98cb79b701918eca3f6ca9823b6c7b7c97b3ecec 
>   core/src/main/scala/kafka/Kafka.scala 
> 77a49e12af6f869e63230162e9f87a7b0b12b610 
>   core/src/main/scala/kafka/controller/KafkaController.scala 
> 66df6d2fbdbdd556da6bea0df84f93e0472c8fbf 
>   core/src/main/scala/kafka/controller/PartitionLeaderSelector.scala 
> 4a31c7271c2d0a4b9e8b28be729340ecfa0696e5 
>   core/src/main/scala/kafka/server/KafkaConfig.scala 
> 6d74983472249eac808d361344c58cc2858ec971 
>   core/src/main/scala/kafka/server/KafkaServer.scala 
> 89200da30a04943f0b9befe84ab17e62b747c8c4 
>   core/src/main/scala/kafka/server/ReplicaFetcherThread.scala 
> 6879e730282185bda3d6bc3659cb15af0672cecf 
>   core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala 
> e63558889272bc76551accdfd554bdafde2e0dd6 
>   core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala 
> 90c0b7a19c7af8e5416e4bdba62b9824f1abd5ab 
>   core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 
> b15237b76def3b234924280fa3fdb25dbb0cc0dc 
>   core/src/test/scala/unit/kafka/admin/AddPartitionsTest.scala 
> 1bf2667f47853585bc33ffb3e28256ec5f24ae84 
>   core/src/test/scala/unit/kafka/admin/AdminTest.scala 
> e28979827110dfbbb92fe5b152e7f1cc973de400 
>   core/src/test/scala/unit/kafka/admin/DeleteTopicTest.scala 
> 33c27678bf8ae8feebcbcdaa4b90a1963157b4a5 
>   core/src/test/scala/unit/kafka/consumer/ConsumerIteratorTest.scala 
> c0355cc0135c6af2e346b4715659353a31723b86 
>   
> core/src/test/scala/unit/kafka/consumer/ZookeeperConsumerConnectorTest.scala 
> a17e8532c44aadf84b8da3a57bcc797a848b5020 
>   core/src/test/scala/unit/kafka/integration/AutoOffsetResetTest.scala 
> 95303e098d40cd790fb370e9b5a47d20860a6da3 
>   core/src/test/scala/unit/kafka/integration/FetcherTest.scala 
> 25845abbcad2e79f56f729e59239b738d3ddbc9d 
>   core/src/test/scala/unit/kafka/integration/PrimitiveApiTest.scala 
> a5386a03b62956bc440b40783247c8cdf7432315 
>   core/src/test/scala/unit/kafka/integration/RollingBounceTest.scala 
> eab4b5f619015af42e4554660eafb5208e72ea33 
>   core/src/test/scala/unit/kafka/integration/TopicMetadataTest.scala 
> 35dc071b1056e775326981573c9618d8046e601d 
>   core/src/test/scala/unit/kafka/integration/UncleanLeaderElectionTest.scala 
> ba3bcdcd1de9843e75e5395dff2fc31b39a5a9d5 
>   
> core/src/test/scala/unit/kafka/javaapi/consumer/ZookeeperConsumerConnectorTest.scala
>  d6248b09bb0f86ee7d3bd0ebce5b99135491453b 
>   core/src/test/scala/unit/kafka/log/LogTest.scala 
> c2dd8eb69da8c0982a0dd20231c6f8bd58eb623e 
>   core/src/test/scala/unit/kafka/log4j/KafkaLog4jAppenderTest.scala 
> 4ea0489c9fd36983fe190491a086b39413f3a9cd 
>   core/src/test/scala/unit/kafka/metrics/MetricsTest.scala 
> 3cf23b3d6d4460535b90cfb36281714788fc681c 
>   core/src/test/scala/unit/kafka/producer/AsyncProducerTest.scala 
> 1db6ac329f7b54e600802c8a623f80d159d4e69b 
>   core/src/test/scala/unit/kafka/producer/ProducerTest.scala 
> ce65dab4910d9182e6774f6ef1a7f45561ec0c23 
>   core/src/test/scala/unit/kafka/producer/SyncProducerTest.scala 
> d60d8e0f49443f4dc8bc2cad6e2f951eda28f5cb 
>   core/src/test/scala/unit/kafka/server/AdvertiseBrokerTest.scala 
> f0c4a56b61b4f081cf4bee799c6e9c523ff45e19 
>   core/src/test/scala/unit/kafka/server/Dyn

[jira] [Created] (KAFKA-1918) System test for ZooKeeper quorum failure scenarios

[Omid Aladini (JIRA)]

Omid Aladini created KAFKA-1918:
---

 Summary: System test for ZooKeeper quorum failure scenarios
 Key: KAFKA-1918
 URL: https://issues.apache.org/jira/browse/KAFKA-1918
 Project: Kafka
  Issue Type: Test
Reporter: Omid Aladini


Following up on the [conversation on the mailing 
list|http://mail-archives.apache.org/mod_mbox/kafka-users/201502.mbox/%3CCAHwHRrX3SAWDUGF5LjU4rrMUsqv%3DtJcyjX7OENeL5C_V5o3tCw%40mail.gmail.com%3E],
 the FAQ writes:

{quote}
Once the Zookeeper quorum is down, brokers could result in a bad state and 
could not normally serve client requests, etc. Although when Zookeeper quorum 
recovers, the Kafka brokers should be able to resume to normal state 
automatically, _there are still a few +corner cases+ the they cannot and a hard 
kill-and-recovery is required to bring it back to normal_. Hence it is 
recommended to closely monitor your zookeeper cluster and provision it so that 
it is performant.
{quote}

As ZK quorum failures are inevitable (due to rolling upgrades of ZK, leader 
hardware failure, etc), it would be great to identify the corner cases (if they 
still exist) and fix them if necessary.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Cannot stop Kafka server if zookeeper is shutdown first

[Jaikiran Pai]

FWIW - the ZkClient project team have merged the pull request that I had 
submitted to allow for timeouts to operations 
https://github.com/sgroschupf/zkclient/pull/29. I heard from Johannes 
(from the ZkClient project team) that they don't have any specific 
release date in mind but are willing to release a new version if/when we 
need one.


-Jaikiran

On Wednesday 04 February 2015 12:33 AM, Gwen Shapira wrote:

So I think the current plan is:
1. Add timeout in zkclient
2. Ask zkclient to release new version (we need it for few other things too)
3. Rebase on new zkclient
4. Fix this jira and the few others than were waiting for the new zkclient

Does that make sense?

Gwen

On Mon, Feb 2, 2015 at 8:33 PM, Jaikiran Pai  wrote:

I just heard back from Stefan, who manages the ZkClient repo and he seems to
be open to have these changes be part of ZkClient project. I'll be creating
a pull request for that project to have it reviewed and merged. Although I
haven't heard of exact release plans, Stefan's reply did indicate that the
project could be released after this change is merged.

-Jaikiran

On Tuesday 03 February 2015 09:03 AM, Jaikiran Pai wrote:

Thanks for pointing to that repo!

I just had a look at it and it appears that the project isn't much active
(going by the lack of activity). The latest contribution is from Gwen and
that was around 3 months back. I haven't found release plans for that
project or a place to ask about it (filing an issue doesn't seem right to
ask this question). So I'll get in touch with the repo owner and see what
his plans for the project are.

-Jaikiran

On Monday 02 February 2015 11:33 PM, Gwen Shapira wrote:

I did!

Thanks for clarifying :)

The client that is part of Zookeeper itself actually does support
timeouts.

On Mon, Feb 2, 2015 at 9:54 AM, Guozhang Wang  wrote:

Hi Jaikiran,

I think Gwen was talking about contributing to ZkClient project:

https://github.com/sgroschupf/zkclient

Guozhang


On Sun, Feb 1, 2015 at 5:30 AM, Jaikiran Pai 
wrote:


Hi Gwen,

Yes, the KafkaZkClient is a wrapper around ZkClient and not a complete
replacement.

As for contributing to Zookeeper, yes that indeed in on my mind, but I
haven't yet had a chance to really look deeper into Zookeeper or get in
touch with their dev team to try and explain this potential improvement
to
them. I have no objection to contributing this or something similar to
Zookeeper directly. I think I should be able to bring this up in the
Zookeeper dev forum, sometime soon in the next few weekends.

-Jaikiran


On Sunday 01 February 2015 11:40 AM, Gwen Shapira wrote:


It looks like the new KafkaZkClient is a wrapper around ZkClient, but
not a replacement. Did I get it right?

I think a wrapper for ZkClient can be useful - for example KAFKA-1664
can also use one.

However, I'm wondering why not contribute the fix directly to ZKClient
project and ask for a release that contains the fix?
This will benefit other users of the project who may also need a
timeout (thats pretty basic...)

As an alternative, if we don't want to collaborate with ZKClient for
some reason, forking the project into Kafka will probably give us more
control than wrappers and without much downside.

Just a thought.

Gwen





On Sat, Jan 31, 2015 at 6:32 AM, Jaikiran Pai

wrote:


Neha, Ewen (and others), my initial attempt to solve this is uploaded
here
https://reviews.apache.org/r/30477/. It solves the shutdown problem
and
now
the server shuts down even when Zookeeper has gone down before the
Kafka
server.

I went with the approach of introducing a custom (enhanced) ZkClient
which
for now allows time outs to be optionally specified for certain
operations.
I intentionally haven't forced the use of this new KafkaZkClient all
over
the code and instead for now have just used it in the KafkaServer.

Does this patch look like something worth using?

-Jaikiran


On Thursday 29 January 2015 10:41 PM, Neha Narkhede wrote:


Ewen is right. ZkClient APIs are blocking and the right fix for this
seems
to be patching ZkClient. At some point, if we find ourselves
fiddling
too
much with ZkClient, it wouldn't hurt to write our own little
zookeeper
client wrapper.

On Thu, Jan 29, 2015 at 12:57 AM, Ewen Cheslack-Postava

wrote:

   Looks like a bug to me -- the underlying ZK library wraps a lot of

blocking
method implementations with waitUntilConnected() calls without any
timeouts. Ideally we could just add a version of
ZkUtils.getController()
with a timeout, but I don't see an easy way to accomplish that with
ZkClient.

There's at least one other call to ZkUtils besides the one in the
stacktrace you gave that would cause the same issue, possibly more
that
aren't directly called in that method. One ugly solution would be
to
use
an
extra thread during shutdown to trigger timeouts, but I'd imagine
we
probably have other threads that could end up blocking in similar
ways.

I filed https://issues.apache.org/jira/browse/KAFKA-1907 to track
the
issue.


On Mon, Jan 

Re: Cannot stop Kafka server if zookeeper is shutdown first

[Harsha]

curator handles sasl connection
https://issues.apache.org/jira/browse/KAFKA-1695

On Wed, Feb 4, 2015, at 06:10 AM, Jaikiran Pai wrote:
> FWIW - the ZkClient project team have merged the pull request that I had 
> submitted to allow for timeouts to operations 
> https://github.com/sgroschupf/zkclient/pull/29. I heard from Johannes 
> (from the ZkClient project team) that they don't have any specific 
> release date in mind but are willing to release a new version if/when we 
> need one.
> 
> -Jaikiran
> 
> On Wednesday 04 February 2015 12:33 AM, Gwen Shapira wrote:
> > So I think the current plan is:
> > 1. Add timeout in zkclient
> > 2. Ask zkclient to release new version (we need it for few other things too)
> > 3. Rebase on new zkclient
> > 4. Fix this jira and the few others than were waiting for the new zkclient
> >
> > Does that make sense?
> >
> > Gwen
> >
> > On Mon, Feb 2, 2015 at 8:33 PM, Jaikiran Pai  
> > wrote:
> >> I just heard back from Stefan, who manages the ZkClient repo and he seems 
> >> to
> >> be open to have these changes be part of ZkClient project. I'll be creating
> >> a pull request for that project to have it reviewed and merged. Although I
> >> haven't heard of exact release plans, Stefan's reply did indicate that the
> >> project could be released after this change is merged.
> >>
> >> -Jaikiran
> >>
> >> On Tuesday 03 February 2015 09:03 AM, Jaikiran Pai wrote:
> >>> Thanks for pointing to that repo!
> >>>
> >>> I just had a look at it and it appears that the project isn't much active
> >>> (going by the lack of activity). The latest contribution is from Gwen and
> >>> that was around 3 months back. I haven't found release plans for that
> >>> project or a place to ask about it (filing an issue doesn't seem right to
> >>> ask this question). So I'll get in touch with the repo owner and see what
> >>> his plans for the project are.
> >>>
> >>> -Jaikiran
> >>>
> >>> On Monday 02 February 2015 11:33 PM, Gwen Shapira wrote:
>  I did!
> 
>  Thanks for clarifying :)
> 
>  The client that is part of Zookeeper itself actually does support
>  timeouts.
> 
>  On Mon, Feb 2, 2015 at 9:54 AM, Guozhang Wang  wrote:
> > Hi Jaikiran,
> >
> > I think Gwen was talking about contributing to ZkClient project:
> >
> > https://github.com/sgroschupf/zkclient
> >
> > Guozhang
> >
> >
> > On Sun, Feb 1, 2015 at 5:30 AM, Jaikiran Pai 
> > wrote:
> >
> >> Hi Gwen,
> >>
> >> Yes, the KafkaZkClient is a wrapper around ZkClient and not a complete
> >> replacement.
> >>
> >> As for contributing to Zookeeper, yes that indeed in on my mind, but I
> >> haven't yet had a chance to really look deeper into Zookeeper or get in
> >> touch with their dev team to try and explain this potential improvement
> >> to
> >> them. I have no objection to contributing this or something similar to
> >> Zookeeper directly. I think I should be able to bring this up in the
> >> Zookeeper dev forum, sometime soon in the next few weekends.
> >>
> >> -Jaikiran
> >>
> >>
> >> On Sunday 01 February 2015 11:40 AM, Gwen Shapira wrote:
> >>
> >>> It looks like the new KafkaZkClient is a wrapper around ZkClient, but
> >>> not a replacement. Did I get it right?
> >>>
> >>> I think a wrapper for ZkClient can be useful - for example KAFKA-1664
> >>> can also use one.
> >>>
> >>> However, I'm wondering why not contribute the fix directly to ZKClient
> >>> project and ask for a release that contains the fix?
> >>> This will benefit other users of the project who may also need a
> >>> timeout (thats pretty basic...)
> >>>
> >>> As an alternative, if we don't want to collaborate with ZKClient for
> >>> some reason, forking the project into Kafka will probably give us more
> >>> control than wrappers and without much downside.
> >>>
> >>> Just a thought.
> >>>
> >>> Gwen
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> On Sat, Jan 31, 2015 at 6:32 AM, Jaikiran Pai
> >>> 
> >>> wrote:
> >>>
>  Neha, Ewen (and others), my initial attempt to solve this is uploaded
>  here
>  https://reviews.apache.org/r/30477/. It solves the shutdown problem
>  and
>  now
>  the server shuts down even when Zookeeper has gone down before the
>  Kafka
>  server.
> 
>  I went with the approach of introducing a custom (enhanced) ZkClient
>  which
>  for now allows time outs to be optionally specified for certain
>  operations.
>  I intentionally haven't forced the use of this new KafkaZkClient all
>  over
>  the code and instead for now have just used it in the KafkaServer.
> 
>  Does this patch look like something worth using?
> 
>  -Jaikiran
> 
> 
> >

Re: kafka.utils.DoublyLinkedList

[Jay Kreps]

Hey Jiangjie,

Both the scala and java doubly linked lists allow O(1) deletion--it isn't
obvious because the api is slightly different from the CS101 linked list
but they do. Are you sure those wouldn't work? Can we remove this class and
use those?

Also the convention for unit tests is if we have a class Foo.scala then we
would have a test FooTest.scala. So you can definitely write a unit test
for classes used by MM without adding it to kafka.utils. The
UtilsTest.scala is really for Utils.scala we shouldn't be adding other
tests there.

-Jay

On Tue, Feb 3, 2015 at 11:46 PM, Jiangjie Qin 
wrote:

> We added this because in redesigned mirror maker we need a raw linked list
> so we can removed an acked offset from a linked list in O(1). Java and
> Scala linked list won¹t work in that case. It was initially put in side
> mirror maker as a private class, but later on we also want to have unit
> test for it, that¹s why it becomes a separate util class and we have unit
> test in UtilsTest.scala.
>
> -Jiangjie (Becket) Qin
>
> On 2/3/15, 9:15 PM, "Jay Kreps"  wrote:
>
> >How come we added this? Is it better than the linked list in java and
> >scala? Also it was added with only one test which is actually put inside
> >UtilsTest.scala which is meant to house tests for Utils.scala.
> >
> >-Jay
>
>

Re: Cannot stop Kafka server if zookeeper is shutdown first

[Dana Powers]

While on the subject of zkclient, also consider KAFKA-1793.  A more
abstract interface to the distributed coordination service that could be
configured to use alternatives like consul or etcd would be very useful
imho.

Dana
FWIW - the ZkClient project team have merged the pull request that I had
submitted to allow for timeouts to operations https://github.com/sgroschupf/
zkclient/pull/29. I heard from Johannes (from the ZkClient project team)
that they don't have any specific release date in mind but are willing to
release a new version if/when we need one.

-Jaikiran

On Wednesday 04 February 2015 12:33 AM, Gwen Shapira wrote:

> So I think the current plan is:
> 1. Add timeout in zkclient
> 2. Ask zkclient to release new version (we need it for few other things
> too)
> 3. Rebase on new zkclient
> 4. Fix this jira and the few others than were waiting for the new zkclient
>
> Does that make sense?
>
> Gwen
>
> On Mon, Feb 2, 2015 at 8:33 PM, Jaikiran Pai 
> wrote:
>
>> I just heard back from Stefan, who manages the ZkClient repo and he seems
>> to
>> be open to have these changes be part of ZkClient project. I'll be
>> creating
>> a pull request for that project to have it reviewed and merged. Although I
>> haven't heard of exact release plans, Stefan's reply did indicate that the
>> project could be released after this change is merged.
>>
>> -Jaikiran
>>
>> On Tuesday 03 February 2015 09:03 AM, Jaikiran Pai wrote:
>>
>>> Thanks for pointing to that repo!
>>>
>>> I just had a look at it and it appears that the project isn't much active
>>> (going by the lack of activity). The latest contribution is from Gwen and
>>> that was around 3 months back. I haven't found release plans for that
>>> project or a place to ask about it (filing an issue doesn't seem right to
>>> ask this question). So I'll get in touch with the repo owner and see what
>>> his plans for the project are.
>>>
>>> -Jaikiran
>>>
>>> On Monday 02 February 2015 11:33 PM, Gwen Shapira wrote:
>>>
 I did!

 Thanks for clarifying :)

 The client that is part of Zookeeper itself actually does support
 timeouts.

 On Mon, Feb 2, 2015 at 9:54 AM, Guozhang Wang 
 wrote:

> Hi Jaikiran,
>
> I think Gwen was talking about contributing to ZkClient project:
>
> https://github.com/sgroschupf/zkclient
>
> Guozhang
>
>
> On Sun, Feb 1, 2015 at 5:30 AM, Jaikiran Pai  >
> wrote:
>
>  Hi Gwen,
>>
>> Yes, the KafkaZkClient is a wrapper around ZkClient and not a complete
>> replacement.
>>
>> As for contributing to Zookeeper, yes that indeed in on my mind, but I
>> haven't yet had a chance to really look deeper into Zookeeper or get
>> in
>> touch with their dev team to try and explain this potential
>> improvement
>> to
>> them. I have no objection to contributing this or something similar to
>> Zookeeper directly. I think I should be able to bring this up in the
>> Zookeeper dev forum, sometime soon in the next few weekends.
>>
>> -Jaikiran
>>
>>
>> On Sunday 01 February 2015 11:40 AM, Gwen Shapira wrote:
>>
>>  It looks like the new KafkaZkClient is a wrapper around ZkClient, but
>>> not a replacement. Did I get it right?
>>>
>>> I think a wrapper for ZkClient can be useful - for example KAFKA-1664
>>> can also use one.
>>>
>>> However, I'm wondering why not contribute the fix directly to
>>> ZKClient
>>> project and ask for a release that contains the fix?
>>> This will benefit other users of the project who may also need a
>>> timeout (thats pretty basic...)
>>>
>>> As an alternative, if we don't want to collaborate with ZKClient for
>>> some reason, forking the project into Kafka will probably give us
>>> more
>>> control than wrappers and without much downside.
>>>
>>> Just a thought.
>>>
>>> Gwen
>>>
>>>
>>>
>>>
>>>
>>> On Sat, Jan 31, 2015 at 6:32 AM, Jaikiran Pai
>>> 
>>> wrote:
>>>
>>>  Neha, Ewen (and others), my initial attempt to solve this is
 uploaded
 here
 https://reviews.apache.org/r/30477/. It solves the shutdown problem
 and
 now
 the server shuts down even when Zookeeper has gone down before the
 Kafka
 server.

 I went with the approach of introducing a custom (enhanced) ZkClient
 which
 for now allows time outs to be optionally specified for certain
 operations.
 I intentionally haven't forced the use of this new KafkaZkClient all
 over
 the code and instead for now have just used it in the KafkaServer.

 Does this patch look like something worth using?

 -Jaikiran


 On Thursday 29 January 2015 10:41 PM, Neha Narkhede wrote:

  Ewen is right. ZkClient A

Re: kafka.utils.DoublyLinkedList

[Joel Koshy]

Jay,

My understanding is that java linked lists (even though they are backed by
a doubly linked list) support O(1) deletion only if you have the index of
the object. If you do a remove(object) it becomes O(n). As for scala lists
- the scaladocs seem to indicate that the doubly linked lists are
deprecated as of 2.11

Joel

On Wednesday, February 4, 2015, Jay Kreps  wrote:

> Hey Jiangjie,
>
> Both the scala and java doubly linked lists allow O(1) deletion--it isn't
> obvious because the api is slightly different from the CS101 linked list
> but they do. Are you sure those wouldn't work? Can we remove this class and
> use those?
>
> Also the convention for unit tests is if we have a class Foo.scala then we
> would have a test FooTest.scala. So you can definitely write a unit test
> for classes used by MM without adding it to kafka.utils. The
> UtilsTest.scala is really for Utils.scala we shouldn't be adding other
> tests there.
>
> -Jay
>
> On Tue, Feb 3, 2015 at 11:46 PM, Jiangjie Qin 
> wrote:
>
> > We added this because in redesigned mirror maker we need a raw linked
> list
> > so we can removed an acked offset from a linked list in O(1). Java and
> > Scala linked list won¹t work in that case. It was initially put in side
> > mirror maker as a private class, but later on we also want to have unit
> > test for it, that¹s why it becomes a separate util class and we have unit
> > test in UtilsTest.scala.
> >
> > -Jiangjie (Becket) Qin
> >
> > On 2/3/15, 9:15 PM, "Jay Kreps" >
> wrote:
> >
> > >How come we added this? Is it better than the linked list in java and
> > >scala? Also it was added with only one test which is actually put inside
> > >UtilsTest.scala which is meant to house tests for Utils.scala.
> > >
> > >-Jay
> >
> >
>


-- 
Sent from Gmail Mobile

[Discussion] Producer Instance and Decouple Kafka Cluster State/TCP Connection Management

[Bhavesh Mistry]

Hi Kafka Dev team,



I would like to discuss the Kafka Cluster State Management and Producer
Instance relationship in 0.8.2.



Current Implementation of Producer ties very closely with Kafka Cluster,
and Topic Metadata management.  So imagine, you have following scenarios:



Application crates multiple instance of Producers for same Kafka cluster
for same topic or more topic (for the same cluster)  to get better
throughput or to avoid blocking calls or avoid synchronization issue ( by
key hash to same partition etc).



It would be great if design and implementation of Cluster State/TCP
connection management and Producer Instance are decoupled with Producer
Instance.



Suppose, we are doing aggregation using Kafka and we wanted to avoid
Dequeue Sync call for performance reason ( I would have to create #
producer = # of partition which can be many each one will manage cluster
and TCP state separately ) .



All I am asking, is it possible to have low level TCP connection pooling/
Cluster State Management and Topic Meta Data Management separately then
encapsulating everything within Producer Instance.



In my case , I would create io thread per partition and dump data one to
one mapping to avoid blocking/sync calls.Having public API for
Cluster/TCP connection   management and Kafka Java Protocol will allow
advance way to achieve higher throughput at expense of CPU/memory which
application control.



I am referring to my experience with
https://issues.apache.org/jira/browse/KAFKA-1710



Let me know you’re taught process.


Thanks,



Bhavesh

Re: kafka.utils.DoublyLinkedList

[Jay Kreps]

Yeah I think the Java LL is worse than that, you actually need to point at
it with an iterator and call remove(). I didn't realize the scala ll was
deprecated. Okay, makes sense. Would be good to move that test out of
UtilsTest.scala, though.

But actually, stepping back, can you guys document/explain what all that mm
code is doing now? It is pretty tricky which makes me worry. Are we keeping
track of each individual message until it is acknowledged in that list?
Can't we just keep the largest input/output offset pair and commit that
periodically and on rebalance?

-Jay

On Wed, Feb 4, 2015 at 8:39 AM, Joel Koshy  wrote:

> Jay,
>
> My understanding is that java linked lists (even though they are backed by
> a doubly linked list) support O(1) deletion only if you have the index of
> the object. If you do a remove(object) it becomes O(n). As for scala lists
> - the scaladocs seem to indicate that the doubly linked lists are
> deprecated as of 2.11
>
> Joel
>
> On Wednesday, February 4, 2015, Jay Kreps  wrote:
>
> > Hey Jiangjie,
> >
> > Both the scala and java doubly linked lists allow O(1) deletion--it isn't
> > obvious because the api is slightly different from the CS101 linked list
> > but they do. Are you sure those wouldn't work? Can we remove this class
> and
> > use those?
> >
> > Also the convention for unit tests is if we have a class Foo.scala then
> we
> > would have a test FooTest.scala. So you can definitely write a unit test
> > for classes used by MM without adding it to kafka.utils. The
> > UtilsTest.scala is really for Utils.scala we shouldn't be adding other
> > tests there.
> >
> > -Jay
> >
> > On Tue, Feb 3, 2015 at 11:46 PM, Jiangjie Qin  >
> > wrote:
> >
> > > We added this because in redesigned mirror maker we need a raw linked
> > list
> > > so we can removed an acked offset from a linked list in O(1). Java and
> > > Scala linked list won¹t work in that case. It was initially put in side
> > > mirror maker as a private class, but later on we also want to have unit
> > > test for it, that¹s why it becomes a separate util class and we have
> unit
> > > test in UtilsTest.scala.
> > >
> > > -Jiangjie (Becket) Qin
> > >
> > > On 2/3/15, 9:15 PM, "Jay Kreps" >
> > wrote:
> > >
> > > >How come we added this? Is it better than the linked list in java and
> > > >scala? Also it was added with only one test which is actually put
> inside
> > > >UtilsTest.scala which is meant to house tests for Utils.scala.
> > > >
> > > >-Jay
> > >
> > >
> >
>
>
> --
> Sent from Gmail Mobile
>

Re: Cannot stop Kafka server if zookeeper is shutdown first

[Joe Stein]

<< A more abstract interface to the distributed coordination service that
could be configured to use alternatives like consul or etcd would be very
useful imho.

+1

If we make the curator changes lets take the time to first build an
interface please we can expose both meta data and async notification
services through it. I have been playing around with etcd 2.0 this last
week since it has been out. I really like some of the concepts like
waitIndex (so you don't miss watches) and the compare and set/delete stuff
is great too. The ectd 2.0 rest call returns and when it does we just have
to fire the same event back into the API like the zookeeper code will do.
Coming up with what that looks like is a KIP as would be changing and using
Curator
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
for those changes. I think as long as the project has two we know it works
for different groups of folks.

I think whoever might be interested in working on that should chime in,
write up the KIP and lets start getting that discussion / release planning
/ work moving ahead.

- Joe Stein

On Wed, Feb 4, 2015 at 11:27 AM, Dana Powers  wrote:

> While on the subject of zkclient, also consider KAFKA-1793.  A more
> abstract interface to the distributed coordination service that could be
> configured to use alternatives like consul or etcd would be very useful
> imho.
>
> Dana
> FWIW - the ZkClient project team have merged the pull request that I had
> submitted to allow for timeouts to operations
> https://github.com/sgroschupf/
> zkclient/pull/29. I heard from Johannes (from the ZkClient project team)
> that they don't have any specific release date in mind but are willing to
> release a new version if/when we need one.
>
> -Jaikiran
>
> On Wednesday 04 February 2015 12:33 AM, Gwen Shapira wrote:
>
> > So I think the current plan is:
> > 1. Add timeout in zkclient
> > 2. Ask zkclient to release new version (we need it for few other things
> > too)
> > 3. Rebase on new zkclient
> > 4. Fix this jira and the few others than were waiting for the new
> zkclient
> >
> > Does that make sense?
> >
> > Gwen
> >
> > On Mon, Feb 2, 2015 at 8:33 PM, Jaikiran Pai 
> > wrote:
> >
> >> I just heard back from Stefan, who manages the ZkClient repo and he
> seems
> >> to
> >> be open to have these changes be part of ZkClient project. I'll be
> >> creating
> >> a pull request for that project to have it reviewed and merged.
> Although I
> >> haven't heard of exact release plans, Stefan's reply did indicate that
> the
> >> project could be released after this change is merged.
> >>
> >> -Jaikiran
> >>
> >> On Tuesday 03 February 2015 09:03 AM, Jaikiran Pai wrote:
> >>
> >>> Thanks for pointing to that repo!
> >>>
> >>> I just had a look at it and it appears that the project isn't much
> active
> >>> (going by the lack of activity). The latest contribution is from Gwen
> and
> >>> that was around 3 months back. I haven't found release plans for that
> >>> project or a place to ask about it (filing an issue doesn't seem right
> to
> >>> ask this question). So I'll get in touch with the repo owner and see
> what
> >>> his plans for the project are.
> >>>
> >>> -Jaikiran
> >>>
> >>> On Monday 02 February 2015 11:33 PM, Gwen Shapira wrote:
> >>>
>  I did!
> 
>  Thanks for clarifying :)
> 
>  The client that is part of Zookeeper itself actually does support
>  timeouts.
> 
>  On Mon, Feb 2, 2015 at 9:54 AM, Guozhang Wang 
>  wrote:
> 
> > Hi Jaikiran,
> >
> > I think Gwen was talking about contributing to ZkClient project:
> >
> > https://github.com/sgroschupf/zkclient
> >
> > Guozhang
> >
> >
> > On Sun, Feb 1, 2015 at 5:30 AM, Jaikiran Pai <
> jai.forums2...@gmail.com
> > >
> > wrote:
> >
> >  Hi Gwen,
> >>
> >> Yes, the KafkaZkClient is a wrapper around ZkClient and not a
> complete
> >> replacement.
> >>
> >> As for contributing to Zookeeper, yes that indeed in on my mind,
> but I
> >> haven't yet had a chance to really look deeper into Zookeeper or get
> >> in
> >> touch with their dev team to try and explain this potential
> >> improvement
> >> to
> >> them. I have no objection to contributing this or something similar
> to
> >> Zookeeper directly. I think I should be able to bring this up in the
> >> Zookeeper dev forum, sometime soon in the next few weekends.
> >>
> >> -Jaikiran
> >>
> >>
> >> On Sunday 01 February 2015 11:40 AM, Gwen Shapira wrote:
> >>
> >>  It looks like the new KafkaZkClient is a wrapper around ZkClient,
> but
> >>> not a replacement. Did I get it right?
> >>>
> >>> I think a wrapper for ZkClient can be useful - for example
> KAFKA-1664
> >>> can also use one.
> >>>
> >>> However, I'm wondering why not contribute the fix directly to
> >>> ZKClient
> >>> project and ask for a release that c

[jira] [Created] (KAFKA-1919) Metadata request issued with no backoff in new producer if there are no topics

[Jay Kreps (JIRA)]

Jay Kreps created KAFKA-1919:


 Summary: Metadata request issued with no backoff in new producer 
if there are no topics
 Key: KAFKA-1919
 URL: https://issues.apache.org/jira/browse/KAFKA-1919
 Project: Kafka
  Issue Type: Bug
Reporter: Jay Kreps


Original report:
We have observed high cpu and high network traffic problem when
1) cluster (0.8.1.1) has no topic
2) KafkaProducer (0.8.2-beta) object is created without sending any traffic

We have observed such problem twice. In both cases, problem went away
immediately after one/any topic is created.

Is this a known issue? Just want to check with the community first before I
spend much time to reproduce it.

I couldn't reproduce the issue with similar setup with unit test code in
IDE. start two brokers with no topic locally on my laptop. create a
KafkaProducer object without sending any msgs. but I only tested with
0.8.2-beta for both broker and producer.

Issue exists in 0.8.2 as well:
I have re-run my unit test with 0.8.2.0. same tight-loop problem happened
after a few mins.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1919) Metadata request issued with no backoff in new producer if there are no topics

[Jay Kreps (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jay Kreps updated KAFKA-1919:
-
Affects Version/s: 0.8.2

> Metadata request issued with no backoff in new producer if there are no topics
> --
>
> Key: KAFKA-1919
> URL: https://issues.apache.org/jira/browse/KAFKA-1919
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2
>Reporter: Jay Kreps
>
> Original report:
> We have observed high cpu and high network traffic problem when
> 1) cluster (0.8.1.1) has no topic
> 2) KafkaProducer (0.8.2-beta) object is created without sending any traffic
> We have observed such problem twice. In both cases, problem went away
> immediately after one/any topic is created.
> Is this a known issue? Just want to check with the community first before I
> spend much time to reproduce it.
> I couldn't reproduce the issue with similar setup with unit test code in
> IDE. start two brokers with no topic locally on my laptop. create a
> KafkaProducer object without sending any msgs. but I only tested with
> 0.8.2-beta for both broker and producer.
> Issue exists in 0.8.2 as well:
> I have re-run my unit test with 0.8.2.0. same tight-loop problem happened
> after a few mins.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1919) Metadata request issued with no backoff in new producer if there are no topics

[Steven Zhen Wu (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14305528#comment-14305528
 ] 

Steven Zhen Wu commented on KAFKA-1919:
---

enabled TRACE level logging. it shows tight loop with metadata request. two 
metadata requests in the same milli-seconds

kafka-producer-network-thread | foo 08:58:34,917 TRACE NetworkClient:315 - 
Ignoring empty metadata response with correlation id 1473532.
kafka-producer-network-thread | foo 08:58:34,917 DEBUG NetworkClient:385 - 
Trying to send metadata request to node -2
kafka-producer-network-thread | foo 08:58:34,917 DEBUG NetworkClient:390 - 
Sending metadata request ClientRequest(expectResponse=true, payload=null, 
request=RequestSend(header={api_key=3,api_version=0,correlation_id=1473533,client_id=foo},
 body={topics=[]})) to node -2
kafka-producer-network-thread | foo 08:58:34,917 TRACE NetworkClient:315 - 
Ignoring empty metadata response with correlation id 1473533.
kafka-producer-network-thread | foo 08:58:34,917 DEBUG NetworkClient:385 - 
Trying to send metadata request to node -2
kafka-producer-network-thread | foo 08:58:34,918 DEBUG NetworkClient:390 - 
Sending metadata request ClientRequest(expectResponse=true, payload=null, 
request=RequestSend(header={api_key=3,api_version=0,correlation_id=1473534,client_id=foo},
 body={topics=[]})) to node -2
kafka-producer-network-thread | foo 08:58:34,918 TRACE NetworkClient:315 - 
Ignoring empty metadata response with correlation id 1473534.
kafka-producer-network-thread | foo 08:58:34,918 DEBUG NetworkClient:385 - 
Trying to send metadata request to node -2
kafka-producer-network-thread | foo 08:58:34,918 DEBUG NetworkClient:390 - 
Sending metadata request ClientRequest(expectResponse=true, payload=null, 
request=RequestSend(header={api_key=3,api_version=0,correlation_id=1473535,client_id=foo},
 body={topics=[]})) to node -2
kafka-producer-network-thread | foo 08:58:34,918 TRACE NetworkClient:315 - 
Ignoring empty metadata response with correlation id 1473535.
kafka-producer-network-thread | foo 08:58:34,918 DEBUG NetworkClient:385 - 
Trying to send metadata request to node -2

> Metadata request issued with no backoff in new producer if there are no topics
> --
>
> Key: KAFKA-1919
> URL: https://issues.apache.org/jira/browse/KAFKA-1919
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2
>Reporter: Jay Kreps
>
> Original report:
> We have observed high cpu and high network traffic problem when
> 1) cluster (0.8.1.1) has no topic
> 2) KafkaProducer (0.8.2-beta) object is created without sending any traffic
> We have observed such problem twice. In both cases, problem went away
> immediately after one/any topic is created.
> Is this a known issue? Just want to check with the community first before I
> spend much time to reproduce it.
> I couldn't reproduce the issue with similar setup with unit test code in
> IDE. start two brokers with no topic locally on my laptop. create a
> KafkaProducer object without sending any msgs. but I only tested with
> 0.8.2-beta for both broker and producer.
> Issue exists in 0.8.2 as well:
> I have re-run my unit test with 0.8.2.0. same tight-loop problem happened
> after a few mins.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [ANNOUNCEMENT] Apache Kafka 0.8.2.0 Released

[Rajasekar Elango]

YaY!. Thanks to Jun and everybody who contributed to this release. We have
been waiting for this release for a while.

Thanks,
Rajasekar Elango (Salesforce.com).

On Tue, Feb 3, 2015 at 8:37 PM, Jun Rao  wrote:

> The Apache Kafka community is pleased to announce the release for Apache
> Kafka 0.8.2.0.
>
> The 0.8.2.0 release introduces many new features, improvements and fixes
> including:
>  - A new Java producer for ease of implementation and enhanced performance.
>  - A Kafka-based offset storage.
>  - Delete topic support.
>  - Per topic configuration of preference for consistency over availability.
>  - Scala 2.11 support and dropping support for Scala 2.8.
>  - LZ4 Compression.
>
> All of the changes in this release can be found:
> https://archive.apache.org/dist/kafka/0.8.2.0/RELEASE_NOTES.html
>
> Apache Kafka is high-throughput, publish-subscribe messaging system
> rethought of as a distributed commit log.
>
> ** Fast => A single Kafka broker can handle hundreds of megabytes of reads
> and
> writes per second from thousands of clients.
>
> ** Scalable => Kafka is designed to allow a single cluster to serve as the
> central data backbone
> for a large organization. It can be elastically and transparently expanded
> without downtime.
> Data streams are partitioned and spread over a cluster of machines to allow
> data streams
> larger than the capability of any single machine and to allow clusters of
> co-ordinated consumers.
>
> ** Durable => Messages are persisted on disk and replicated within the
> cluster to prevent
> data loss. Each broker can handle terabytes of messages without performance
> impact.
>
> ** Distributed by Design => Kafka has a modern cluster-centric design that
> offers
> strong durability and fault-tolerance guarantees.
>
> You can download the release from: http://kafka.apache.org/downloads.html
>
> We welcome your help and feedback. For more information on how to
> report problems, and to get involved, visit the project website at
> http://kafka.apache.org/
>
> Thanks,
>
> Jun
>



-- 
Thanks,
Raja.

[jira] [Comment Edited] (KAFKA-1919) Metadata request issued with no backoff in new producer if there are no topics

[Steven Zhen Wu (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14305528#comment-14305528
 ] 

Steven Zhen Wu edited comment on KAFKA-1919 at 2/4/15 6:11 PM:
---

enabled TRACE level logging. it shows tight loop with metadata request. two 
metadata requests in the same milli-seconds
{code}
kafka-producer-network-thread | foo 08:58:34,917 TRACE NetworkClient:315 - 
Ignoring empty metadata response with correlation id 1473532.
kafka-producer-network-thread | foo 08:58:34,917 DEBUG NetworkClient:385 - 
Trying to send metadata request to node -2
kafka-producer-network-thread | foo 08:58:34,917 DEBUG NetworkClient:390 - 
Sending metadata request ClientRequest(expectResponse=true, payload=null, 
request=RequestSend(header={api_key=3,api_version=0,correlation_id=1473533,client_id=foo},
 body={topics=[]})) to node -2
kafka-producer-network-thread | foo 08:58:34,917 TRACE NetworkClient:315 - 
Ignoring empty metadata response with correlation id 1473533.
kafka-producer-network-thread | foo 08:58:34,917 DEBUG NetworkClient:385 - 
Trying to send metadata request to node -2
kafka-producer-network-thread | foo 08:58:34,918 DEBUG NetworkClient:390 - 
Sending metadata request ClientRequest(expectResponse=true, payload=null, 
request=RequestSend(header={api_key=3,api_version=0,correlation_id=1473534,client_id=foo},
 body={topics=[]})) to node -2
kafka-producer-network-thread | foo 08:58:34,918 TRACE NetworkClient:315 - 
Ignoring empty metadata response with correlation id 1473534.
kafka-producer-network-thread | foo 08:58:34,918 DEBUG NetworkClient:385 - 
Trying to send metadata request to node -2
kafka-producer-network-thread | foo 08:58:34,918 DEBUG NetworkClient:390 - 
Sending metadata request ClientRequest(expectResponse=true, payload=null, 
request=RequestSend(header={api_key=3,api_version=0,correlation_id=1473535,client_id=foo},
 body={topics=[]})) to node -2
kafka-producer-network-thread | foo 08:58:34,918 TRACE NetworkClient:315 - 
Ignoring empty metadata response with correlation id 1473535.
kafka-producer-network-thread | foo 08:58:34,918 DEBUG NetworkClient:385 - 
Trying to send metadata request to node -2
{code}


was (Author: stevenz3wu):
enabled TRACE level logging. it shows tight loop with metadata request. two 
metadata requests in the same milli-seconds

kafka-producer-network-thread | foo 08:58:34,917 TRACE NetworkClient:315 - 
Ignoring empty metadata response with correlation id 1473532.
kafka-producer-network-thread | foo 08:58:34,917 DEBUG NetworkClient:385 - 
Trying to send metadata request to node -2
kafka-producer-network-thread | foo 08:58:34,917 DEBUG NetworkClient:390 - 
Sending metadata request ClientRequest(expectResponse=true, payload=null, 
request=RequestSend(header={api_key=3,api_version=0,correlation_id=1473533,client_id=foo},
 body={topics=[]})) to node -2
kafka-producer-network-thread | foo 08:58:34,917 TRACE NetworkClient:315 - 
Ignoring empty metadata response with correlation id 1473533.
kafka-producer-network-thread | foo 08:58:34,917 DEBUG NetworkClient:385 - 
Trying to send metadata request to node -2
kafka-producer-network-thread | foo 08:58:34,918 DEBUG NetworkClient:390 - 
Sending metadata request ClientRequest(expectResponse=true, payload=null, 
request=RequestSend(header={api_key=3,api_version=0,correlation_id=1473534,client_id=foo},
 body={topics=[]})) to node -2
kafka-producer-network-thread | foo 08:58:34,918 TRACE NetworkClient:315 - 
Ignoring empty metadata response with correlation id 1473534.
kafka-producer-network-thread | foo 08:58:34,918 DEBUG NetworkClient:385 - 
Trying to send metadata request to node -2
kafka-producer-network-thread | foo 08:58:34,918 DEBUG NetworkClient:390 - 
Sending metadata request ClientRequest(expectResponse=true, payload=null, 
request=RequestSend(header={api_key=3,api_version=0,correlation_id=1473535,client_id=foo},
 body={topics=[]})) to node -2
kafka-producer-network-thread | foo 08:58:34,918 TRACE NetworkClient:315 - 
Ignoring empty metadata response with correlation id 1473535.
kafka-producer-network-thread | foo 08:58:34,918 DEBUG NetworkClient:385 - 
Trying to send metadata request to node -2

> Metadata request issued with no backoff in new producer if there are no topics
> --
>
> Key: KAFKA-1919
> URL: https://issues.apache.org/jira/browse/KAFKA-1919
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2
>Reporter: Jay Kreps
>
> Original report:
> We have observed high cpu and high network traffic problem when
> 1) cluster (0.8.1.1) has no topic
> 2) KafkaProducer (0.8.2-beta) object is created without sending any traffic
> We have observed such problem twice. In both cases, problem went away
> immediately after one/any topic is created.
> Is this

[jira] [Commented] (KAFKA-1730) add the doc for the new java producer in 0.8.2

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14305733#comment-14305733
 ] 

Jun Rao commented on KAFKA-1730:


[~omkreddy], could you add those mbeans for the new producer as a patch to 
https://svn.apache.org/repos/asf/kafka/site/082/ops.html ? Thanks,

> add the doc for the new java producer in 0.8.2
> --
>
> Key: KAFKA-1730
> URL: https://issues.apache.org/jira/browse/KAFKA-1730
> Project: Kafka
>  Issue Type: Sub-task
>Affects Versions: 0.8.2
>Reporter: Jun Rao
>Priority: Blocker
> Fix For: 0.8.2
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [KIP-DISCUSSION] Mirror Maker Enhancement

[Bhavesh Mistry]

Hi Jiangjie,

Thanks for entertaining my question so far.  Last question, I have is about
serialization of message key.  If the key de-serialization (Class) is not
present at the MM instance, then does it use raw byte hashcode to determine
the partition ?  How are you going to address the situation where key needs
to be de-serialization and get actual hashcode needs to be computed  ?.


Thanks,

Bhavesh

On Fri, Jan 30, 2015 at 1:41 PM, Jiangjie Qin 
wrote:

> Hi Bhavesh,
>
> Please see inline comments.
>
> Jiangjie (Becket) Qin
>
> On 1/29/15, 7:00 PM, "Bhavesh Mistry"  wrote:
>
> >Hi Jiangjie,
> >
> >Thanks for the input.
> >
> >a) Is MM will  producer ack will be attach to Producer Instance or per
> >topic.  Use case is that one instance of MM
> >needs to handle both strong ack and also ack=0 for some topic.  Or it
> >would
> >be better to set-up another instance of MM.
> The acks setting is producer level setting instead of topic level setting.
> In this case you probably need to set up another instance.
> >
> >b) Regarding TCP connections, Why does #producer instance attach to TCP
> >connection.  Is it possible to use Broker Connection TCP Pool, producer
> >will just checkout TCP connection  to Broker.  So, # of Producer Instance
> >does not correlation to Brokers Connection.  Is this possible ?
> In new producer, each producer maintains a connection to each broker
> within the producer instance. Making producer instances to share the TCP
> connections is a very big change to the current design, so I suppose we
> won’t be able to do that.
> >
> >
> >Thanks,
> >
> >Bhavesh
> >
> >On Thu, Jan 29, 2015 at 11:50 AM, Jiangjie Qin  >
> >wrote:
> >
> >> Hi Bhavesh,
> >>
> >> I think it is the right discussion to have when we are talking about the
> >> new new design for MM.
> >> Please see the inline comments.
> >>
> >> Jiangjie (Becket) Qin
> >>
> >> On 1/28/15, 10:48 PM, "Bhavesh Mistry" 
> >>wrote:
> >>
> >> >Hi Jiangjie,
> >> >
> >> >I just wanted to let you know about our use case and stress the point
> >>that
> >> >local data center broker cluster have fewer partitions than the
> >> >destination
> >> >offline broker cluster. Just because we do the batch pull from CAMUS
> >>and
> >> >in
> >> >order to drain data faster than the injection rate (from four DCs for
> >>same
> >> >topic).
> >> Keeping the same partition number in source and target cluster will be
> >>an
> >> option but will not be enforced by default.
> >> >
> >> >We are facing following issues (probably due to configuration):
> >> >
> >> >1)  We occasionally loose data due to message batch size is too
> >>large
> >> >(2MB) on target data (we are using old producer but I think new
> >>producer
> >> >will solve this problem to some extend).
> >> We do see this issue in LinkedIn as well. New producer also might have
> >> this issue. There are some proposal of solutions, but no real work
> >>started
> >> yet. For now, as a workaround, setting a more aggressive batch size on
> >> producer side should work.
> >> >2)  Since only one instance is set to MM data,  we are not able to
> >> >set-up ack per topic instead ack is attached to producer instance.
> >> I don’t quite get the question here.
> >> >3)  How are you going to address two phase commit problem if ack is
> >> >set
> >> >to strongest, but auto commit is on for consumer (meaning producer does
> >> >not
> >> >get ack,  but consumer auto committed offset that message).  Is there
> >> >transactional (Kafka transaction is in process) based ack and commit
> >> >offset
> >> >?
> >> Auto offset commit should be turned off in this case. The offset will
> >>only
> >> be committed once by the offset commit thread. So there is no two phase
> >> commit.
> >> >4)  How are you planning to avoid duplicated message?  ( Is
> >> >brokergoing
> >> >have moving window of message collected and de-dupe ?)  Possibly, we
> >>get
> >> >this from retry set to 5…?
> >> We are not trying to completely avoid duplicates. The duplicates will
> >> still be there if:
> >> 1. Producer retries on failure.
> >> 2. Mirror maker is hard killed.
> >> Currently, dedup is expected to be done by user if necessary.
> >> >5)  Last, is there any warning or any thing you can provide insight
> >> >from MM component about data injection rate into destination
> >>partitions is
> >> >NOT evenly distributed regardless  of  keyed or non-keyed message
> >>(Hence
> >> >there is ripple effect such as data not arriving late, or data is
> >>arriving
> >> >out of order in  intern of time stamp  and early some time, and CAMUS
> >> >creates huge number of file count on HDFS due to uneven injection rate
> >>.
> >> >Camus Job is  configured to run every 3 minutes.)
> >> I think uneven data distribution is typically caused by server side
> >> unbalance, instead of something mirror maker could control. In new
> >>mirror
> >> maker, however, there is a customizable message handler, that might be
> >> able to help a little bit. In me

[jira] [Commented] (KAFKA-1856) Add PreCommit Patch Testing

[Gwen Shapira (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14305833#comment-14305833
 ] 

Gwen Shapira commented on KAFKA-1856:
-

Here's what I did and the result:

{code}
gshapira-MBP:kafka gshapira$ python ./dev-utils/test-patch.py --defect 
KAFKA-1809 --output patch-process --run-tests 

Defect: KAFKA-1809
INFO: URL = https://issues.apache.org/jira/rest/api/2/issue/KAFKA-1809, 
Username = gwenshap, data = None, headers = {}
No Affected or Fixed version found in JIRA
INFO: Guessed branch as trunk
INFO: URL = https://issues.apache.org/jira/browse/KAFKA-1809, Username = 
gwenshap, data = None, headers = {}
INFO: URL = 
https://issues.apache.org/jira/secure/attachment/12696246/KAFKA-1809_2015-02-03_10%3A52%3A36.patch,
 Username = gwenshap, data = None, headers = {}
INFO: Executing gradle 1>patch-process/bootstrap.txt 2>&1
INFO: Executing ./gradlew clean 1>patch-process/clean.txt 2>&1
INFO: Executing git checkout trunk
Already on 'trunk'
Your branch is up-to-date with 'origin/trunk'.
INFO: Executing git clean -d -f
Removing dev-utils/
Removing gradle/wrapper/
Removing patch-process/
INFO: Executing git reset --hard HEAD
HEAD is now at 1c6d5bb KAFKA-1915: Add checkstyle for java code.
INFO: Executing git fetch origin
INFO: Executing git merge --ff-only origin/trunk
Already up-to-date.
INFO: Executing git apply -p1 < patch-process/KAFKA-1809.patch 
1>patch-process/apply.txt 2>&1
/bin/sh: patch-process/KAFKA-1809.patch: No such file or directory
Traceback (most recent call last):
  File "./dev-utils/test-patch.py", line 420, in 
  File "./dev-utils/test-patch.py", line 205, in git_apply
  File "./dev-utils/test-patch.py", line 280, in fatal
  File "./dev-utils/test-patch.py", line 344, in log_and_exit
  File "./dev-utils/test-patch.py", line 37, in write_file
IOError: [Errno 2] No such file or directory: 'patch-process/jira-comment.txt'
{code}

I think we are just missing something like:
{code}
if options.output_dir and not os.path.isdir(options.output_dir):
  os.makedirs(options.output_dir)
{code}

> Add PreCommit Patch Testing
> ---
>
> Key: KAFKA-1856
> URL: https://issues.apache.org/jira/browse/KAFKA-1856
> Project: Kafka
>  Issue Type: Task
>Reporter: Ashish Kumar Singh
>Assignee: Ashish Kumar Singh
> Attachments: KAFKA-1856.patch, KAFKA-1856_2015-01-18_21:43:56.patch
>
>
> h1. Kafka PreCommit Patch Testing - *Don't wait for it to break*
> h2. Motivation
> *With great power comes great responsibility* - Uncle Ben. As Kafka user list 
> is growing, mechanism to ensure quality of the product is required. Quality 
> becomes hard to measure and maintain in an open source project, because of a 
> wide community of contributors. Luckily, Kafka is not the first open source 
> project and can benefit from learnings of prior projects.
> PreCommit tests are the tests that are run for each patch that gets attached 
> to an open JIRA. Based on tests results, test execution framework, test bot, 
> +1 or -1 the patch. Having PreCommit tests take the load off committers to 
> look at or test each patch.
> h2. Tests in Kafka
> h3. Unit and Integraiton Tests
> [Unit and Integration 
> tests|https://cwiki.apache.org/confluence/display/KAFKA/Kafka+0.9+Unit+and+Integration+Tests]
>  are cardinal to help contributors to avoid breaking existing functionalities 
> while adding new functionalities or fixing older ones. These tests, atleast 
> the ones relevant to the changes, must be run by contributors before 
> attaching a patch to a JIRA.
> h3. System Tests
> [System 
> tests|https://cwiki.apache.org/confluence/display/KAFKA/Kafka+System+Tests] 
> are much wider tests that, unlike unit tests, focus on end-to-end scenarios 
> and not some specific method or class.
> h2. Apache PreCommit tests
> Apache provides a mechanism to automatically build a project and run a series 
> of tests whenever a patch is uploaded to a JIRA. Based on test execution, the 
> test framework will comment with a +1 or -1 on the JIRA.
> You can read more about the framework here:
> http://wiki.apache.org/general/PreCommitBuilds
> h2. Plan
> # Create a test-patch.py script (similar to the one used in Flume, Sqoop and 
> other projects) that will take a jira as a parameter, apply on the 
> appropriate branch, build the project, run tests and report results. This 
> script should be committed into the Kafka code-base. To begin with, this will 
> only run unit tests. We can add code sanity checks, system_tests, etc in the 
> future.
> # Create a jenkins job for running the test (as described in 
> http://wiki.apache.org/general/PreCommitBuilds) and validate that it works 
> manually. This must be done by a committer with Jenkins access.
> # Ask someone with access to https://builds.apache.org/job/PreCommit-Admin/ 
> to add Kafka to

Re: [KIP-DISCUSSION] Mirror Maker Enhancement

[Jiangjie Qin]

In mirror maker we do not do de-serialization on the messages. Mirror
maker use source TopicPartition hash to chose a producer to send messages
from the same source partition. The partition those messages end up with
are decided by Partitioner class in KafkaProducer (assuming you are using
the new producer), which uses hash code of bytes[].

If deserialization is needed, it has to be done in message handler.

Thanks.

Jiangjie (Becket) Qin

On 2/4/15, 11:33 AM, "Bhavesh Mistry"  wrote:

>Hi Jiangjie,
>
>Thanks for entertaining my question so far.  Last question, I have is
>about
>serialization of message key.  If the key de-serialization (Class) is not
>present at the MM instance, then does it use raw byte hashcode to
>determine
>the partition ?  How are you going to address the situation where key
>needs
>to be de-serialization and get actual hashcode needs to be computed  ?.
>
>
>Thanks,
>
>Bhavesh
>
>On Fri, Jan 30, 2015 at 1:41 PM, Jiangjie Qin 
>wrote:
>
>> Hi Bhavesh,
>>
>> Please see inline comments.
>>
>> Jiangjie (Becket) Qin
>>
>> On 1/29/15, 7:00 PM, "Bhavesh Mistry" 
>>wrote:
>>
>> >Hi Jiangjie,
>> >
>> >Thanks for the input.
>> >
>> >a) Is MM will  producer ack will be attach to Producer Instance or per
>> >topic.  Use case is that one instance of MM
>> >needs to handle both strong ack and also ack=0 for some topic.  Or it
>> >would
>> >be better to set-up another instance of MM.
>> The acks setting is producer level setting instead of topic level
>>setting.
>> In this case you probably need to set up another instance.
>> >
>> >b) Regarding TCP connections, Why does #producer instance attach to TCP
>> >connection.  Is it possible to use Broker Connection TCP Pool, producer
>> >will just checkout TCP connection  to Broker.  So, # of Producer
>>Instance
>> >does not correlation to Brokers Connection.  Is this possible ?
>> In new producer, each producer maintains a connection to each broker
>> within the producer instance. Making producer instances to share the TCP
>> connections is a very big change to the current design, so I suppose we
>> won’t be able to do that.
>> >
>> >
>> >Thanks,
>> >
>> >Bhavesh
>> >
>> >On Thu, Jan 29, 2015 at 11:50 AM, Jiangjie Qin
>>> >
>> >wrote:
>> >
>> >> Hi Bhavesh,
>> >>
>> >> I think it is the right discussion to have when we are talking about
>>the
>> >> new new design for MM.
>> >> Please see the inline comments.
>> >>
>> >> Jiangjie (Becket) Qin
>> >>
>> >> On 1/28/15, 10:48 PM, "Bhavesh Mistry" 
>> >>wrote:
>> >>
>> >> >Hi Jiangjie,
>> >> >
>> >> >I just wanted to let you know about our use case and stress the
>>point
>> >>that
>> >> >local data center broker cluster have fewer partitions than the
>> >> >destination
>> >> >offline broker cluster. Just because we do the batch pull from CAMUS
>> >>and
>> >> >in
>> >> >order to drain data faster than the injection rate (from four DCs
>>for
>> >>same
>> >> >topic).
>> >> Keeping the same partition number in source and target cluster will
>>be
>> >>an
>> >> option but will not be enforced by default.
>> >> >
>> >> >We are facing following issues (probably due to configuration):
>> >> >
>> >> >1)  We occasionally loose data due to message batch size is too
>> >>large
>> >> >(2MB) on target data (we are using old producer but I think new
>> >>producer
>> >> >will solve this problem to some extend).
>> >> We do see this issue in LinkedIn as well. New producer also might
>>have
>> >> this issue. There are some proposal of solutions, but no real work
>> >>started
>> >> yet. For now, as a workaround, setting a more aggressive batch size
>>on
>> >> producer side should work.
>> >> >2)  Since only one instance is set to MM data,  we are not able
>>to
>> >> >set-up ack per topic instead ack is attached to producer instance.
>> >> I don’t quite get the question here.
>> >> >3)  How are you going to address two phase commit problem if
>>ack is
>> >> >set
>> >> >to strongest, but auto commit is on for consumer (meaning producer
>>does
>> >> >not
>> >> >get ack,  but consumer auto committed offset that message).  Is
>>there
>> >> >transactional (Kafka transaction is in process) based ack and commit
>> >> >offset
>> >> >?
>> >> Auto offset commit should be turned off in this case. The offset will
>> >>only
>> >> be committed once by the offset commit thread. So there is no two
>>phase
>> >> commit.
>> >> >4)  How are you planning to avoid duplicated message?  ( Is
>> >> >brokergoing
>> >> >have moving window of message collected and de-dupe ?)  Possibly, we
>> >>get
>> >> >this from retry set to 5…?
>> >> We are not trying to completely avoid duplicates. The duplicates will
>> >> still be there if:
>> >> 1. Producer retries on failure.
>> >> 2. Mirror maker is hard killed.
>> >> Currently, dedup is expected to be done by user if necessary.
>> >> >5)  Last, is there any warning or any thing you can provide
>>insight
>> >> >from MM component about data injection rate into destination
>> >>partitions is
>> >

[jira] [Updated] (KAFKA-1646) Improve consumer read performance for Windows

[Jakob Homan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jakob Homan updated KAFKA-1646:
---
Assignee: xueqiang wang

> Improve consumer read performance for Windows
> -
>
> Key: KAFKA-1646
> URL: https://issues.apache.org/jira/browse/KAFKA-1646
> Project: Kafka
>  Issue Type: Improvement
>  Components: log
>Affects Versions: 0.8.1.1
> Environment: Windows
>Reporter: xueqiang wang
>Assignee: xueqiang wang
>  Labels: newbie, patch
> Attachments: Improve consumer read performance for Windows.patch, 
> KAFKA-1646-truncate-off-trailing-zeros-on-broker-restart-if-bro.patch, 
> KAFKA-1646_20141216_163008.patch
>
>
> This patch is for Window platform only. In Windows platform, if there are 
> more than one replicas writing to disk, the segment log files will not be 
> consistent in disk and then consumer reading performance will be dropped down 
> greatly. This fix allocates more disk spaces when rolling a new segment, and 
> then it will improve the consumer reading performance in NTFS file system.
> This patch doesn't affect file allocation of other filesystems, for it only 
> adds statements like 'if(Os.iswindow)' or adds methods used on Windows.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (KAFKA-1646) Improve consumer read performance for Windows

[Jakob Homan (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14231258#comment-14231258
 ] 

Jakob Homan edited comment on KAFKA-1646 at 2/4/15 8:32 PM:


Hi, Jun. 
I also came from same team with Xueqiang & Honghai. 
Just as Jay mentioned before, recover the log is also necessary when the broker 
is stopped gracefully. Since we just recover the activeSegment (only one 
LogSegment), so it would not cost a lot of time. 
And I just write some code to test the performance of LogSegment.recover(),
{code}
public class TestLogSegment {
public static void main(String[] args) throws Exception {
String dirTemplate = args.length > 0? args[0] : 
"D:\\C\\scp2\\tachyon\\logs\\testbroker\\mvlogs-";
int logFileNums = args.length > 1? Integer.parseInt(args[1]) : 10;

long startOffset = 0L;
int indexIntervalBytes = 4096;
int maxIndexSize = 10485760;
long initFileSize = 536870912L;
int maxMessageSize = 100;

long totalTime = 0L;
for (int i=0; i 0? args[0] : 
"D:\\C\\scp2\\tachyon\\logs\\testbroker\\mvlogs-";
int logFileNums = args.length > 1? Integer.parseInt(args[1]) : 10;

long startOffset = 0L;
int indexIntervalBytes = 4096;
int maxIndexSize = 10485760;
long initFileSize = 536870912L;
int maxMessageSize = 100;

long totalTime = 0L;
for (int i=0; i Improve consumer read performance for Windows
> -
>
> Key: KAFKA-1646
> URL: https://issues.apache.org/jira/browse/KAFKA-1646
> Project: Kafka
>  Issue Type: Improvement
>  Components: log
>Affects Versions: 0.8.1.1
> Environment: Windows
>Reporter: xueqiang wang
>Assignee: xueqiang wang
>  Labels: newbie, patch
> Attachments: Improve consumer read performance for Windows.patch, 
> KAFKA-1646-truncate-off-trailing-zeros-on-broker-restart-if-bro.patch, 
> KAFKA-1646_20141216_163008.patch
>
>
> This patch is for Window platform only. In Windows platform, if there are 
> more than one replicas writing to disk, the segment log files will not be 
> consistent in disk and then consumer reading performance will be dropped down 
> greatly. This fix allocates more disk spaces when rolling a new segment, and 
> then it will improve the consumer reading performance in NTFS file system.
> This patch doesn't affect file allocation of other filesystems, for it only 
> adds statements like 'if(Os.iswindow)' or adds methods used on Windows.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (KAFKA-1646) Improve consumer read performance for Windows

[Jakob Homan (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14179731#comment-14179731
 ] 

Jakob Homan edited comment on KAFKA-1646 at 2/4/15 8:36 PM:


Hey, Jay, sorry for late. I have done a test and find there is no pause when 
rolling new log segment. The time for creating an 1G file and 1K file is almost 
identical: all about 1ms.
Here is the test code which creating 10 files of 1G:
{code}
public static void main(String[] args) throws Exception  {
String filePre = "d:\\temp\\file";
long startTime, elapsedTime;
startTime = System.currentTimeMillis();
try {
long initFileSize = 102400l;
for (int i = 0; i < 10; i++) {
RandomAccessFile randomAccessFile = new 
RandomAccessFile(filePre + i, "rw");
randomAccessFile.setLength(initFileSize);
randomAccessFile.getChannel();
}
elapsedTime = System.currentTimeMillis() - startTime;
System.out.format("elapsedTime: %2d ms", elapsedTime);
} catch (Exception exception) { }
}
{code}
The result is: elapsedTime: 14 ms



was (Author: xueqiang):
Hey, Jay, sorry for late. I have done a test and find there is no pause when 
rolling new log segment. The time for creating an 1G file and 1K file is almost 
identical: all about 1ms.
Here is the test code which creating 10 files of 1G:
public static void main(String[] args) throws Exception  {
String filePre = "d:\\temp\\file";
long startTime, elapsedTime;
startTime = System.currentTimeMillis();
try {
long initFileSize = 102400l;
for (int i = 0; i < 10; i++) {
RandomAccessFile randomAccessFile = new 
RandomAccessFile(filePre + i, "rw");
randomAccessFile.setLength(initFileSize);
randomAccessFile.getChannel();
}
elapsedTime = System.currentTimeMillis() - startTime;
System.out.format("elapsedTime: %2d ms", elapsedTime);
} catch (Exception exception) { }
}

The result is: elapsedTime: 14 ms


> Improve consumer read performance for Windows
> -
>
> Key: KAFKA-1646
> URL: https://issues.apache.org/jira/browse/KAFKA-1646
> Project: Kafka
>  Issue Type: Improvement
>  Components: log
>Affects Versions: 0.8.1.1
> Environment: Windows
>Reporter: xueqiang wang
>Assignee: xueqiang wang
>  Labels: newbie, patch
> Attachments: Improve consumer read performance for Windows.patch, 
> KAFKA-1646-truncate-off-trailing-zeros-on-broker-restart-if-bro.patch, 
> KAFKA-1646_20141216_163008.patch
>
>
> This patch is for Window platform only. In Windows platform, if there are 
> more than one replicas writing to disk, the segment log files will not be 
> consistent in disk and then consumer reading performance will be dropped down 
> greatly. This fix allocates more disk spaces when rolling a new segment, and 
> then it will improve the consumer reading performance in NTFS file system.
> This patch doesn't affect file allocation of other filesystems, for it only 
> adds statements like 'if(Os.iswindow)' or adds methods used on Windows.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: kafka.utils.DoublyLinkedList

[Jiangjie Qin]

Hi Jay,

The approach you suggested is exactly what we did at beginning. But Jun
mentioned a case in RB that could cause problem.

"Suppose there are 2 target partitions. Offset 1,2,4,5 can be routed to
partition 1 and offset 3 can be routed to partition 2. When all messages
are sent, you will see partition 1 with offset 5 and partition 2 with
offset 3. Since offset 3 and 5 are not consecutive, are you just going to
commit offset 3? If so, you may be stuck in offset 3 forever if no new
messages  are sent to partition 2."


I’ll update the Mirror Maker document later today.

BTW, actually even remove(ind index) in Java LinkedList still takes O(n)
to remove a node.

public E remove(int index) {
  checkElementIndex(index);
  return unlink(node(index));
}

Node node(int index) {
  // assert isElementIndex(index);

  if (index < (size >> 1)) {
Node x = first;
for (int i = 0; i < index; i++)
  x = x.next;
return x;
  } else {
Node x = last;
for (int i = size - 1; i > index; i--)
  x = x.prev;
return x;
  }
}




Thanks.

Jiangjie (Becket) Qin


On 2/4/15, 9:00 AM, "Jay Kreps"  wrote:

>Yeah I think the Java LL is worse than that, you actually need to point at
>it with an iterator and call remove(). I didn't realize the scala ll was
>deprecated. Okay, makes sense. Would be good to move that test out of
>UtilsTest.scala, though.
>
>But actually, stepping back, can you guys document/explain what all that
>mm
>code is doing now? It is pretty tricky which makes me worry. Are we
>keeping
>track of each individual message until it is acknowledged in that list?
>Can't we just keep the largest input/output offset pair and commit that
>periodically and on rebalance?
>
>-Jay
>
>On Wed, Feb 4, 2015 at 8:39 AM, Joel Koshy  wrote:
>
>> Jay,
>>
>> My understanding is that java linked lists (even though they are backed
>>by
>> a doubly linked list) support O(1) deletion only if you have the index
>>of
>> the object. If you do a remove(object) it becomes O(n). As for scala
>>lists
>> - the scaladocs seem to indicate that the doubly linked lists are
>> deprecated as of 2.11
>>
>> Joel
>>
>> On Wednesday, February 4, 2015, Jay Kreps  wrote:
>>
>> > Hey Jiangjie,
>> >
>> > Both the scala and java doubly linked lists allow O(1) deletion--it
>>isn't
>> > obvious because the api is slightly different from the CS101 linked
>>list
>> > but they do. Are you sure those wouldn't work? Can we remove this
>>class
>> and
>> > use those?
>> >
>> > Also the convention for unit tests is if we have a class Foo.scala
>>then
>> we
>> > would have a test FooTest.scala. So you can definitely write a unit
>>test
>> > for classes used by MM without adding it to kafka.utils. The
>> > UtilsTest.scala is really for Utils.scala we shouldn't be adding other
>> > tests there.
>> >
>> > -Jay
>> >
>> > On Tue, Feb 3, 2015 at 11:46 PM, Jiangjie Qin
>>> >
>> > wrote:
>> >
>> > > We added this because in redesigned mirror maker we need a raw
>>linked
>> > list
>> > > so we can removed an acked offset from a linked list in O(1). Java
>>and
>> > > Scala linked list won¹t work in that case. It was initially put in
>>side
>> > > mirror maker as a private class, but later on we also want to have
>>unit
>> > > test for it, that¹s why it becomes a separate util class and we have
>> unit
>> > > test in UtilsTest.scala.
>> > >
>> > > -Jiangjie (Becket) Qin
>> > >
>> > > On 2/3/15, 9:15 PM, "Jay Kreps" >
>> > wrote:
>> > >
>> > > >How come we added this? Is it better than the linked list in java
>>and
>> > > >scala? Also it was added with only one test which is actually put
>> inside
>> > > >UtilsTest.scala which is meant to house tests for Utils.scala.
>> > > >
>> > > >-Jay
>> > >
>> > >
>> >
>>
>>
>> --
>> Sent from Gmail Mobile
>>

[jira] [Commented] (KAFKA-1646) Improve consumer read performance for Windows

[Jakob Homan (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14305923#comment-14305923
 ] 

Jakob Homan commented on KAFKA-1646:


This is a big performance hit on Windows, and it looks like the concerns about 
the fix's impact on normal performance have been addressed.  Any further 
concerns? This is looking ready to go in.

> Improve consumer read performance for Windows
> -
>
> Key: KAFKA-1646
> URL: https://issues.apache.org/jira/browse/KAFKA-1646
> Project: Kafka
>  Issue Type: Improvement
>  Components: log
>Affects Versions: 0.8.1.1
> Environment: Windows
>Reporter: xueqiang wang
>Assignee: xueqiang wang
>  Labels: newbie, patch
> Attachments: Improve consumer read performance for Windows.patch, 
> KAFKA-1646-truncate-off-trailing-zeros-on-broker-restart-if-bro.patch, 
> KAFKA-1646_20141216_163008.patch
>
>
> This patch is for Window platform only. In Windows platform, if there are 
> more than one replicas writing to disk, the segment log files will not be 
> consistent in disk and then consumer reading performance will be dropped down 
> greatly. This fix allocates more disk spaces when rolling a new segment, and 
> then it will improve the consumer reading performance in NTFS file system.
> This patch doesn't affect file allocation of other filesystems, for it only 
> adds statements like 'if(Os.iswindow)' or adds methods used on Windows.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (KAFKA-1919) Metadata request issued with no backoff in new producer if there are no topics

[Steven Zhen Wu (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14305966#comment-14305966
 ] 

Steven Zhen Wu edited comment on KAFKA-1919 at 2/4/15 9:12 PM:
---

this is the producer metrics at the beginning of the test. it's all idle.
{code}
main 13:06:54,212  INFO EmptyTopicTest:52 - [request-latency-avg] NaN
main 13:06:54,213  INFO EmptyTopicTest:52 - [outgoing-byte-rate] NaN
main 13:06:54,213  INFO EmptyTopicTest:52 - [batch-size-avg] NaN
main 13:06:54,213  INFO EmptyTopicTest:52 - [connection-close-rate] NaN
main 13:06:54,213  INFO EmptyTopicTest:52 - [record-size-max] -Infinity
main 13:06:54,214  INFO EmptyTopicTest:52 - [waiting-threads] 0.0
main 13:06:54,214  INFO EmptyTopicTest:52 - [record-queue-time-avg] NaN
main 13:06:54,214  INFO EmptyTopicTest:52 - [connection-count] 0.0
main 13:06:54,215  INFO EmptyTopicTest:52 - [metadata-age] 120.032
main 13:06:54,215  INFO EmptyTopicTest:52 - [request-size-max] -Infinity
main 13:06:54,215  INFO EmptyTopicTest:52 - [records-per-request-avg] NaN
main 13:06:54,216  INFO EmptyTopicTest:52 - [record-retry-rate] NaN
main 13:06:54,216  INFO EmptyTopicTest:52 - [record-send-rate] NaN
main 13:06:54,216  INFO EmptyTopicTest:52 - [buffer-total-bytes] 3.3554432E7
main 13:06:54,216  INFO EmptyTopicTest:52 - [network-io-rate] NaN
main 13:06:54,217  INFO EmptyTopicTest:52 - [response-rate] NaN
main 13:06:54,217  INFO EmptyTopicTest:52 - [io-wait-time-ns-avg] NaN
main 13:06:54,217  INFO EmptyTopicTest:52 - [compression-rate-avg] NaN
main 13:06:54,217  INFO EmptyTopicTest:52 - [incoming-byte-rate] NaN
main 13:06:54,217  INFO EmptyTopicTest:52 - [bufferpool-wait-ratio] NaN
main 13:06:54,218  INFO EmptyTopicTest:52 - [connection-creation-rate] NaN
main 13:06:54,218  INFO EmptyTopicTest:52 - [buffer-available-bytes] 3.3554432E7
main 13:06:54,218  INFO EmptyTopicTest:52 - [record-queue-time-max] -Infinity
main 13:06:54,218  INFO EmptyTopicTest:52 - [record-size-avg] NaN
main 13:06:54,219  INFO EmptyTopicTest:52 - [record-error-rate] NaN
main 13:06:54,219  INFO EmptyTopicTest:52 - [request-latency-max] -Infinity
main 13:06:54,219  INFO EmptyTopicTest:52 - [request-rate] NaN
main 13:06:54,219  INFO EmptyTopicTest:52 - [batch-size-max] -Infinity
main 13:06:54,220  INFO EmptyTopicTest:52 - [io-time-ns-avg] NaN
main 13:06:54,220  INFO EmptyTopicTest:52 - [select-rate] NaN
main 13:06:54,220  INFO EmptyTopicTest:52 - [io-ratio] NaN
main 13:06:54,220  INFO EmptyTopicTest:52 - [requests-in-flight] 0.0
main 13:06:54,220  INFO EmptyTopicTest:52 - [io-wait-ratio] NaN
main 13:06:54,221  INFO EmptyTopicTest:52 - [request-size-avg] NaN
{code}


was (Author: stevenz3wu):
this is the producer metrics at the beginning of the test. it's all idle.

main 13:06:54,212  INFO EmptyTopicTest:52 - [request-latency-avg] NaN
main 13:06:54,213  INFO EmptyTopicTest:52 - [outgoing-byte-rate] NaN
main 13:06:54,213  INFO EmptyTopicTest:52 - [batch-size-avg] NaN
main 13:06:54,213  INFO EmptyTopicTest:52 - [connection-close-rate] NaN
main 13:06:54,213  INFO EmptyTopicTest:52 - [record-size-max] -Infinity
main 13:06:54,214  INFO EmptyTopicTest:52 - [waiting-threads] 0.0
main 13:06:54,214  INFO EmptyTopicTest:52 - [record-queue-time-avg] NaN
main 13:06:54,214  INFO EmptyTopicTest:52 - [connection-count] 0.0
main 13:06:54,215  INFO EmptyTopicTest:52 - [metadata-age] 120.032
main 13:06:54,215  INFO EmptyTopicTest:52 - [request-size-max] -Infinity
main 13:06:54,215  INFO EmptyTopicTest:52 - [records-per-request-avg] NaN
main 13:06:54,216  INFO EmptyTopicTest:52 - [record-retry-rate] NaN
main 13:06:54,216  INFO EmptyTopicTest:52 - [record-send-rate] NaN
main 13:06:54,216  INFO EmptyTopicTest:52 - [buffer-total-bytes] 3.3554432E7
main 13:06:54,216  INFO EmptyTopicTest:52 - [network-io-rate] NaN
main 13:06:54,217  INFO EmptyTopicTest:52 - [response-rate] NaN
main 13:06:54,217  INFO EmptyTopicTest:52 - [io-wait-time-ns-avg] NaN
main 13:06:54,217  INFO EmptyTopicTest:52 - [compression-rate-avg] NaN
main 13:06:54,217  INFO EmptyTopicTest:52 - [incoming-byte-rate] NaN
main 13:06:54,217  INFO EmptyTopicTest:52 - [bufferpool-wait-ratio] NaN
main 13:06:54,218  INFO EmptyTopicTest:52 - [connection-creation-rate] NaN
main 13:06:54,218  INFO EmptyTopicTest:52 - [buffer-available-bytes] 3.3554432E7
main 13:06:54,218  INFO EmptyTopicTest:52 - [record-queue-time-max] -Infinity
main 13:06:54,218  INFO EmptyTopicTest:52 - [record-size-avg] NaN
main 13:06:54,219  INFO EmptyTopicTest:52 - [record-error-rate] NaN
main 13:06:54,219  INFO EmptyTopicTest:52 - [request-latency-max] -Infinity
main 13:06:54,219  INFO EmptyTopicTest:52 - [request-rate] NaN
main 13:06:54,219  INFO EmptyTopicTest:52 - [batch-size-max] -Infinity
main 13:06:54,220  INFO EmptyTopicTest:52 - [io-time-ns-avg] NaN
main 13:06:54,220  INFO EmptyTopicTest:52 - [select-rate] NaN
main 13:06:54,220  INFO EmptyTopicTest:52 - [io-ratio]

[jira] [Commented] (KAFKA-1919) Metadata request issued with no backoff in new producer if there are no topics

[Steven Zhen Wu (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14305966#comment-14305966
 ] 

Steven Zhen Wu commented on KAFKA-1919:
---

this is the producer metrics at the beginning of the test. it's all idle.

main 13:06:54,212  INFO EmptyTopicTest:52 - [request-latency-avg] NaN
main 13:06:54,213  INFO EmptyTopicTest:52 - [outgoing-byte-rate] NaN
main 13:06:54,213  INFO EmptyTopicTest:52 - [batch-size-avg] NaN
main 13:06:54,213  INFO EmptyTopicTest:52 - [connection-close-rate] NaN
main 13:06:54,213  INFO EmptyTopicTest:52 - [record-size-max] -Infinity
main 13:06:54,214  INFO EmptyTopicTest:52 - [waiting-threads] 0.0
main 13:06:54,214  INFO EmptyTopicTest:52 - [record-queue-time-avg] NaN
main 13:06:54,214  INFO EmptyTopicTest:52 - [connection-count] 0.0
main 13:06:54,215  INFO EmptyTopicTest:52 - [metadata-age] 120.032
main 13:06:54,215  INFO EmptyTopicTest:52 - [request-size-max] -Infinity
main 13:06:54,215  INFO EmptyTopicTest:52 - [records-per-request-avg] NaN
main 13:06:54,216  INFO EmptyTopicTest:52 - [record-retry-rate] NaN
main 13:06:54,216  INFO EmptyTopicTest:52 - [record-send-rate] NaN
main 13:06:54,216  INFO EmptyTopicTest:52 - [buffer-total-bytes] 3.3554432E7
main 13:06:54,216  INFO EmptyTopicTest:52 - [network-io-rate] NaN
main 13:06:54,217  INFO EmptyTopicTest:52 - [response-rate] NaN
main 13:06:54,217  INFO EmptyTopicTest:52 - [io-wait-time-ns-avg] NaN
main 13:06:54,217  INFO EmptyTopicTest:52 - [compression-rate-avg] NaN
main 13:06:54,217  INFO EmptyTopicTest:52 - [incoming-byte-rate] NaN
main 13:06:54,217  INFO EmptyTopicTest:52 - [bufferpool-wait-ratio] NaN
main 13:06:54,218  INFO EmptyTopicTest:52 - [connection-creation-rate] NaN
main 13:06:54,218  INFO EmptyTopicTest:52 - [buffer-available-bytes] 3.3554432E7
main 13:06:54,218  INFO EmptyTopicTest:52 - [record-queue-time-max] -Infinity
main 13:06:54,218  INFO EmptyTopicTest:52 - [record-size-avg] NaN
main 13:06:54,219  INFO EmptyTopicTest:52 - [record-error-rate] NaN
main 13:06:54,219  INFO EmptyTopicTest:52 - [request-latency-max] -Infinity
main 13:06:54,219  INFO EmptyTopicTest:52 - [request-rate] NaN
main 13:06:54,219  INFO EmptyTopicTest:52 - [batch-size-max] -Infinity
main 13:06:54,220  INFO EmptyTopicTest:52 - [io-time-ns-avg] NaN
main 13:06:54,220  INFO EmptyTopicTest:52 - [select-rate] NaN
main 13:06:54,220  INFO EmptyTopicTest:52 - [io-ratio] NaN
main 13:06:54,220  INFO EmptyTopicTest:52 - [requests-in-flight] 0.0
main 13:06:54,220  INFO EmptyTopicTest:52 - [io-wait-ratio] NaN
main 13:06:54,221  INFO EmptyTopicTest:52 - [request-size-avg] NaN

> Metadata request issued with no backoff in new producer if there are no topics
> --
>
> Key: KAFKA-1919
> URL: https://issues.apache.org/jira/browse/KAFKA-1919
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2
>Reporter: Jay Kreps
>
> Original report:
> We have observed high cpu and high network traffic problem when
> 1) cluster (0.8.1.1) has no topic
> 2) KafkaProducer (0.8.2-beta) object is created without sending any traffic
> We have observed such problem twice. In both cases, problem went away
> immediately after one/any topic is created.
> Is this a known issue? Just want to check with the community first before I
> spend much time to reproduce it.
> I couldn't reproduce the issue with similar setup with unit test code in
> IDE. start two brokers with no topic locally on my laptop. create a
> KafkaProducer object without sending any msgs. but I only tested with
> 0.8.2-beta for both broker and producer.
> Issue exists in 0.8.2 as well:
> I have re-run my unit test with 0.8.2.0. same tight-loop problem happened
> after a few mins.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1919) Metadata request issued with no backoff in new producer if there are no topics

[Steven Zhen Wu (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14305971#comment-14305971
 ] 

Steven Zhen Wu commented on KAFKA-1919:
---

this is the producer metrics when cpu shoot up to 100% on one core. it clears 
shows request rate shoot up to the roof.
 [request-rate] 18716.091686350177

{code}
main 13:10:54,254  INFO EmptyTopicTest:52 - [request-latency-avg] NaN
main 13:10:54,254  INFO EmptyTopicTest:52 - [outgoing-byte-rate] 
393038.5293628348
main 13:10:54,254  INFO EmptyTopicTest:52 - [batch-size-avg] NaN
main 13:10:54,254  INFO EmptyTopicTest:52 - [connection-close-rate] NaN
main 13:10:54,254  INFO EmptyTopicTest:52 - [record-size-max] -Infinity
main 13:10:54,255  INFO EmptyTopicTest:52 - [waiting-threads] 0.0
main 13:10:54,255  INFO EmptyTopicTest:52 - [record-queue-time-avg] NaN
main 13:10:54,255  INFO EmptyTopicTest:52 - [connection-count] 1.0
main 13:10:54,255  INFO EmptyTopicTest:52 - [metadata-age] 360.072
main 13:10:54,255  INFO EmptyTopicTest:52 - [request-size-max] 21.0
main 13:10:54,255  INFO EmptyTopicTest:52 - [records-per-request-avg] NaN
main 13:10:54,255  INFO EmptyTopicTest:52 - [record-retry-rate] NaN
main 13:10:54,256  INFO EmptyTopicTest:52 - [record-send-rate] NaN
main 13:10:54,256  INFO EmptyTopicTest:52 - [buffer-total-bytes] 3.3554432E7
main 13:10:54,256  INFO EmptyTopicTest:52 - [network-io-rate] 37432.278670526
main 13:10:54,256  INFO EmptyTopicTest:52 - [response-rate] 18715.822784810127
main 13:10:54,256  INFO EmptyTopicTest:52 - [io-wait-time-ns-avg] 
19766.866064623417
main 13:10:54,256  INFO EmptyTopicTest:52 - [compression-rate-avg] NaN
main 13:10:54,257  INFO EmptyTopicTest:52 - [incoming-byte-rate] 
224586.38952733087
main 13:10:54,257  INFO EmptyTopicTest:52 - [bufferpool-wait-ratio] NaN
main 13:10:54,257  INFO EmptyTopicTest:52 - [connection-creation-rate] NaN
main 13:10:54,257  INFO EmptyTopicTest:52 - [buffer-available-bytes] 3.3554432E7
main 13:10:54,257  INFO EmptyTopicTest:52 - [record-queue-time-max] -Infinity
main 13:10:54,257  INFO EmptyTopicTest:52 - [record-size-avg] NaN
main 13:10:54,258  INFO EmptyTopicTest:52 - [record-error-rate] NaN
main 13:10:54,258  INFO EmptyTopicTest:52 - [request-latency-max] -Infinity
main 13:10:54,258  INFO EmptyTopicTest:52 - [request-rate] 18716.091686350177
main 13:10:54,258  INFO EmptyTopicTest:52 - [batch-size-max] -Infinity
main 13:10:54,258  INFO EmptyTopicTest:52 - [io-time-ns-avg] 5109.803392094824
main 13:10:54,258  INFO EmptyTopicTest:52 - [select-rate] 37431.806091235536
main 13:10:54,259  INFO EmptyTopicTest:52 - [io-ratio] 0.19126365661468897
main 13:10:54,259  INFO EmptyTopicTest:52 - [requests-in-flight] 1.0
main 13:10:54,259  INFO EmptyTopicTest:52 - [io-wait-ratio] 0.7398999567776041
main 13:10:54,259  INFO EmptyTopicTest:52 - [request-size-avg] 21.0
{code}

> Metadata request issued with no backoff in new producer if there are no topics
> --
>
> Key: KAFKA-1919
> URL: https://issues.apache.org/jira/browse/KAFKA-1919
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2
>Reporter: Jay Kreps
>
> Original report:
> We have observed high cpu and high network traffic problem when
> 1) cluster (0.8.1.1) has no topic
> 2) KafkaProducer (0.8.2-beta) object is created without sending any traffic
> We have observed such problem twice. In both cases, problem went away
> immediately after one/any topic is created.
> Is this a known issue? Just want to check with the community first before I
> spend much time to reproduce it.
> I couldn't reproduce the issue with similar setup with unit test code in
> IDE. start two brokers with no topic locally on my laptop. create a
> KafkaProducer object without sending any msgs. but I only tested with
> 0.8.2-beta for both broker and producer.
> Issue exists in 0.8.2 as well:
> I have re-run my unit test with 0.8.2.0. same tight-loop problem happened
> after a few mins.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Cannot stop Kafka server if zookeeper is shutdown first

[Gwen Shapira]

Hi,

KAFKA-1155 is likely Zookeeper and not the specific client.
I believe the rest are already fixed in ZKClient and its a matter of asking
them to release, rebase our code and make sure the issues are resolved (or
that we use the features ZKClient added to resolve them).

I'm a fan of Curator, but its not exactly a drop-in replacement for
ZKClient (the APIs are slightly different, if we even decide to just use
the APIs and not the recipes). I suspect that replacing ZKClient with
Curator is a large project. Perhaps too large to resolve 3 issues that are
already resolved in ZKClient.

What are the benefits you guys see in the replacement?

Gwen


On Tue, Feb 3, 2015 at 10:42 PM, Guozhang Wang  wrote:

> Now may be a good time.
>
> We could verify if Curator has fixed the known issues we have seen so far,
> an incomplete list would be:
>
> KAFKA-1082 
> KAFKA-1155 
> KAFKA-1907 
> KAFKA-992 
>
>
>
> Guozhang
>
> On Tue, Feb 3, 2015 at 10:21 PM, Ashish Singh  wrote:
>
> > +1 on using curator.
> >
> > On Tue, Feb 3, 2015 at 10:09 PM, Manikumar Reddy 
> > wrote:
> >
> > > I think we should consider to moving to  apache curator (KAFKA-873).
> > > Curator is now more mature and a apache top-level project.
> > >
> > >
> > > On Wed, Feb 4, 2015 at 11:29 AM, Harsha  wrote:
> > >
> > > > Any reason not to go with apache curator http://curator.apache.org/
> .
> > > > -Harsha
> > > > On Tue, Feb 3, 2015, at 09:55 PM, Guozhang Wang wrote:
> > > > > I am also +1 on Neha's suggestion that "At some point, if we find
> > > > > ourselves
> > > > > fiddling too much with ZkClient, it wouldn't hurt to write our own
> > > little
> > > > > zookeeper client wrapper." since we have accumulated a bunch of
> > issues
> > > > > with
> > > > > zkClient which takes long time be resolved if ever, so we ended up
> > have
> > > > > some hacky way handling zkClient errors.
> > > > >
> > > > > Guozhang
> > > > >
> > > > > On Tue, Feb 3, 2015 at 7:47 PM, Jaikiran Pai <
> > jai.forums2...@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > Yes, that's the plan :)
> > > > > >
> > > > > > -Jaikiran
> > > > > >
> > > > > > On Wednesday 04 February 2015 12:33 AM, Gwen Shapira wrote:
> > > > > >
> > > > > >> So I think the current plan is:
> > > > > >> 1. Add timeout in zkclient
> > > > > >> 2. Ask zkclient to release new version (we need it for few other
> > > > things
> > > > > >> too)
> > > > > >> 3. Rebase on new zkclient
> > > > > >> 4. Fix this jira and the few others than were waiting for the
> new
> > > > zkclient
> > > > > >>
> > > > > >> Does that make sense?
> > > > > >>
> > > > > >> Gwen
> > > > > >>
> > > > > >> On Mon, Feb 2, 2015 at 8:33 PM, Jaikiran Pai <
> > > > jai.forums2...@gmail.com>
> > > > > >> wrote:
> > > > > >>
> > > > > >>> I just heard back from Stefan, who manages the ZkClient repo
> and
> > he
> > > > > >>> seems to
> > > > > >>> be open to have these changes be part of ZkClient project. I'll
> > be
> > > > > >>> creating
> > > > > >>> a pull request for that project to have it reviewed and merged.
> > > > Although
> > > > > >>> I
> > > > > >>> haven't heard of exact release plans, Stefan's reply did
> indicate
> > > > that
> > > > > >>> the
> > > > > >>> project could be released after this change is merged.
> > > > > >>>
> > > > > >>> -Jaikiran
> > > > > >>>
> > > > > >>> On Tuesday 03 February 2015 09:03 AM, Jaikiran Pai wrote:
> > > > > >>>
> > > > >  Thanks for pointing to that repo!
> > > > > 
> > > > >  I just had a look at it and it appears that the project isn't
> > much
> > > > >  active
> > > > >  (going by the lack of activity). The latest contribution is
> from
> > > > Gwen
> > > > >  and
> > > > >  that was around 3 months back. I haven't found release plans
> for
> > > > that
> > > > >  project or a place to ask about it (filing an issue doesn't
> seem
> > > > right
> > > > >  to
> > > > >  ask this question). So I'll get in touch with the repo owner
> and
> > > see
> > > > >  what
> > > > >  his plans for the project are.
> > > > > 
> > > > >  -Jaikiran
> > > > > 
> > > > >  On Monday 02 February 2015 11:33 PM, Gwen Shapira wrote:
> > > > > 
> > > > > > I did!
> > > > > >
> > > > > > Thanks for clarifying :)
> > > > > >
> > > > > > The client that is part of Zookeeper itself actually does
> > support
> > > > > > timeouts.
> > > > > >
> > > > > > On Mon, Feb 2, 2015 at 9:54 AM, Guozhang Wang <
> > > wangg...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > >> Hi Jaikiran,
> > > > > >>
> > > > > >> I think Gwen was talking about contributing to ZkClient
> > project:
> > > > > >>
> > > > > >> https://github.com/sgroschupf/zkclient
> > > > > >>
> > > > > >> Guozhang
> 

[jira] [Created] (KAFKA-1920) Add a metric to count client side errors in BrokerTopicMetrics

[Aditya Auradkar (JIRA)]

Aditya Auradkar created KAFKA-1920:
--

 Summary: Add a metric to count client side errors in 
BrokerTopicMetrics
 Key: KAFKA-1920
 URL: https://issues.apache.org/jira/browse/KAFKA-1920
 Project: Kafka
  Issue Type: Improvement
Reporter: Aditya Auradkar
Assignee: Aditya Auradkar


Currently the BrokerTopicMetrics count only "failures" across all topics and 
for individual topics. Should we consider adding a metric to count the number 
of client side errors?
This essentially counts the number of bad requests per topic.





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 29893: Patch for KAFKA-1856

[Ashish Singh]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29893/
---

(Updated Feb. 4, 2015, 10:57 p.m.)


Review request for kafka.


Bugs: KAFKA-1856
https://issues.apache.org/jira/browse/KAFKA-1856


Repository: kafka


Description
---

KAFKA-1856: Add PreCommit Patch Testing


Diffs (updated)
-

  dev-utils/test-patch.py PRE-CREATION 

Diff: https://reviews.apache.org/r/29893/diff/


Testing
---

Tested on KAFKA-1664, 
https://issues.apache.org/jira/browse/KAFKA-1664?focusedCommentId=14277439&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14277439

How to run:
python dev-utils/test-patch.py --defect KAFKA-1664 --username  
--password  --run-tests --post-results


Thanks,

Ashish Singh

[jira] [Updated] (KAFKA-1856) Add PreCommit Patch Testing

[Ashish Kumar Singh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Kumar Singh updated KAFKA-1856:
--
Attachment: KAFKA-1856_2015-02-04_14:57:05.patch

> Add PreCommit Patch Testing
> ---
>
> Key: KAFKA-1856
> URL: https://issues.apache.org/jira/browse/KAFKA-1856
> Project: Kafka
>  Issue Type: Task
>Reporter: Ashish Kumar Singh
>Assignee: Ashish Kumar Singh
> Attachments: KAFKA-1856.patch, KAFKA-1856_2015-01-18_21:43:56.patch, 
> KAFKA-1856_2015-02-04_14:57:05.patch
>
>
> h1. Kafka PreCommit Patch Testing - *Don't wait for it to break*
> h2. Motivation
> *With great power comes great responsibility* - Uncle Ben. As Kafka user list 
> is growing, mechanism to ensure quality of the product is required. Quality 
> becomes hard to measure and maintain in an open source project, because of a 
> wide community of contributors. Luckily, Kafka is not the first open source 
> project and can benefit from learnings of prior projects.
> PreCommit tests are the tests that are run for each patch that gets attached 
> to an open JIRA. Based on tests results, test execution framework, test bot, 
> +1 or -1 the patch. Having PreCommit tests take the load off committers to 
> look at or test each patch.
> h2. Tests in Kafka
> h3. Unit and Integraiton Tests
> [Unit and Integration 
> tests|https://cwiki.apache.org/confluence/display/KAFKA/Kafka+0.9+Unit+and+Integration+Tests]
>  are cardinal to help contributors to avoid breaking existing functionalities 
> while adding new functionalities or fixing older ones. These tests, atleast 
> the ones relevant to the changes, must be run by contributors before 
> attaching a patch to a JIRA.
> h3. System Tests
> [System 
> tests|https://cwiki.apache.org/confluence/display/KAFKA/Kafka+System+Tests] 
> are much wider tests that, unlike unit tests, focus on end-to-end scenarios 
> and not some specific method or class.
> h2. Apache PreCommit tests
> Apache provides a mechanism to automatically build a project and run a series 
> of tests whenever a patch is uploaded to a JIRA. Based on test execution, the 
> test framework will comment with a +1 or -1 on the JIRA.
> You can read more about the framework here:
> http://wiki.apache.org/general/PreCommitBuilds
> h2. Plan
> # Create a test-patch.py script (similar to the one used in Flume, Sqoop and 
> other projects) that will take a jira as a parameter, apply on the 
> appropriate branch, build the project, run tests and report results. This 
> script should be committed into the Kafka code-base. To begin with, this will 
> only run unit tests. We can add code sanity checks, system_tests, etc in the 
> future.
> # Create a jenkins job for running the test (as described in 
> http://wiki.apache.org/general/PreCommitBuilds) and validate that it works 
> manually. This must be done by a committer with Jenkins access.
> # Ask someone with access to https://builds.apache.org/job/PreCommit-Admin/ 
> to add Kafka to the list of projects PreCommit-Admin triggers.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1856) Add PreCommit Patch Testing

[Ashish Kumar Singh (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14306164#comment-14306164
 ] 

Ashish Kumar Singh commented on KAFKA-1856:
---

Updated reviewboard https://reviews.apache.org/r/29893/
 against branch trunk

> Add PreCommit Patch Testing
> ---
>
> Key: KAFKA-1856
> URL: https://issues.apache.org/jira/browse/KAFKA-1856
> Project: Kafka
>  Issue Type: Task
>Reporter: Ashish Kumar Singh
>Assignee: Ashish Kumar Singh
> Attachments: KAFKA-1856.patch, KAFKA-1856_2015-01-18_21:43:56.patch, 
> KAFKA-1856_2015-02-04_14:57:05.patch
>
>
> h1. Kafka PreCommit Patch Testing - *Don't wait for it to break*
> h2. Motivation
> *With great power comes great responsibility* - Uncle Ben. As Kafka user list 
> is growing, mechanism to ensure quality of the product is required. Quality 
> becomes hard to measure and maintain in an open source project, because of a 
> wide community of contributors. Luckily, Kafka is not the first open source 
> project and can benefit from learnings of prior projects.
> PreCommit tests are the tests that are run for each patch that gets attached 
> to an open JIRA. Based on tests results, test execution framework, test bot, 
> +1 or -1 the patch. Having PreCommit tests take the load off committers to 
> look at or test each patch.
> h2. Tests in Kafka
> h3. Unit and Integraiton Tests
> [Unit and Integration 
> tests|https://cwiki.apache.org/confluence/display/KAFKA/Kafka+0.9+Unit+and+Integration+Tests]
>  are cardinal to help contributors to avoid breaking existing functionalities 
> while adding new functionalities or fixing older ones. These tests, atleast 
> the ones relevant to the changes, must be run by contributors before 
> attaching a patch to a JIRA.
> h3. System Tests
> [System 
> tests|https://cwiki.apache.org/confluence/display/KAFKA/Kafka+System+Tests] 
> are much wider tests that, unlike unit tests, focus on end-to-end scenarios 
> and not some specific method or class.
> h2. Apache PreCommit tests
> Apache provides a mechanism to automatically build a project and run a series 
> of tests whenever a patch is uploaded to a JIRA. Based on test execution, the 
> test framework will comment with a +1 or -1 on the JIRA.
> You can read more about the framework here:
> http://wiki.apache.org/general/PreCommitBuilds
> h2. Plan
> # Create a test-patch.py script (similar to the one used in Flume, Sqoop and 
> other projects) that will take a jira as a parameter, apply on the 
> appropriate branch, build the project, run tests and report results. This 
> script should be committed into the Kafka code-base. To begin with, this will 
> only run unit tests. We can add code sanity checks, system_tests, etc in the 
> future.
> # Create a jenkins job for running the test (as described in 
> http://wiki.apache.org/general/PreCommitBuilds) and validate that it works 
> manually. This must be done by a committer with Jenkins access.
> # Ask someone with access to https://builds.apache.org/job/PreCommit-Admin/ 
> to add Kafka to the list of projects PreCommit-Admin triggers.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1856) Add PreCommit Patch Testing

[Ashish Kumar Singh (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14306178#comment-14306178
 ] 

Ashish Kumar Singh commented on KAFKA-1856:
---

[~gwenshap] so the issue was that you were specifying {{output}} to be inside 
your kafka root dir and while cleaning up kafka before applying patch, {{git 
reset HEAD --hard}}, hard reset removes the dir. I have made a few 
modifications to accommodate this. Let me know if it still does not work.

I will try out the checkstyle and if it works fine then I will try to include 
that as part of this.

> Add PreCommit Patch Testing
> ---
>
> Key: KAFKA-1856
> URL: https://issues.apache.org/jira/browse/KAFKA-1856
> Project: Kafka
>  Issue Type: Task
>Reporter: Ashish Kumar Singh
>Assignee: Ashish Kumar Singh
> Attachments: KAFKA-1856.patch, KAFKA-1856_2015-01-18_21:43:56.patch, 
> KAFKA-1856_2015-02-04_14:57:05.patch
>
>
> h1. Kafka PreCommit Patch Testing - *Don't wait for it to break*
> h2. Motivation
> *With great power comes great responsibility* - Uncle Ben. As Kafka user list 
> is growing, mechanism to ensure quality of the product is required. Quality 
> becomes hard to measure and maintain in an open source project, because of a 
> wide community of contributors. Luckily, Kafka is not the first open source 
> project and can benefit from learnings of prior projects.
> PreCommit tests are the tests that are run for each patch that gets attached 
> to an open JIRA. Based on tests results, test execution framework, test bot, 
> +1 or -1 the patch. Having PreCommit tests take the load off committers to 
> look at or test each patch.
> h2. Tests in Kafka
> h3. Unit and Integraiton Tests
> [Unit and Integration 
> tests|https://cwiki.apache.org/confluence/display/KAFKA/Kafka+0.9+Unit+and+Integration+Tests]
>  are cardinal to help contributors to avoid breaking existing functionalities 
> while adding new functionalities or fixing older ones. These tests, atleast 
> the ones relevant to the changes, must be run by contributors before 
> attaching a patch to a JIRA.
> h3. System Tests
> [System 
> tests|https://cwiki.apache.org/confluence/display/KAFKA/Kafka+System+Tests] 
> are much wider tests that, unlike unit tests, focus on end-to-end scenarios 
> and not some specific method or class.
> h2. Apache PreCommit tests
> Apache provides a mechanism to automatically build a project and run a series 
> of tests whenever a patch is uploaded to a JIRA. Based on test execution, the 
> test framework will comment with a +1 or -1 on the JIRA.
> You can read more about the framework here:
> http://wiki.apache.org/general/PreCommitBuilds
> h2. Plan
> # Create a test-patch.py script (similar to the one used in Flume, Sqoop and 
> other projects) that will take a jira as a parameter, apply on the 
> appropriate branch, build the project, run tests and report results. This 
> script should be committed into the Kafka code-base. To begin with, this will 
> only run unit tests. We can add code sanity checks, system_tests, etc in the 
> future.
> # Create a jenkins job for running the test (as described in 
> http://wiki.apache.org/general/PreCommitBuilds) and validate that it works 
> manually. This must be done by a committer with Jenkins access.
> # Ask someone with access to https://builds.apache.org/job/PreCommit-Admin/ 
> to add Kafka to the list of projects PreCommit-Admin triggers.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 29831: Patch for KAFKA-1476

[Onur Karaman]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29831/
---

(Updated Feb. 4, 2015, 11:41 p.m.)


Review request for kafka.


Bugs: KAFKA-1476
https://issues.apache.org/jira/browse/KAFKA-1476


Repository: kafka


Description
---

Merged in work for KAFKA-1476 and sub-task KAFKA-1826


Diffs (updated)
-

  bin/kafka-consumer-groups.sh PRE-CREATION 
  core/src/main/scala/kafka/admin/AdminUtils.scala 
28b12c7b89a56c113b665fbde1b95f873f8624a3 
  core/src/main/scala/kafka/admin/ConsumerGroupCommand.scala PRE-CREATION 
  core/src/main/scala/kafka/utils/ZkUtils.scala 
c14bd455b6642f5e6eb254670bef9f57ae41d6cb 
  core/src/test/scala/unit/kafka/admin/DeleteConsumerGroupTest.scala 
PRE-CREATION 
  core/src/test/scala/unit/kafka/admin/DeleteTopicTest.scala 
33c27678bf8ae8feebcbcdaa4b90a1963157b4a5 
  core/src/test/scala/unit/kafka/utils/TestUtils.scala 
54755e8dd3f23ced313067566cd4ea867f8a496e 

Diff: https://reviews.apache.org/r/29831/diff/


Testing
---


Thanks,

Onur Karaman

[jira] [Updated] (KAFKA-1476) Get a list of consumer groups

[Onur Karaman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Onur Karaman updated KAFKA-1476:

Attachment: KAFKA-1476_2015-02-04_15:41:50.patch

> Get a list of consumer groups
> -
>
> Key: KAFKA-1476
> URL: https://issues.apache.org/jira/browse/KAFKA-1476
> Project: Kafka
>  Issue Type: Wish
>  Components: tools
>Affects Versions: 0.8.1.1
>Reporter: Ryan Williams
>Assignee: Balaji Seshadri
>  Labels: newbie
> Fix For: 0.9.0
>
> Attachments: ConsumerCommand.scala, KAFKA-1476-LIST-GROUPS.patch, 
> KAFKA-1476-RENAME.patch, KAFKA-1476-REVIEW-COMMENTS.patch, KAFKA-1476.patch, 
> KAFKA-1476.patch, KAFKA-1476.patch, KAFKA-1476.patch, 
> KAFKA-1476_2014-11-10_11:58:26.patch, KAFKA-1476_2014-11-10_12:04:01.patch, 
> KAFKA-1476_2014-11-10_12:06:35.patch, KAFKA-1476_2014-12-05_12:00:12.patch, 
> KAFKA-1476_2015-01-12_16:22:26.patch, KAFKA-1476_2015-01-12_16:31:20.patch, 
> KAFKA-1476_2015-01-13_10:36:18.patch, KAFKA-1476_2015-01-15_14:30:04.patch, 
> KAFKA-1476_2015-01-22_02:32:52.patch, KAFKA-1476_2015-01-30_11:09:59.patch, 
> KAFKA-1476_2015-02-04_15:41:50.patch, 
> sample-kafka-consumer-groups-sh-output-1-23-2015.txt, 
> sample-kafka-consumer-groups-sh-output.txt
>
>
> It would be useful to have a way to get a list of consumer groups currently 
> active via some tool/script that ships with kafka. This would be helpful so 
> that the system tools can be explored more easily.
> For example, when running the ConsumerOffsetChecker, it requires a group 
> option
> bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --topic test --group 
> ?
> But, when just getting started with kafka, using the console producer and 
> consumer, it is not clear what value to use for the group option.  If a list 
> of consumer groups could be listed, then it would be clear what value to use.
> Background:
> http://mail-archives.apache.org/mod_mbox/kafka-users/201405.mbox/%3cCAOq_b1w=slze5jrnakxvak0gu9ctdkpazak1g4dygvqzbsg...@mail.gmail.com%3e



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1476) Get a list of consumer groups

[Onur Karaman (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14306232#comment-14306232
 ] 

Onur Karaman commented on KAFKA-1476:
-

Updated reviewboard https://reviews.apache.org/r/29831/diff/
 against branch origin/trunk

> Get a list of consumer groups
> -
>
> Key: KAFKA-1476
> URL: https://issues.apache.org/jira/browse/KAFKA-1476
> Project: Kafka
>  Issue Type: Wish
>  Components: tools
>Affects Versions: 0.8.1.1
>Reporter: Ryan Williams
>Assignee: Balaji Seshadri
>  Labels: newbie
> Fix For: 0.9.0
>
> Attachments: ConsumerCommand.scala, KAFKA-1476-LIST-GROUPS.patch, 
> KAFKA-1476-RENAME.patch, KAFKA-1476-REVIEW-COMMENTS.patch, KAFKA-1476.patch, 
> KAFKA-1476.patch, KAFKA-1476.patch, KAFKA-1476.patch, 
> KAFKA-1476_2014-11-10_11:58:26.patch, KAFKA-1476_2014-11-10_12:04:01.patch, 
> KAFKA-1476_2014-11-10_12:06:35.patch, KAFKA-1476_2014-12-05_12:00:12.patch, 
> KAFKA-1476_2015-01-12_16:22:26.patch, KAFKA-1476_2015-01-12_16:31:20.patch, 
> KAFKA-1476_2015-01-13_10:36:18.patch, KAFKA-1476_2015-01-15_14:30:04.patch, 
> KAFKA-1476_2015-01-22_02:32:52.patch, KAFKA-1476_2015-01-30_11:09:59.patch, 
> KAFKA-1476_2015-02-04_15:41:50.patch, 
> sample-kafka-consumer-groups-sh-output-1-23-2015.txt, 
> sample-kafka-consumer-groups-sh-output.txt
>
>
> It would be useful to have a way to get a list of consumer groups currently 
> active via some tool/script that ships with kafka. This would be helpful so 
> that the system tools can be explored more easily.
> For example, when running the ConsumerOffsetChecker, it requires a group 
> option
> bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --topic test --group 
> ?
> But, when just getting started with kafka, using the console producer and 
> consumer, it is not clear what value to use for the group option.  If a list 
> of consumer groups could be listed, then it would be clear what value to use.
> Background:
> http://mail-archives.apache.org/mod_mbox/kafka-users/201405.mbox/%3cCAOq_b1w=slze5jrnakxvak0gu9ctdkpazak1g4dygvqzbsg...@mail.gmail.com%3e



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1856) Add PreCommit Patch Testing

[Ashish Kumar Singh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Kumar Singh updated KAFKA-1856:
--
Attachment: KAFKA-1856_2015-02-04_15:44:47.patch

> Add PreCommit Patch Testing
> ---
>
> Key: KAFKA-1856
> URL: https://issues.apache.org/jira/browse/KAFKA-1856
> Project: Kafka
>  Issue Type: Task
>Reporter: Ashish Kumar Singh
>Assignee: Ashish Kumar Singh
> Attachments: KAFKA-1856.patch, KAFKA-1856_2015-01-18_21:43:56.patch, 
> KAFKA-1856_2015-02-04_14:57:05.patch, KAFKA-1856_2015-02-04_15:44:47.patch
>
>
> h1. Kafka PreCommit Patch Testing - *Don't wait for it to break*
> h2. Motivation
> *With great power comes great responsibility* - Uncle Ben. As Kafka user list 
> is growing, mechanism to ensure quality of the product is required. Quality 
> becomes hard to measure and maintain in an open source project, because of a 
> wide community of contributors. Luckily, Kafka is not the first open source 
> project and can benefit from learnings of prior projects.
> PreCommit tests are the tests that are run for each patch that gets attached 
> to an open JIRA. Based on tests results, test execution framework, test bot, 
> +1 or -1 the patch. Having PreCommit tests take the load off committers to 
> look at or test each patch.
> h2. Tests in Kafka
> h3. Unit and Integraiton Tests
> [Unit and Integration 
> tests|https://cwiki.apache.org/confluence/display/KAFKA/Kafka+0.9+Unit+and+Integration+Tests]
>  are cardinal to help contributors to avoid breaking existing functionalities 
> while adding new functionalities or fixing older ones. These tests, atleast 
> the ones relevant to the changes, must be run by contributors before 
> attaching a patch to a JIRA.
> h3. System Tests
> [System 
> tests|https://cwiki.apache.org/confluence/display/KAFKA/Kafka+System+Tests] 
> are much wider tests that, unlike unit tests, focus on end-to-end scenarios 
> and not some specific method or class.
> h2. Apache PreCommit tests
> Apache provides a mechanism to automatically build a project and run a series 
> of tests whenever a patch is uploaded to a JIRA. Based on test execution, the 
> test framework will comment with a +1 or -1 on the JIRA.
> You can read more about the framework here:
> http://wiki.apache.org/general/PreCommitBuilds
> h2. Plan
> # Create a test-patch.py script (similar to the one used in Flume, Sqoop and 
> other projects) that will take a jira as a parameter, apply on the 
> appropriate branch, build the project, run tests and report results. This 
> script should be committed into the Kafka code-base. To begin with, this will 
> only run unit tests. We can add code sanity checks, system_tests, etc in the 
> future.
> # Create a jenkins job for running the test (as described in 
> http://wiki.apache.org/general/PreCommitBuilds) and validate that it works 
> manually. This must be done by a committer with Jenkins access.
> # Ask someone with access to https://builds.apache.org/job/PreCommit-Admin/ 
> to add Kafka to the list of projects PreCommit-Admin triggers.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1856) Add PreCommit Patch Testing

[Ashish Kumar Singh (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14306237#comment-14306237
 ] 

Ashish Kumar Singh commented on KAFKA-1856:
---

Updated reviewboard https://reviews.apache.org/r/29893/
 against branch trunk

> Add PreCommit Patch Testing
> ---
>
> Key: KAFKA-1856
> URL: https://issues.apache.org/jira/browse/KAFKA-1856
> Project: Kafka
>  Issue Type: Task
>Reporter: Ashish Kumar Singh
>Assignee: Ashish Kumar Singh
> Attachments: KAFKA-1856.patch, KAFKA-1856_2015-01-18_21:43:56.patch, 
> KAFKA-1856_2015-02-04_14:57:05.patch, KAFKA-1856_2015-02-04_15:44:47.patch
>
>
> h1. Kafka PreCommit Patch Testing - *Don't wait for it to break*
> h2. Motivation
> *With great power comes great responsibility* - Uncle Ben. As Kafka user list 
> is growing, mechanism to ensure quality of the product is required. Quality 
> becomes hard to measure and maintain in an open source project, because of a 
> wide community of contributors. Luckily, Kafka is not the first open source 
> project and can benefit from learnings of prior projects.
> PreCommit tests are the tests that are run for each patch that gets attached 
> to an open JIRA. Based on tests results, test execution framework, test bot, 
> +1 or -1 the patch. Having PreCommit tests take the load off committers to 
> look at or test each patch.
> h2. Tests in Kafka
> h3. Unit and Integraiton Tests
> [Unit and Integration 
> tests|https://cwiki.apache.org/confluence/display/KAFKA/Kafka+0.9+Unit+and+Integration+Tests]
>  are cardinal to help contributors to avoid breaking existing functionalities 
> while adding new functionalities or fixing older ones. These tests, atleast 
> the ones relevant to the changes, must be run by contributors before 
> attaching a patch to a JIRA.
> h3. System Tests
> [System 
> tests|https://cwiki.apache.org/confluence/display/KAFKA/Kafka+System+Tests] 
> are much wider tests that, unlike unit tests, focus on end-to-end scenarios 
> and not some specific method or class.
> h2. Apache PreCommit tests
> Apache provides a mechanism to automatically build a project and run a series 
> of tests whenever a patch is uploaded to a JIRA. Based on test execution, the 
> test framework will comment with a +1 or -1 on the JIRA.
> You can read more about the framework here:
> http://wiki.apache.org/general/PreCommitBuilds
> h2. Plan
> # Create a test-patch.py script (similar to the one used in Flume, Sqoop and 
> other projects) that will take a jira as a parameter, apply on the 
> appropriate branch, build the project, run tests and report results. This 
> script should be committed into the Kafka code-base. To begin with, this will 
> only run unit tests. We can add code sanity checks, system_tests, etc in the 
> future.
> # Create a jenkins job for running the test (as described in 
> http://wiki.apache.org/general/PreCommitBuilds) and validate that it works 
> manually. This must be done by a committer with Jenkins access.
> # Ask someone with access to https://builds.apache.org/job/PreCommit-Admin/ 
> to add Kafka to the list of projects PreCommit-Admin triggers.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 29893: Patch for KAFKA-1856

[Ashish Singh]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29893/
---

(Updated Feb. 4, 2015, 11:44 p.m.)


Review request for kafka.


Bugs: KAFKA-1856
https://issues.apache.org/jira/browse/KAFKA-1856


Repository: kafka


Description
---

KAFKA-1856: Add PreCommit Patch Testing


Diffs (updated)
-

  dev-utils/test-patch.py PRE-CREATION 

Diff: https://reviews.apache.org/r/29893/diff/


Testing
---

Tested on KAFKA-1664, 
https://issues.apache.org/jira/browse/KAFKA-1664?focusedCommentId=14277439&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14277439

How to run:
python dev-utils/test-patch.py --defect KAFKA-1664 --username  
--password  --run-tests --post-results


Thanks,

Ashish Singh

[jira] [Commented] (KAFKA-1856) Add PreCommit Patch Testing

[Ashish Kumar Singh (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14306241#comment-14306241
 ] 

Ashish Kumar Singh commented on KAFKA-1856:
---

[~gwenshap] added checkstyle checks as part of pre-commit patch testing.

> Add PreCommit Patch Testing
> ---
>
> Key: KAFKA-1856
> URL: https://issues.apache.org/jira/browse/KAFKA-1856
> Project: Kafka
>  Issue Type: Task
>Reporter: Ashish Kumar Singh
>Assignee: Ashish Kumar Singh
> Attachments: KAFKA-1856.patch, KAFKA-1856_2015-01-18_21:43:56.patch, 
> KAFKA-1856_2015-02-04_14:57:05.patch, KAFKA-1856_2015-02-04_15:44:47.patch
>
>
> h1. Kafka PreCommit Patch Testing - *Don't wait for it to break*
> h2. Motivation
> *With great power comes great responsibility* - Uncle Ben. As Kafka user list 
> is growing, mechanism to ensure quality of the product is required. Quality 
> becomes hard to measure and maintain in an open source project, because of a 
> wide community of contributors. Luckily, Kafka is not the first open source 
> project and can benefit from learnings of prior projects.
> PreCommit tests are the tests that are run for each patch that gets attached 
> to an open JIRA. Based on tests results, test execution framework, test bot, 
> +1 or -1 the patch. Having PreCommit tests take the load off committers to 
> look at or test each patch.
> h2. Tests in Kafka
> h3. Unit and Integraiton Tests
> [Unit and Integration 
> tests|https://cwiki.apache.org/confluence/display/KAFKA/Kafka+0.9+Unit+and+Integration+Tests]
>  are cardinal to help contributors to avoid breaking existing functionalities 
> while adding new functionalities or fixing older ones. These tests, atleast 
> the ones relevant to the changes, must be run by contributors before 
> attaching a patch to a JIRA.
> h3. System Tests
> [System 
> tests|https://cwiki.apache.org/confluence/display/KAFKA/Kafka+System+Tests] 
> are much wider tests that, unlike unit tests, focus on end-to-end scenarios 
> and not some specific method or class.
> h2. Apache PreCommit tests
> Apache provides a mechanism to automatically build a project and run a series 
> of tests whenever a patch is uploaded to a JIRA. Based on test execution, the 
> test framework will comment with a +1 or -1 on the JIRA.
> You can read more about the framework here:
> http://wiki.apache.org/general/PreCommitBuilds
> h2. Plan
> # Create a test-patch.py script (similar to the one used in Flume, Sqoop and 
> other projects) that will take a jira as a parameter, apply on the 
> appropriate branch, build the project, run tests and report results. This 
> script should be committed into the Kafka code-base. To begin with, this will 
> only run unit tests. We can add code sanity checks, system_tests, etc in the 
> future.
> # Create a jenkins job for running the test (as described in 
> http://wiki.apache.org/general/PreCommitBuilds) and validate that it works 
> manually. This must be done by a committer with Jenkins access.
> # Ask someone with access to https://builds.apache.org/job/PreCommit-Admin/ 
> to add Kafka to the list of projects PreCommit-Admin triggers.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-1921) Delete offsets for a group with kafka offset storage

[Onur Karaman (JIRA)]

Onur Karaman created KAFKA-1921:
---

 Summary: Delete offsets for a group with kafka offset storage
 Key: KAFKA-1921
 URL: https://issues.apache.org/jira/browse/KAFKA-1921
 Project: Kafka
  Issue Type: Improvement
Reporter: Onur Karaman


There is currently no way to delete offsets for a consumer group when using 
kafka offset storage.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 29831: Patch for KAFKA-1476

[Onur Karaman]

> On Feb. 2, 2015, 5:42 p.m., Neha Narkhede wrote:
> > Onur, we should be able to check in after these review comments are 
> > addressed. Also, how would deleting offsets for a group work when the 
> > offset storage is Kafka? It's fine to not address it in this patch. Can you 
> > please create a JIRA to handle that?

Clark brought up some suggestions on the tool:
- describe currently just shows the log end offset. It might also be useful to 
show the log start offset.
- describe should ideally compare results (current offset, lag, and maybe 
ownership) using both kafka and zookeeper side-by-side.

If we were to follow through with these suggestions, should they be applied in 
a later patch?

As for kafka offset storage: I think that's out of the scope of this patch. I 
made a separate jira for that: https://issues.apache.org/jira/browse/KAFKA-1921


- Onur


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29831/#review70581
---


On Feb. 4, 2015, 11:41 p.m., Onur Karaman wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/29831/
> ---
> 
> (Updated Feb. 4, 2015, 11:41 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1476
> https://issues.apache.org/jira/browse/KAFKA-1476
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Merged in work for KAFKA-1476 and sub-task KAFKA-1826
> 
> 
> Diffs
> -
> 
>   bin/kafka-consumer-groups.sh PRE-CREATION 
>   core/src/main/scala/kafka/admin/AdminUtils.scala 
> 28b12c7b89a56c113b665fbde1b95f873f8624a3 
>   core/src/main/scala/kafka/admin/ConsumerGroupCommand.scala PRE-CREATION 
>   core/src/main/scala/kafka/utils/ZkUtils.scala 
> c14bd455b6642f5e6eb254670bef9f57ae41d6cb 
>   core/src/test/scala/unit/kafka/admin/DeleteConsumerGroupTest.scala 
> PRE-CREATION 
>   core/src/test/scala/unit/kafka/admin/DeleteTopicTest.scala 
> 33c27678bf8ae8feebcbcdaa4b90a1963157b4a5 
>   core/src/test/scala/unit/kafka/utils/TestUtils.scala 
> 54755e8dd3f23ced313067566cd4ea867f8a496e 
> 
> Diff: https://reviews.apache.org/r/29831/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Onur Karaman
> 
>

Re: Detecting lost connection in high level consumer

[harikiran]

Hey! Did anyone get a chance to read this?

Thanks!!
On Mon, Jan 19, 2015 at 1:26 AM harikiran  wrote:

> Hi
>
> I am using the 0811 Kafka High level consumer and I have configured "
> consumer.timeout.ms" to a value that is not -1, say 5000ms.
>
> I create the consumer iterator and invoke hasNext() method on it.
>
> Irrespective of whether kafka broker was shutdown or there was no message
> written to kafka, I see a ConsumerTimeOut exception after 5000ms.
>
> My goal is to detect lost connection and reconnect but I cannot figure out
> a way.
>
> Any kind of help is appreciated.
>
> Thanks
> Hari
>

Re: Review Request 29831: Patch for KAFKA-1476

[Onur Karaman]

> On Jan. 23, 2015, 4:47 p.m., Neha Narkhede wrote:
> > core/src/main/scala/kafka/admin/ConsumerGroupCommand.scala, line 114
> > 
> >
> > How is this tool going to behave if the consumer's offset information 
> > is stored in kafka, not zookeeper?
> > 
> > The assumption of the user would be to handle that case transparently 
> > as well.

Addressed this in a later comment. Adding the JIRA to this for completeness: 
https://issues.apache.org/jira/browse/KAFKA-1921


- Onur


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29831/#review69400
---


On Feb. 4, 2015, 11:41 p.m., Onur Karaman wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/29831/
> ---
> 
> (Updated Feb. 4, 2015, 11:41 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1476
> https://issues.apache.org/jira/browse/KAFKA-1476
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Merged in work for KAFKA-1476 and sub-task KAFKA-1826
> 
> 
> Diffs
> -
> 
>   bin/kafka-consumer-groups.sh PRE-CREATION 
>   core/src/main/scala/kafka/admin/AdminUtils.scala 
> 28b12c7b89a56c113b665fbde1b95f873f8624a3 
>   core/src/main/scala/kafka/admin/ConsumerGroupCommand.scala PRE-CREATION 
>   core/src/main/scala/kafka/utils/ZkUtils.scala 
> c14bd455b6642f5e6eb254670bef9f57ae41d6cb 
>   core/src/test/scala/unit/kafka/admin/DeleteConsumerGroupTest.scala 
> PRE-CREATION 
>   core/src/test/scala/unit/kafka/admin/DeleteTopicTest.scala 
> 33c27678bf8ae8feebcbcdaa4b90a1963157b4a5 
>   core/src/test/scala/unit/kafka/utils/TestUtils.scala 
> 54755e8dd3f23ced313067566cd4ea867f8a496e 
> 
> Diff: https://reviews.apache.org/r/29831/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Onur Karaman
> 
>

[jira] [Commented] (KAFKA-1760) Implement new consumer client

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14306324#comment-14306324
 ] 

Guozhang Wang commented on KAFKA-1760:
--

OK, the first one observed so far is that the consumer metadata responses does 
not actually help update the metadata due to some protocol bug:

In the below example, although the returned node id is 1, it is read as 
-2147483648 as always (and Node.toString did not print the id since it is < 0), 
and later responses with node id 2 will not change consumer's coordinator 
destination.

{code}
[2015-02-04 12:43:46,251] TRACE [KafkaApi-1] Sending consumer metadata 
ConsumerMetadataResponse(Some(id:1,host:localhost,port:46148),0,27) for 
correlation id 27 to client consumer-1. (kafka.server.KafkaApis:36)
[2015-02-04 12:43:46,252] TRACE Processor id 1 selection time = 4558827 ns 
(kafka.network.Processor:36)
[2015-02-04 12:43:46,252] TRACE Socket server received response to send, 
registering for write: 
Response(1,Request(1,sun.nio.ch.SelectionKeyImpl@4bd15081,null,1423082626245,/127.0.0.1:51454),kafka.network.BoundedByteBufferSend@40aaecb5,SendAction)
 (kafka.network.Processor:36)
[2015-02-04 12:43:46,252] TRACE Processor id 1 selection time = 18275 ns 
(kafka.network.Processor:36)
[2015-02-04 12:43:46,252] TRACE 29 bytes written to /127.0.0.1:51454 using 
key sun.nio.ch.SelectionKeyImpl@4bd15081 (kafka.network.Processor:36)
[2015-02-04 12:43:46,253] TRACE Completed request:Name: 
ConsumerMetadataRequest; Version: 0; CorrelationId: 27; ClientId: consumer-1; 
Group: my-test from client 
/127.0.0.1:51454;totalTime:7,requestQueueTime:2,localTime:5,remoteTime:0,responseQueueTime:0,sendTime:0
 (kafka.request.logger:85)
[2015-02-04 12:43:46,253] TRACE Finished writing, registering for read on 
connection /127.0.0.1:51454 (kafka.network.Processor:36)
[2015-02-04 12:43:46,253] DEBUG Found consumer co-ordinator: 
Node(localhost, 46148) (org.apache.kafka.clients.consumer.KafkaConsumer:1259)
[2015-02-04 12:43:46,253] DEBUG Initiating connection to node -2147483648 
at localhost:46148. (org.apache.kafka.clients.NetworkClient:484)

{code}

> Implement new consumer client
> -
>
> Key: KAFKA-1760
> URL: https://issues.apache.org/jira/browse/KAFKA-1760
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Jay Kreps
>Assignee: Jay Kreps
> Fix For: 0.8.3
>
> Attachments: KAFKA-1760.patch, KAFKA-1760_2015-01-11_16:57:15.patch, 
> KAFKA-1760_2015-01-18_19:10:13.patch, KAFKA-1760_2015-01-21_08:42:20.patch, 
> KAFKA-1760_2015-01-22_10:03:26.patch, KAFKA-1760_2015-01-22_20:21:56.patch, 
> KAFKA-1760_2015-01-23_13:13:00.patch, KAFKA-1760_2015-01-29_03:20:20.patch
>
>
> Implement a consumer client.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1760) Implement new consumer client

[Jay Kreps (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14306332#comment-14306332
 ] 

Jay Kreps commented on KAFKA-1760:
--

Actually shameful to say but I'm not sure this is a bug but rather a "feature".

We need to give the client it's own connection to the co-ordinator, but the 
existing client maintains a single connection per node id. My hack for this is 
to give the coordinator connection the node id -MAX_INT:
{code}
if (response.errorCode() == Errors.NONE.code())
return new Node(Integer.MIN_VALUE, response.node().host(), 
response.node().port());
{code}
I would agree this is somewhat egregious, but I really am not sure about how we 
would change the client to model connections and nodes differently for a 
non-hack to work and still be usable.

> Implement new consumer client
> -
>
> Key: KAFKA-1760
> URL: https://issues.apache.org/jira/browse/KAFKA-1760
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Jay Kreps
>Assignee: Jay Kreps
> Fix For: 0.8.3
>
> Attachments: KAFKA-1760.patch, KAFKA-1760_2015-01-11_16:57:15.patch, 
> KAFKA-1760_2015-01-18_19:10:13.patch, KAFKA-1760_2015-01-21_08:42:20.patch, 
> KAFKA-1760_2015-01-22_10:03:26.patch, KAFKA-1760_2015-01-22_20:21:56.patch, 
> KAFKA-1760_2015-01-23_13:13:00.patch, KAFKA-1760_2015-01-29_03:20:20.patch
>
>
> Implement a consumer client.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Detecting lost connection in high level consumer

[harikiran]

Thanks Guozhang and Jun for your replies.

On Wed, Feb 4, 2015 at 4:06 PM, harikiran  wrote:

> Hey! Did anyone get a chance to read this?
>
> Thanks!!
>
> On Mon, Jan 19, 2015 at 1:26 AM harikiran  wrote:
>
>> Hi
>>
>> I am using the 0811 Kafka High level consumer and I have configured "
>> consumer.timeout.ms" to a value that is not -1, say 5000ms.
>>
>> I create the consumer iterator and invoke hasNext() method on it.
>>
>> Irrespective of whether kafka broker was shutdown or there was no message
>> written to kafka, I see a ConsumerTimeOut exception after 5000ms.
>>
>> My goal is to detect lost connection and reconnect but I cannot figure
>> out a way.
>>
>> Any kind of help is appreciated.
>>
>> Thanks
>> Hari
>>
>

[jira] [Commented] (KAFKA-1760) Implement new consumer client

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14306358#comment-14306358
 ] 

Guozhang Wang commented on KAFKA-1760:
--

[~jkreps], I think this is a bug, because later on when it tries to send any 
coordinator requests it will use just the node id, not the host/port:

{code}
RequestSend send = new RequestSend(this.consumerCoordinator.id(), header, 
request);
{code}

and like I mentioned before, when coordinator migrates, the updated coordinator 
info ended up not getting to consumer's network client, and hence it still 
tries to talk to the old guy.

> Implement new consumer client
> -
>
> Key: KAFKA-1760
> URL: https://issues.apache.org/jira/browse/KAFKA-1760
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Jay Kreps
>Assignee: Jay Kreps
> Fix For: 0.8.3
>
> Attachments: KAFKA-1760.patch, KAFKA-1760_2015-01-11_16:57:15.patch, 
> KAFKA-1760_2015-01-18_19:10:13.patch, KAFKA-1760_2015-01-21_08:42:20.patch, 
> KAFKA-1760_2015-01-22_10:03:26.patch, KAFKA-1760_2015-01-22_20:21:56.patch, 
> KAFKA-1760_2015-01-23_13:13:00.patch, KAFKA-1760_2015-01-29_03:20:20.patch
>
>
> Implement a consumer client.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1919) Metadata request issued with no backoff in new producer if there are no topics

[Jay Kreps (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14306357#comment-14306357
 ] 

Jay Kreps commented on KAFKA-1919:
--

Here is what I think is happening: in 0.8.1 the metadata request just gives 
back nodes for the requested topics. But in this case there are no requested 
topics so there are no nodes. When we get a metadata response with no nodes we 
actually ignore it, because if we update our view of the cluster with one that 
has no machines then we have nowhere to even make metadata requests. 
Unfortunately by not updating the cluster we are bypassing the backoff the 
prevents another request from being issued.

I suspect this doesn't happen against a 0.8.2 server because in 0.8.2 we are 
giving back all the nodes no matter what in the metadata response specifically 
to avoid every running into a situation where there is no one to issue metadata 
requests against. I think the fix is to "update" the metadata with the existing 
metadata not the new (empty) metadata so as to guarantee that the backoff will 
always come into effect.

[~stevenz3wu] I have a patch that I think will fix this if I understand the 
root cause. Want to give it a shot?

> Metadata request issued with no backoff in new producer if there are no topics
> --
>
> Key: KAFKA-1919
> URL: https://issues.apache.org/jira/browse/KAFKA-1919
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2
>Reporter: Jay Kreps
>
> Original report:
> We have observed high cpu and high network traffic problem when
> 1) cluster (0.8.1.1) has no topic
> 2) KafkaProducer (0.8.2-beta) object is created without sending any traffic
> We have observed such problem twice. In both cases, problem went away
> immediately after one/any topic is created.
> Is this a known issue? Just want to check with the community first before I
> spend much time to reproduce it.
> I couldn't reproduce the issue with similar setup with unit test code in
> IDE. start two brokers with no topic locally on my laptop. create a
> KafkaProducer object without sending any msgs. but I only tested with
> 0.8.2-beta for both broker and producer.
> Issue exists in 0.8.2 as well:
> I have re-run my unit test with 0.8.2.0. same tight-loop problem happened
> after a few mins.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1919) Metadata request issued with no backoff in new producer if there are no topics

[Jay Kreps (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jay Kreps updated KAFKA-1919:
-
Attachment: KAFKA-1919-v1.patch

> Metadata request issued with no backoff in new producer if there are no topics
> --
>
> Key: KAFKA-1919
> URL: https://issues.apache.org/jira/browse/KAFKA-1919
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2
>Reporter: Jay Kreps
> Attachments: KAFKA-1919-v1.patch
>
>
> Original report:
> We have observed high cpu and high network traffic problem when
> 1) cluster (0.8.1.1) has no topic
> 2) KafkaProducer (0.8.2-beta) object is created without sending any traffic
> We have observed such problem twice. In both cases, problem went away
> immediately after one/any topic is created.
> Is this a known issue? Just want to check with the community first before I
> spend much time to reproduce it.
> I couldn't reproduce the issue with similar setup with unit test code in
> IDE. start two brokers with no topic locally on my laptop. create a
> KafkaProducer object without sending any msgs. but I only tested with
> 0.8.2-beta for both broker and producer.
> Issue exists in 0.8.2 as well:
> I have re-run my unit test with 0.8.2.0. same tight-loop problem happened
> after a few mins.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1476) Get a list of consumer groups

[Onur Karaman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Onur Karaman updated KAFKA-1476:

Attachment: KAFKA-1476_2015-02-04_18:03:15.patch

> Get a list of consumer groups
> -
>
> Key: KAFKA-1476
> URL: https://issues.apache.org/jira/browse/KAFKA-1476
> Project: Kafka
>  Issue Type: Wish
>  Components: tools
>Affects Versions: 0.8.1.1
>Reporter: Ryan Williams
>Assignee: Balaji Seshadri
>  Labels: newbie
> Fix For: 0.9.0
>
> Attachments: ConsumerCommand.scala, KAFKA-1476-LIST-GROUPS.patch, 
> KAFKA-1476-RENAME.patch, KAFKA-1476-REVIEW-COMMENTS.patch, KAFKA-1476.patch, 
> KAFKA-1476.patch, KAFKA-1476.patch, KAFKA-1476.patch, 
> KAFKA-1476_2014-11-10_11:58:26.patch, KAFKA-1476_2014-11-10_12:04:01.patch, 
> KAFKA-1476_2014-11-10_12:06:35.patch, KAFKA-1476_2014-12-05_12:00:12.patch, 
> KAFKA-1476_2015-01-12_16:22:26.patch, KAFKA-1476_2015-01-12_16:31:20.patch, 
> KAFKA-1476_2015-01-13_10:36:18.patch, KAFKA-1476_2015-01-15_14:30:04.patch, 
> KAFKA-1476_2015-01-22_02:32:52.patch, KAFKA-1476_2015-01-30_11:09:59.patch, 
> KAFKA-1476_2015-02-04_15:41:50.patch, KAFKA-1476_2015-02-04_18:03:15.patch, 
> sample-kafka-consumer-groups-sh-output-1-23-2015.txt, 
> sample-kafka-consumer-groups-sh-output.txt
>
>
> It would be useful to have a way to get a list of consumer groups currently 
> active via some tool/script that ships with kafka. This would be helpful so 
> that the system tools can be explored more easily.
> For example, when running the ConsumerOffsetChecker, it requires a group 
> option
> bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --topic test --group 
> ?
> But, when just getting started with kafka, using the console producer and 
> consumer, it is not clear what value to use for the group option.  If a list 
> of consumer groups could be listed, then it would be clear what value to use.
> Background:
> http://mail-archives.apache.org/mod_mbox/kafka-users/201405.mbox/%3cCAOq_b1w=slze5jrnakxvak0gu9ctdkpazak1g4dygvqzbsg...@mail.gmail.com%3e



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1476) Get a list of consumer groups

[Onur Karaman (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14306485#comment-14306485
 ] 

Onur Karaman commented on KAFKA-1476:
-

Updated reviewboard https://reviews.apache.org/r/29831/diff/
 against branch origin/trunk

> Get a list of consumer groups
> -
>
> Key: KAFKA-1476
> URL: https://issues.apache.org/jira/browse/KAFKA-1476
> Project: Kafka
>  Issue Type: Wish
>  Components: tools
>Affects Versions: 0.8.1.1
>Reporter: Ryan Williams
>Assignee: Balaji Seshadri
>  Labels: newbie
> Fix For: 0.9.0
>
> Attachments: ConsumerCommand.scala, KAFKA-1476-LIST-GROUPS.patch, 
> KAFKA-1476-RENAME.patch, KAFKA-1476-REVIEW-COMMENTS.patch, KAFKA-1476.patch, 
> KAFKA-1476.patch, KAFKA-1476.patch, KAFKA-1476.patch, 
> KAFKA-1476_2014-11-10_11:58:26.patch, KAFKA-1476_2014-11-10_12:04:01.patch, 
> KAFKA-1476_2014-11-10_12:06:35.patch, KAFKA-1476_2014-12-05_12:00:12.patch, 
> KAFKA-1476_2015-01-12_16:22:26.patch, KAFKA-1476_2015-01-12_16:31:20.patch, 
> KAFKA-1476_2015-01-13_10:36:18.patch, KAFKA-1476_2015-01-15_14:30:04.patch, 
> KAFKA-1476_2015-01-22_02:32:52.patch, KAFKA-1476_2015-01-30_11:09:59.patch, 
> KAFKA-1476_2015-02-04_15:41:50.patch, KAFKA-1476_2015-02-04_18:03:15.patch, 
> sample-kafka-consumer-groups-sh-output-1-23-2015.txt, 
> sample-kafka-consumer-groups-sh-output.txt
>
>
> It would be useful to have a way to get a list of consumer groups currently 
> active via some tool/script that ships with kafka. This would be helpful so 
> that the system tools can be explored more easily.
> For example, when running the ConsumerOffsetChecker, it requires a group 
> option
> bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --topic test --group 
> ?
> But, when just getting started with kafka, using the console producer and 
> consumer, it is not clear what value to use for the group option.  If a list 
> of consumer groups could be listed, then it would be clear what value to use.
> Background:
> http://mail-archives.apache.org/mod_mbox/kafka-users/201405.mbox/%3cCAOq_b1w=slze5jrnakxvak0gu9ctdkpazak1g4dygvqzbsg...@mail.gmail.com%3e



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 29831: Patch for KAFKA-1476

[Onur Karaman]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29831/
---

(Updated Feb. 5, 2015, 2:03 a.m.)


Review request for kafka.


Bugs: KAFKA-1476
https://issues.apache.org/jira/browse/KAFKA-1476


Repository: kafka


Description
---

Merged in work for KAFKA-1476 and sub-task KAFKA-1826


Diffs (updated)
-

  bin/kafka-consumer-groups.sh PRE-CREATION 
  core/src/main/scala/kafka/admin/AdminUtils.scala 
28b12c7b89a56c113b665fbde1b95f873f8624a3 
  core/src/main/scala/kafka/admin/ConsumerGroupCommand.scala PRE-CREATION 
  core/src/main/scala/kafka/utils/ZkUtils.scala 
c14bd455b6642f5e6eb254670bef9f57ae41d6cb 
  core/src/test/scala/unit/kafka/admin/DeleteConsumerGroupTest.scala 
PRE-CREATION 
  core/src/test/scala/unit/kafka/admin/DeleteTopicTest.scala 
33c27678bf8ae8feebcbcdaa4b90a1963157b4a5 
  core/src/test/scala/unit/kafka/utils/TestUtils.scala 
54755e8dd3f23ced313067566cd4ea867f8a496e 

Diff: https://reviews.apache.org/r/29831/diff/


Testing
---


Thanks,

Onur Karaman

Re: Review Request 30063: Patch for KAFKA-1840

[Guozhang Wang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30063/#review71132
---



core/src/main/scala/kafka/tools/MirrorMaker.scala


How about use the "embeded handler config file" as we did for producer and 
consumer, this gives us more flexibility as well as better operatibility.



core/src/main/scala/kafka/tools/MirrorMaker.scala


Is there an expected use case for one-to-many handlers?


- Guozhang Wang


On Feb. 1, 2015, 8:17 a.m., Jiangjie Qin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30063/
> ---
> 
> (Updated Feb. 1, 2015, 8:17 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1840
> https://issues.apache.org/jira/browse/KAFKA-1840
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Merge branch 'trunk' of http://git-wip-us.apache.org/repos/asf/kafka into 
> KAFKA-1840
> 
> 
> Addressed Joel's comments
> 
> 
> Allow message handler to specify partitions for produce
> 
> 
> Addressed Joel's comments
> 
> 
> Addressed previous reviews.
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/tools/MirrorMaker.scala 
> 81ae205ef7b2050d0152f29f8da7dd91b17b8b00 
> 
> Diff: https://reviews.apache.org/r/30063/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jiangjie Qin
> 
>

Re: Review Request 30083: Patch for KAFKA-1839

[Guozhang Wang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30083/#review71133
---

Ship it!



core/src/main/scala/kafka/consumer/PartitionAssignor.scala


Copy the comment as in RangeAssignor here.


- Guozhang Wang


On Jan. 20, 2015, 9:28 p.m., Jiangjie Qin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30083/
> ---
> 
> (Updated Jan. 20, 2015, 9:28 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1839
> https://issues.apache.org/jira/browse/KAFKA-1839
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Modified unit test.
> 
> 
> Minor code change
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/consumer/PartitionAssignor.scala 
> e6ff7683a0df4a7d221e949767e57c34703d5aad 
>   core/src/main/scala/kafka/consumer/ZookeeperConsumerConnector.scala 
> 191a8677444e53b043e9ad6e94c5a9191c32599e 
>   core/src/main/scala/kafka/javaapi/consumer/ConsumerRebalanceListener.java 
> facf509841918bdcf92a271d29f6c825ff99345a 
>   core/src/main/scala/kafka/tools/MirrorMaker.scala 
> 5cbc8103e33a0a234d158c048e5314e841da6249 
>   core/src/test/scala/unit/kafka/consumer/PartitionAssignorTest.scala 
> 24954de66ccc5158696166b7e2aabad0f1b1f287 
>   
> core/src/test/scala/unit/kafka/consumer/ZookeeperConsumerConnectorTest.scala 
> a17e8532c44aadf84b8da3a57bcc797a848b5020 
> 
> Diff: https://reviews.apache.org/r/30083/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jiangjie Qin
> 
>

[jira] [Created] (KAFKA-1922) Move doubly linked list test out of UtilsTest

[Joel Koshy (JIRA)]

Joel Koshy created KAFKA-1922:
-

 Summary: Move doubly linked list test out of UtilsTest
 Key: KAFKA-1922
 URL: https://issues.apache.org/jira/browse/KAFKA-1922
 Project: Kafka
  Issue Type: Bug
Reporter: Joel Koshy
Assignee: Jiangjie Qin


Per mailing list discussion, this should ideally be stand-alone outside of 
UtilsTest



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 30063: Patch for KAFKA-1840

[Jiangjie Qin]

> On Feb. 5, 2015, 2:15 a.m., Guozhang Wang wrote:
> > core/src/main/scala/kafka/tools/MirrorMaker.scala, lines 687-689
> > 
> >
> > Is there an expected use case for one-to-many handlers?

Yes, for example some user might want to accumulate a trasaction until it 
either finishes or aborts.


> On Feb. 5, 2015, 2:15 a.m., Guozhang Wang wrote:
> > core/src/main/scala/kafka/tools/MirrorMaker.scala, lines 182-186
> > 
> >
> > How about use the "embeded handler config file" as we did for producer 
> > and consumer, this gives us more flexibility as well as better 
> > operatibility.

I also felt a little bit less strict for this handler argument and thought over 
this. For producer/consumer config are all key-value pairs, but for handler it 
might not be the case. In current way, we can let user to decide whether they 
want to use a config file (they can always put a file name as commandline 
argument) or just use some simple commandline config parameters without 
creating a config file. So we let user interpret what the command line 
argument. What do you think?


- Jiangjie


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30063/#review71132
---


On Feb. 1, 2015, 8:17 a.m., Jiangjie Qin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30063/
> ---
> 
> (Updated Feb. 1, 2015, 8:17 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1840
> https://issues.apache.org/jira/browse/KAFKA-1840
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Merge branch 'trunk' of http://git-wip-us.apache.org/repos/asf/kafka into 
> KAFKA-1840
> 
> 
> Addressed Joel's comments
> 
> 
> Allow message handler to specify partitions for produce
> 
> 
> Addressed Joel's comments
> 
> 
> Addressed previous reviews.
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/tools/MirrorMaker.scala 
> 81ae205ef7b2050d0152f29f8da7dd91b17b8b00 
> 
> Diff: https://reviews.apache.org/r/30063/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jiangjie Qin
> 
>

[jira] [Commented] (KAFKA-1919) Metadata request issued with no backoff in new producer if there are no topics

[Steven Zhen Wu (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14306617#comment-14306617
 ] 

Steven Zhen Wu commented on KAFKA-1919:
---

this does happen with 0.8.2.0 server. In my local unit test, both producer and 
server are 0.8.2.0.

[~jkreps] will try the patch later this week. 

> Metadata request issued with no backoff in new producer if there are no topics
> --
>
> Key: KAFKA-1919
> URL: https://issues.apache.org/jira/browse/KAFKA-1919
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2
>Reporter: Jay Kreps
> Attachments: KAFKA-1919-v1.patch
>
>
> Original report:
> We have observed high cpu and high network traffic problem when
> 1) cluster (0.8.1.1) has no topic
> 2) KafkaProducer (0.8.2-beta) object is created without sending any traffic
> We have observed such problem twice. In both cases, problem went away
> immediately after one/any topic is created.
> Is this a known issue? Just want to check with the community first before I
> spend much time to reproduce it.
> I couldn't reproduce the issue with similar setup with unit test code in
> IDE. start two brokers with no topic locally on my laptop. create a
> KafkaProducer object without sending any msgs. but I only tested with
> 0.8.2-beta for both broker and producer.
> Issue exists in 0.8.2 as well:
> I have re-run my unit test with 0.8.2.0. same tight-loop problem happened
> after a few mins.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1922) Move doubly linked list test out of UtilsTest

[Jiangjie Qin (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14306665#comment-14306665
 ] 

Jiangjie Qin commented on KAFKA-1922:
-

Created reviewboard https://reviews.apache.org/r/30651/diff/
 against branch origin/trunk

> Move doubly linked list test out of UtilsTest
> -
>
> Key: KAFKA-1922
> URL: https://issues.apache.org/jira/browse/KAFKA-1922
> Project: Kafka
>  Issue Type: Bug
>Reporter: Joel Koshy
>Assignee: Jiangjie Qin
> Attachments: KAFKA-1922.patch
>
>
> Per mailing list discussion, this should ideally be stand-alone outside of 
> UtilsTest



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1922) Move doubly linked list test out of UtilsTest

[Jiangjie Qin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jiangjie Qin updated KAFKA-1922:

Status: Patch Available  (was: Open)

> Move doubly linked list test out of UtilsTest
> -
>
> Key: KAFKA-1922
> URL: https://issues.apache.org/jira/browse/KAFKA-1922
> Project: Kafka
>  Issue Type: Bug
>Reporter: Joel Koshy
>Assignee: Jiangjie Qin
> Attachments: KAFKA-1922.patch
>
>
> Per mailing list discussion, this should ideally be stand-alone outside of 
> UtilsTest



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 30651: Patch for KAFKA-1922

[Jiangjie Qin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30651/
---

Review request for kafka.


Bugs: KAFKA-1922
https://issues.apache.org/jira/browse/KAFKA-1922


Repository: kafka


Description
---

patch for KAFKA-1922. Move test for DoublyLinkedList out of UtilTests.scala


Diffs
-

  core/src/test/scala/unit/kafka/utils/DoublyLinkedListTest.scala PRE-CREATION 
  core/src/test/scala/unit/kafka/utils/UtilsTest.scala 
8c3797a964a2760a00ed60530d1a48e3da473769 

Diff: https://reviews.apache.org/r/30651/diff/


Testing
---


Thanks,

Jiangjie Qin

[jira] [Updated] (KAFKA-1922) Move doubly linked list test out of UtilsTest

[Jiangjie Qin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jiangjie Qin updated KAFKA-1922:

Attachment: KAFKA-1922.patch

> Move doubly linked list test out of UtilsTest
> -
>
> Key: KAFKA-1922
> URL: https://issues.apache.org/jira/browse/KAFKA-1922
> Project: Kafka
>  Issue Type: Bug
>Reporter: Joel Koshy
>Assignee: Jiangjie Qin
> Attachments: KAFKA-1922.patch
>
>
> Per mailing list discussion, this should ideally be stand-alone outside of 
> UtilsTest



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1730) add the doc for the new java producer in 0.8.2

[Manikumar Reddy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar Reddy updated KAFKA-1730:
---
Attachment: new-producer-metrics.patch

Attaching a new producer metrics/mbean doc.

> add the doc for the new java producer in 0.8.2
> --
>
> Key: KAFKA-1730
> URL: https://issues.apache.org/jira/browse/KAFKA-1730
> Project: Kafka
>  Issue Type: Sub-task
>Affects Versions: 0.8.2
>Reporter: Jun Rao
>Priority: Blocker
> Fix For: 0.8.2
>
> Attachments: new-producer-metrics.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1451) Broker stuck due to leader election race

[justdebugit (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14306753#comment-14306753
 ] 

justdebugit commented on KAFKA-1451:


I think the patch dose not RESOLVE the problem,it will be happen again when zk 
event notification request arrive long after the /controller   has  deleted.
kafkaController.onControllerResignation method has executed,but elect action is 
return when it find that "/controller" znode has already been created

> Broker stuck due to leader election race 
> -
>
> Key: KAFKA-1451
> URL: https://issues.apache.org/jira/browse/KAFKA-1451
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.1.1
>Reporter: Maciek Makowski
>Assignee: Manikumar Reddy
>Priority: Minor
>  Labels: newbie
> Fix For: 0.8.2
>
> Attachments: KAFKA-1451.patch, KAFKA-1451_2014-07-28_20:27:32.patch, 
> KAFKA-1451_2014-07-29_10:13:23.patch
>
>
> h3. Symptoms
> The broker does not become available due to being stuck in an infinite loop 
> while electing leader. This can be recognised by the following line being 
> repeatedly written to server.log:
> {code}
> [2014-05-14 04:35:09,187] INFO I wrote this conflicted ephemeral node 
> [{"version":1,"brokerid":1,"timestamp":"1400060079108"}] at /controller a 
> while back in a different session, hence I will backoff for this node to be 
> deleted by Zookeeper and retry (kafka.utils.ZkUtils$)
> {code}
> h3. Steps to Reproduce
> In a single kafka 0.8.1.1 node, single zookeeper 3.4.6 (but will likely 
> behave the same with the ZK version included in Kafka distribution) node 
> setup:
> # start both zookeeper and kafka (in any order)
> # stop zookeeper
> # stop kafka
> # start kafka
> # start zookeeper
> h3. Likely Cause
> {{ZookeeperLeaderElector}} subscribes to data changes on startup, and then 
> triggers an election. if the deletion of ephemeral {{/controller}} node 
> associated with previous zookeeper session of the broker happens after 
> subscription to changes in new session, election will be invoked twice, once 
> from {{startup}} and once from {{handleDataDeleted}}:
> * {{startup}}: acquire {{controllerLock}}
> * {{startup}}: subscribe to data changes
> * zookeeper: delete {{/controller}} since the session that created it timed 
> out
> * {{handleDataDeleted}}: {{/controller}} was deleted
> * {{handleDataDeleted}}: wait on {{controllerLock}}
> * {{startup}}: elect -- writes {{/controller}}
> * {{startup}}: release {{controllerLock}}
> * {{handleDataDeleted}}: acquire {{controllerLock}}
> * {{handleDataDeleted}}: elect -- attempts to write {{/controller}} and then 
> gets into infinite loop as a result of conflict
> {{createEphemeralPathExpectConflictHandleZKBug}} assumes that the existing 
> znode was written from different session, which is not true in this case; it 
> was written from the same session. That adds to the confusion.
> h3. Suggested Fix
> In {{ZookeeperLeaderElector.startup}} first run {{elect}} and then subscribe 
> to data changes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)