[jira] [Commented] (KAFKA-1789) Issue with Async producer
[ https://issues.apache.org/jira/browse/KAFKA-1789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14228255#comment-14228255 ] devendra tagare commented on KAFKA-1789: Hi, The crux of the issue we are facing is that the enqueues are taking longer than 20ms. The call that is taking more than 20ms is the producer.send() call. ~Dev > Issue with Async producer > - > > Key: KAFKA-1789 > URL: https://issues.apache.org/jira/browse/KAFKA-1789 > Project: Kafka > Issue Type: Bug >Affects Versions: 0.8.0, 0.8.1 >Reporter: devendra tagare >Priority: Critical > > Hi, > We are using an async producer to send data to a kafka cluster.The event rate > at peak is around 250 events/second of size 25KB each. > In the producer code base we have added specific debug statements to capture > the time taken to create a producer,create a keyed message with a byte > payload & send the message. > We have added the below properties to the producerConfig > queue.enqueue.timeout.ms=20 > send.buffer.bytes=1024000 > topic.metadata.refresh.interval.ms=3 > Based on the documentation, producer.send() queues the message on the async > producer's queue. > So, ideally if the queue is full then the enqueue operation should result in > an kafka.common.QueueFullException in 20 ms. > The logs indicate that the enqueue operation is taking more than 20ms (takes > around 250ms) without throwing any exceptions. > Is there any other property that could conflict with queue.enqueue.timeout.ms > which is causing this behavior ? > Or is it possible that the queue is not full & yet the producer.send() call > is still taking around 200ms under peak load ? > Also, could you suggest any other alternatives so that we can either enforce > a timeout or throw an exception in-case the async producer is taking more > than a specified amount of time. > Regards, > Dev -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1781) Readme should specify that Gradle 2.0 is required for initial bootstrap
[ https://issues.apache.org/jira/browse/KAFKA-1781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14228633#comment-14228633 ] Joe Stein commented on KAFKA-1781: -- +1 to double commit KAFKA-1624 and add the gradle version requirement for JDK 8 in README > Readme should specify that Gradle 2.0 is required for initial bootstrap > --- > > Key: KAFKA-1781 > URL: https://issues.apache.org/jira/browse/KAFKA-1781 > Project: Kafka > Issue Type: Bug > Components: build >Affects Versions: 0.8.2 >Reporter: Jean-Francois Im >Priority: Blocker > Fix For: 0.8.2 > > Attachments: gradle-2.0-readme.patch > > > Current README.md says "You need to have gradle installed." > As the bootstrap procedure doesn't work with gradle 1.12, this needs to say > that 2.0 or greater is needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KAFKA-1802) Add a new type of request for the discovery of the controller
Joe Stein created KAFKA-1802: Summary: Add a new type of request for the discovery of the controller Key: KAFKA-1802 URL: https://issues.apache.org/jira/browse/KAFKA-1802 Project: Kafka Issue Type: Sub-task Reporter: Joe Stein Fix For: 0.8.3 The goal here is like meta data discovery is for producer so CLI can find which broker it should send the rest of its admin requests too. Any broker can respond to this specific AdminMeta RQ/RP but only the controller broker should be responding to Admin message otherwise that broker should respond to any admin message with the response for what the controller is. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 27818: Patch for KAFKA-328
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27818/ --- (Updated Nov. 29, 2014, 7:07 a.m.) Review request for kafka. Bugs: KAFKA-328 https://issues.apache.org/jira/browse/KAFKA-328 Repository: kafka Description --- KAFKA-328 Write unit test for kafka server startup and shutdown API - Review Comments Diffs (updated) - core/src/test/scala/unit/kafka/server/ServerShutdownTest.scala 1bfb501b2f29c50f3fc5f930fdaad02e03b91e4f core/src/test/scala/unit/kafka/server/ServerStartupTest.scala a0ed4855f2550a0eb2e363dd2fccd8377a9ac172 Diff: https://reviews.apache.org/r/27818/diff/ Testing --- Thanks, Balaji Seshadri
[jira] [Commented] (KAFKA-328) Write unit test for kafka server startup and shutdown API
[ https://issues.apache.org/jira/browse/KAFKA-328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14228655#comment-14228655 ] BalajiSeshadri commented on KAFKA-328: -- Updated reviewboard https://reviews.apache.org/r/27818/diff/ against branch origin/trunk > Write unit test for kafka server startup and shutdown API > -- > > Key: KAFKA-328 > URL: https://issues.apache.org/jira/browse/KAFKA-328 > Project: Kafka > Issue Type: Bug >Reporter: Neha Narkhede >Assignee: BalajiSeshadri > Labels: newbie > Attachments: KAFKA-328-FORMATTED.patch, > KAFKA-328-REVIEW-COMMENTS.patch, KAFKA-328.patch, KAFKA-328.patch, > KAFKA-328_2014-11-10_11:05:58.patch, KAFKA-328_2014-11-29_00:08:05.patch > > > Background discussion in KAFKA-320 > People often try to embed KafkaServer in an application that ends up calling > startup() and shutdown() repeatedly and sometimes in odd ways. To ensure this > works correctly we have to be very careful about cleaning up resources. This > is a good practice for making unit tests reliable anyway. > A good first step would be to add some unit tests on startup and shutdown to > cover various cases: > 1. A Kafka server can startup if it is not already starting up, if it is not > currently being shutdown, or if it hasn't been already started > 2. A Kafka server can shutdown if it is not already shutting down, if it is > not currently starting up, or if it hasn't been already shutdown. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-328) Write unit test for kafka server startup and shutdown API
[ https://issues.apache.org/jira/browse/KAFKA-328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BalajiSeshadri updated KAFKA-328: - Attachment: KAFKA-328_2014-11-29_00:08:05.patch > Write unit test for kafka server startup and shutdown API > -- > > Key: KAFKA-328 > URL: https://issues.apache.org/jira/browse/KAFKA-328 > Project: Kafka > Issue Type: Bug >Reporter: Neha Narkhede >Assignee: BalajiSeshadri > Labels: newbie > Attachments: KAFKA-328-FORMATTED.patch, > KAFKA-328-REVIEW-COMMENTS.patch, KAFKA-328.patch, KAFKA-328.patch, > KAFKA-328_2014-11-10_11:05:58.patch, KAFKA-328_2014-11-29_00:08:05.patch > > > Background discussion in KAFKA-320 > People often try to embed KafkaServer in an application that ends up calling > startup() and shutdown() repeatedly and sometimes in odd ways. To ensure this > works correctly we have to be very careful about cleaning up resources. This > is a good practice for making unit tests reliable anyway. > A good first step would be to add some unit tests on startup and shutdown to > cover various cases: > 1. A Kafka server can startup if it is not already starting up, if it is not > currently being shutdown, or if it hasn't been already started > 2. A Kafka server can shutdown if it is not already shutting down, if it is > not currently starting up, or if it hasn't been already shutdown. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-328) Write unit test for kafka server startup and shutdown API
[ https://issues.apache.org/jira/browse/KAFKA-328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14228657#comment-14228657 ] BalajiSeshadri commented on KAFKA-328: -- [~nehanarkhede] Please find updated review board. > Write unit test for kafka server startup and shutdown API > -- > > Key: KAFKA-328 > URL: https://issues.apache.org/jira/browse/KAFKA-328 > Project: Kafka > Issue Type: Bug >Reporter: Neha Narkhede >Assignee: BalajiSeshadri > Labels: newbie > Attachments: KAFKA-328-FORMATTED.patch, KAFKA-328-REVIEW-11-29.patch, > KAFKA-328-REVIEW-COMMENTS.patch, KAFKA-328.patch, KAFKA-328.patch, > KAFKA-328_2014-11-10_11:05:58.patch, KAFKA-328_2014-11-29_00:08:05.patch > > > Background discussion in KAFKA-320 > People often try to embed KafkaServer in an application that ends up calling > startup() and shutdown() repeatedly and sometimes in odd ways. To ensure this > works correctly we have to be very careful about cleaning up resources. This > is a good practice for making unit tests reliable anyway. > A good first step would be to add some unit tests on startup and shutdown to > cover various cases: > 1. A Kafka server can startup if it is not already starting up, if it is not > currently being shutdown, or if it hasn't been already started > 2. A Kafka server can shutdown if it is not already shutting down, if it is > not currently starting up, or if it hasn't been already shutdown. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-328) Write unit test for kafka server startup and shutdown API
[ https://issues.apache.org/jira/browse/KAFKA-328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BalajiSeshadri updated KAFKA-328: - Attachment: KAFKA-328-REVIEW-11-29.patch > Write unit test for kafka server startup and shutdown API > -- > > Key: KAFKA-328 > URL: https://issues.apache.org/jira/browse/KAFKA-328 > Project: Kafka > Issue Type: Bug >Reporter: Neha Narkhede >Assignee: BalajiSeshadri > Labels: newbie > Attachments: KAFKA-328-FORMATTED.patch, KAFKA-328-REVIEW-11-29.patch, > KAFKA-328-REVIEW-COMMENTS.patch, KAFKA-328.patch, KAFKA-328.patch, > KAFKA-328_2014-11-10_11:05:58.patch, KAFKA-328_2014-11-29_00:08:05.patch > > > Background discussion in KAFKA-320 > People often try to embed KafkaServer in an application that ends up calling > startup() and shutdown() repeatedly and sometimes in odd ways. To ensure this > works correctly we have to be very careful about cleaning up resources. This > is a good practice for making unit tests reliable anyway. > A good first step would be to add some unit tests on startup and shutdown to > cover various cases: > 1. A Kafka server can startup if it is not already starting up, if it is not > currently being shutdown, or if it hasn't been already started > 2. A Kafka server can shutdown if it is not already shutting down, if it is > not currently starting up, or if it hasn't been already shutdown. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
