[jira] [Created] (KAFKA-843) Re-add the release-zip sbt target
Cosmin Lehene created KAFKA-843: --- Summary: Re-add the release-zip sbt target Key: KAFKA-843 URL: https://issues.apache.org/jira/browse/KAFKA-843 Project: Kafka Issue Type: Sub-task Components: packaging Reporter: Cosmin Lehene Fix For: 0.8 This target has been removed a while ago. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: How to create a deployable release?
Thanks,
I added https://issues.apache.org/jira/browse/KAFKA-843
Cosmin
On 4/1/13 8:57 PM, "Neha Narkhede" wrote:
>Yes, feel free to add a subtask to the release JIRA.
>
>Thanks,
>Neha
>
>On Mon, Apr 1, 2013 at 10:46 AM, Jun Rao wrote:
>> Probably we should add this as a subtask for the 0.8 beta release?
>>
>> Thanks,
>>
>> Jun
>>
>> On Mon, Apr 1, 2013 at 8:46 AM, Neha Narkhede
>>wrote:
>>
>>> Cosmin,
>>>
>>> The release-zip target worked for a short while but has been broken
>>> for a long time now. Its purpose was to create a deployable release
>>> tarball that has all the dependencies
>>> , configs and scripts. Patches are welcome!
>>>
>>> Thanks,
>>> Neha
>>>
>>> On Mon, Apr 1, 2013 at 8:14 AM, Cosmin Lehene
>>>wrote:
>>> > Hi,
>>> >
>>> > Reading the documentation it seems there should be a release-zip task
>>> ("release-zip : Create all the jars, run unit tests and create a
>>>deployable
>>> release zip")
>>> > I can't see it, nor can I find the equivalent. Any hints?
>>> >
>>> > Thanks,
>>> > Cosmin
>>> >
>>>
[jira] [Commented] (KAFKA-835) Update 0.8 configs on the website
[ https://issues.apache.org/jira/browse/KAFKA-835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13619743#comment-13619743 ] Cosmin Lehene commented on KAFKA-835: - What exactly needs to be updated (wiki or something in the source repo)? One change I noticed was brokerid renamed to broker.id > Update 0.8 configs on the website > - > > Key: KAFKA-835 > URL: https://issues.apache.org/jira/browse/KAFKA-835 > Project: Kafka > Issue Type: Sub-task >Affects Versions: 0.8 >Reporter: Neha Narkhede > > We renamed existing configs and also added new configs in 0.8. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (KAFKA-835) Update 0.8 configs on the website
[ https://issues.apache.org/jira/browse/KAFKA-835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13619743#comment-13619743 ] Cosmin Lehene edited comment on KAFKA-835 at 4/2/13 12:29 PM: -- What exactly needs to be updated (wiki or something in the source repo)? One change that I noticed was brokerid renamed to broker.id was (Author: clehene): What exactly needs to be updated (wiki or something in the source repo)? One change I noticed was brokerid renamed to broker.id > Update 0.8 configs on the website > - > > Key: KAFKA-835 > URL: https://issues.apache.org/jira/browse/KAFKA-835 > Project: Kafka > Issue Type: Sub-task >Affects Versions: 0.8 >Reporter: Neha Narkhede > > We renamed existing configs and also added new configs in 0.8. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-296) Update Go Client to new version of Go
[ https://issues.apache.org/jira/browse/KAFKA-296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13619786#comment-13619786 ] David Arthur commented on KAFKA-296: I thought we were moving clients out of the main project? > Update Go Client to new version of Go > - > > Key: KAFKA-296 > URL: https://issues.apache.org/jira/browse/KAFKA-296 > Project: Kafka > Issue Type: Improvement > Components: clients >Affects Versions: 0.7.1 >Reporter: AaronR > Fix For: 0.8 > > Attachments: go1updates.patch, kafka296-v2.patch, > kafka296v3.git.patch, kafka296_v4.patch > > > go (http://golang.org) is close to releasing a new release of go (1.0) which > requires updates to the client, in the meantime most of the go community has > moved to this version. > This change contains: > * language changes to existing client (os.Error, time. Signals, etc) > * removes the Makefile's (no longer used by go) > * It also runs "go fmt" (formats it in standard go) which are most of the > lines changes, from spaces to tabs. > * updates the import path to allow for "go get" installs (don't need to get > source and build) > Not sure which versions this should apply to, but i think it should go to 0.7 > and newer. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-296) Update Go Client to new version of Go
[ https://issues.apache.org/jira/browse/KAFKA-296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13619789#comment-13619789 ] Jeffrey Damick commented on KAFKA-296: -- this is a very old ticket that was never completed. > Update Go Client to new version of Go > - > > Key: KAFKA-296 > URL: https://issues.apache.org/jira/browse/KAFKA-296 > Project: Kafka > Issue Type: Improvement > Components: clients >Affects Versions: 0.7.1 >Reporter: AaronR > Fix For: 0.8 > > Attachments: go1updates.patch, kafka296-v2.patch, > kafka296v3.git.patch, kafka296_v4.patch > > > go (http://golang.org) is close to releasing a new release of go (1.0) which > requires updates to the client, in the meantime most of the go community has > moved to this version. > This change contains: > * language changes to existing client (os.Error, time. Signals, etc) > * removes the Makefile's (no longer used by go) > * It also runs "go fmt" (formats it in standard go) which are most of the > lines changes, from spaces to tabs. > * updates the import path to allow for "go get" installs (don't need to get > source and build) > Not sure which versions this should apply to, but i think it should go to 0.7 > and newer. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-835) Update 0.8 configs on the website
[ https://issues.apache.org/jira/browse/KAFKA-835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13619836#comment-13619836 ] Tudor Scurtu commented on KAFKA-835: brokerid -> broker.id hostname -> host.name num.threads -> num.network.threads, num.io.threads socket.send.buffer -> socket.send.buffer.bytes socket.receive.buffer -> socket.receive.buffer.bytes max.socket.request.bytes -> socket.request.max.bytes log.dir -> log.dir[s] topic.partition.count.map -> X topic.flush.intervals.ms -> log.flush.intervals.ms.per.topic log.flush.interval -> log.flush.interval.messages log.default.flush.interval.ms -> log.flush.interval.ms topic.flush.intervals.ms -> log.flush.intervals.ms.per.topic log.default.flush.scheduler.interval.ms -> X log.retention.size -> log.retention.bytes log.file.size -> log.segment.bytes enable.zookeeper -> X zk.connectiontimeout.ms -> zk.connection.timeout.ms Hope I got them all. > Update 0.8 configs on the website > - > > Key: KAFKA-835 > URL: https://issues.apache.org/jira/browse/KAFKA-835 > Project: Kafka > Issue Type: Sub-task >Affects Versions: 0.8 >Reporter: Neha Narkhede > > We renamed existing configs and also added new configs in 0.8. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-316) disallow recursively compressed message
[ https://issues.apache.org/jira/browse/KAFKA-316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Arthur updated KAFKA-316: --- Attachment: 0001-KAFKA-316-Disallow-MessageSets-within-MessageSets.patch > disallow recursively compressed message > --- > > Key: KAFKA-316 > URL: https://issues.apache.org/jira/browse/KAFKA-316 > Project: Kafka > Issue Type: Improvement > Components: core >Reporter: Jun Rao > Fix For: 0.8 > > Attachments: > 0001-KAFKA-316-Disallow-MessageSets-within-MessageSets.patch > > > Currently, it is possible to create a compressed Message that contains a set > of Messages, each of which is further compressed. Support recursively > compressed messages has little benefit and can complicates the on disk > storage format. We should probably disallow this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-316) disallow recursively compressed message
[ https://issues.apache.org/jira/browse/KAFKA-316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Arthur updated KAFKA-316: --- Status: Patch Available (was: Open) Add a "depth" variable to keep track of recursion level. If depth > 0, throw an InvalidMessageException > disallow recursively compressed message > --- > > Key: KAFKA-316 > URL: https://issues.apache.org/jira/browse/KAFKA-316 > Project: Kafka > Issue Type: Improvement > Components: core >Reporter: Jun Rao > Fix For: 0.8 > > Attachments: > 0001-KAFKA-316-Disallow-MessageSets-within-MessageSets.patch > > > Currently, it is possible to create a compressed Message that contains a set > of Messages, each of which is further compressed. Support recursively > compressed messages has little benefit and can complicates the on disk > storage format. We should probably disallow this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-665) Outgoing responses delayed on a busy Kafka broker
[ https://issues.apache.org/jira/browse/KAFKA-665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13619914#comment-13619914 ] David Arthur commented on KAFKA-665: Maybe RequestChannel.sendRequest and RequestChannel.requestQueue should have a timeout instead of blocking forever? > Outgoing responses delayed on a busy Kafka broker > -- > > Key: KAFKA-665 > URL: https://issues.apache.org/jira/browse/KAFKA-665 > Project: Kafka > Issue Type: Bug >Affects Versions: 0.8 >Reporter: Neha Narkhede >Priority: Critical > Labels: replication-performance > Fix For: 0.8 > > > In a long running test, I observed that after a few hours of operation, few > requests start timing out, mainly because they spent very long time sitting > in the response queue - > [2012-12-07 22:05:56,670] TRACE Completed request with correlation id 3965966 > and client : TopicMetadataRequest:4009, queueTime:1, localTime:28, > remoteTime:0, sendTime:3980 (kafka.network.RequestChannel$) > [2012-12-07 22:04:12,046] TRACE Completed request with correlation id 3962561 > and client : TopicMetadataRequest:3449, queueTime:0, localTime:29, > remoteTime:0, sendTime:3420 (kafka.network.RequestChannel$) > [2012-12-07 22:05:56,670] TRACE Completed request with correlation id 3965966 > and client : TopicMetadataRequest:4009, queueTime:1, localTime:28, > remoteTime:0, sendTime:3980 (kafka.network.RequestChannel$) > We might have a problem in the way we process outgoing responses. Basically, > if the processor thread blocks on enqueuing requests in the request queue, it > doesn't come around to processing its responses which are ready to go out. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Work started] (KAFKA-842) Mirror maker can lose some messages during shutdown
[ https://issues.apache.org/jira/browse/KAFKA-842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KAFKA-842 started by Neha Narkhede. > Mirror maker can lose some messages during shutdown > --- > > Key: KAFKA-842 > URL: https://issues.apache.org/jira/browse/KAFKA-842 > Project: Kafka > Issue Type: Bug > Components: tools >Affects Versions: 0.8 >Reporter: Neha Narkhede >Assignee: Neha Narkhede >Priority: Critical > > On shutdown, mirror maker can lose a few messages. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-842) Mirror maker can lose some messages during shutdown
[ https://issues.apache.org/jira/browse/KAFKA-842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-842: Attachment: kafka-842.patch Mirror maker was losing few messages even on a clean shutdown. This is because, on shutdown, we only waited for the embedded producer to send data to an async producer. The messages were lost from the async producer's queue since we didn't call producer.close() that flushes the outstanding messages in the producer's queue. This patch fixes the bug. The system test failed due to this bug and another bug in the system test itself. Will file a separate bug to fix the test. > Mirror maker can lose some messages during shutdown > --- > > Key: KAFKA-842 > URL: https://issues.apache.org/jira/browse/KAFKA-842 > Project: Kafka > Issue Type: Bug > Components: tools >Affects Versions: 0.8 >Reporter: Neha Narkhede >Assignee: Neha Narkhede >Priority: Critical > Attachments: kafka-842.patch > > > On shutdown, mirror maker can lose a few messages. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-842) Mirror maker can lose some messages during shutdown
[ https://issues.apache.org/jira/browse/KAFKA-842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-842: Status: Patch Available (was: In Progress) > Mirror maker can lose some messages during shutdown > --- > > Key: KAFKA-842 > URL: https://issues.apache.org/jira/browse/KAFKA-842 > Project: Kafka > Issue Type: Bug > Components: tools >Affects Versions: 0.8 >Reporter: Neha Narkhede >Assignee: Neha Narkhede >Priority: Critical > Attachments: kafka-842.patch > > > On shutdown, mirror maker can lose a few messages. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (KAFKA-844) System Test - Mirror Maker cases enhancements
John Fung created KAFKA-844: --- Summary: System Test - Mirror Maker cases enhancements Key: KAFKA-844 URL: https://issues.apache.org/jira/browse/KAFKA-844 Project: Kafka Issue Type: Task Reporter: John Fung The following changes are needed to enhance System Test Mirror Maker cases: 1. Auto create topic 2. Specify num.partitions = 1 3. Wait longer after mirror maker is terminated by kill -15 4. Add ConsoleConsumer to consume data from source cluster for validation reference -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-844) System Test - Mirror Maker cases enhancements
[ https://issues.apache.org/jira/browse/KAFKA-844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-844: Status: Patch Available (was: Open) > System Test - Mirror Maker cases enhancements > - > > Key: KAFKA-844 > URL: https://issues.apache.org/jira/browse/KAFKA-844 > Project: Kafka > Issue Type: Task >Reporter: John Fung > Attachments: kafka-844-v1.patch > > > The following changes are needed to enhance System Test Mirror Maker cases: > 1. Auto create topic > 2. Specify num.partitions = 1 > 3. Wait longer after mirror maker is terminated by kill -15 > 4. Add ConsoleConsumer to consume data from source cluster for validation > reference -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-844) System Test - Mirror Maker cases enhancements
[ https://issues.apache.org/jira/browse/KAFKA-844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-844: Attachment: kafka-844-v1.patch > System Test - Mirror Maker cases enhancements > - > > Key: KAFKA-844 > URL: https://issues.apache.org/jira/browse/KAFKA-844 > Project: Kafka > Issue Type: Task >Reporter: John Fung > Attachments: kafka-844-v1.patch > > > The following changes are needed to enhance System Test Mirror Maker cases: > 1. Auto create topic > 2. Specify num.partitions = 1 > 3. Wait longer after mirror maker is terminated by kill -15 > 4. Add ConsoleConsumer to consume data from source cluster for validation > reference -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-840) Controller tries to perform preferred replica election on failover before state machines have started up
[ https://issues.apache.org/jira/browse/KAFKA-840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13619965#comment-13619965 ] Neha Narkhede commented on KAFKA-840: - Thanks for the patch! One issue is that the !hasShutdown check should be within the lock, otherwise the state machine can end up trying to respond to a zookeeper listener even after it has shutdown. Today, this cannot happen since we close the zkclient before the controller shuts down, but this is a defensive check which will protect us if the shutdown order changes in the future. > Controller tries to perform preferred replica election on failover before > state machines have started up > > > Key: KAFKA-840 > URL: https://issues.apache.org/jira/browse/KAFKA-840 > Project: Kafka > Issue Type: Bug >Affects Versions: 0.8 >Reporter: Swapnil Ghike >Assignee: Swapnil Ghike >Priority: Blocker > Labels: bugs > Fix For: 0.8 > > Attachments: kafka-840-after-commit-v1.patch, kafka-840.patch, > kafka-840-v2.patch > > > If the admin/preferred_replica_election path is non-empty when a new > controller starts, the controller attempts to perform preferred replica > election before the partition and replica state machine have been > initialized. In this case, the controller will try to make invalid state > transitions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-840) Controller tries to perform preferred replica election on failover before state machines have started up
[ https://issues.apache.org/jira/browse/KAFKA-840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-840: Component/s: controller > Controller tries to perform preferred replica election on failover before > state machines have started up > > > Key: KAFKA-840 > URL: https://issues.apache.org/jira/browse/KAFKA-840 > Project: Kafka > Issue Type: Bug > Components: controller >Affects Versions: 0.8 >Reporter: Swapnil Ghike >Assignee: Swapnil Ghike >Priority: Blocker > Labels: bugs > Fix For: 0.8 > > Attachments: kafka-840-after-commit-v1.patch, kafka-840.patch, > kafka-840-v2.patch > > > If the admin/preferred_replica_election path is non-empty when a new > controller starts, the controller attempts to perform preferred replica > election before the partition and replica state machine have been > initialized. In this case, the controller will try to make invalid state > transitions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-844) System Test - Mirror Maker cases enhancements
[ https://issues.apache.org/jira/browse/KAFKA-844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13619980#comment-13619980 ] Neha Narkhede commented on KAFKA-844: - Thanks for the patch, John! I think changes 1and 4 make sense. However, I don't think we should make change #2. This might hide some issues that show up with multiple partitions. Let's set the number of partitions to at least 2 or 3 and bump up producer retries if required. Also with the fix for KAFKA-842, I don't think you need change #3 as well, let's confirm that. > System Test - Mirror Maker cases enhancements > - > > Key: KAFKA-844 > URL: https://issues.apache.org/jira/browse/KAFKA-844 > Project: Kafka > Issue Type: Task >Reporter: John Fung > Attachments: kafka-844-v1.patch > > > The following changes are needed to enhance System Test Mirror Maker cases: > 1. Auto create topic > 2. Specify num.partitions = 1 > 3. Wait longer after mirror maker is terminated by kill -15 > 4. Add ConsoleConsumer to consume data from source cluster for validation > reference -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-840) Controller tries to perform preferred replica election on failover before state machines have started up
[ https://issues.apache.org/jira/browse/KAFKA-840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13620165#comment-13620165 ] Swapnil Ghike commented on KAFKA-840: - Makes sense. Should we also de-register the topic/broker change listeners when the partition/replica state machines are shutting down? > Controller tries to perform preferred replica election on failover before > state machines have started up > > > Key: KAFKA-840 > URL: https://issues.apache.org/jira/browse/KAFKA-840 > Project: Kafka > Issue Type: Bug > Components: controller >Affects Versions: 0.8 >Reporter: Swapnil Ghike >Assignee: Swapnil Ghike >Priority: Blocker > Labels: bugs > Fix For: 0.8 > > Attachments: kafka-840-after-commit-v1.patch, kafka-840.patch, > kafka-840-v2.patch > > > If the admin/preferred_replica_election path is non-empty when a new > controller starts, the controller attempts to perform preferred replica > election before the partition and replica state machine have been > initialized. In this case, the controller will try to make invalid state > transitions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-840) Controller tries to perform preferred replica election on failover before state machines have started up
[ https://issues.apache.org/jira/browse/KAFKA-840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13620167#comment-13620167 ] Neha Narkhede commented on KAFKA-840: - That would make sense, but with the current shutdown logic of the server, zkclient is closed before controller is shutdown. So doing that will raise an error. > Controller tries to perform preferred replica election on failover before > state machines have started up > > > Key: KAFKA-840 > URL: https://issues.apache.org/jira/browse/KAFKA-840 > Project: Kafka > Issue Type: Bug > Components: controller >Affects Versions: 0.8 >Reporter: Swapnil Ghike >Assignee: Swapnil Ghike >Priority: Blocker > Labels: bugs > Fix For: 0.8 > > Attachments: kafka-840-after-commit-v1.patch, kafka-840.patch, > kafka-840-v2.patch > > > If the admin/preferred_replica_election path is non-empty when a new > controller starts, the controller attempts to perform preferred replica > election before the partition and replica state machine have been > initialized. In this case, the controller will try to make invalid state > transitions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-840) Controller tries to perform preferred replica election on failover before state machines have started up
[ https://issues.apache.org/jira/browse/KAFKA-840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Swapnil Ghike updated KAFKA-840: Attachment: kafka-840-after-commit-v2.patch Cool, attached a new patch 'after-commit-v2'. > Controller tries to perform preferred replica election on failover before > state machines have started up > > > Key: KAFKA-840 > URL: https://issues.apache.org/jira/browse/KAFKA-840 > Project: Kafka > Issue Type: Bug > Components: controller >Affects Versions: 0.8 >Reporter: Swapnil Ghike >Assignee: Swapnil Ghike >Priority: Blocker > Labels: bugs > Fix For: 0.8 > > Attachments: kafka-840-after-commit-v1.patch, > kafka-840-after-commit-v2.patch, kafka-840.patch, kafka-840-v2.patch > > > If the admin/preferred_replica_election path is non-empty when a new > controller starts, the controller attempts to perform preferred replica > election before the partition and replica state machine have been > initialized. In this case, the controller will try to make invalid state > transitions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-840) Controller tries to perform preferred replica election on failover before state machines have started up
[ https://issues.apache.org/jira/browse/KAFKA-840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13620190#comment-13620190 ] Neha Narkhede commented on KAFKA-840: - +1 on the latest patch, thanks Swapnil! > Controller tries to perform preferred replica election on failover before > state machines have started up > > > Key: KAFKA-840 > URL: https://issues.apache.org/jira/browse/KAFKA-840 > Project: Kafka > Issue Type: Bug > Components: controller >Affects Versions: 0.8 >Reporter: Swapnil Ghike >Assignee: Swapnil Ghike >Priority: Blocker > Labels: bugs > Fix For: 0.8 > > Attachments: kafka-840-after-commit-v1.patch, > kafka-840-after-commit-v2.patch, kafka-840.patch, kafka-840-v2.patch > > > If the admin/preferred_replica_election path is non-empty when a new > controller starts, the controller attempts to perform preferred replica > election before the partition and replica state machine have been > initialized. In this case, the controller will try to make invalid state > transitions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-844) System Test - Mirror Maker cases enhancements
[ https://issues.apache.org/jira/browse/KAFKA-844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-844: Attachment: kafka-844-v2.patch > System Test - Mirror Maker cases enhancements > - > > Key: KAFKA-844 > URL: https://issues.apache.org/jira/browse/KAFKA-844 > Project: Kafka > Issue Type: Task >Reporter: John Fung > Attachments: kafka-844-v1.patch, kafka-844-v2.patch > > > The following changes are needed to enhance System Test Mirror Maker cases: > 1. Auto create topic > 2. Specify num.partitions = 1 > 3. Wait longer after mirror maker is terminated by kill -15 > 4. Add ConsoleConsumer to consume data from source cluster for validation > reference -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-844) System Test - Mirror Maker cases enhancements
[ https://issues.apache.org/jira/browse/KAFKA-844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13620202#comment-13620202 ] John Fung commented on KAFKA-844: - Thanks Neha for reviewing. The num.partitions is set to 5 and this patch is assumed to be applied on top of KAFKA-842. > System Test - Mirror Maker cases enhancements > - > > Key: KAFKA-844 > URL: https://issues.apache.org/jira/browse/KAFKA-844 > Project: Kafka > Issue Type: Task >Reporter: John Fung > Attachments: kafka-844-v1.patch, kafka-844-v2.patch > > > The following changes are needed to enhance System Test Mirror Maker cases: > 1. Auto create topic > 2. Specify num.partitions = 1 > 3. Wait longer after mirror maker is terminated by kill -15 > 4. Add ConsoleConsumer to consume data from source cluster for validation > reference -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-844) System Test - Mirror Maker cases enhancements
[ https://issues.apache.org/jira/browse/KAFKA-844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-844: Description: The following changes are needed to enhance System Test Mirror Maker cases: 1. Auto create topic 2. Specify num.partitions = 1 3. Add ConsoleConsumer to consume data from source cluster for validation reference was: The following changes are needed to enhance System Test Mirror Maker cases: 1. Auto create topic 2. Specify num.partitions = 1 3. Wait longer after mirror maker is terminated by kill -15 4. Add ConsoleConsumer to consume data from source cluster for validation reference > System Test - Mirror Maker cases enhancements > - > > Key: KAFKA-844 > URL: https://issues.apache.org/jira/browse/KAFKA-844 > Project: Kafka > Issue Type: Task >Reporter: John Fung > Attachments: kafka-844-v1.patch, kafka-844-v2.patch > > > The following changes are needed to enhance System Test Mirror Maker cases: > 1. Auto create topic > 2. Specify num.partitions = 1 > 3. Add ConsoleConsumer to consume data from source cluster for validation > reference -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (KAFKA-844) System Test - Mirror Maker cases enhancements
[ https://issues.apache.org/jira/browse/KAFKA-844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13620202#comment-13620202 ] John Fung edited comment on KAFKA-844 at 4/2/13 7:59 PM: - Thanks Neha for reviewing. The num.partitions is set to 5 and this patch is assumed to be applied on top of KAFKA-842. (original #3 (wait longer after mirror maker is terminated) is removed. was (Author: jfung): Thanks Neha for reviewing. The num.partitions is set to 5 and this patch is assumed to be applied on top of KAFKA-842. > System Test - Mirror Maker cases enhancements > - > > Key: KAFKA-844 > URL: https://issues.apache.org/jira/browse/KAFKA-844 > Project: Kafka > Issue Type: Task >Reporter: John Fung > Attachments: kafka-844-v1.patch, kafka-844-v2.patch > > > The following changes are needed to enhance System Test Mirror Maker cases: > 1. Auto create topic > 2. Specify num.partitions = 1 > 3. Add ConsoleConsumer to consume data from source cluster for validation > reference -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-844) System Test - Mirror Maker cases enhancements
[ https://issues.apache.org/jira/browse/KAFKA-844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-844: Attachment: (was: kafka-844-v2.patch) > System Test - Mirror Maker cases enhancements > - > > Key: KAFKA-844 > URL: https://issues.apache.org/jira/browse/KAFKA-844 > Project: Kafka > Issue Type: Task >Reporter: John Fung > Attachments: kafka-844-v1.patch > > > The following changes are needed to enhance System Test Mirror Maker cases: > 1. Auto create topic > 2. Specify num.partitions = 1 > 3. Add ConsoleConsumer to consume data from source cluster for validation > reference -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-844) System Test - Mirror Maker cases enhancements
[ https://issues.apache.org/jira/browse/KAFKA-844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-844: Attachment: kafka-844-v2.patch > System Test - Mirror Maker cases enhancements > - > > Key: KAFKA-844 > URL: https://issues.apache.org/jira/browse/KAFKA-844 > Project: Kafka > Issue Type: Task >Reporter: John Fung > Attachments: kafka-844-v1.patch, kafka-844-v2.patch > > > The following changes are needed to enhance System Test Mirror Maker cases: > 1. Auto create topic > 2. Specify num.partitions = 1 > 3. Add ConsoleConsumer to consume data from source cluster for validation > reference -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-826) Make Kafka 0.8 depend on metrics 2.2.0 instead of 3.x
[ https://issues.apache.org/jira/browse/KAFKA-826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13620363#comment-13620363 ] Jun Rao commented on KAFKA-826: --- Otis, The reason that we want to downgrade metrics to 2.x is because quite a few people feel this is the most stable version that they can depend on. Do you know how stable is this beta release? Will there be any API or bean name change btw the final 3.x release and this beta release? In general, do people feel comfortable about using the metrics 3.0 beta release? > Make Kafka 0.8 depend on metrics 2.2.0 instead of 3.x > - > > Key: KAFKA-826 > URL: https://issues.apache.org/jira/browse/KAFKA-826 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 0.8 >Reporter: Neha Narkhede >Assignee: Jun Rao >Priority: Blocker > Labels: build, kafka-0.8, metrics > Attachments: kafka-fix-for-826.patch, kafka-fix-for-826-take2.patch > > > In order to mavenize Kafka 0.8, we have to depend on metrics 2.2.0 since > metrics 3.x is a huge change as well as not an officially supported release. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-826) Make Kafka 0.8 depend on metrics 2.2.0 instead of 3.x
[ https://issues.apache.org/jira/browse/KAFKA-826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13620428#comment-13620428 ] Swapnil Ghike commented on KAFKA-826: - Dragos, thanks for the input. We can definitely use your patch if we decide to downgrade metrics to 2.2.0. Could you please address the following comments and re-submit the patch? 1. Could you move the metrics-core, metrics-annotations and zkclient library dependences to core/build.sbt? You will need to replace the old organization and version number of zkclient there with the one you added to KafkaBuild. 2. Could you append the corresponding paths to class path in bin/kafka-run-class.sh? Also please remove core/lib/*.jar from class path since we are moving the three jars there to ivy. 3. git rm core/lib for the same reason mentioned in 2. Apart from your patch, we will need to fix the csv reporter separately. > Make Kafka 0.8 depend on metrics 2.2.0 instead of 3.x > - > > Key: KAFKA-826 > URL: https://issues.apache.org/jira/browse/KAFKA-826 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 0.8 >Reporter: Neha Narkhede >Assignee: Jun Rao >Priority: Blocker > Labels: build, kafka-0.8, metrics > Attachments: kafka-fix-for-826.patch, kafka-fix-for-826-take2.patch > > > In order to mavenize Kafka 0.8, we have to depend on metrics 2.2.0 since > metrics 3.x is a huge change as well as not an officially supported release. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-826) Make Kafka 0.8 depend on metrics 2.2.0 instead of 3.x
[ https://issues.apache.org/jira/browse/KAFKA-826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Swapnil Ghike updated KAFKA-826: Assignee: Dragos Manolescu (was: Jun Rao) > Make Kafka 0.8 depend on metrics 2.2.0 instead of 3.x > - > > Key: KAFKA-826 > URL: https://issues.apache.org/jira/browse/KAFKA-826 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 0.8 >Reporter: Neha Narkhede >Assignee: Dragos Manolescu >Priority: Blocker > Labels: build, kafka-0.8, metrics > Attachments: kafka-fix-for-826.patch, kafka-fix-for-826-take2.patch > > > In order to mavenize Kafka 0.8, we have to depend on metrics 2.2.0 since > metrics 3.x is a huge change as well as not an officially supported release. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: C/C++ Client
Yes - we would be interested in doing that. I have been spending most of my time over the past couple weeks on the C++ library (currently, only for the producer). It is reasonably stable, although it has not been tried and tested in production. I can start with publishing a wiki describing the design/implementation and list out future work that others can contribute to if there is interest, but I'll quickly summarize some of its goals here: - Use non-blocking I/O for sending requests/receiving responses. This mitigates the request-response RTT latency that the 0.8 protocol introduces (if acks != 0). - C-compliant API, so that other languages e.g., Ruby/Python can wrap the library. I understand there are people working on 0.8 clients in those languages so this is just an alternative approach. - Non-blocking variants for all operations in the API: there are some use-cases where blocking in the user-thread is unacceptable. - Keep third-party libraries to a minimum to make porting to other platforms easier. There are two options in pursuing this: 1 - Contribute it as part of the Kafka project. i.e., re-introduce the clients directory. We removed it because none of the committers were original authors of any of the non-Java clients and did not have the bandwidth/background to review patches for those clients. 2 - Maintain it externally. I was going to also suggest "sub-project" as a third option, but I heard from Jakob that it is no longer encouraged by the board and a client by itself would not be a substantial sub-project. If we want to do (1), we would obviously go through the standard contribution process: i.e., create a jira for it and provide a patch that people can review; and other implementations will also be considered if contributors provide patches. (2) is also fine, although one benefit with (1) is that it would be reasonable to use the existing infrastructure (mailing lists, issue tracking) for the Kafka project. Any preferences/thoughts? Thanks, Joel On Mon, Apr 1, 2013 at 8:22 AM, mrevilgnome wrote: > Any interest in open sourcing it now and picking up contributors? > > > On Mon, Apr 1, 2013 at 7:54 AM, Jun Rao wrote: > > > At LinkedIn, we are also building a native C producer client for 0.8. It > > uses non-blocking socket I/O to improve the producer throughput. We plan > to > > open source it when it's fully tested, hopefully in a couple of months. > > > > Thanks, > > > > Jun > > > > On Wed, Mar 27, 2013 at 8:48 PM, Matthew Stump > >wrote: > > > > > Howdy, > > > > > > I'm considering the use of Kafka in the rewrite of a big legacy > product. > > A > > > good chunk of the back end code is going to be written in C++ (large in > > > memory data-structures). The two possible options available to me for > > > clients appear to be: > > > > > > https://github.com/edenhill/librdkafka > > > > > > and > > > > > > https://github.com/quipo/kafka-cpp > > > > > > The problem is that librdkafka currently only works on Linux due to the > > use > > > of the AF_NETLINK API, and thread local storage. There may be other > > issues, > > > but I just started playing with it today and that's what I've > discovered > > > thus far. > > > > > > kafka-cpp is incomplete (no consumer) and it looks unused. > > > > > > For either I would need to hop in and do some significant work. Is > there > > > any client I'm missing that can shorten my path? > > > > > > If I adopt one of these projects (lets say kafka-cpp) am I better off > > > implementing the 0.8 protocol? I'de like to have something running in > > > staging a couple months from now. How far out is 0.8? > > > > > > Thanks, > > > Matt Stump > > > > > >
[jira] [Commented] (KAFKA-826) Make Kafka 0.8 depend on metrics 2.2.0 instead of 3.x
[ https://issues.apache.org/jira/browse/KAFKA-826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13620566#comment-13620566 ] Otis Gospodnetic commented on KAFKA-826: [~junrao] - I'm on the ML for that metrics lib and a few weeks (1-2 months?) ago Coda wrote an email saying he was not happy with a number of things in 2.x that he completely reworked in 3.x. I don't recall the details, but I recall feeling like 3.x is the version I'd use if I had to pick between 2.x and 3.x. Unless Kafka 0.8 is going to be released within the next week or two, I'd personally go with 3.x, unless you are open moving from 2.x to 3.x in the near future in 0.8.1. > Make Kafka 0.8 depend on metrics 2.2.0 instead of 3.x > - > > Key: KAFKA-826 > URL: https://issues.apache.org/jira/browse/KAFKA-826 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 0.8 >Reporter: Neha Narkhede >Assignee: Dragos Manolescu >Priority: Blocker > Labels: build, kafka-0.8, metrics > Attachments: kafka-fix-for-826.patch, kafka-fix-for-826-take2.patch > > > In order to mavenize Kafka 0.8, we have to depend on metrics 2.2.0 since > metrics 3.x is a huge change as well as not an officially supported release. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
