[jira] [Resolved] (KAFKA-631) Implement log compaction

[ https://issues.apache.org/jira/browse/KAFKA-631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps resolved KAFKA-631. - Resolution: Fixed > Implement log compaction > > > Key: KAFKA

[jira] [Commented] (KAFKA-631) Implement log compaction

[ https://issues.apache.org/jira/browse/KAFKA-631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13565483#comment-13565483 ] Jay Kreps commented on KAFKA-631: - Cool, checked in with those fixes. Filed KAFKA-739, KAFK

[jira] [Commented] (KAFKA-695) Broker shuts down due to attempt to read a closed index file

[ https://issues.apache.org/jira/browse/KAFKA-695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13565524#comment-13565524 ] Jun Rao commented on KAFKA-695: --- Just realized there are a couple of other corner cases that

[jira] [Updated] (KAFKA-734) Migration tool needs a revamp, it was poorly written and has many performance bugs

[ https://issues.apache.org/jira/browse/KAFKA-734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-734: Status: Patch Available (was: Open) > Migration tool needs a revamp, it was poorly written and

[jira] [Updated] (KAFKA-734) Migration tool needs a revamp, it was poorly written and has many performance bugs

[ https://issues.apache.org/jira/browse/KAFKA-734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-734: Attachment: kafka-734-v2.patch Thanks for the review, Jun! 1. Moved it to the migration tool. Howev

[jira] [Updated] (KAFKA-671) DelayedProduce requests should not hold full producer request data

[ https://issues.apache.org/jira/browse/KAFKA-671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sriram Subramanian updated KAFKA-671: - Attachment: outOfMemFix-v2-rebase.patch rebased > DelayedProduce requests

[jira] [Updated] (KAFKA-738) correlationId is not set in FetchRequest in AbstractFetcherThread

[ https://issues.apache.org/jira/browse/KAFKA-738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-738: -- Attachment: kafka-738.patch Attach patch v3. Having a separate reset() method adds more work to the client. Cha

[jira] [Commented] (KAFKA-738) correlationId is not set in FetchRequest in AbstractFetcherThread

[ https://issues.apache.org/jira/browse/KAFKA-738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13565631#comment-13565631 ] Neha Narkhede commented on KAFKA-738: - Thanks for the new patch. Some more comments -

Re: 0.8 wire protocol for inter-broker communication

I don't think this is actually that hard to handle, you just need a config to enable the new fields: Step 1: Implement optional support for the new field with some option that controls whether it is used Step 2: Push all servers, still using the old format. Step 3: Now enable the new field on serve

[jira] [Commented] (KAFKA-671) DelayedProduce requests should not hold full producer request data

[ https://issues.apache.org/jira/browse/KAFKA-671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13565679#comment-13565679 ] Neha Narkhede commented on KAFKA-671: - Thanks for the v2 patch, few review comments -

[jira] [Created] (KAFKA-742) Existing directories under the Kafka data directory without any data cause process to not start

Chris Curtin created KAFKA-742: -- Summary: Existing directories under the Kafka data directory without any data cause process to not start Key: KAFKA-742 URL: https://issues.apache.org/jira/browse/KAFKA-742

[jira] [Commented] (KAFKA-742) Existing directories under the Kafka data directory without any data cause process to not start

[ https://issues.apache.org/jira/browse/KAFKA-742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13565696#comment-13565696 ] Jay Kreps commented on KAFKA-742: - Getting an error is not really bad since we assume we ha

[jira] [Commented] (KAFKA-742) Existing directories under the Kafka data directory without any data cause process to not start

[ https://issues.apache.org/jira/browse/KAFKA-742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13565709#comment-13565709 ] Chris Curtin commented on KAFKA-742: Thanks Jay, a better error is a good idea for this

[jira] [Closed] (KAFKA-720) Migration tool halts

[ https://issues.apache.org/jira/browse/KAFKA-720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede closed KAFKA-720. --- > Migration tool halts > > > Key: KAFKA-720 > URL

[jira] [Updated] (KAFKA-649) Cleanup log4j logging

[ https://issues.apache.org/jira/browse/KAFKA-649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-649: Labels: (was: p4) > Cleanup log4j logging > - > > Key: KAF

[jira] [Commented] (KAFKA-736) Add an option to the 0.8 producer to mimic 0.7 producer behavior

[ https://issues.apache.org/jira/browse/KAFKA-736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13565762#comment-13565762 ] Neha Narkhede commented on KAFKA-736: - Another approach is to reset the interest ops to

[jira] [Updated] (KAFKA-568) LogTest started failing after KAFKA-506

[ https://issues.apache.org/jira/browse/KAFKA-568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-568: Labels: unit-test (was: bugs) > LogTest started failing after KAFKA-506 >

[jira] [Resolved] (KAFKA-719) Kafka broker shuts down due to irrecoverable IO error

[ https://issues.apache.org/jira/browse/KAFKA-719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede resolved KAFKA-719. - Resolution: Fixed > Kafka broker shuts down due to irrecoverable IO error > --

[jira] [Closed] (KAFKA-719) Kafka broker shuts down due to irrecoverable IO error

[ https://issues.apache.org/jira/browse/KAFKA-719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede closed KAFKA-719. --- > Kafka broker shuts down due to irrecoverable IO error >

[jira] [Updated] (KAFKA-738) correlationId is not set in FetchRequest in AbstractFetcherThread

[ https://issues.apache.org/jira/browse/KAFKA-738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-738: -- Attachment: kafka-738_v3.patch Sorry, attached the wrong patch. Attach the correct one v3 this time.

[jira] [Commented] (KAFKA-738) correlationId is not set in FetchRequest in AbstractFetcherThread

[ https://issues.apache.org/jira/browse/KAFKA-738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13565888#comment-13565888 ] Neha Narkhede commented on KAFKA-738: - +1 on the latest patch > correl

[jira] [Commented] (KAFKA-738) correlationId is not set in FetchRequest in AbstractFetcherThread

[ https://issues.apache.org/jira/browse/KAFKA-738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13565965#comment-13565965 ] Swapnil Ghike commented on KAFKA-738: - Though with this patch, the requestMap will be c

[jira] [Commented] (KAFKA-671) DelayedProduce requests should not hold full producer request data

[ https://issues.apache.org/jira/browse/KAFKA-671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13565980#comment-13565980 ] Sriram Subramanian commented on KAFKA-671: -- 1. You can ignore the changes in syste

[jira] [Commented] (KAFKA-738) correlationId is not set in FetchRequest in AbstractFetcherThread

[ https://issues.apache.org/jira/browse/KAFKA-738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13565992#comment-13565992 ] Swapnil Ghike commented on KAFKA-738: - +1 > correlationId is not set i

[jira] [Updated] (KAFKA-671) DelayedProduce requests should not hold full producer request data

[ https://issues.apache.org/jira/browse/KAFKA-671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sriram Subramanian updated KAFKA-671: - Attachment: outOfMemFix-v3.patch > DelayedProduce requests should not hold full produc

[jira] [Updated] (KAFKA-738) correlationId is not set in FetchRequest in AbstractFetcherThread

[ https://issues.apache.org/jira/browse/KAFKA-738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-738: -- Resolution: Fixed Fix Version/s: 0.8 Status: Resolved (was: Patch Available) Thanks for the r

[jira] [Closed] (KAFKA-738) correlationId is not set in FetchRequest in AbstractFetcherThread

[ https://issues.apache.org/jira/browse/KAFKA-738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede closed KAFKA-738. --- > correlationId is not set in FetchRequest in AbstractFetcherThread >

[jira] [Updated] (KAFKA-737) System Test : Disable shallow.iterator in Mirror Maker test cases to make compression work correctly

[ https://issues.apache.org/jira/browse/KAFKA-737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-737: -- Resolution: Fixed Fix Version/s: 0.8 Status: Resolved (was: Patch Available) Thanks for the p

[jira] [Closed] (KAFKA-737) System Test : Disable shallow.iterator in Mirror Maker test cases to make compression work correctly

[ https://issues.apache.org/jira/browse/KAFKA-737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede closed KAFKA-737. --- > System Test : Disable shallow.iterator in Mirror Maker test cases to make > compression work correc

[jira] [Updated] (KAFKA-736) Add an option to the 0.8 producer to mimic 0.7 producer behavior

[ https://issues.apache.org/jira/browse/KAFKA-736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-736: Attachment: kafka-736-v1.patch Went with the simpler approach of resetting the interest ops on the p

Question about offsets

Fiddling with my Python client on 0.8, noticed something has changed with offsets. It seems that instead of a byte offset in the log file, the offset is now a logical one. I had a few questions about this: 1) How is the byte offset determined by the broker? Since messages are not fixed width

Re: Question about offsets

Yes, offsets are now logical 0,1,2,3... Some details on this change are here: https://cwiki.apache.org/confluence/display/KAFKA/Keyed+Messages+Proposal https://issues.apache.org/jira/browse/KAFKA-506 https://issues.apache.org/jira/browse/KAFKA-631 There was some previous threads on this that disc

[jira] [Reopened] (KAFKA-683) Fix correlation ids in all requests sent to kafka

[ https://issues.apache.org/jira/browse/KAFKA-683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps reopened KAFKA-683: - Guys, this patch adds request specific stuff to RequestChannel. RequestChannel is really not the place to be

[jira] [Commented] (KAFKA-671) DelayedProduce requests should not hold full producer request data

[ https://issues.apache.org/jira/browse/KAFKA-671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13566188#comment-13566188 ] Jay Kreps commented on KAFKA-671: - Took a look at this. Looks reasonable. Other atrocities

[jira] [Commented] (KAFKA-683) Fix correlation ids in all requests sent to kafka

[ https://issues.apache.org/jira/browse/KAFKA-683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13566205#comment-13566205 ] Neha Narkhede commented on KAFKA-683: - Understand and agree with you here. However, we

[jira] [Commented] (KAFKA-736) Add an option to the 0.8 producer to mimic 0.7 producer behavior

[ https://issues.apache.org/jira/browse/KAFKA-736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13566215#comment-13566215 ] Jay Kreps commented on KAFKA-736: - This is basically good, but the comments and naming are

[jira] [Commented] (KAFKA-671) DelayedProduce requests should not hold full producer request data

[ https://issues.apache.org/jira/browse/KAFKA-671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13566222#comment-13566222 ] Sriram Subramanian commented on KAFKA-671: -- The issue is that even if we pass only

[jira] [Commented] (KAFKA-683) Fix correlation ids in all requests sent to kafka

[ https://issues.apache.org/jira/browse/KAFKA-683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13566234#comment-13566234 ] Jay Kreps commented on KAFKA-683: - Yeah, I am on the fence about the metrics stuff, maybe i

[jira] [Commented] (KAFKA-671) DelayedProduce requests should not hold full producer request data

[ https://issues.apache.org/jira/browse/KAFKA-671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13566240#comment-13566240 ] Jay Kreps commented on KAFKA-671: - Okay let's sync up. I think requestObj is the devil. :-)