0.8 Protocol Status

2012-12-06 Thread Jay Kreps
Okay I have fixed all the problems people found that I am aware of: 1. Correlation id used throughout 2. Brokers in metadata api not repeated per-partition 3. Removed creator id from broker because it is silly 4. All versions baselined to 0 (both ApiVersions and the message "magic" version) 5. Remo

[jira] [Commented] (KAFKA-374) Move to java CRC32 implementation

2012-12-06 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13526171#comment-13526171 ] Jay Kreps commented on KAFKA-374: - Ooo very nice. > Move to java CRC32 imp

[jira] [Commented] (KAFKA-374) Move to java CRC32 implementation

2012-12-06 Thread David Arthur (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13526136#comment-13526136 ] David Arthur commented on KAFKA-374: I pulled in the pure-Java implementation from Hado

[jira] [Commented] (KAFKA-642) Protocol tweaks for 0.8

2012-12-06 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13526125#comment-13526125 ] Jay Kreps commented on KAFKA-642: - It also baselines all versions, including the message ma

[jira] [Updated] (KAFKA-642) Protocol tweaks for 0.8

2012-12-06 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps updated KAFKA-642: Attachment: KAFKA-642-remove-response-versions.patch This patch removes the response version. There was a di

[jira] [Created] (KAFKA-661) Prevent a shutting down broker from re-entering the ISR

2012-12-06 Thread Joel Koshy (JIRA)
Joel Koshy created KAFKA-661: Summary: Prevent a shutting down broker from re-entering the ISR Key: KAFKA-661 URL: https://issues.apache.org/jira/browse/KAFKA-661 Project: Kafka Issue Type: Bug

[jira] [Closed] (KAFKA-633) AdminTest.testShutdownBroker fails

2012-12-06 Thread Joel Koshy (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Koshy closed KAFKA-633. Checked into 0.8. Will file separate jira to protect the ISR from re-expansion. > AdminTest.tes

[jira] [Resolved] (KAFKA-633) AdminTest.testShutdownBroker fails

2012-12-06 Thread Joel Koshy (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Koshy resolved KAFKA-633. -- Resolution: Fixed > AdminTest.testShutdownBroker fails > -- > >

[jira] [Commented] (KAFKA-633) AdminTest.testShutdownBroker fails

2012-12-06 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13526028#comment-13526028 ] Jun Rao commented on KAFKA-633: --- Thanks for the patch. +1. > AdminTest.testS

[jira] [Updated] (KAFKA-633) AdminTest.testShutdownBroker fails

2012-12-06 Thread Joel Koshy (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Koshy updated KAFKA-633: - Attachment: KAFKA-633-v1.patch This is a timing issue that affects low-volume topics (and in this case an

[jira] [Updated] (KAFKA-660) Change (topic, partition) tuples to TopicAndPartition

2012-12-06 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-660: Affects Version/s: (was: 0.8) 0.8.1 > Change (topic, partition) tuple

[jira] [Created] (KAFKA-660) Change (topic, partition) tuples to TopicAndPartition

2012-12-06 Thread Swapnil Ghike (JIRA)
Swapnil Ghike created KAFKA-660: --- Summary: Change (topic, partition) tuples to TopicAndPartition Key: KAFKA-660 URL: https://issues.apache.org/jira/browse/KAFKA-660 Project: Kafka Issue Type: B

[jira] [Commented] (KAFKA-654) Irrecoverable error while trying to roll a segment that already exists

2012-12-06 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13512081#comment-13512081 ] Neha Narkhede commented on KAFKA-654: - The bug is inside Log.truncateAndStartWithNewOff

[jira] Subscription: outstanding kafka patches

2012-12-06 Thread jira
Issue Subscription Filter: outstanding kafka patches (54 issues) The list of outstanding kafka patches Subscriber: kafka-mailing-list Key Summary KAFKA-645 Create a shell script to run System Test with DEBUG details and "tee" console output to a file https://issues.a

[jira] [Closed] (KAFKA-201) Support for mirroring from multiple sources

2012-12-06 Thread Joel Koshy (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Koshy closed KAFKA-201. > Support for mirroring from multiple sources > --- > >

[jira] [Resolved] (KAFKA-201) Support for mirroring from multiple sources

2012-12-06 Thread Joel Koshy (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Koshy resolved KAFKA-201. -- Resolution: Fixed Yes - this should be addressed by MirrorMaker although the user should ensure that th

Re: mirror maker wiki

2012-12-06 Thread Joel Koshy
Good point - fixed. On Wed, Dec 5, 2012 at 9:22 PM, Jay Kreps wrote: > Hey your wiki on mirror maker is very good. It doesn't actually say how to > run it, though, does it? > > -Jay >

[jira] [Commented] (KAFKA-654) Irrecoverable error while trying to roll a segment that already exists

2012-12-06 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13511601#comment-13511601 ] Jay Kreps commented on KAFKA-654: - I think this is almost certainly a bug in the log. I wil

[jira] [Commented] (KAFKA-654) Irrecoverable error while trying to roll a segment that already exists

2012-12-06 Thread Joel Koshy (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13511579#comment-13511579 ] Joel Koshy commented on KAFKA-654: -- FWIW, it should be easy to reproduce this - I just saw

[jira] [Updated] (KAFKA-330) Add delete topic support

2012-12-06 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps updated KAFKA-330: Labels: features project (was: features) > Add delete topic support > - > >

[jira] [Commented] (KAFKA-631) Implement log compaction

2012-12-06 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13511573#comment-13511573 ] Jun Rao commented on KAFKA-631: --- Prashanth, Will you be interested in KAFKA-330 (delete topi

[jira] [Commented] (KAFKA-330) Add delete topic support

2012-12-06 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13511570#comment-13511570 ] Jun Rao commented on KAFKA-330: --- The delete topic logic can follow the same logic in partitio

[jira] [Commented] (KAFKA-642) Protocol tweaks for 0.8

2012-12-06 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13511571#comment-13511571 ] Jay Kreps commented on KAFKA-642: - Ack, you're right. > Protocol tweaks fo

[jira] [Commented] (KAFKA-642) Protocol tweaks for 0.8

2012-12-06 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13511549#comment-13511549 ] Jun Rao commented on KAFKA-642: --- 60. What you said makes sense. But from IDE, that call maps

Fun Projects

2012-12-06 Thread Jay Kreps
A common question is what Kafka projects would be fun and aren't being worked on by anyone else right now. Most of the committers are working hard on getting 0.8 stable. So the most useful thing anyone could do is download the 0.8 branch and bang on it until it breaks and then help us track it dow

[jira] [Updated] (KAFKA-656) Add Quotas to Kafka

2012-12-06 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps updated KAFKA-656: Labels: project (was: ) > Add Quotas to Kafka > --- > > Key: KAFKA-656

[jira] [Updated] (KAFKA-657) Add an API to commit offsets

2012-12-06 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps updated KAFKA-657: Labels: project (was: ) > Add an API to commit offsets > > >

[jira] [Updated] (KAFKA-559) Garbage collect old consumer metadata entries

2012-12-06 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps updated KAFKA-559: Labels: project (was: ) > Garbage collect old consumer metadata entries > -

[jira] [Updated] (KAFKA-252) Generalize getOffsetsBefore API to a new more general API getLeaderMetadata

2012-12-06 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps updated KAFKA-252: Labels: project (was: replication) > Generalize getOffsetsBefore API to a new more general API getLeade

[jira] [Updated] (KAFKA-554) Move all per-topic configuration into ZK and add to the CreateTopicCommand

2012-12-06 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps updated KAFKA-554: Labels: project (was: ) > Move all per-topic configuration into ZK and add to the CreateTopicCommand >

[jira] [Updated] (KAFKA-266) Kafka web console

2012-12-06 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps updated KAFKA-266: Labels: project (was: ) > Kafka web console > - > > Key: KAFKA-266 >

[jira] [Updated] (KAFKA-658) Implement "Exact Mirroring" functionality in mirror maker

2012-12-06 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps updated KAFKA-658: Labels: project (was: ) > Implement "Exact Mirroring" functionality in mirror maker > -

[jira] [Updated] (KAFKA-560) Garbage Collect obsolete topics

2012-12-06 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps updated KAFKA-560: Labels: project (was: ) > Garbage Collect obsolete topics > --- > >

[jira] [Created] (KAFKA-659) Support request pipelining in the network server

2012-12-06 Thread Jay Kreps (JIRA)
Jay Kreps created KAFKA-659: --- Summary: Support request pipelining in the network server Key: KAFKA-659 URL: https://issues.apache.org/jira/browse/KAFKA-659 Project: Kafka Issue Type: New Feature

[jira] [Created] (KAFKA-658) Implement "Exact Mirroring" functionality in mirror maker

2012-12-06 Thread Jay Kreps (JIRA)
Jay Kreps created KAFKA-658: --- Summary: Implement "Exact Mirroring" functionality in mirror maker Key: KAFKA-658 URL: https://issues.apache.org/jira/browse/KAFKA-658 Project: Kafka Issue Type: New F

[jira] [Updated] (KAFKA-554) Move all per-topic configuration into ZK and add to the CreateTopicCommand

2012-12-06 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps updated KAFKA-554: Description: We have a number of per-topic configurations that control message retention and flush interval

[jira] [Updated] (KAFKA-252) Generalize getOffsetsBefore API to a new more general API getLeaderMetadata

2012-12-06 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps updated KAFKA-252: Description: The relevant discussion is here - https://issues.apache.org/jira/browse/KAFKA-238?focusedComme

[jira] [Created] (KAFKA-657) Add an API to commit offsets

2012-12-06 Thread Jay Kreps (JIRA)
Jay Kreps created KAFKA-657: --- Summary: Add an API to commit offsets Key: KAFKA-657 URL: https://issues.apache.org/jira/browse/KAFKA-657 Project: Kafka Issue Type: New Feature Reporter:

[jira] [Created] (KAFKA-656) Add Quotas to Kafka

2012-12-06 Thread Jay Kreps (JIRA)
Jay Kreps created KAFKA-656: --- Summary: Add Quotas to Kafka Key: KAFKA-656 URL: https://issues.apache.org/jira/browse/KAFKA-656 Project: Kafka Issue Type: New Feature Components: core

[jira] [Updated] (KAFKA-266) Kafka web console

2012-12-06 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps updated KAFKA-266: Summary: Kafka web console (was: Kafka web console design) > Kafka web console > - > >

[jira] [Resolved] (KAFKA-629) Generalize the getOffsets call

2012-12-06 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps resolved KAFKA-629. - Resolution: Duplicate > Generalize the getOffsets call > -- > >

[jira] [Resolved] (KAFKA-223) Add metadata requests to Kafka broker

2012-12-06 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps resolved KAFKA-223. - Resolution: Fixed We have added an API like this in 0.8. It is described here: https://cwiki.apache.org/c

[jira] [Resolved] (KAFKA-271) Modify new FetchResponse object to remove the initial offset field

2012-12-06 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps resolved KAFKA-271. - Resolution: Fixed Addressed this as part of KAFKA-642. > Modify new FetchResponse object

[jira] [Assigned] (KAFKA-271) Modify new FetchResponse object to remove the initial offset field

2012-12-06 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps reassigned KAFKA-271: --- Assignee: Jay Kreps > Modify new FetchResponse object to remove the initial offset field > ---

[jira] [Updated] (KAFKA-252) Generalize getOffsetsBefore API to a new more general API getLeaderMetadata

2012-12-06 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps updated KAFKA-252: Priority: Major (was: Critical) Description: The relevant discussion is here - https://issues.apach

[jira] [Commented] (KAFKA-642) Protocol tweaks for 0.8

2012-12-06 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13511489#comment-13511489 ] Jay Kreps commented on KAFKA-642: - 60. This isn't an implict conversion. In java the method

[jira] [Updated] (KAFKA-595) Decouple producer side compression from server-side compression.

2012-12-06 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps updated KAFKA-595: Labels: feature (was: feature features) Description: In 0.7 Kafka always appended messages to the

[jira] [Issue Comment Deleted] (KAFKA-595) Decouple producer side compression from server-side compression.

2012-12-06 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps updated KAFKA-595: Comment: was deleted (was: I also think this should be a post 0.8 feature.) > Decouple producer side co

[jira] [Issue Comment Deleted] (KAFKA-595) Decouple producer side compression from server-side compression.

2012-12-06 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps updated KAFKA-595: Comment: was deleted (was: I think saying it is unnecessary is perhaps overstating it. It depends what you

[jira] [Updated] (KAFKA-328) Write unit test for kafka server startup and shutdown API

2012-12-06 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps updated KAFKA-328: Labels: newbie (was: ) > Write unit test for kafka server startup and shutdown API > -

[jira] [Updated] (KAFKA-328) Improve kafka server startup and shutdown API

2012-12-06 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps updated KAFKA-328: Description: Background discussion in KAFKA-320 People often try to embed KafkaServer in an application tha

[jira] [Updated] (KAFKA-328) Write unit test for kafka server startup and shutdown API

2012-12-06 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps updated KAFKA-328: Summary: Write unit test for kafka server startup and shutdown API (was: Improve kafka server startup and

[jira] [Commented] (KAFKA-631) Implement log compaction

2012-12-06 Thread Prashanth Menon (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13511430#comment-13511430 ] Prashanth Menon commented on KAFKA-631: --- Sure, pass the ideas along onto the distribu

[jira] [Resolved] (KAFKA-235) Add a 'log.file.age' configuration parameter to force rotation of log files after they've reached a certain age

2012-12-06 Thread Swapnil Ghike (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Swapnil Ghike resolved KAFKA-235. - Resolution: Fixed Fix Version/s: 0.7.2 0.8 > Add a 'log.file.age' co

[jira] [Comment Edited] (KAFKA-235) Add a 'log.file.age' configuration parameter to force rotation of log files after they've reached a certain age

2012-12-06 Thread Swapnil Ghike (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13511208#comment-13511208 ] Swapnil Ghike edited comment on KAFKA-235 at 12/6/12 7:59 AM: --