Re: Question about offsets

2013-01-30 Thread David Arthur
Thanks, Jay. Makes sense +1 for documentation. I think in general the wiki is a bit unorganized, makes it hard to know what is the "current" info vs old proposals. -David On 1/30/13 12:01 AM, Jay Kreps wrote: Yes, offsets are now logical 0,1,2,3... Some details on this change are here: http

Troubles with compressed message set

2013-01-30 Thread David Arthur
I'm working on a client and I'm running into difficulties with compressed message sets. I am able to produce them fine, but when I go to fetch them, things seem strange. I am sending a message who's value is a compressed message set. The inner message set contains a single message. Specificall

[jira] [Commented] (KAFKA-683) Fix correlation ids in all requests sent to kafka

2013-01-30 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13566563#comment-13566563 ] Jun Rao commented on KAFKA-683: --- Yes, the main changes were made by me in kafka-203. If we wa

[jira] [Created] (KAFKA-743) PreferredReplicaLeaderElectionCommand has command line error

2013-01-30 Thread Jun Rao (JIRA)
Jun Rao created KAFKA-743: - Summary: PreferredReplicaLeaderElectionCommand has command line error Key: KAFKA-743 URL: https://issues.apache.org/jira/browse/KAFKA-743 Project: Kafka Issue Type: Bug

[jira] [Updated] (KAFKA-743) PreferredReplicaLeaderElectionCommand has command line error

2013-01-30 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-743: -- Status: Patch Available (was: Open) > PreferredReplicaLeaderElectionCommand has command line error > --

[jira] [Updated] (KAFKA-743) PreferredReplicaLeaderElectionCommand has command line error

2013-01-30 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-743: -- Attachment: kafka-743.patch Attach a fix. > PreferredReplicaLeaderElectionCommand has command l

Re: Troubles with compressed message set

2013-01-30 Thread Jay Kreps
Hi David, MessageSets aren't size delimited because that format is shared with producer/consumer and the kafka log itself. The log itself is just a big sequence of messages and any subset that begins and ends at valid message boundaries is a valid message set. This means that message sets are size

[jira] [Commented] (KAFKA-743) PreferredReplicaLeaderElectionCommand has command line error

2013-01-30 Thread Maxime Brugidou (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13566574#comment-13566574 ] Maxime Brugidou commented on KAFKA-743: --- Same thing for CheckReassignmentStatus and R

Re: order guarantee failure: better/best effort?

2013-01-30 Thread Jay Kreps
Hey Ben, Sorry to have dropped this, somehow the holidays wiped it from my memory. Long story short, you were right (probably not surprising to you). Neha reproduced this problem as part of another fix and has a patch. We will get the fix into 0.8 in the next few days: https://issues.apache.org/ji

[jira] [Created] (KAFKA-744) Bad response time for fetch request

2013-01-30 Thread Danny Yeshurun (JIRA)
Danny Yeshurun created KAFKA-744: Summary: Bad response time for fetch request Key: KAFKA-744 URL: https://issues.apache.org/jira/browse/KAFKA-744 Project: Kafka Issue Type: Bug R

[jira] [Commented] (KAFKA-683) Fix correlation ids in all requests sent to kafka

2013-01-30 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13566599#comment-13566599 ] Jay Kreps commented on KAFKA-683: - The design was that the network layer is fully generic a

[jira] [Updated] (KAFKA-713) Update Hadoop producer for Kafka 0.8 changes

2013-01-30 Thread Sam Shah (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Shah updated KAFKA-713: --- Attachment: (was: KAFKA-713.patch) > Update Hadoop producer for Kafka 0.8 changes > --

[jira] [Updated] (KAFKA-713) Update Hadoop producer for Kafka 0.8 changes

2013-01-30 Thread Sam Shah (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Shah updated KAFKA-713: --- Attachment: KAFKA-713.patch Updated patch with Neha's suggestions. I default to GZip compression with the Had

[jira] [Updated] (KAFKA-695) Broker shuts down due to attempt to read a closed index file

2013-01-30 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-695: -- Attachment: kafka-695_followup.patch Attach a follow up patch. 1. Make ExpiredRequestReaper a non-daemon thread

[jira] [Reopened] (KAFKA-695) Broker shuts down due to attempt to read a closed index file

2013-01-30 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao reopened KAFKA-695: --- reopen it to deal with followup issue.s > Broker shuts down due to attempt to read a closed index

[jira] [Updated] (KAFKA-736) Add an option to the 0.8 producer to mimic 0.7 producer behavior

2013-01-30 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-736: Attachment: kafka-736-v2.patch Thanks for the review, these are good points. I see your point about

[jira] [Commented] (KAFKA-736) Add an option to the 0.8 producer to mimic 0.7 producer behavior

2013-01-30 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13566883#comment-13566883 ] Jay Kreps commented on KAFKA-736: - 1. It would be good to remove getShutdownReceive() too.

Re: Troubles with compressed message set

2013-01-30 Thread David Arthur
Jay, Figured it out. In Message.scala, CurrentMagicValue is set to 0; should be 2. This was causing my client to attempt to decode it as a v0 message. Changing the value to 2 solved my problem. Seems like a trivial change, so I'll let you decided if you want a Jira or not. From my previous

Re: Troubles with compressed message set

2013-01-30 Thread David Arthur
On 1/30/13 11:18 AM, Jay Kreps wrote: Hi David, MessageSets aren't size delimited because that format is shared with producer/consumer and the kafka log itself. The log itself is just a big sequence of messages and any subset that begins and ends at valid message boundaries is a valid message s

Re: Troubles with compressed message set

2013-01-30 Thread Jay Kreps
Ah, yes, the decision since we were making a breaking change to the protocol to just reset all the versions to 0. But if you are trying to handle both cases from the same code that will be hard. -Jay On Wed, Jan 30, 2013 at 12:59 PM, David Arthur wrote: > Jay, > > Figured it out. > > In Messag

Re: Troubles with compressed message set

2013-01-30 Thread David Arthur
On 1/30/13 4:17 PM, Jay Kreps wrote: Ah, yes, the decision since we were making a breaking change to the protocol to just reset all the versions to 0. Excellent, I shall purge my 0.7x compatible code at once :) But if you are trying to handle both cases from the same code that will be hard. -

kafka pull request: KAFKA-294

2013-01-30 Thread fsaintjacques
GitHub user fsaintjacques opened a pull request: https://github.com/apache/kafka/pull/2 KAFKA-294 This issue can be caused by a non-existing path but also a misunderstanding from the config file. A short example will help the user. You can merge this pull request into a Git reposit

[jira] [Created] (KAFKA-745) Remove getShutdownReceive() and other kafka specific code from the RequestChannel

2013-01-30 Thread Neha Narkhede (JIRA)
Neha Narkhede created KAFKA-745: --- Summary: Remove getShutdownReceive() and other kafka specific code from the RequestChannel Key: KAFKA-745 URL: https://issues.apache.org/jira/browse/KAFKA-745 Project: