Re: Consumer behavior when message exceeds fetch.message.max.bytes

2013-07-31 Thread Jun Rao
server: replica.fetch.max.bytes should be >= message.max.bytes. Otherwise, the follower will get stuck when replicating data from the leader. Thanks, Jun On Wed, Jul 31, 2013 at 10:10 AM, Sam Meder wrote: > I also noticed that there are two properties related to messages size on > the server:

Re: Consumer behavior when message exceeds fetch.message.max.bytes

2013-07-31 Thread Jun Rao
If you look at ConsumerIterator, we will throw MessageSizeTooLargeException to the caller if the fetch size is too small. Thanks, Jun On Wed, Jul 31, 2013 at 8:25 AM, Sam Meder wrote: > We're expecting to occasionally have to deal with pretty large messages > being sent to Kafka. We will of co

Re: [VOTE] Logo runoff vote

2013-07-31 Thread Jun Rao
+1 for 296 Thanks, Jun On Wed, Jul 31, 2013 at 1:34 PM, Jay Kreps wrote: > Okay folks we did a survey on the JIRA (KAFKA-982) and it was more or less > a tie between two logos: > > https://issues.apache.org/jira/secure/attachment/12593545/296.png > https://issues.apache.org/jira/secure/attach

[jira] [Updated] (KAFKA-992) Double Check on Broker Registration to Avoid False NodeExist Exception

2013-07-31 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-992: Attachment: KAFKA-992.v1.patch > Double Check on Broker Registration to Avoid False NodeExist Ex

[jira] [Updated] (KAFKA-649) Cleanup log4j logging

2013-07-31 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-649: Attachment: KAFKA-649.v5.patch One left-bracket mistyping fix. > Cleanup log4j logg

[jira] [Commented] (KAFKA-881) Kafka broker not respecting log.roll.hours

2013-07-31 Thread Dan F (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13725974#comment-13725974 ] Dan F commented on KAFKA-881: - I am not necessarily opposed, but I don't understand what "cross

Re: [VOTE] Logo runoff vote

2013-07-31 Thread Ashish
+1 for 298 On Thu, Aug 1, 2013 at 2:04 AM, Jay Kreps wrote: > Okay folks we did a survey on the JIRA (KAFKA-982) and it was more or less > a tie between two logos: > > https://issues.apache.org/jira/secure/attachment/12593545/296.png > https://issues.apache.org/jira/secure/attachment/12593547/2

Re: [VOTE] Logo runoff vote

2013-07-31 Thread Neha Narkhede
+1 on 296, after looking at the website mockup. Thanks, Neha On Wed, Jul 31, 2013 at 3:09 PM, Benjamin Black wrote: > +1 on 296 > > Not even close > On Jul 31, 2013 1:34 PM, "Jay Kreps" wrote: > > > Okay folks we did a survey on the JIRA (KAFKA-982) and it was more or > less > > a tie between

Re: [VOTE] Logo runoff vote

2013-07-31 Thread Benjamin Black
+1 on 296 Not even close On Jul 31, 2013 1:34 PM, "Jay Kreps" wrote: > Okay folks we did a survey on the JIRA (KAFKA-982) and it was more or less > a tie between two logos: > > https://issues.apache.org/jira/secure/attachment/12593545/296.png > https://issues.apache.org/jira/secure/attachment/12

Re: [VOTE] Logo runoff vote

2013-07-31 Thread Prashanth Menon
+1 296 - Prashanth On Wed, Jul 31, 2013 at 4:36 PM, Joe Stein wrote: > +1 298 > > // Joestein > > On Wed, Jul 31, 2013 at 4:34 PM, Jay Kreps wrote: > > > +1 296 > > > > -Jay > > > > > > On Wed, Jul 31, 2013 at 1:34 PM, Jay Kreps wrote: > > > > > Okay folks we did a survey on the JIRA (KAFKA-

Re: [VOTE] Logo runoff vote

2013-07-31 Thread Joe Stein
+1 298 // Joestein On Wed, Jul 31, 2013 at 4:34 PM, Jay Kreps wrote: > +1 296 > > -Jay > > > On Wed, Jul 31, 2013 at 1:34 PM, Jay Kreps wrote: > > > Okay folks we did a survey on the JIRA (KAFKA-982) and it was more or > less > > a tie between two logos: > > > > https://issues.apache.org/jira/

[VOTE] Logo runoff vote

2013-07-31 Thread Jay Kreps
Okay folks we did a survey on the JIRA (KAFKA-982) and it was more or less a tie between two logos: https://issues.apache.org/jira/secure/attachment/12593545/296.png https://issues.apache.org/jira/secure/attachment/12593547/298.jpeg Personally I like both, but we need to pick one. So let's quickl

Re: [VOTE] Logo runoff vote

2013-07-31 Thread Jay Kreps
+1 296 -Jay On Wed, Jul 31, 2013 at 1:34 PM, Jay Kreps wrote: > Okay folks we did a survey on the JIRA (KAFKA-982) and it was more or less > a tie between two logos: > > https://issues.apache.org/jira/secure/attachment/12593545/296.png > https://issues.apache.org/jira/secure/attachment/1259354

[jira] [Updated] (KAFKA-982) Logo for Kafka

2013-07-31 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps updated KAFKA-982: Attachment: 298-screenshot.png > Logo for Kafka > -- > > Key: KAFKA-982 >

[jira] [Updated] (KAFKA-982) Logo for Kafka

2013-07-31 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps updated KAFKA-982: Attachment: 296-screenshot.png > Logo for Kafka > -- > > Key: KAFKA-982 >

[jira] [Updated] (KAFKA-982) Logo for Kafka

2013-07-31 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps updated KAFKA-982: Attachment: (was: 290-screenshot.png) > Logo for Kafka > -- > > Key: KAF

[jira] [Updated] (KAFKA-982) Logo for Kafka

2013-07-31 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps updated KAFKA-982: Attachment: (was: 285-screenshot.png) > Logo for Kafka > -- > > Key: KAF

[jira] [Updated] (KAFKA-982) Logo for Kafka

2013-07-31 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps updated KAFKA-982: Attachment: 285-screenshot.png > Logo for Kafka > -- > > Key: KAFKA-982 >

[jira] [Updated] (KAFKA-982) Logo for Kafka

2013-07-31 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps updated KAFKA-982: Attachment: 290-screenshot.png > Logo for Kafka > -- > > Key: KAFKA-982 >

[jira] [Updated] (KAFKA-649) Cleanup log4j logging

2013-07-31 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-649: Attachment: KAFKA-649.v4.patch > Cleanup log4j logging > - > >

[jira] [Commented] (KAFKA-649) Cleanup log4j logging

2013-07-31 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13725485#comment-13725485 ] Guozhang Wang commented on KAFKA-649: - Thanks for the comments Jun. 30,31,32,33. Done.

[jira] [Updated] (KAFKA-955) After a leader change, messages sent with ack=0 are lost

2013-07-31 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-955: Attachment: KAFKA-955.v2.patch > After a leader change, messages sent with ack=0 are lost >

[jira] [Commented] (KAFKA-955) After a leader change, messages sent with ack=0 are lost

2013-07-31 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13725452#comment-13725452 ] Guozhang Wang commented on KAFKA-955: - Thanks for the comments Jun. 1,2,3. Done.

Re: Consumer behavior when message exceeds fetch.message.max.bytes

2013-07-31 Thread Sam Meder
I also noticed that there are two properties related to messages size on the server: replica.fetch.max.bytes and message.max.bytes. What happens when replica.fetch.max.bytes is lower than message.max.bytes? Should there even be two properties? /Sam On Jul 31, 2013, at 5:25 PM, Sam Meder wrote

[jira] [Commented] (KAFKA-984) Avoid a full rebalance in cases when a new topic is discovered but container/broker set stay the same

2013-07-31 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13725425#comment-13725425 ] Jun Rao commented on KAFKA-984: --- Thanks for patch v2. Some comments: 20. It seems that we sh

[jira] [Commented] (KAFKA-881) Kafka broker not respecting log.roll.hours

2013-07-31 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13725399#comment-13725399 ] Jun Rao commented on KAFKA-881: --- Thanks for the patch. Would it be simpler to just test that

[jira] [Commented] (KAFKA-967) Use key range in ProducerPerformance

2013-07-31 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13725381#comment-13725381 ] Jun Rao commented on KAFKA-967: --- Thanks for the patch. We probably should only set the key if

[jira] [Updated] (KAFKA-991) Reduce the queue size in hadoop producer

2013-07-31 Thread Swapnil Ghike (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Swapnil Ghike updated KAFKA-991: Description: Currently the queue.size in hadoop producer is 10MB. This means that the KafkaRecordWr

[jira] [Commented] (KAFKA-649) Cleanup log4j logging

2013-07-31 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13725351#comment-13725351 ] Jun Rao commented on KAFKA-649: --- Thanks for patch v3. 30. KafkaApi.readMessageSets(): We don

Consumer behavior when message exceeds fetch.message.max.bytes

2013-07-31 Thread Sam Meder
We're expecting to occasionally have to deal with pretty large messages being sent to Kafka. We will of course set the fetch size appropriately high, but are concerned about the behavior when the message exceeds the fetch size. As far as I can tell the current behavior when a message that is too

[jira] [Commented] (KAFKA-955) After a leader change, messages sent with ack=0 are lost

2013-07-31 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13725324#comment-13725324 ] Jun Rao commented on KAFKA-955: --- Thanks for the patch. Some comments: 1. SocketServer: We sh

Re: fsync management patch

2013-07-31 Thread Prashanth Menon
Hi Jay, I can probably look at this either this week or on the weekend. On Mon, Jul 29, 2013 at 7:37 PM, Jay Kreps wrote: > It would be great if someone could take a look at this: > https://issues.apache.org/jira/browse/KAFKA-615 > > -Jay >

[jira] [Created] (KAFKA-993) Offset Management API is either broken or mis-documented

2013-07-31 Thread Evan Huus (JIRA)
Evan Huus created KAFKA-993: --- Summary: Offset Management API is either broken or mis-documented Key: KAFKA-993 URL: https://issues.apache.org/jira/browse/KAFKA-993 Project: Kafka Issue Type: Bug

[jira] [Updated] (KAFKA-993) Offset Management API is either broken or mis-documented

2013-07-31 Thread Evan Huus (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Evan Huus updated KAFKA-993: Description: I am in the process of building a set of Go client bindings for the new 0.8 protocol (https:/