Re: New Consumer API discussion

I think you are saying both, i.e. if you have committed on a partition it returns you that value but if you haven't it does a remote lookup? Correct. The other argument for making committed batched is that commit() is batched, so there is symmetry. position() and seek() are always in memory chan

Re: New Consumer API discussion

Conceptually, do the position methods only apply to topics you've subscribed to, or do they apply to all topics in the cluster? E.g., could I retrieve or set the committed position of any partition? The positive use case for having access to all partition information would be to setup an active m

Re: New Consumer API discussion

Hey Neha, I actually wasn't proposing the name TopicOffsetPosition, that was just a typo. I meant TopicPartitionOffset, and I was just referencing what was in the javadoc. So to restate my proposal without the typo, using just the existing classes (that naming is a separate question): long posi

[jira] [Resolved] (KAFKA-1263) Snazzy up the README markdown for better visibility on github

[ https://issues.apache.org/jira/browse/KAFKA-1263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Stein resolved KAFKA-1263. -- Resolution: Fixed double committed to 0.8.1 and trunk > Snazzy up the README markdown for better visib

[jira] [Updated] (KAFKA-1264) Make ConsoleProducer compression codec configurable

[ https://issues.apache.org/jira/browse/KAFKA-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kieren Hynd updated KAFKA-1264: --- Attachment: kafka-1264-v1.patch Patch changes '{{compress}}' boolean option for the same '{{compressi

[jira] [Created] (KAFKA-1264) Make ConsoleProducer compression codec configurable

Kieren Hynd created KAFKA-1264: -- Summary: Make ConsoleProducer compression codec configurable Key: KAFKA-1264 URL: https://issues.apache.org/jira/browse/KAFKA-1264 Project: Kafka Issue Type: Imp

Re: New Consumer API discussion

Hi Neha, 6. It seems like #4 can be avoided by using Map> Long> or Map as the argument type. > > How? lastCommittedOffsets() is independent of positions(). I'm not sure I > understood your suggestion. I think of subscription as you're subscribing to a Set of TopicPartitions. Because the argume

[jira] [Commented] (KAFKA-1243) Gradle issues for release

[ https://issues.apache.org/jira/browse/KAFKA-1243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13900925#comment-13900925 ] Clark Breyman commented on KAFKA-1243: -- Perhaps this is worthy of a separate ticket,

Re: Review Request 17006: Check if there are hanging producer threads in stopping all entities

> On Feb. 13, 2014, 10:40 p.m., Joel Koshy wrote: > > system_test/utils/kafka_system_test_utils.py, line 937 > > > > > > This does not seem right (or I may be misunderstanding the issue in this > > jira). > >

Re: Review Request 17006: Check if there are hanging producer threads in stopping all entities

> On Feb. 13, 2014, 10:40 p.m., Joel Koshy wrote: > > system_test/utils/kafka_system_test_utils.py, line 937 > > > > > > This does not seem right (or I may be misunderstanding the issue in this > > jira). > >

Re: Metrics in new producer

Not requiring the client to link Coda/Yammer metrics sounds like a compelling reason to pivot to new interfaces. If that's the agreed direction, I'm hoping that we'd get the choice of backend to provide (e.g. facade on Yammer metrics for those with an investment in that) rather than force the new b

[jira] [Updated] (KAFKA-1260) Integration Test for New Producer Part II: Broker Failure Handling

[ https://issues.apache.org/jira/browse/KAFKA-1260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-1260: - Attachment: KAFKA-1260_2014-02-13_15:14:21.patch > Integration Test for New Producer Part II: Bro

[jira] [Commented] (KAFKA-1260) Integration Test for New Producer Part II: Broker Failure Handling

[ https://issues.apache.org/jira/browse/KAFKA-1260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13900867#comment-13900867 ] Guozhang Wang commented on KAFKA-1260: -- Updated reviewboard https://reviews.apache.or

Re: Review Request 18102: Integration Test for New Producer: Part II

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/18102/ --- (Updated Feb. 13, 2014, 11:16 p.m.) Review request for kafka. Bugs: KAFKA-126

Re: Review Request 18102: Integration Test for New Producer: Part II

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/18102/ --- (Updated Feb. 13, 2014, 11:14 p.m.) Review request for kafka. Bugs: KAFKA-126

Re: Review Request 18102: Integration Test for New Producer: Part II

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/18102/ --- (Updated Feb. 13, 2014, 11:13 p.m.) Review request for kafka. Bugs: KAFKA-126

Review Request 18102: Integration Test for New Producer: Part II

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/18102/ --- Review request for kafka. Bugs: KAFKA-1260 https://issues.apache.org/jira/b

[jira] [Commented] (KAFKA-1260) Integration Test for New Producer Part II: Broker Failure Handling

[ https://issues.apache.org/jira/browse/KAFKA-1260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13900856#comment-13900856 ] Guozhang Wang commented on KAFKA-1260: -- Created reviewboard https://reviews.apache.or

[jira] [Updated] (KAFKA-1260) Integration Test for New Producer Part II: Broker Failure Handling

[ https://issues.apache.org/jira/browse/KAFKA-1260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-1260: - Attachment: KAFKA-1260.patch > Integration Test for New Producer Part II: Broker Failure Handling

Re: Review Request 16718: Patch for KAFKA-1180

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/16718/#review34433 --- core/src/main/scala/kafka/utils/Utils.scala

[jira] [Resolved] (KAFKA-1188) Stale LeaderAndIsr request could be handled by the broker on Controller failover

[ https://issues.apache.org/jira/browse/KAFKA-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede resolved KAFKA-1188. -- Resolution: Fixed Assignee: Guozhang Wang Fixed the minor suggestions and checked in. >

Re: Review Request 17006: Check if there are hanging producer threads in stopping all entities

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/17006/#review34432 --- system_test/utils/kafka_system_test_utils.py

Re: Metrics in new producer

Can we leave metrics and have multiple supported KafkaMetricsGroup implementing a yammer based implementation? ProducerRequestStats with your configured analytics group? On Thu, Feb 13, 2014 at 11:37 AM, Jay Kreps wrote: > I think we discussed the scala/java stuff more fully previously. > Essen

Re: Review Request 18092: Patch for KAFKA-1263

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/18092/#review34428 --- Ship it! Ship It! - Neha Narkhede On Feb. 13, 2014, 7:46 p.m., J

Re: Review Request 16360: Patch for KAFKA-1188

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/16360/#review34406 --- Ship it! Only minor comments core/src/main/scala/kafka/cluster/P

Re: New Consumer API discussion

2. It returns a list of results. But how can you use the list? The only way to use the list is to make a map of tp=>offset and then look up results in this map (or do a for loop over the list for the partition you want). I recommend that if this is an in-memory check we just do one at a time. E.g.

Re: Error Handling on New Producer

Sounds good to me. -Jay On Thu, Feb 13, 2014 at 1:28 PM, Guozhang Wang wrote: > OK, unless someone else is against the second option I will change the code > accordingly. > > > On Thu, Feb 13, 2014 at 10:15 AM, Jay Kreps wrote: > > > Yeah I agree, I think we just have to be very careful that

Re: Error Handling on New Producer

OK, unless someone else is against the second option I will change the code accordingly. On Thu, Feb 13, 2014 at 10:15 AM, Jay Kreps wrote: > Yeah I agree, I think we just have to be very careful that the right errors > are thrown versus stuffed in the future. > > -Jay > > > On Thu, Feb 13, 201

Re: New Consumer API discussion

Hey guys, One thing that bugs me is the lack of symmetric for the different position calls. The way I see it there are two positions we maintain: the fetch position and the last commit position. There are two things you can do to these positions: get the current value or change the current value.

Re: Review Request 16718: Patch for KAFKA-1180

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/16718/#review34422 --- Ship it! Ship It! - Guozhang Wang On Feb. 13, 2014, 8:24 p.m., J

[jira] [Commented] (KAFKA-1180) WhiteList topic filter gets a NullPointerException on complex Regex

[ https://issues.apache.org/jira/browse/KAFKA-1180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13900700#comment-13900700 ] Joe Stein commented on KAFKA-1180: -- version 3,4,5 were some issue with my local branches.

[jira] [Commented] (KAFKA-1180) WhiteList topic filter gets a NullPointerException on complex Regex

[ https://issues.apache.org/jira/browse/KAFKA-1180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13900698#comment-13900698 ] Joe Stein commented on KAFKA-1180: -- Updated reviewboard https://reviews.apache.org/r/1671

[jira] [Updated] (KAFKA-1180) WhiteList topic filter gets a NullPointerException on complex Regex

[ https://issues.apache.org/jira/browse/KAFKA-1180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Stein updated KAFKA-1180: - Attachment: KAFKA-1180_2014-02-13_15:23:28.patch > WhiteList topic filter gets a NullPointerException on

Re: Review Request 16718: Patch for KAFKA-1180

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/16718/ --- (Updated Feb. 13, 2014, 8:24 p.m.) Review request for kafka. Bugs: KAFKA-1180

[jira] [Commented] (KAFKA-1180) WhiteList topic filter gets a NullPointerException on complex Regex

[ https://issues.apache.org/jira/browse/KAFKA-1180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13900697#comment-13900697 ] Joe Stein commented on KAFKA-1180: -- Updated reviewboard https://reviews.apache.org/r/1671

[jira] [Updated] (KAFKA-1180) WhiteList topic filter gets a NullPointerException on complex Regex

[ https://issues.apache.org/jira/browse/KAFKA-1180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Stein updated KAFKA-1180: - Attachment: KAFKA-1180_2014-02-13_15:21:51.patch > WhiteList topic filter gets a NullPointerException on

Re: Review Request 16718: Patch for KAFKA-1180

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/16718/ --- (Updated Feb. 13, 2014, 8:21 p.m.) Review request for kafka. Bugs: KAFKA-1180

Re: Review Request 16718: Patch for KAFKA-1180

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/16718/ --- (Updated Feb. 13, 2014, 8:16 p.m.) Review request for kafka. Bugs: KAFKA-1180

[jira] [Updated] (KAFKA-1180) WhiteList topic filter gets a NullPointerException on complex Regex

[ https://issues.apache.org/jira/browse/KAFKA-1180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Stein updated KAFKA-1180: - Attachment: KAFKA-1180_2014-02-13_15:13:17.patch > WhiteList topic filter gets a NullPointerException on

[jira] [Commented] (KAFKA-1180) WhiteList topic filter gets a NullPointerException on complex Regex

[ https://issues.apache.org/jira/browse/KAFKA-1180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13900685#comment-13900685 ] Joe Stein commented on KAFKA-1180: -- Updated reviewboard https://reviews.apache.org/r/1671

Re: Review Request 16718: Patch for KAFKA-1180

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/16718/ --- (Updated Feb. 13, 2014, 8:13 p.m.) Review request for kafka. Bugs: KAFKA-1180

[jira] [Commented] (KAFKA-1180) WhiteList topic filter gets a NullPointerException on complex Regex

[ https://issues.apache.org/jira/browse/KAFKA-1180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13900661#comment-13900661 ] Joe Stein commented on KAFKA-1180: -- The last patch uploaded just now is rebased on 0.8.1

[jira] [Commented] (KAFKA-1180) WhiteList topic filter gets a NullPointerException on complex Regex

[ https://issues.apache.org/jira/browse/KAFKA-1180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13900660#comment-13900660 ] Joe Stein commented on KAFKA-1180: -- Updated reviewboard https://reviews.apache.org/r/1671

[jira] [Updated] (KAFKA-1180) WhiteList topic filter gets a NullPointerException on complex Regex

[ https://issues.apache.org/jira/browse/KAFKA-1180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Stein updated KAFKA-1180: - Attachment: KAFKA-1180_2014-02-13_14:50:40.patch > WhiteList topic filter gets a NullPointerException on

Re: Review Request 16718: Patch for KAFKA-1180

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/16718/ --- (Updated Feb. 13, 2014, 7:50 p.m.) Review request for kafka. Bugs: KAFKA-1180

Re: Review Request 16718: Patch for KAFKA-1180

> On Feb. 10, 2014, 10:58 p.m., Guozhang Wang wrote: > > core/src/test/scala/unit/kafka/consumer/TopicFilterTest.scala, line 45 > > > > > > Just caught my eye: this does not test anything right? It was like that before

[jira] [Updated] (KAFKA-1263) Snazzy up the README markdown for better visibility on github

[ https://issues.apache.org/jira/browse/KAFKA-1263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Stein updated KAFKA-1263: - Attachment: KAFKA-1263.patch > Snazzy up the README markdown for better visibility on github > --

[jira] [Commented] (KAFKA-1263) Snazzy up the README markdown for better visibility on github

[ https://issues.apache.org/jira/browse/KAFKA-1263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13900653#comment-13900653 ] Joe Stein commented on KAFKA-1263: -- Created reviewboard https://reviews.apache.org/r/1809

Review Request 18092: Patch for KAFKA-1263

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/18092/ --- Review request for kafka. Bugs: KAFKA-1263 https://issues.apache.org/jira/b

[jira] [Resolved] (KAFKA-1248) jars are missing from maven upload that were previously there

[ https://issues.apache.org/jira/browse/KAFKA-1248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Stein resolved KAFKA-1248. -- Resolution: Won't Fix agree with Jun this is not necessary > jars are missing from maven upload that w

[jira] [Updated] (KAFKA-1245) the jar files and pom are not being signed so nexus is failing to publish them

[ https://issues.apache.org/jira/browse/KAFKA-1245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Stein updated KAFKA-1245: - Attachment: KAFKA-1245_2014-02-13_14:41:18.patch > the jar files and pom are not being signed so nexus is

[jira] [Commented] (KAFKA-1245) the jar files and pom are not being signed so nexus is failing to publish them

[ https://issues.apache.org/jira/browse/KAFKA-1245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13900648#comment-13900648 ] Joe Stein commented on KAFKA-1245: -- The artifacts were successfully re-staged to https:/

[jira] [Commented] (KAFKA-1245) the jar files and pom are not being signed so nexus is failing to publish them

[ https://issues.apache.org/jira/browse/KAFKA-1245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13900647#comment-13900647 ] Joe Stein commented on KAFKA-1245: -- Updated reviewboard https://reviews.apache.org/r/1787

[jira] [Created] (KAFKA-1263) Snazzy up the README markdown for better visibility on github

Joe Stein created KAFKA-1263: Summary: Snazzy up the README markdown for better visibility on github Key: KAFKA-1263 URL: https://issues.apache.org/jira/browse/KAFKA-1263 Project: Kafka Issue Ty

Re: Review Request 17879: Patch for KAFKA-1245

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/17879/ --- (Updated Feb. 13, 2014, 7:41 p.m.) Review request for kafka. Bugs: KAFKA-1245

Review Request 18091: Patch for 17879

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/18091/ --- Review request for kafka. Bugs: 17879 https://issues.apache.org/jira/browse

0.8.1 README.md

Hi, was just making some updates into the 0.8.1 README.md for KAFKA-1245 and noticed the license is at the top and the formatting with the markdown is a little clunky as a result of that and some other items. Considering this is the first thing folks see when going to github I thought we can chang

Re: The consumer performance test seems to be totally broken

Synced up offline about this. Makes more sense now. On Thu, Feb 13, 2014 at 10:15 AM, Jay Kreps wrote: > Read my email again :-) > > -Jay > > > On Thu, Feb 13, 2014 at 9:43 AM, Neha Narkhede >wrote: > > > Did you try this on an otherwise empty topic? > > > > Yes and it blocks until --messages

Re: Blocking Behavior of Metadata Refresh on New Producer

Yeah this is a good one to discuss. Current send will block in two conditions: 1. You are beginning a new data batch and have run out of buffer memory and block.on.buffer.full=true 2. Regardless of block.on.buffer.full the first request for each topic will block on fetching metadata that contains

[jira] Subscription: outstanding kafka patches

Issue Subscription Filter: outstanding kafka patches (88 issues) The list of outstanding kafka patches Subscriber: kafka-mailing-list Key Summary KAFKA-1255 Offset in RecordMetadata is Incorrect with New Producer Ack = -1 https://issues.apache.org/jira/browse/KAFKA-1255 KAFKA-

Blocking Behavior of Metadata Refresh on New Producer

I think we have discussed this issue before but I just want to bring it up again to get confirmed. Currently when the send() call is triggered, first of all the cluster information will be fetched for the specified topic, and if such information is not available yet the send call will block on ref

[jira] [Resolved] (KAFKA-1261) Make metadata refresh configurable

[ https://issues.apache.org/jira/browse/KAFKA-1261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps resolved KAFKA-1261. -- Resolution: Fixed > Make metadata refresh configurable > -- > >

[jira] [Resolved] (KAFKA-1257) New producer runs into out of order producer responses

[ https://issues.apache.org/jira/browse/KAFKA-1257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps resolved KAFKA-1257. -- Resolution: Fixed > New producer runs into out of order producer responses > --

Re: Error Handling on New Producer

Yeah I agree, I think we just have to be very careful that the right errors are thrown versus stuffed in the future. -Jay On Thu, Feb 13, 2014 at 9:24 AM, Neha Narkhede wrote: > Alternately we could throw user errors from send. These you wouldn't need > to catch because they should not happen (

Re: The consumer performance test seems to be totally broken

Read my email again :-) -Jay On Thu, Feb 13, 2014 at 9:43 AM, Neha Narkhede wrote: > Did you try this on an otherwise empty topic? > > Yes and it blocks until --messages are read. But again, my interpretation > of that config has been - "read exactly --messages # of messages, not less > not mo

Re: The consumer performance test seems to be totally broken

Did you try this on an otherwise empty topic? Yes and it blocks until --messages are read. But again, my interpretation of that config has been - "read exactly --messages # of messages, not less not more". Thanks, Neha On Mon, Feb 10, 2014 at 9:31 AM, Jay Kreps wrote: > Hmm, I could be wrong

Re: Error Handling on New Producer

Alternately we could throw user errors from send. These you wouldn't need to catch because they should not happen (and if they do it is a bug, not something you should really catch and handle). The only concern here is that we have to be very careful to ensure we categorize each exception correctly

Re: Metrics in new producer

I think we discussed the scala/java stuff more fully previously. Essentially the client is embedded everywhere. Scala is very incompatible with itself so this makes it very hard to use for people using anything else in scala. Also Scala stack traces are very confusing. Basically we thought plain ja

Re: Kafka http endpoint

This is really cool! -Jay On Wed, Feb 12, 2014 at 1:36 PM, Sasha Klizhentas < sasha.klizhen...@mailgunhq.com> wrote: > JFYI: > > Kafka-http endpoint + docs as a separate project: > > https://github.com/mailgun/kafka-http > > Thanks, > Sasha >

Re: Error Handling on New Producer

Some context on this. Originally (i.e. in the first patch) I threw any exception I could including all class B exceptions directly from the send() call and only threw api exceptions with the future. My concern with this approach was just the complexity of the error handling. You end up needing to

Re: Metrics in new producer

The issue with metrics like message size is that max_mesage_size is a server side config, not a client side one. Even if it is on the client side, different clients can have different distributions (e.g., uniform in the whole range, dense in the low, mid or high range). So, I am not sure there is a

Re: Metrics in new producer

Jay, pretty impressive how you just write a 'quick version' like that :) Not to get off-topic but why didn't you write this in scala? On Wed, Feb 12, 2014 at 6:54 PM, Joel Koshy wrote: > I have not had a chance to review the new metrics code and its > features carefully (apart from your write-