Re: [DISCUSSION] adding the serializer api back to the new java producer

Jun, Thanks for summarizing this - it helps confirm for me that I did not misunderstand anything in this thread so far; and that I disagree with the premise that the steps in using the current byte-oriented API is cumbersome or inflexible. It involves instantiating the K-V serializers in code (as

[jira] [Commented] (KAFKA-1499) Broker-side compression configuration

[ https://issues.apache.org/jira/browse/KAFKA-1499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14249536#comment-14249536 ] Joel Koshy commented on KAFKA-1499: --- Thank you for the ping, and sorry about the delayed

[jira] [Commented] (KAFKA-1819) Cleaner gets confused about deleted and re-created topics

[ https://issues.apache.org/jira/browse/KAFKA-1819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14249530#comment-14249530 ] Joel Koshy commented on KAFKA-1819: --- [~gwenshap] that's right - that's what I meant by "

Re: Review Request 29072: rebased for 0.8.2 branch

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29072/#review65296 --- I found the same review comments I had on trunk. Can you update this

[jira] [Commented] (KAFKA-1070) Auto-assign node id

[ https://issues.apache.org/jira/browse/KAFKA-1070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14249492#comment-14249492 ] Neha Narkhede commented on KAFKA-1070: -- Thanks for the updated patch [~sriharsha]. Al

Build failed in Jenkins: Kafka-trunk #355

See Changes: [neha.narkhede] KAFKA-742: Existing directories under the Kafka data directory without any data cause process to not start; reviewed by Neha Narkhede -- [...truncated 1516 lines...]

Re: Review Request 23702: Patch for KAFKA-1070

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23702/#review65293 --- core/src/main/scala/kafka/common/GenerateBrokerIdException.scala

Re: Review Request 23702: Patch for KAFKA-1070

> On Dec. 17, 2014, 4:58 a.m., Neha Narkhede wrote: > > Overall, this looks good. How about adding some unit tests before checking > > it in? Please ignore this. I was looking at only part of the patch. - Neha --- This is an automatica

Re: Review Request 23702: Patch for KAFKA-1070

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23702/#review65289 --- Overall, this looks good. How about adding some unit tests before ch

[jira] [Updated] (KAFKA-742) Existing directories under the Kafka data directory without any data cause process to not start

[ https://issues.apache.org/jira/browse/KAFKA-742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-742: Resolution: Fixed Status: Resolved (was: Patch Available) Thanks for the patch, [~singhashis

[jira] [Commented] (KAFKA-1819) Cleaner gets confused about deleted and re-created topics

[ https://issues.apache.org/jira/browse/KAFKA-1819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14249451#comment-14249451 ] Gwen Shapira commented on KAFKA-1819: - The recovery checkpoints are currently handled

Re: Review Request 29030: KAFKA-742: Existing directories under the Kafka data directory without any data cause process to not start

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29030/#review65287 --- Ship it! Ship It! - Neha Narkhede On Dec. 16, 2014, 6:27 p.m., A

[jira] [Commented] (KAFKA-1819) Cleaner gets confused about deleted and re-created topics

[ https://issues.apache.org/jira/browse/KAFKA-1819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14249443#comment-14249443 ] Neha Narkhede commented on KAFKA-1819: -- [~jjkoshy] Shouldn't we remove the deleted to

[jira] [Comment Edited] (KAFKA-1819) Cleaner gets confused about deleted and re-created topics

[ https://issues.apache.org/jira/browse/KAFKA-1819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14249443#comment-14249443 ] Neha Narkhede edited comment on KAFKA-1819 at 12/17/14 4:08 AM:

[jira] [Comment Edited] (KAFKA-1788) producer record can stay in RecordAccumulator forever if leader is no available

[ https://issues.apache.org/jira/browse/KAFKA-1788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14249266#comment-14249266 ] Bhavesh Mistry edited comment on KAFKA-1788 at 12/17/14 1:26 AM: ---

[jira] [Commented] (KAFKA-1788) producer record can stay in RecordAccumulator forever if leader is no available

[ https://issues.apache.org/jira/browse/KAFKA-1788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14249266#comment-14249266 ] Bhavesh Mistry commented on KAFKA-1788: --- [~jkreps], Can we just take quick look at

[jira] [Commented] (KAFKA-1788) producer record can stay in RecordAccumulator forever if leader is no available

[ https://issues.apache.org/jira/browse/KAFKA-1788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14249255#comment-14249255 ] Jay Kreps commented on KAFKA-1788: -- Currently the producer supports either blocking or dr

[jira] [Created] (KAFKA-1822) Add "echo" request

Gwen Shapira created KAFKA-1822: --- Summary: Add "echo" request Key: KAFKA-1822 URL: https://issues.apache.org/jira/browse/KAFKA-1822 Project: Kafka Issue Type: Improvement Components:

[jira] [Commented] (KAFKA-1788) producer record can stay in RecordAccumulator forever if leader is no available

[ https://issues.apache.org/jira/browse/KAFKA-1788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14249131#comment-14249131 ] Ewen Cheslack-Postava commented on KAFKA-1788: -- [~bpot] that sounds right, I'

[jira] [Commented] (KAFKA-1479) Logs filling up while Kafka ReplicaFetcherThread tries to retrieve partition info for deleted topics

[ https://issues.apache.org/jira/browse/KAFKA-1479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14249071#comment-14249071 ] Arup Malakar commented on KAFKA-1479: - For people who may stumble upon this JIRA, the

[jira] [Commented] (KAFKA-1806) broker can still expose uncommitted data to a consumer

[ https://issues.apache.org/jira/browse/KAFKA-1806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14248993#comment-14248993 ] lokesh Birla commented on KAFKA-1806: - This problem occurs multiple times in server.lo

[jira] [Commented] (KAFKA-1806) broker can still expose uncommitted data to a consumer

[ https://issues.apache.org/jira/browse/KAFKA-1806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14248764#comment-14248764 ] Evan Huus commented on KAFKA-1806: -- Sarama client maintainer here (via https://github.co

Re: API Annotations

Hey Gwen, We discussed this a bit about this when starting on the new clients. We were super sloppy about this in initial Kafka development--single jar, no real differentiation between public and private apis. The plan was something like the following: 1. Start to consider this with the new clie

Re: [DISCUSSION] adding the serializer api back to the new java producer

Joel, With a byte array interface, of course there is nothing that one can't do. However, the real question is that whether we want to encourage people to use it this way or not. Being able to flow just bytes is definitely easier to get started. That's why many early adopters choose to do it that

Re: Review Request 24704: Patch for KAFKA-1499

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24704/#review65223 --- Ship it! Ship It! - Jonathan Creasy On Dec. 16, 2014, 5:10 p.m.,

Re: Review Request 24704: Patch for KAFKA-1499

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24704/#review65221 --- core/src/main/scala/kafka/log/Log.scala

[jira] [Updated] (KAFKA-742) Existing directories under the Kafka data directory without any data cause process to not start

[ https://issues.apache.org/jira/browse/KAFKA-742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Kumar Singh updated KAFKA-742: - Attachment: KAFKA-742.1.patch Makes sense. Updated patch. > Existing directories under the

Re: Review Request 29030: KAFKA-742: Existing directories under the Kafka data directory without any data cause process to not start

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29030/ --- (Updated Dec. 16, 2014, 6:27 p.m.) Review request for kafka. Changes ---

Re: API Annotations

This is definitely an area we can improve upon, thanks for bringing it up Gwen. We should also decide on structure for how we publish too that has not been something I have been consistent about for each release that should be standard. We can publish to SVN but I always feel like drift happens th

[jira] [Commented] (KAFKA-1819) Cleaner gets confused about deleted and re-created topics

[ https://issues.apache.org/jira/browse/KAFKA-1819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14248602#comment-14248602 ] Gwen Shapira commented on KAFKA-1819: - Thanks for pointing the similar issue [~jjkoshy

[jira] [Commented] (KAFKA-1070) Auto-assign node id

[ https://issues.apache.org/jira/browse/KAFKA-1070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14248599#comment-14248599 ] Sriharsha Chintalapani commented on KAFKA-1070: --- [~nehanarkhede] Can you ple

[jira] [Commented] (KAFKA-1646) Improve consumer read performance for Windows

[ https://issues.apache.org/jira/browse/KAFKA-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14248597#comment-14248597 ] Sriharsha Chintalapani commented on KAFKA-1646: --- Thanks [~qixia] for the rev

Re: Review Request 29091: Patch for KAFKA-1646

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29091/#review65216 --- Ship it! Overall it looks good to me. - Sriharsha Chintalapani O

API Annotations

Hi, Kafka has public APIs in Java and Scala, intended for use by external developers. In addition, Kafka also exposes many public methods that are intended to use within Kafka but are not intended to be called by external developers. Also, some of the external APIs are less stable than others (the

[jira] [Commented] (KAFKA-1499) Broker-side compression configuration

[ https://issues.apache.org/jira/browse/KAFKA-1499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14248520#comment-14248520 ] Manikumar Reddy commented on KAFKA-1499: [~jjkoshy] can you review this patch? >

Re: Review Request 24704: Patch for KAFKA-1499

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24704/ --- (Updated Dec. 16, 2014, 5:10 p.m.) Review request for kafka. Bugs: KAFKA-1499

[jira] [Commented] (KAFKA-1499) Broker-side compression configuration

[ https://issues.apache.org/jira/browse/KAFKA-1499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14248503#comment-14248503 ] Manikumar Reddy commented on KAFKA-1499: Updated reviewboard https://reviews.apach

[jira] [Updated] (KAFKA-1499) Broker-side compression configuration

[ https://issues.apache.org/jira/browse/KAFKA-1499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar Reddy updated KAFKA-1499: --- Attachment: KAFKA-1499_2014-12-16_22:39:10.patch > Broker-side compression configuration > ---

Re: Review Request 25995: Patch for KAFKA-1650

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25995/ --- (Updated Dec. 16, 2014, 4:03 p.m.) Review request for kafka. Bugs: KAFKA-1650

[jira] [Commented] (KAFKA-1650) Mirror Maker could lose data on unclean shutdown.

[ https://issues.apache.org/jira/browse/KAFKA-1650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14248407#comment-14248407 ] Jiangjie Qin commented on KAFKA-1650: - Updated reviewboard https://reviews.apache.org/

[jira] [Updated] (KAFKA-1650) Mirror Maker could lose data on unclean shutdown.

[ https://issues.apache.org/jira/browse/KAFKA-1650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiangjie Qin updated KAFKA-1650: Attachment: KAFKA-1650_2014-12-16_08:03:45.patch > Mirror Maker could lose data on unclean shutdown.

Optimistic locking

I'm trying to design a system that uses Kafka as its primary data store by persisting immutable events into a topic and keeping a secondary index in another data store. The secondary index would store the "entities". Each event would pertain to some "entity", e.g. a user, and those entities are sto

[jira] [Commented] (KAFKA-1646) Improve consumer read performance for Windows

[ https://issues.apache.org/jira/browse/KAFKA-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14247936#comment-14247936 ] Qianlin Xia commented on KAFKA-1646: Updated reviewboard https://reviews.apache.org/r/

[jira] [Updated] (KAFKA-1646) Improve consumer read performance for Windows

[ https://issues.apache.org/jira/browse/KAFKA-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qianlin Xia updated KAFKA-1646: --- Attachment: KAFKA-1646_20141216_163008.patch > Improve consumer read performance for Windows > ---

Re: Review Request 29091: Patch for KAFKA-1646

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29091/ --- (Updated Dec. 16, 2014, 8:30 a.m.) Review request for kafka. Bugs: KAFKA-1646

Re: Review Request 29091: Patch for KAFKA-1646

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29091/ --- (Updated Dec. 16, 2014, 8:22 a.m.) Review request for kafka. Bugs: KAFKA-1646

Re: Review Request 29091: Patch for KAFKA-1646

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29091/ --- (Updated Dec. 16, 2014, 8:18 a.m.) Review request for kafka. Bugs: KAFKA-1646

[jira] [Commented] (KAFKA-1819) Cleaner gets confused about deleted and re-created topics

[ https://issues.apache.org/jira/browse/KAFKA-1819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14247925#comment-14247925 ] Joel Koshy commented on KAFKA-1819: --- This is a similar stack-trace to KAFKA-1641 but tha

Re: Review Request 29091: Patch for KAFKA-1646

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29091/ --- (Updated Dec. 16, 2014, 8:13 a.m.) Review request for kafka. Bugs: KAFKA-1646