[jira] [Updated] (KAFKA-1791) Corrupt index after safe shutdown and restart

[ https://issues.apache.org/jira/browse/KAFKA-1791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vamsi Subhash Achanta updated KAFKA-1791: - Attachment: 0233.log Attached the log file. > Corrupt index after

[jira] [Commented] (KAFKA-1461) Replica fetcher thread does not implement any back-off behavior

[ https://issues.apache.org/jira/browse/KAFKA-1461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14225799#comment-14225799 ] Nicolae Marasoiu commented on KAFKA-1461: - I agree to give to someone else, did no

[jira] [Commented] (KAFKA-1507) Using GetOffsetShell against non-existent topic creates the topic unintentionally

[ https://issues.apache.org/jira/browse/KAFKA-1507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14225727#comment-14225727 ] Sriharsha Chintalapani commented on KAFKA-1507: --- [~junrao] I've seen few us

Re: Review Request 23702: Patch for KAFKA-1070

> On Nov. 23, 2014, 8:35 p.m., Neha Narkhede wrote: > > core/src/main/scala/kafka/utils/ZkUtils.scala, line 714 > > > > > > Why not use persistent sequential nodes instead? if I used createPersistentSequential it keep

[jira] [Commented] (KAFKA-1070) Auto-assign node id

[ https://issues.apache.org/jira/browse/KAFKA-1070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14225725#comment-14225725 ] Sriharsha Chintalapani commented on KAFKA-1070: --- [~nehanarkhede] Thanks for

[jira] [Commented] (KAFKA-1070) Auto-assign node id

[ https://issues.apache.org/jira/browse/KAFKA-1070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14225723#comment-14225723 ] Sriharsha Chintalapani commented on KAFKA-1070: --- Updated reviewboard https:/

[jira] [Updated] (KAFKA-1070) Auto-assign node id

[ https://issues.apache.org/jira/browse/KAFKA-1070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sriharsha Chintalapani updated KAFKA-1070: -- Attachment: KAFKA-1070_2014-11-25_20:29:37.patch > Auto-assign node id > ---

Re: Review Request 23702: Patch for KAFKA-1070

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23702/ --- (Updated Nov. 26, 2014, 4:29 a.m.) Review request for kafka. Bugs: KAFKA-1070

Re: Does Kafka Producer service ?

If you want to separate your main application from producer logic you can use or write something similar to https://github.com/mozilla-metrics/bagheera . Basically have a service that provides an REST api which your main application can call and has a Kafkaproducer to write to kafka. -Harsha On

Re: Does Kafka Producer service ?

Assuming that you use a single producer in the async mode, the Kafka overhead should be limited to a single thread. Using a cheaper compression codec such as snappy will also help reducing the CPU load. Thanks, Jun On Tue, Nov 25, 2014 at 12:02 PM, Krishna Raj wrote: > Hi Jun, > > Thanks for r

Re: Review Request 28121: Patch for KAFKA-1780

> On Nov. 25, 2014, 12:26 a.m., Neha Narkhede wrote: > > core/src/main/scala/kafka/utils/IteratorTemplate.scala, line 46 > > > > > > The change is larger than I expected it to be. What is the reason we > > couldn't cha

[jira] [Commented] (KAFKA-1419) cross build for scala 2.11

[ https://issues.apache.org/jira/browse/KAFKA-1419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14225564#comment-14225564 ] Jun Rao commented on KAFKA-1419: This is being tracked in KAFKA-1054. > cross build for s

[jira] [Commented] (KAFKA-1791) Corrupt index after safe shutdown and restart

[ https://issues.apache.org/jira/browse/KAFKA-1791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14225560#comment-14225560 ] Jun Rao commented on KAFKA-1791: Could you attach the .log file? > Corrupt index after sa

[jira] [Commented] (KAFKA-1748) Decouple system test cluster resources definition from service definitions

[ https://issues.apache.org/jira/browse/KAFKA-1748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14225549#comment-14225549 ] Neha Narkhede commented on KAFKA-1748: -- [~guozhang], [~jjkoshy] Any chance someone at

[jira] [Updated] (KAFKA-1788) producer record can stay in RecordAccumulator forever if leader is no available

[ https://issues.apache.org/jira/browse/KAFKA-1788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1788: - Affects Version/s: (was: 0.8.3) 0.8.2 > producer record can stay in Rec

[jira] [Commented] (KAFKA-1646) Improve consumer read performance for Windows

[ https://issues.apache.org/jira/browse/KAFKA-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14225536#comment-14225536 ] Jun Rao commented on KAFKA-1646: The bug fix patch forces log recovery on clean shutdown.

[jira] [Updated] (KAFKA-1788) producer record can stay in RecordAccumulator forever if leader is no available

[ https://issues.apache.org/jira/browse/KAFKA-1788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1788: - Fix Version/s: 0.8.3 > producer record can stay in RecordAccumulator forever if leader is no > av

[jira] [Updated] (KAFKA-1788) producer record can stay in RecordAccumulator forever if leader is no available

[ https://issues.apache.org/jira/browse/KAFKA-1788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1788: - Component/s: producer > producer record can stay in RecordAccumulator forever if leader is no >

[jira] [Updated] (KAFKA-1788) producer record can stay in RecordAccumulator forever if leader is no available

[ https://issues.apache.org/jira/browse/KAFKA-1788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1788: - Labels: newbie++ (was: ) > producer record can stay in RecordAccumulator forever if leader is no

[jira] [Updated] (KAFKA-1783) Missing slash in documentation for the Zookeeper paths in ZookeeperConsumerConnector

[ https://issues.apache.org/jira/browse/KAFKA-1783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1783: - Reviewer: Guozhang Wang [~guozhang] Assigning to you for review. Feel free to reassign. > Missing

[jira] [Commented] (KAFKA-1751) handle "broker not exists" and "topic not exists" scenarios

[ https://issues.apache.org/jira/browse/KAFKA-1751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14225528#comment-14225528 ] Neha Narkhede commented on KAFKA-1751: -- [~Dmitry Pekar] I've reviewed your latest pat

[jira] [Updated] (KAFKA-1796) Sanity check partition command line tools

[ https://issues.apache.org/jira/browse/KAFKA-1796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1796: - Fix Version/s: (was: 0.0.9) 0.8.3 > Sanity check partition command line too

[jira] [Updated] (KAFKA-1796) Sanity check partition command line tools

[ https://issues.apache.org/jira/browse/KAFKA-1796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1796: - Labels: newbie (was: ) > Sanity check partition command line tools >

[jira] [Commented] (KAFKA-1796) Sanity check partition command line tools

[ https://issues.apache.org/jira/browse/KAFKA-1796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14225520#comment-14225520 ] Neha Narkhede commented on KAFKA-1796: -- [~guozhang] Can you add more details to the d

[jira] [Commented] (KAFKA-1790) Remote controlled shutdown was removed

[ https://issues.apache.org/jira/browse/KAFKA-1790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14225519#comment-14225519 ] Jun Rao commented on KAFKA-1790: The way that the controlled shutdown works now requires c

Re: Review Request 28108: KAFKA-1664: Kafka does not properly parse multiple ZK nodes with non-root chroot

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28108/#review63083 --- Can you add a few test cases? - Neha Narkhede On Nov. 23, 2014, 5

[jira] [Commented] (KAFKA-1752) add --replace-broker option

[ https://issues.apache.org/jira/browse/KAFKA-1752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14225512#comment-14225512 ] Neha Narkhede commented on KAFKA-1752: -- [~Dmitry Pekar], It is true that we do not ne

[jira] [Commented] (KAFKA-1774) REPL and Shell Client for Admin Message RQ/RP

[ https://issues.apache.org/jira/browse/KAFKA-1774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14225497#comment-14225497 ] Jun Rao commented on KAFKA-1774: We probably don't want the client library to drag in depe

[jira] [Commented] (KAFKA-1772) Add an Admin message type for request response

[ https://issues.apache.org/jira/browse/KAFKA-1772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14225475#comment-14225475 ] Jun Rao commented on KAFKA-1772: Thanks for the patch. A few comments on the request/respo

[jira] [Created] (KAFKA-1800) KafkaException was not recorded at the per-topic metrics

Guozhang Wang created KAFKA-1800: Summary: KafkaException was not recorded at the per-topic metrics Key: KAFKA-1800 URL: https://issues.apache.org/jira/browse/KAFKA-1800 Project: Kafka Issue

[jira] [Updated] (KAFKA-1723) make the metrics name in new producer more standard

[ https://issues.apache.org/jira/browse/KAFKA-1723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-1723: --- Priority: Blocker (was: Major) Fix Version/s: (was: 0.8.3) 0.8.2 Since changi

[jira] [Updated] (KAFKA-1799) ProducerConfig.METRIC_REPORTER_CLASSES_CONFIG doesn't work

[ https://issues.apache.org/jira/browse/KAFKA-1799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-1799: --- Priority: Blocker (was: Major) Affects Version/s: 0.8.2 Fix Version/s: 0.8.2 Marking thi

[jira] [Commented] (KAFKA-1799) ProducerConfig.METRIC_REPORTER_CLASSES_CONFIG doesn't work

[ https://issues.apache.org/jira/browse/KAFKA-1799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14225374#comment-14225374 ] Jun Rao commented on KAFKA-1799: This is because ConfigDef.parseType() assumes that if the

[jira] [Created] (KAFKA-1799) ProducerConfig.METRIC_REPORTER_CLASSES_CONFIG doesn't work

Jun Rao created KAFKA-1799: -- Summary: ProducerConfig.METRIC_REPORTER_CLASSES_CONFIG doesn't work Key: KAFKA-1799 URL: https://issues.apache.org/jira/browse/KAFKA-1799 Project: Kafka Issue Type: Bug

[jira] [Commented] (KAFKA-1794) Make config and config defaults accessible to clients

[ https://issues.apache.org/jira/browse/KAFKA-1794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14225349#comment-14225349 ] Guozhang Wang commented on KAFKA-1794: -- One more note is that the new producer config

[jira] [Updated] (KAFKA-1667) topic-level configuration not validated

[ https://issues.apache.org/jira/browse/KAFKA-1667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-1667: --- Resolution: Fixed Fix Version/s: 0.8.3 Assignee: Dmytro Kostiuchenko Status: Resolve

[jira] [Created] (KAFKA-1798) ConfigDef.parseType() should throw exception on invalid boolean value

Jun Rao created KAFKA-1798: -- Summary: ConfigDef.parseType() should throw exception on invalid boolean value Key: KAFKA-1798 URL: https://issues.apache.org/jira/browse/KAFKA-1798 Project: Kafka Issu

[jira] [Commented] (KAFKA-1273) Brokers should make sure replica.fetch.max.bytes >= message.max.bytes

[ https://issues.apache.org/jira/browse/KAFKA-1273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14225322#comment-14225322 ] Sriharsha Chintalapani commented on KAFKA-1273: --- [~nehanarkhede] Yes. workin

[jira] [Updated] (KAFKA-1273) Brokers should make sure replica.fetch.max.bytes >= message.max.bytes

[ https://issues.apache.org/jira/browse/KAFKA-1273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1273: - [~sriharsha] Do you want to take a look? > Brokers should make sure replica.fetch.max.bytes >= messag

[jira] [Commented] (KAFKA-1654) Provide a way to override server configuration from command line

[ https://issues.apache.org/jira/browse/KAFKA-1654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14225282#comment-14225282 ] Neha Narkhede commented on KAFKA-1654: -- Thanks for your contributions, [~jarcec]. Loo

[jira] [Commented] (KAFKA-1273) Brokers should make sure replica.fetch.max.bytes >= message.max.bytes

[ https://issues.apache.org/jira/browse/KAFKA-1273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14225215#comment-14225215 ] Patrick Lucas commented on KAFKA-1273: -- I ran into this on 0.8.1.1. Perhaps if you o

[jira] [Commented] (KAFKA-1461) Replica fetcher thread does not implement any back-off behavior

[ https://issues.apache.org/jira/browse/KAFKA-1461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14225166#comment-14225166 ] Joe Stein commented on KAFKA-1461: -- [~n...@museglobal.ro] are you working on this patch?

[jira] [Updated] (KAFKA-1461) Replica fetcher thread does not implement any back-off behavior

[ https://issues.apache.org/jira/browse/KAFKA-1461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Stein updated KAFKA-1461: - Fix Version/s: 0.8.3 > Replica fetcher thread does not implement any back-off behavior > -

Re: Does Kafka Producer service ?

Hi Jun, Thanks for replying back on this. Appreciated. I do understand that the Kafka Client just needs a protocol compatibility with the Application which is producing the messages. To clarity a bit more: I witnessed a scenario where a large scale website uses the Kafka Library in their Web Ap

[SECURITY DISCUSSION] Refactoring Brokers to support multiple ports

Hi Everyone, One of the pre-requisites we have for supporting multiple security protocols (SSL, Kerberos) is to support them on separate ports. This is done in KAFKA-1684 (The SSL Patch), but that patch addresses several different issues - Multiple ports, enriching the channels, SSL implementatio

Re: Does Kafka Producer service ?

Could you be a bit more specific about the issue? As long as there is protocol compatibility btw the Kafka client and the broker, upgrading the Kafka client library should be easy, right? Thanks, Jun On Mon, Nov 24, 2014 at 3:57 PM, Krishna Raj wrote: > Hello Amazing Kafka Creators & User, > >

[jira] [Commented] (KAFKA-1795) OOME - high level kafka consumer

[ https://issues.apache.org/jira/browse/KAFKA-1795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14224911#comment-14224911 ] Jiangjie Qin commented on KAFKA-1795: - Which GC tuning are you using? Are you using G1

[jira] [Commented] (KAFKA-1789) Issue with Async producer

[ https://issues.apache.org/jira/browse/KAFKA-1789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14224907#comment-14224907 ] Jiangjie Qin commented on KAFKA-1789: - Do you mean that the enqueue took 250ms or the

Re: Review Request 27430: Fix KAFKA-1720

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27430/#review63003 --- core/src/main/scala/kafka/server/ReplicaManager.scala

Re: [DISCUSSION] adding the serializer api back to the new java producer

The old consumer already takes a deserializer when creating streams. So you plug in your decoder there. Thanks, Jun On Tue, Nov 25, 2014 at 8:29 AM, Manikumar Reddy wrote: > +1 for this change. > > what about de-serializer class in 0.8.2? Say i am using new producer with > Avro and old consu

Re: [DISCUSSION] adding the serializer api back to the new java producer

Shiomi, Sorry, at that time, I didn't realize that we would be better off with an api change. Yes, it sucks that we have to break the api. However, if we have to change it, it's better to do it now rather than later. Note that if you want to just produce byte[] to Kafka, you can still do that wit

Re: [DISCUSSION] adding the serializer api back to the new java producer

Hey Shlomi, I agree that we just blew this one from a timing perspective. We ideally should have thought this through in the original api discussion. But as we really started to think about this area we realized that the existing api made it really hard to provide a simple way package of serializa

Re: [DISCUSSION] adding the serializer api back to the new java producer

How will mix bag will work with Consumer side ? Entire site can not be rolled at once so Consumer will have to deals with New and Old Serialize Bytes ? This could be app team responsibility. Are you guys targeting 0.8.2 release, which may break customer who are already using new producer API (be

Re: [DISCUSSION] adding the serializer api back to the new java producer

+1 for this change. what about de-serializer class in 0.8.2? Say i am using new producer with Avro and old consumer combination. then i need to give custom Decoder implementation for Avro right?. On Tue, Nov 25, 2014 at 9:19 PM, Joe Stein wrote: > The serializer is an expected use of the prod

Re: [DISCUSSION] adding the serializer api back to the new java producer

The serializer is an expected use of the producer/consumer now and think we should continue that support in the new client. As far as breaking the API it is why we released the 0.8.2-beta to help get through just these type of blocking issues in a way that the community at large could be involved i

Re: [DISCUSSION] adding the serializer api back to the new java producer

+1 on this change — APIs are forever. As much as we’d love to see 0.8.2 release ASAP, it is important to get this right. -JW > On Nov 24, 2014, at 5:58 PM, Jun Rao wrote: > > Hi, Everyone, > > I'd like to start a discussion on whether it makes sense to add the > serializer api back to the new

[jira] [Commented] (KAFKA-1667) topic-level configuration not validated

[ https://issues.apache.org/jira/browse/KAFKA-1667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14224394#comment-14224394 ] Dmytro Kostiuchenko commented on KAFKA-1667: Updated reviewboard https://revie

[jira] [Updated] (KAFKA-1667) topic-level configuration not validated

[ https://issues.apache.org/jira/browse/KAFKA-1667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmytro Kostiuchenko updated KAFKA-1667: --- Attachment: KAFKA-1667_2014-11-25_12:03:56.patch > topic-level configuration not vali

Re: Review Request 27634: Patch for KAFKA-1667

> On Nov. 21, 2014, 7:46 p.m., Jun Rao wrote: > > core/src/main/scala/kafka/log/LogConfig.scala, lines 176-177 > > > > > > This is a useful genenal check for boolean type. Could we include the > > validation in Config

Re: Review Request 27634: Patch for KAFKA-1667

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27634/ --- (Updated Nov. 25, 2014, 11:04 a.m.) Review request for kafka. Bugs: KAFKA-166

Re: [DISCUSSION] adding the serializer api back to the new java producer

Jun, while just a humble user, I would like to recall that it was just 6 days ago that you told me on the user list that the producer is stable when I asked what producer to go with and if the new producer is production stable (you can still see that email down the list). maybe I miss something, bu