Re: Make documentation part of new features acceptance criteria?

I like the release criteria idea. Perhaps add them to coding guide or the developer section on wiki? WRT feature completeness, I didn't think about having a doc for each one, but rather updating the existing doc or the CHANGES.txt file (we don't have one yet) with a note when adding new configurat

[jira] [Created] (KAFKA-972) MetadataRequest returns stale list of brokers

Vinicius Carvalho created KAFKA-972: --- Summary: MetadataRequest returns stale list of brokers Key: KAFKA-972 URL: https://issues.apache.org/jira/browse/KAFKA-972 Project: Kafka Issue Type: B

[jira] [Updated] (KAFKA-615) Avoid fsync on log segment roll

[ https://issues.apache.org/jira/browse/KAFKA-615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps updated KAFKA-615: Attachment: KAFKA-615-v3.patch Patch version v3: - Found a call to flush the index in Log.roll(). Removed th

[jira] [Commented] (KAFKA-969) Need to prevent failure of rebalance when there are no brokers available when consumer comes up

[ https://issues.apache.org/jira/browse/KAFKA-969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13705954#comment-13705954 ] Guozhang Wang commented on KAFKA-969: - I think this is related to how we would enforce

[jira] [Closed] (KAFKA-969) Need to prevent failure of rebalance when there are no brokers available when consumer comes up

[ https://issues.apache.org/jira/browse/KAFKA-969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sriram Subramanian closed KAFKA-969. > Need to prevent failure of rebalance when there are no brokers available when > consumer

[jira] [Commented] (KAFKA-969) Need to prevent failure of rebalance when there are no brokers available when consumer comes up

[ https://issues.apache.org/jira/browse/KAFKA-969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13705971#comment-13705971 ] Sam Meder commented on KAFKA-969: - I feel that Kafka servers and clients should really not

Re: having problem with 0.8 gzip compression

Joel, Would you mind point me to how I would be able to enable the trace logs in the producer and broker? Thanks, Scott On Wed, Jul 10, 2013 at 5:33 PM, Joel Koshy wrote: > Weird - I tried your exact code and it worked for me (although I was > using 0.8 head and not the beta). Can you re-run

[jira] Subscription: outstanding kafka patches

Issue Subscription Filter: outstanding kafka patches (75 issues) The list of outstanding kafka patches Subscriber: kafka-mailing-list Key Summary KAFKA-967 Use key range in ProducerPerformance https://issues.apache.org/jira/browse/KAFKA-967 KAFKA-956 High-level cons

Added an "operations" page for 0.8 to site

Probably it is very incomplete. I copied over much of the content from 0.7 that is still applicable, but I added the following: 1. A section on data centers and mirroring 2. Removed the cut-and-pasted scala code in the monitoring section and just gave a high-level description of what we monitor 3.

[jira] [Updated] (KAFKA-718) kafka-run-class.sh should use reasonable gc settings

[ https://issues.apache.org/jira/browse/KAFKA-718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps updated KAFKA-718: Attachment: KAFKA-718-v2.patch Hey Ashwanth, thanks for the patch. I applied it to 0.8 and I want to abuse

KAFKA-718

I included a number of "good housekeeping" fixes in with the GC settings fix here: https://issues.apache.org/jira/browse/KAFKA-718 I think these should go into 0.8 as they are sloppy, confusing, or in the case of the GC stuff likely to lead to production problems. -Jay

Re: Make documentation part of new features acceptance criteria?

Cool, let's do that then. I think we will likely be in a better state in a week--Sriram is working on updating the design page which is the last big outstanding thing that is out of date. I do totally love the hbase documentation, they are doing it right. -Jay On Thu, Jul 11, 2013 at 2:47 AM,

Re: Added an "operations" page for 0.8 to site

Ack, thanks! Fixed. -Jay On Thu, Jul 11, 2013 at 2:22 PM, Roger Hoover wrote: > Thank you, Jay. > > When talking about flush rates, I think you mean the opposite of what was > said here: > > "However very high application flush rates can lead to high latency when > the flush does occur." > > sh

How to release

Cosmin had a good point about adding documentation to the release steps and critieria. I don't think we actually have a written set of release criteria and steps. Joe, you've done the last few releases I wonder if you could give the steps you went through? All any suggestions on the things that s

Re: How to release

yup yup, i have an old ticket for it already https://issues.apache.org/jira/browse/KAFKA-394 btw, i submitted 0.8.0-beta1 to maven central yesterday morning. like watching the paint to dry for it to show up =8^) will send email once i see it On Thu, Jul 11, 2013 at 5:46 PM, Jay Kreps wrote: >

Re: Make documentation part of new features acceptance criteria?

I must say that the HBASE site is just awesome! They even have multi language support for documentation. On 7/11/13 2:34 PM, "Jay Kreps" wrote: >Cool, let's do that then. > >I think we will likely be in a better state in a week--Sriram is working >on >updating the design page which is the last b

[jira] [Commented] (KAFKA-943) Move all configuration key string to constants

[ https://issues.apache.org/jira/browse/KAFKA-943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13706288#comment-13706288 ] Jay Kreps commented on KAFKA-943: - I cleaned up the bad config strings I saw in our example

[jira] [Commented] (KAFKA-898) Add a KafkaMetricsReporter that wraps Librato's reporter

[ https://issues.apache.org/jira/browse/KAFKA-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13706300#comment-13706300 ] Jay Kreps commented on KAFKA-898: - Hey Scott, thanks for the patch! I think it would be be

KAFKA-885

Any objection to this patch for 0.8? We seem to be creating our jar twice in different places. Seems like it should fall under basic housekeeping issues: https://issues.apache.org/jira/browse/KAFKA-885

[jira] [Commented] (KAFKA-877) Still getting kafka.common.NotLeaderForPartitionException

[ https://issues.apache.org/jira/browse/KAFKA-877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13706302#comment-13706302 ] Jay Kreps commented on KAFKA-877: - Is this still happening? If not I want to mark it closed

Re: KAFKA-885

good housecleaning in my book +1 On Thu, Jul 11, 2013 at 6:01 PM, Jay Kreps wrote: > Any objection to this patch for 0.8? We seem to be creating our jar twice > in different places. Seems like it should fall under basic housekeeping > issues: > > https://issues.apache.org/jira/browse/KAFKA-885

[jira] [Commented] (KAFKA-877) Still getting kafka.common.NotLeaderForPartitionException

[ https://issues.apache.org/jira/browse/KAFKA-877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13706314#comment-13706314 ] BalajiSeshadri commented on KAFKA-877: -- Please close it,we will reopen it if it happen

[jira] [Updated] (KAFKA-836) Update quickstart for Kafka 0.8

[ https://issues.apache.org/jira/browse/KAFKA-836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps updated KAFKA-836: Resolution: Fixed Status: Resolved (was: Patch Available) Done! > Update quickstar

[jira] [Commented] (KAFKA-717) scala 2.10 build support

[ https://issues.apache.org/jira/browse/KAFKA-717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13706326#comment-13706326 ] Jay Kreps commented on KAFKA-717: - Yeah we can do it in 0.8.1 which should happen fairly qu

[jira] [Commented] (KAFKA-705) Controlled shutdown doesn't seem to work on more than one broker in a cluster

[ https://issues.apache.org/jira/browse/KAFKA-705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13706330#comment-13706330 ] Jay Kreps commented on KAFKA-705: - Joel, this is done, no? > Controlled sh

Running 0.8 against 0.7 data and zk

This JIRA has a patch but I thought we already did something like this: https://issues.apache.org/jira/browse/KAFKA-686 Am I just confused? -Jay

[jira] [Resolved] (KAFKA-682) java.lang.OutOfMemoryError: Java heap space

[ https://issues.apache.org/jira/browse/KAFKA-682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps resolved KAFKA-682. - Resolution: Fixed Marking resolved as we fixed a 0.8 bug that impacted memory and improved the default GC

[jira] [Updated] (KAFKA-649) Cleanup log4j logging

[ https://issues.apache.org/jira/browse/KAFKA-649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps updated KAFKA-649: Resolution: Fixed Status: Resolved (was: Patch Available) Looks fixed. > Cleanup l

[jira] [Commented] (KAFKA-677) Retention process gives exception if an empty segment is chosen for collection

[ https://issues.apache.org/jira/browse/KAFKA-677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13706338#comment-13706338 ] Jay Kreps commented on KAFKA-677: - Jun, is this still relevant? > Retentio

[jira] [Commented] (KAFKA-898) Add a KafkaMetricsReporter that wraps Librato's reporter

[ https://issues.apache.org/jira/browse/KAFKA-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13706342#comment-13706342 ] Scott Clasen commented on KAFKA-898: Hey Jay Makes sense. I had seen a bunch of q

[jira] [Resolved] (KAFKA-547) The ConsumerStats MBean name should include the groupid

[ https://issues.apache.org/jira/browse/KAFKA-547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps resolved KAFKA-547. - Resolution: Fixed At this point I think we should pass on this for 0.8 since we anyway have it in trunk.

[jira] [Resolved] (KAFKA-465) Performance test scripts - refactoring leftovers from tools to perf package

[ https://issues.apache.org/jira/browse/KAFKA-465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps resolved KAFKA-465. - Resolution: Fixed As mentioned I think this was fixed in KAFKA-425. > Performance test sc

[jira] [Commented] (KAFKA-397) kafka.common.InvalidMessageSizeException: null

[ https://issues.apache.org/jira/browse/KAFKA-397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13706352#comment-13706352 ] Jay Kreps commented on KAFKA-397: - So is there anything we should be fixing here?

[jira] [Resolved] (KAFKA-399) 0.7.1 seems to show less performance than 0.7.0

[ https://issues.apache.org/jira/browse/KAFKA-399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps resolved KAFKA-399. - Resolution: Won't Fix 0.8 is different enough that I think it is safe to mark this "won't fix".

[jira] [Commented] (KAFKA-717) scala 2.10 build support

[ https://issues.apache.org/jira/browse/KAFKA-717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13706354#comment-13706354 ] Matei Zaharia commented on KAFKA-717: - Sounds good! Glad to hear you guys are looking a

[jira] [Commented] (KAFKA-318) update zookeeper dependency to 3.3.5

[ https://issues.apache.org/jira/browse/KAFKA-318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13706359#comment-13706359 ] Jay Kreps commented on KAFKA-318: - I recommend we just take this on trunk. It will get in 0

[jira] [Updated] (KAFKA-852) Remove clientId from OffsetFetchResponse and OffsetCommitResponse

[ https://issues.apache.org/jira/browse/KAFKA-852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps updated KAFKA-852: Resolution: Fixed Status: Resolved (was: Patch Available) Sorry for lagging on this. Committed.

[jira] [Resolved] (KAFKA-278) Issues partitioning a new topic

[ https://issues.apache.org/jira/browse/KAFKA-278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps resolved KAFKA-278. - Resolution: Fixed This should be fixed in 0.8 where we handle partitioning properly. > Is

[jira] [Commented] (KAFKA-273) Occassional GZIP errors on the server while writing compressed data to disk

[ https://issues.apache.org/jira/browse/KAFKA-273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13706377#comment-13706377 ] Jay Kreps commented on KAFKA-273: - Is this still happening? > Occassional

[jira] [Commented] (KAFKA-242) Subsequent calls of ConsumerConnector.createMessageStreams cause Consumer offset to be incorrect

[ https://issues.apache.org/jira/browse/KAFKA-242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13706381#comment-13706381 ] Jay Kreps commented on KAFKA-242: - Does this bug still exist in 0.8? > Sub

Re: How to release

Oh, cool. -Jay On Thu, Jul 11, 2013 at 2:50 PM, Joe Stein wrote: > yup yup, i have an old ticket for it already > https://issues.apache.org/jira/browse/KAFKA-394 > > btw, i submitted 0.8.0-beta1 to maven central yesterday morning. like > watching the paint to dry for it to show up =8^) will s

[jira] [Resolved] (KAFKA-877) Still getting kafka.common.NotLeaderForPartitionException

[ https://issues.apache.org/jira/browse/KAFKA-877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps resolved KAFKA-877. - Resolution: Cannot Reproduce Will reopen if it reoccurs. > Still getting kafka.common.Not

[jira] [Commented] (KAFKA-898) Add a KafkaMetricsReporter that wraps Librato's reporter

[ https://issues.apache.org/jira/browse/KAFKA-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13706392#comment-13706392 ] Jay Kreps commented on KAFKA-898: - We can dump it in contrib, but I think what we have foun

[jira] [Updated] (KAFKA-898) Add a KafkaMetricsReporter that wraps Librato's reporter

[ https://issues.apache.org/jira/browse/KAFKA-898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps updated KAFKA-898: Resolution: Won't Fix Status: Resolved (was: Patch Available) Will be happier elsewhere.

Re: having problem with 0.8 gzip compression

Ok, the problem solved, I think it might be because some of the jar files that I was using were "OLD". I was building the producer and consumer under the 0.7 environment except swapping out the kafka jar file. Now, I created a whole new environment and pull in all the jar files from the 0.8. Th

[jira] [Commented] (KAFKA-898) Add a KafkaMetricsReporter that wraps Librato's reporter

[ https://issues.apache.org/jira/browse/KAFKA-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13706415#comment-13706415 ] Scott Clasen commented on KAFKA-898: Cool ill see if the librato folks want it, if not

[jira] [Closed] (KAFKA-898) Add a KafkaMetricsReporter that wraps Librato's reporter

[ https://issues.apache.org/jira/browse/KAFKA-898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Scott Clasen closed KAFKA-898. -- > Add a KafkaMetricsReporter that wraps Librato's reporter > ---

[jira] [Updated] (KAFKA-718) kafka-run-class.sh should use reasonable gc settings

[ https://issues.apache.org/jira/browse/KAFKA-718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps updated KAFKA-718: Attachment: KAFKA-718-v3.patch Separate out a few more command line variables (GC, GC logging, SCALA_VERSION

[jira] [Updated] (KAFKA-637) Separate log4j environment variable from KAFKA_OPTS in kafka-run-class.sh

[ https://issues.apache.org/jira/browse/KAFKA-637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps updated KAFKA-637: Resolution: Fixed Assignee: Jay Kreps Status: Resolved (was: Patch Available) This is sensibl

[jira] [Commented] (KAFKA-718) kafka-run-class.sh should use reasonable gc settings

[ https://issues.apache.org/jira/browse/KAFKA-718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13706461#comment-13706461 ] Ashwanth Fernando commented on KAFKA-718: - Thanks [~jkreps]!! Can we mark this as r

[jira] [Comment Edited] (KAFKA-718) kafka-run-class.sh should use reasonable gc settings

[ https://issues.apache.org/jira/browse/KAFKA-718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13706461#comment-13706461 ] Ashwanth Fernando edited comment on KAFKA-718 at 7/11/13 11:40 PM: --

[jira] [Commented] (KAFKA-946) Kafka Hadoop Consumer fails when verifying message checksum

[ https://issues.apache.org/jira/browse/KAFKA-946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13706465#comment-13706465 ] Jay Kreps commented on KAFKA-946: - >From Richard: Sorry, It took a while to remember the c

[jira] [Comment Edited] (KAFKA-718) kafka-run-class.sh should use reasonable gc settings

[ https://issues.apache.org/jira/browse/KAFKA-718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13706461#comment-13706461 ] Ashwanth Fernando edited comment on KAFKA-718 at 7/11/13 11:40 PM: --

[jira] [Commented] (KAFKA-718) kafka-run-class.sh should use reasonable gc settings

[ https://issues.apache.org/jira/browse/KAFKA-718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13706474#comment-13706474 ] Jay Kreps commented on KAFKA-718: - Since 0.8 is somewhat locked down I am just waiting on c

Re: Added an "operations" page for 0.8 to site

Thank you, Jay. When talking about flush rates, I think you mean the opposite of what was said here: "However very high application flush rates can lead to high latency when the flush does occur." should be However very low application flush rates (infrequent flushes) can lead to high latency w

KAFKA-718

Hi Kafka team, I submitted the patch. Can you let me know how to take this forward? Thanks, Ashwanth

[jira] [Updated] (KAFKA-957) MirrorMaker needs to preserve the key in the source cluster

[ https://issues.apache.org/jira/browse/KAFKA-957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-957: Attachment: KAFKA-957.v1.patch 4. Add a ByteArrayPartitioner since Array[Byte].hashCode will result

[jira] [Updated] (KAFKA-957) MirrorMaker needs to preserve the key in the source cluster

[ https://issues.apache.org/jira/browse/KAFKA-957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-957: Status: Patch Available (was: Open) > MirrorMaker needs to preserve the key in the source clust

[jira] [Commented] (KAFKA-885) sbt package builds two kafka jars

[ https://issues.apache.org/jira/browse/KAFKA-885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13706664#comment-13706664 ] Jun Rao commented on KAFKA-885: --- Thanks for the patch. Is there away to not generating the to

Re: Running 0.8 against 0.7 data and zk

No. If you point an 0.8 broker to a ZK with 0.7 layout, it will still hit some weird exception, maybe not exactly the same as specified in the jira. Thanks, Jun On Thu, Jul 11, 2013 at 3:15 PM, Jay Kreps wrote: > This JIRA has a patch but I thought we already did something like this: > https:

[jira] [Commented] (KAFKA-885) sbt package builds two kafka jars

[ https://issues.apache.org/jira/browse/KAFKA-885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13706711#comment-13706711 ] Ashwanth Fernando commented on KAFKA-885: - Let me see if we can do that. Will get b

[jira] [Comment Edited] (KAFKA-885) sbt package builds two kafka jars

[ https://issues.apache.org/jira/browse/KAFKA-885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13706711#comment-13706711 ] Ashwanth Fernando edited comment on KAFKA-885 at 7/12/13 6:58 AM: ---