Re: Consumer Offset Migration Tool

There is an old jira on getting the old import/export ZK offsets tool to work for Kafka-based offsets . Ideally, that should be done in the kafka-consumer-groups tool. We could have kafka-consumer-groups tool support import/export for ZK-based offse

This is not a spam - ISR issues

Hi, Any comments on this? Ciprian. Lähettäjä: Pascu, Ciprian (Nokia - FI/Espoo) Lähetetty: 9. elokuuta 2016 15:34:10 Vastaanottaja: dev@kafka.apache.org Aihe: ISR issues Hi, We are testing Kafka cluster with high traffic loads (2+ messages/second) and

[jira] [Assigned] (KAFKA-3123) Follower Broker cannot start if offsets are already out of range

[ https://issues.apache.org/jira/browse/KAFKA-3123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Soumyajit Sahu reassigned KAFKA-3123: - Assignee: Soumyajit Sahu (was: Neha Narkhede) > Follower Broker cannot start if offsets

[jira] [Created] (KAFKA-4027) Leader for a cetain partition unavailable forever

tu nguyen khac created KAFKA-4027: - Summary: Leader for a cetain partition unavailable forever Key: KAFKA-4027 URL: https://issues.apache.org/jira/browse/KAFKA-4027 Project: Kafka Issue Type:

Re: [VOTE] KIP-70: Revise Partition Assignment Semantics on New Consumer's Subscription Change

+1 (binding), thanks for working on this Vahid. @Dana - See https://cwiki.apache.org/confluence/display/KAFKA/Bylaws re: binding/non-binding, although I now notice that we specify criteria (lazy majority) on the KIP overview https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Propo

Re: Consumer Offset Migration Tool

Hi Jun, Exactly what Gwen said. I am assuming shutdown old consumers, migrate offsets, start new consumers. This is best for cases where you are migrating from the old clients to the new clients without ever using dual commit in the old client. Because the new clients can't coordinate with the old

Re: Consumer Offset Migration Tool

Jun, Grant's use-case is about migrating from old-consumer-committing-to-ZK to new-consumer-committing-to-Kafka (which is what happens if you upgrade Flume, and maybe other stream processing systems too). This seems to require shutting down all instances in any case. Gwen On Tue, Aug 9, 2016 at

Re: Consumer Offset Migration Tool

Hi, Grant, For your tool to work, do you expect all consumer instances in the consumer group to be stopped before copying the offsets? Some applications may not want to do that. Thanks, Jun On Tue, Aug 9, 2016 at 10:01 AM, Grant Henke wrote: > I had to write a simple offset migration tool an

[jira] [Commented] (KAFKA-3123) Follower Broker cannot start if offsets are already out of range

[ https://issues.apache.org/jira/browse/KAFKA-3123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15414528#comment-15414528 ] ASF GitHub Bot commented on KAFKA-3123: --- GitHub user soumyajit-sahu opened a pull re

[GitHub] kafka pull request #1716: KAFKA-3123: Make abortAndPauseCleaning() a no-op w...

GitHub user soumyajit-sahu opened a pull request: https://github.com/apache/kafka/pull/1716 KAFKA-3123: Make abortAndPauseCleaning() a no-op when state is already LogCleaningPaused The function of LogCleanerManager.abortAndPauseCleaning() is to mark a TopicAndPartition as LogCleani

[jira] [Commented] (KAFKA-3123) Follower Broker cannot start if offsets are already out of range

[ https://issues.apache.org/jira/browse/KAFKA-3123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15414500#comment-15414500 ] Soumyajit Sahu commented on KAFKA-3123: --- I hit another (similar) exception while try

[jira] [Closed] (KAFKA-3448) Support zone index in IPv6 regex

[ https://issues.apache.org/jira/browse/KAFKA-3448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Soumyajit Sahu closed KAFKA-3448. - > Support zone index in IPv6 regex > > > Key: KAFKA-3

[jira] [Closed] (KAFKA-2305) Cluster Monitoring and Management UI

[ https://issues.apache.org/jira/browse/KAFKA-2305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Soumyajit Sahu closed KAFKA-2305. - > Cluster Monitoring and Management UI > > > Key:

[DISCUSS] Time-based releases for Apache Kafka

Dear Kafka Developers and Users, In the past, our releases have been quite unpredictable. We'll notice that a large number of nice features made it in (or are close), someone would suggest a release and we'd do it. This is fun, but makes planning really hard - we saw it during the last release whi

Re: [DISCUSS] KIP-73: Replication Quotas

When there are several unthrottled replicas, we could also just do what's suggested in KIP-74. The client is responsible for reordering the partitions and the leader fills in the bytes to those partitions in order, up to the quota limit. We could also do what you suggested. If quota is exceeded, i

[jira] [Commented] (KAFKA-4022) TopicCommand is using default max.message.bytes instead of broker's setting

[ https://issues.apache.org/jira/browse/KAFKA-4022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15414281#comment-15414281 ] Tomasz Trębski commented on KAFKA-4022: --- Yes, that generally my concern here that cu

Re: [DISCUSS] KIP-73: Replication Quotas

> > > > On the leader side, one challenge is related to the fairness issue that Ben > brought up. The question is what if the fetch response limit is filled up > by the throttled replicas? If this happens constantly, we will delay the > progress of those un-throttled replicas. However, I think we c

Re: [DISCUSS] KIP-4 ACL Admin Schema

Grant, Thanks for the reply. I had one inline reply below. On Mon, Aug 8, 2016 at 2:44 PM, Grant Henke wrote: > Thank you for the feedback everyone. Below I respond to the last batch of > emails: > > You mention that "delete" actions > > will get processed before "add" actions, which makes sens

Re: [DISCUSS] KIP-73: Replication Quotas

Joel, Mayuresh, Thanks for the suggestion. Yes, I think we could use a single replica fetch thread to handle both throttled and unthrottled replicas. On the follower side, this is easy. If the quota is exceeded, we remember the amount of the delay needed, take those throttled replicas off the fet

[jira] [Comment Edited] (KAFKA-4022) TopicCommand is using default max.message.bytes instead of broker's setting

[ https://issues.apache.org/jira/browse/KAFKA-4022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15413806#comment-15413806 ] Vahid Hashemian edited comment on KAFKA-4022 at 8/9/16 7:47 PM:

Re: [DISCUSS] KIP-73: Replication Quotas

Nice write up Ben. I agree with Joel for keeping this simple by excluding the partitions from the fetch request/response when the quota is violated at the follower or leader instead of having a separate set of threads for handling the quota and non quota cases. Even though its different from the c

[jira] [Commented] (KAFKA-3936) Validate user parameters as early as possible

[ https://issues.apache.org/jira/browse/KAFKA-3936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15413983#comment-15413983 ] ASF GitHub Bot commented on KAFKA-3936: --- Github user asfgit closed the pull request

[jira] [Updated] (KAFKA-3936) Validate user parameters as early as possible

[ https://issues.apache.org/jira/browse/KAFKA-3936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-3936: - Resolution: Fixed Fix Version/s: 0.10.1.0 Status: Resolved (was: Patch Available

[GitHub] kafka pull request #1711: KAFKA-3936: Validate parameters as early as possib...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/1711 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[GitHub] kafka pull request #1709: MINOR: Doc individual partition must fit on the se...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/1709 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

Re: [VOTE] KIP-70: Revise Partition Assignment Semantics on New Consumer's Subscription Change

+1. On Tue, Aug 9, 2016 at 10:06 AM, Jun Rao wrote: > Vahid, > > Thanks for the clear explanation in the KIP. +1 > > Jun > > On Mon, Aug 8, 2016 at 11:53 AM, Vahid S Hashemian < > vahidhashem...@us.ibm.com> wrote: > > > I would like to initiate the voting process for KIP-70 ( > > https://cwiki.a

Jenkins build is back to normal : kafka-trunk-jdk8 #808

See

Re: [DISCUSS] KIP-73: Replication Quotas

On the need for both leader/follower throttling: that makes sense - thanks for clarifying. For completeness, can we add this detail to the doc - say, after the quote that I pasted earlier? >From an implementation perspective though: I’m still interested in the simplicity of not having to add separ

Re: [DISCUSS] KIP-74: Add Fetch Response Size Limit in Bytes

Hi, Andrey, Thanks for the proposal. It looks good overall. Some minor comments. 1. It seems that it's bit weird that fetch.partition.max.bytes is a broker level configuration while fetch.limit.bytes is a client side configuration. Intuitively, it seems both should be set by the client? If we do

Re: [VOTE] KIP-70: Revise Partition Assignment Semantics on New Consumer's Subscription Change

Vahid, Thanks for the clear explanation in the KIP. +1 Jun On Mon, Aug 8, 2016 at 11:53 AM, Vahid S Hashemian < vahidhashem...@us.ibm.com> wrote: > I would like to initiate the voting process for KIP-70 ( > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > 70%3A+Revise+Partition+Assignme

Consumer Offset Migration Tool

I had to write a simple offset migration tool and I wanted to get feedback on whether or not this would be a useful addition to Apache Kafka. Currently the path to upgrade from the zookeeper offsets to the Kafka offset (and often the Scala to Java client) is via dual commit. The process is documen

[GitHub] kafka pull request #1715: WIP: Add a consumer offset migration tool

GitHub user granthenke opened a pull request: https://github.com/apache/kafka/pull/1715 WIP: Add a consumer offset migration tool Please see the dev mailing list for context. You can merge this pull request into a Git repository by running: $ git pull https://github.com/granth

[jira] [Comment Edited] (KAFKA-2629) Enable getting SSL password from an executable rather than passing plaintext password

[ https://issues.apache.org/jira/browse/KAFKA-2629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15413820#comment-15413820 ] Bharat Viswanadham edited comment on KAFKA-2629 at 8/9/16 4:51 PM: -

[jira] [Commented] (KAFKA-2629) Enable getting SSL password from an executable rather than passing plaintext password

[ https://issues.apache.org/jira/browse/KAFKA-2629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15413820#comment-15413820 ] Bharat Viswanadham commented on KAFKA-2629: --- [~singhashish] What is the final ap

[jira] [Commented] (KAFKA-2629) Enable getting SSL password from an executable rather than passing plaintext password

[ https://issues.apache.org/jira/browse/KAFKA-2629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15413818#comment-15413818 ] Bharat Viswanadham commented on KAFKA-2629: --- [~singhashish] Thank you for update

Re: [DISCUSS] KIP-73: Replication Quotas

Just to elaborate on what Ben said why we need throttling on both the leader and the follower side. If we only have throttling on the follower side, consider a case that we add 5 more new brokers and want to move some replicas from existing brokers over to those 5 brokers. Each of those broker is

[jira] [Commented] (KAFKA-4022) TopicCommand is using default max.message.bytes instead of broker's setting

[ https://issues.apache.org/jira/browse/KAFKA-4022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15413806#comment-15413806 ] Vahid Hashemian commented on KAFKA-4022: [~kornicameis...@gmail.com] Yes, I see wh

[jira] [Assigned] (KAFKA-4022) TopicCommand is using default max.message.bytes instead of broker's setting

[ https://issues.apache.org/jira/browse/KAFKA-4022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vahid Hashemian reassigned KAFKA-4022: -- Assignee: Vahid Hashemian > TopicCommand is using default max.message.bytes instead of

[jira] [Updated] (KAFKA-3934) Start scripts enable GC by default with no way to disable

[ https://issues.apache.org/jira/browse/KAFKA-3934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewen Cheslack-Postava updated KAFKA-3934: - Resolution: Fixed Fix Version/s: 0.10.1.0 Status: Resolved (was:

[jira] [Commented] (KAFKA-3934) Start scripts enable GC by default with no way to disable

[ https://issues.apache.org/jira/browse/KAFKA-3934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15413794#comment-15413794 ] ASF GitHub Bot commented on KAFKA-3934: --- Github user asfgit closed the pull request

[GitHub] kafka pull request #1631: KAFKA-3934: Start scripts enable GC by default wit...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/1631 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

Re: Review request for KAFKA-3600

Provided wrong link to PR, here is the PR for KAFKA-3600. On Tue, Aug 9, 2016 at 9:45 AM, Ashish Singh wrote: > Hey Guys, > > KAFKA-3600 was part of > KIP-35's proposal. KAFKA-3307 >

[GitHub] kafka pull request #1487: MINOR: kafka-run-class.sh now runs under Cygwin.

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/1487 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

Review request for KAFKA-3600

Hey Guys, KAFKA-3600 was part of KIP-35's proposal. KAFKA-3307 , adding ApiVersionsRequest/Response, was committed to 0.10.0.0, but KAFKA-3600, enhancing java clients, is still under review. Here i

[jira] [Commented] (KAFKA-2629) Enable getting SSL password from an executable rather than passing plaintext password

[ https://issues.apache.org/jira/browse/KAFKA-2629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15413738#comment-15413738 ] Ashish K Singh commented on KAFKA-2629: --- [~bharatviswa] I would like this to go in.

Re: [VOTE] KIP-70: Revise Partition Assignment Semantics on New Consumer's Subscription Change

+1 On Mon, Aug 8, 2016 at 11:53 AM, Vahid S Hashemian < vahidhashem...@us.ibm.com> wrote: > I would like to initiate the voting process for KIP-70 ( > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > 70%3A+Revise+Partition+Assignment+Semantics+on+New+ > Consumer%27s+Subscription+Change >

[jira] [Updated] (KAFKA-3934) Start scripts enable GC by default with no way to disable

[ https://issues.apache.org/jira/browse/KAFKA-3934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KAFKA-3934: --- Summary: Start scripts enable GC by default with no way to disable (was: kafka-server-start.sh enable

Re: [DISCUSS] KIP-72 Allow Sizing Incoming Request Queue in Bytes

Radi, Yes, I got the benefit of bounding the request queue by bytes. My concern is the following if we don't change the behavior of processor blocking on queue full. If the broker truly doesn't have enough memory for buffering outstanding requests from all connections, we have to either hit OOM o

Re: [DISCUSS] KIP-72 Allow Sizing Incoming Request Queue in Bytes

I like the simplicity of the approach and can see that it is an improvement over the current implementation in typical scenarios. But I would like to see Jun's proposal to mute sockets explored further. With the proposal in the KIP to limit queue size, I am not sure how to calculate the total memor

Re: [VOTE] KIP-70: Revise Partition Assignment Semantics on New Consumer's Subscription Change

+1 (still confused about bindings) On Tue, Aug 9, 2016 at 6:24 AM, Ismael Juma wrote: > Thanks for the KIP. +1 (binding) > > Ismael > > On Mon, Aug 8, 2016 at 7:53 PM, Vahid S Hashemian > wrote: > >> I would like to initiate the voting process for KIP-70 ( >> https://cwiki.apache.org/confluence/d

Re: [VOTE] KIP-70: Revise Partition Assignment Semantics on New Consumer's Subscription Change

Thanks for the KIP. +1 (binding) Ismael On Mon, Aug 8, 2016 at 7:53 PM, Vahid S Hashemian wrote: > I would like to initiate the voting process for KIP-70 ( > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > 70%3A+Revise+Partition+Assignment+Semantics+on+New+ > Consumer%27s+Subscription+

Re: [VOTE] KIP-70: Revise Partition Assignment Semantics on New Consumer's Subscription Change

Vahid, Thanks for the KIP. +1 (non-binding) On Mon, Aug 8, 2016 at 8:09 PM, Jason Gustafson wrote: > Thanks Vahid. +1 from me (non-binding). > > On Mon, Aug 8, 2016 at 11:53 AM, Vahid S Hashemian < > vahidhashem...@us.ibm.com> wrote: > > > I would like to initiate the voting process for KIP-70

ISR issues

Hi, We are testing Kafka cluster with high traffic loads (2+ messages/second) and we encounter quite frequently issues with brokers dropping persistently from ISR for some partitions. Looking at the code, I noticed in class AbstractFetcherThread , processFetchRequest method, that a KafkaExc

Re: [DISCUSS] KIP-55: Secure quotas for authenticated users

Hi Tom, Have updated the KIP wiki. Will submit a PR later this week. Regards, Rajini On Tue, Aug 9, 2016 at 12:30 PM, Tom Crayford wrote: > Seeing as voting passed on this, can somebody with access update the wiki? > > Is there code for this KIP in a PR somewhere that needs merging? > > Thank

Re: [DISCUSS] KIP-73: Replication Quotas

Hi Joel Thanks for taking the time to look at this. Appreciated. Regarding throttling on both leader and follower, this proposal covers a more general solution which can guarantee a quota, even when a rebalance operation produces an asymmetric profile of load. This means administrators don’t n

Re: [DISCUSS] KIP-55: Secure quotas for authenticated users

Seeing as voting passed on this, can somebody with access update the wiki? Is there code for this KIP in a PR somewhere that needs merging? Thanks Tom Crayford Heroku Kafka On Friday, 1 July 2016, Rajini Sivaram wrote: > Thank you, Jun. > > Hi all, > > Please let me know if you have any commen

[jira] [Commented] (KAFKA-4022) TopicCommand is using default max.message.bytes instead of broker's setting

[ https://issues.apache.org/jira/browse/KAFKA-4022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15413088#comment-15413088 ] Tomasz Trębski commented on KAFKA-4022: --- [~vahid] , so maybe I was misguided by the