[jira] [Commented] (KAFKA-2431) Test SSL/TLS impact on performance

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15182739#comment-15182739
 ] 

Ismael Juma commented on KAFKA-2431:


Hi [~ssuo], an OpenSSL-based implementation is not being developed right now. 
See KAFKA-2561 for the information we collected.

> Test SSL/TLS impact on performance
> --
>
> Key: KAFKA-2431
> URL: https://issues.apache.org/jira/browse/KAFKA-2431
> Project: Kafka
>  Issue Type: Sub-task
>  Components: security
>Reporter: Ismael Juma
>Assignee: Ben Stopford
>Priority: Blocker
> Fix For: 0.9.0.0
>
>
> Test new Producer and new Consumer performance with and without SSL/TLS once 
> the SSL/TLS branch is integrated.
> The ideal scenario is that SSL/TLS would not have an impact if disabled. When 
> enabled, there will be some overhead (encryption and the inability to use 
> `SendFile`) and it will be good to quantify it. The encryption overhead is 
> reduced if recent JDKs are used with CPUs that support AES-specific 
> instructions (https://en.wikipedia.org/wiki/AES_instruction_set).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3334) First message on new topic not actually being sent, no exception thrown

[Aleksandar Stojadinovic (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15182767#comment-15182767
 ] 

Aleksandar Stojadinovic commented on KAFKA-3334:


Sure :) .

> First message on new topic not actually being sent, no exception thrown
> ---
>
> Key: KAFKA-3334
> URL: https://issues.apache.org/jira/browse/KAFKA-3334
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.9.0.0
> Environment: Linux, Java
>Reporter: Aleksandar Stojadinovic
>Assignee: Ashish K Singh
>
> Although I've seen this issue pop around the internet in a few forms, I'm not 
> sure it is yet properly fixed. 
> When publishing to a new topic, with auto create-enabled, the java client 
> (0.9.0) shows this WARN message in the log, and the message is not sent 
> obviously:
> org.apache.kafka.clients.NetworkClient - Error while fetching metadata with 
> correlation id 0 : {file.topic=LEADER_NOT_AVAILABLE}
> In the meantime I see in the console the message that a log for partition is 
> created. The next messages are patched through normally, but the first one is 
> never sent. No exception is ever thrown, either by calling get on the future, 
> or with the async usage, like everything is perfect.
> I notice when I leave my application blocked on the get call, in the 
> debugger, then the message may be processed, but with significant delay. This 
> is consistent with another issue I found for the python client. Also, if I 
> call partitionsFor previously, the topic is created and the message is sent. 
> But it seems silly to call it every time, just to mitigate this issue.
> {code}
> Future recordMetadataFuture = producer.send(new 
> ProducerRecord<>(topic, key, file));
> RecordMetadata recordMetadata = recordMetadataFuture.get(30, 
> TimeUnit.SECONDS);
> {code}
> I hope I'm clear enough.
> Related similar (but not same) issues:
> https://issues.apache.org/jira/browse/KAFKA-1124
> https://github.com/dpkp/kafka-python/issues/150
> http://stackoverflow.com/questions/35187933/how-to-resolve-leader-not-available-kafka-error-when-trying-to-consume



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3326) Error: A JNI error has occurred, please check your installation and try again

[iyaozhen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15182774#comment-15182774
 ] 

iyaozhen commented on KAFKA-3326:
-

Oh no. It is my problem.

MD5 (kafka_2.11-0.9.0.1.tgz) = dfc0601ad7bb815e21e1697bc2d8c967 But 
https://dist.apache.org/repos/dist/release/kafka/0.9.0.1/kafka_2.11-0.9.0.1.tgz.md5
 is 'B71E5CBC78165C1CA483279C27402663'.

> Error: A JNI error has occurred, please check your installation and try again
> -
>
> Key: KAFKA-3326
> URL: https://issues.apache.org/jira/browse/KAFKA-3326
> Project: Kafka
>  Issue Type: Bug
>  Components: config
>Affects Versions: 0.9.0.1
>Reporter: iyaozhen
>
> I use kafka start at 
> doc(http://kafka.apache.org/documentation.html#quickstart):
> [kafka_2.11-0.9.0.1] bin/kafka-server-start.sh config/server.properties
> Error: A JNI error has occurred, please check your installation and try again
> Exception in thread "main" java.lang.NoClassDefFoundError: 
> org/apache/log4j/Logger
>   at java.lang.Class.getDeclaredMethods0(Native Method)
>   at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
>   at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
>   at java.lang.Class.getMethod0(Class.java:3018)
>   at java.lang.Class.getMethod(Class.java:1784)
>   at 
> sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
>   at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
> Caused by: java.lang.ClassNotFoundException: org.apache.log4j.Logger
>   at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>   ... 7 more
> java -version
> java version "1.8.0_66"
> Java(TM) SE Runtime Environment (build 1.8.0_66-b17)
> Java HotSpot(TM) 64-Bit Server VM (build 25.66-b17, mixed mode)
> echo $CLASSPATH
> .:/home/work/yaozhen/jdk1.8.0_66/lib/dt.jar:/home/work/yaozhen/jdk1.8.0_66/lib/tools.jar
> My JAVA is poor, I am no idea to resolve it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: MINOR: streams javadoc corrections

[omkreddy]

GitHub user omkreddy opened a pull request:

https://github.com/apache/kafka/pull/1019

MINOR: streams javadoc corrections



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/omkreddy/kafka JAVADOC

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1019.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1019


commit 7ceac7955c95812cac7530348edeba6ed8e8a1af
Author: Manikumar reddy O 
Date:   2016-03-07T10:51:44Z

MINOR: streams javadoc corrections




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Closed] (KAFKA-2354) setting log.dirs property makes tools fail if there is a comma

[Michael Graff (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Graff closed KAFKA-2354.


> setting log.dirs property makes tools fail if there is a comma
> --
>
> Key: KAFKA-2354
> URL: https://issues.apache.org/jira/browse/KAFKA-2354
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.8.2.1
> Environment: centos
>Reporter: Michael Graff
>
> If one sets log.dirs=/u1/kafka,/u2/kafka, the tools fail to run:
> kafka-topics --describe --zookeeper localhost/kafka
> Error: Could not find or load main class .u1.kafka,
> The broker will start, however.  If the tools are run from a machine without 
> multiple entries in log.dirs, it works.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-3342) https://cwiki.apache.org/confluence/display/KAFKA/Log+Compaction has log.cleaner.min.cleanable.ratio listed twice in error

[Michael Graff (JIRA)]

Michael Graff created KAFKA-3342:


 Summary: 
https://cwiki.apache.org/confluence/display/KAFKA/Log+Compaction has 
log.cleaner.min.cleanable.ratio listed twice in error
 Key: KAFKA-3342
 URL: https://issues.apache.org/jira/browse/KAFKA-3342
 Project: Kafka
  Issue Type: Bug
Reporter: Michael Graff
Priority: Minor


https://cwiki.apache.org/confluence/display/KAFKA/Log+Compaction has 
log.cleaner.min.cleanable.ratio listed both for what it should describe, and 
duplicated again for the setting for the number of threads.  This is in the 
table near the end of the page.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3064) Improve resync method to waste less time and data transfer

[Michael Graff (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183141#comment-15183141
 ] 

Michael Graff commented on KAFKA-3064:
--

As we have added two more brokers (bringing our total to 4) we see this is less 
and less of an issue since multiple sources can saturate the receiver quickly.  
However, as we are using Kafka in two-broker mirrored pairs in some other 
locations, this is still an issue.


> Improve resync method to waste less time and data transfer
> --
>
> Key: KAFKA-3064
> URL: https://issues.apache.org/jira/browse/KAFKA-3064
> Project: Kafka
>  Issue Type: Improvement
>  Components: controller, network
>Affects Versions: 0.8.2.1, 0.9.0.0
>Reporter: Michael Graff
>Assignee: Neha Narkhede
>
> We have several topics which are large (65 GB per partition) with 12 
> partitions.  Data rates into each topic vary, but in general each one has its 
> own rate.
> After a raid rebuild, we are pulling all the data over to the newly rebuild 
> raid.  This takes forever, and has yet to complete after nearly 8 hours.
> Here are my observations:
> (1)  The Kafka broker seems to pull from all topics on all partitions at the 
> same time, starting at the oldest message.
> (2)  When you divide total disk bandwidth available across all partitions 
> (really, only 48 of which have significant amounts of data, about 65 * 12 = 
> 780 GB each topic) the ingest rate of 36 out of 48 of them is higher than the 
> available bandwidth.
> (3)  The effect of (2) is that one topic SLOWLY catches up, while the other 4 
> topics continue to retrieve data at 75% of the bandwidth, just to toss it 
> away because the source broker has discarded it already.
> (4)  Eventually that one topic catches up, and the remaining bandwidth is 
> then divided into the remaining 36 partitions, one group of which starts to 
> catch up again.
> What I want to see is a way to say “don’t transfer more than X partitions at 
> the same time” and ideally a priority rule that says, “Transfer partitions 
> you are responsible for first, then transfer ones you are not.  Also, 
> transfer these first, then those, but no more than 1 topic at a time”
> What I REALLY want is for Kafka to track the new data (track the head of the 
> log) and then ask for the tail in chunks.  Ideally this would request from 
> the source, “what is the next logical older starting point?” and then start 
> there.  This way, the transfer basically becomes a file transfer of the log 
> stored on the source’s disk. Once that block is retrieved, it moves on to the 
> next oldest.  This way, there is almost zero waste as both the head and tail 
> grow, but the tail runs the risk of losing the final chunk only.  Thus, 
> bandwidth is not significantly wasted.
> All this changes the ISR check to be is “am I caught up on head AND tail?” 
> when the tail part is implied right now.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[VOTE] Release plan - Kafka 0.10.0

[Gwen Shapira]

Greetings Kafka Developer Community,

As you all know, we have few big features that are almost complete
(Timestamps! Interceptors! Streams!). It is time to start planning our
next release.

I suggest the following:
* Cut branches on March 21st
* Publish the first release candidate the next day
* Start testing, finding important issues, fixing them, rolling out new releases
* And eventually get a release candidate that we all agree is awesome
enough to release. Hopefully this won't take too many iterations :)

Note that this is a 2 weeks heads-up on branch cutting. After we cut
branches, we will try to minimize cherrypicks to just critical bugs
(because last major release was a bit insane).
Therefore,  if you have a feature that you really want to see in
0.10.0 - you'll need to have it committed by March 21st. As a curtesy
to the release manager, if you have features that you are not planning
on getting in for 0.10.0, please change the "fix version" field in
JIRA accordingly.

I will send a heads-up few days before cutting branches, to give
everyone a chance to get stragglers in.

The vote will be open for 72 hours.
All in favor, please reply with +1.

Gwen Shapira

Re: [VOTE] Release plan - Kafka 0.10.0

[Joe Stein]

+1

quick question/definition for the release cut (assuming vote passes) your
proposing please.

Critical bug fixes for new features and regression or just regression and
new feature can get pulled if not working right if less impactful to-do so?
Understandably that is dependent on the feature and/or fix but we have a
bunch on the plan for
https://cwiki.apache.org/confluence/display/KAFKA/Future+release+plan and
is that the hard freeze? I think the producer and consumer interceptors are
part of streams so maybe just an update on that?

+1 the timeline seems manageable and adjusting for what is released or not
doesn't affect the approach so g2g lots to get out

for 0.11 I would like to suggest trying to nominate or if volunteer always
good a release manager along with what is being worked on and collaborate
around for different KIP so folks know where to contribute and work
downstream too operational.

~ Joe Stein

On Mon, Mar 7, 2016 at 12:27 PM, Gwen Shapira  wrote:

> Greetings Kafka Developer Community,
>
> As you all know, we have few big features that are almost complete
> (Timestamps! Interceptors! Streams!). It is time to start planning our
> next release.
>
> I suggest the following:
> * Cut branches on March 21st
> * Publish the first release candidate the next day
> * Start testing, finding important issues, fixing them, rolling out new
> releases
> * And eventually get a release candidate that we all agree is awesome
> enough to release. Hopefully this won't take too many iterations :)
>
> Note that this is a 2 weeks heads-up on branch cutting. After we cut
> branches, we will try to minimize cherrypicks to just critical bugs
> (because last major release was a bit insane).
> Therefore,  if you have a feature that you really want to see in
> 0.10.0 - you'll need to have it committed by March 21st. As a curtesy
> to the release manager, if you have features that you are not planning
> on getting in for 0.10.0, please change the "fix version" field in
> JIRA accordingly.
>
> I will send a heads-up few days before cutting branches, to give
> everyone a chance to get stragglers in.
>
> The vote will be open for 72 hours.
> All in favor, please reply with +1.
>
> Gwen Shapira
>

[jira] [Updated] (KAFKA-3297) More optimally balanced partition assignment strategy (new consumer)

[Andrew Olson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Olson updated KAFKA-3297:

Fix Version/s: 0.10.0.0

> More optimally balanced partition assignment strategy (new consumer)
> 
>
> Key: KAFKA-3297
> URL: https://issues.apache.org/jira/browse/KAFKA-3297
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Andrew Olson
>Assignee: Andrew Olson
> Fix For: 0.10.0.0
>
>
> While the roundrobin partition assignment strategy is an improvement over the 
> range strategy, when the consumer topic subscriptions are not identical 
> (previously disallowed but will be possible as of KAFKA-2172) it can produce 
> heavily skewed assignments. As suggested 
> [here|https://issues.apache.org/jira/browse/KAFKA-2172?focusedCommentId=14530767&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14530767]
>  it would be nice to have a strategy that attempts to assign an equal number 
> of partitions to each consumer in a group, regardless of how similar their 
> individual topic subscriptions are. We can accomplish this by tracking the 
> number of partitions assigned to each consumer, and having the partition 
> assignment loop assign each partition to a consumer interested in that topic 
> with the least number of partitions assigned. 
> Additionally, we can optimize the distribution fairness by adjusting the 
> partition assignment order:
> * Topics with fewer consumers are assigned first.
> * In the event of a tie for least consumers, the topic with more partitions 
> is assigned first.
> The general idea behind these two rules is to keep the most flexible 
> assignment choices available as long as possible by starting with the most 
> constrained partitions/consumers.
> This JIRA addresses the new consumer. For the original high-level consumer, 
> see KAFKA-2435.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2435) More optimally balanced partition assignment strategy

[Andrew Olson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Olson updated KAFKA-2435:

Fix Version/s: 0.10.0.0

> More optimally balanced partition assignment strategy
> -
>
> Key: KAFKA-2435
> URL: https://issues.apache.org/jira/browse/KAFKA-2435
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Andrew Olson
>Assignee: Andrew Olson
> Fix For: 0.10.0.0
>
> Attachments: KAFKA-2435.patch
>
>
> While the roundrobin partition assignment strategy is an improvement over the 
> range strategy, when the consumer topic subscriptions are not identical 
> (previously disallowed but will be possible as of KAFKA-2172) it can produce 
> heavily skewed assignments. As suggested 
> [here|https://issues.apache.org/jira/browse/KAFKA-2172?focusedCommentId=14530767&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14530767]
>  it would be nice to have a strategy that attempts to assign an equal number 
> of partitions to each consumer in a group, regardless of how similar their 
> individual topic subscriptions are. We can accomplish this by tracking the 
> number of partitions assigned to each consumer, and having the partition 
> assignment loop assign each partition to a consumer interested in that topic 
> with the least number of partitions assigned. 
> Additionally, we can optimize the distribution fairness by adjusting the 
> partition assignment order:
> * Topics with fewer consumers are assigned first.
> * In the event of a tie for least consumers, the topic with more partitions 
> is assigned first.
> The general idea behind these two rules is to keep the most flexible 
> assignment choices available as long as possible by starting with the most 
> constrained partitions/consumers.
> This JIRA addresses the original high-level consumer. For the new consumer, 
> see KAFKA-3297.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2434) remove roundrobin identical topic constraint in consumer coordinator (old API)

[Andrew Olson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Olson updated KAFKA-2434:

Fix Version/s: 0.10.0.0

> remove roundrobin identical topic constraint in consumer coordinator (old API)
> --
>
> Key: KAFKA-2434
> URL: https://issues.apache.org/jira/browse/KAFKA-2434
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Andrew Olson
>Assignee: Andrew Olson
> Fix For: 0.10.0.0
>
> Attachments: KAFKA-2434.patch
>
>
> The roundrobin strategy algorithm improvement made in KAFKA-2196 should be 
> applied to the original high-level consumer API as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] Deprecating the old Scala producers for 0.10.0.0

[Joel Koshy]

+1

On Thu, Mar 3, 2016 at 2:36 PM, Ismael Juma  wrote:

> Hi all,
>
> The new Java producer was introduced in 0.8.2.0 (released in February
> 2015). It has become the default implementation for various tools since
> 0.9.0.0 (released in October 2015) and it is the only implementation with
> support for the security features introduced in 0.9.0.0.
>
> Given this, I think we should deprecate the old Scala producers for
> 0.10.0.0 by adding @deprecated annotations in the code and updating the the
> documentation to encourage usage of the new Java producer. This would give
> our users a stronger signal regarding our plans to focus on the new Java
> producer going forward.
>
> Note that this proposal is only about deprecating the old Scala producers
> as,
> in my opinion, it is too early to do the same for the old Scala consumers.
> The
> new Java consumer was only introduced in 0.9.0.0 and it's still marked as
> beta. It would be good to have a full release cycle where the new consumer
> is no longer in beta before we deprecate the old consumers. We are hoping
> to remove the beta label for the consumer for 0.10.0.0, but that's a
> separate discussion.
>
> With regards to removal of the deprecated producers, the current thinking
> is to remove all Scala clients at the same time, so it will take at least
> two non bug-fix release cycles (it could take longer depending on users'
> feedback).
>
> The feedback was mostly positive in the discuss thread although some points
> were raised about deprecating the old producers before deprecating the old
> consumers:
>
>
> http://search-hadoop.com/m/uyzND1KVJJmcbgAf2&subj=+DISCUSS+Deprecating+the+old+Scala+producers+for+the+next+release
>
> The JIRA for tracking this is KAFKA-2982.
>
> The vote will run for 72 hours.
>
> Thanks,
> Ismael
>

Re: [DISCUSS] KIP-49: Fair Partition Assignment Strategy

[Joel Koshy]

Hey Andrew,

Thanks for adding the example. No I don't think we have a jira open for
that issue. I'm not sure if we need to fix it in roundrobin (now that it's
already out there and some may be using it) vs. just going with your new
"fair" strategy and maybe add a new explicit roundrobinv2.

Thanks

Joel,

On Mon, Feb 29, 2016 at 7:53 AM, Olson,Andrew  wrote:

> Thanks for the feedback. I have added a concrete example to the document
> that I think illustrates the benefit relatively well.
>
> The observation about scaling the workload of individual consumers is
> certainly valid. I had not really considered this. Our primary concern is
> being able to gradually roll out consumption configuration changes in a
> minimally disruptive fashion, including load-balancing. If the round robin
> strategy can be enhanced to adequately handle that use case, we would be
> happy. Is there a Jira open for the "flaw" that you mentioned?
>
>
>
>
> On 2/26/16, 7:22 PM, "Joel Koshy"  wrote:
>
> >Hi Andrew,
> >
> >Thanks for the wiki. Just a couple of comments:
> >
> >   - The disruptive config change issue that you mentioned is pretty much
> a
> >   non-issue in the new consumer due to central assignment.
> >   - Optional: but it may be helpful to add a concrete example.
> >   - More of an orthogonal observation than a comment: with heavily skewed
> >   subscriptions fairness is sort of moot. i.e., people would generally
> scale
> >   up or down subscription counts with the express purpose of
> >   reducing/increasing load on those instances.
> >   - WRT roundrobin we later realized a significant flaw in the way we lay
> >   out partitions: we originally wanted to randomize the partition layout
> to
> >   reduce the likelihood of most partitions of the same topic from ending
> up
> >   on a given consumer which is important if you have a few very large
> topics.
> >   Unfortunately we used hashCode - which does a splendid job of clumping
> >   partitions from the same topic together :( We can probably just "fix"
> that
> >   in the new consumer's roundrobin assignor.
> >
> >Thanks,
> >
> >Joel
> >
> >
> >On Fri, Feb 26, 2016 at 2:32 PM, Olson,Andrew  wrote:
> >
> >> Here is a proposal for a new partition assignment strategy,
> >>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-49+-+Fair+Partition+Assignment+Strategy
> >>
> >> This KIP corresponds to these two pending pull requests,
> >> https://github.com/apache/kafka/pull/146
> >> https://github.com/apache/kafka/pull/979
> >>
> >> thanks,
> >> Andrew
> >>
> >> CONFIDENTIALITY NOTICE This message and any included attachments are
> from
> >> Cerner Corporation and are intended only for the addressee. The
> information
> >> contained in this message is confidential and may constitute inside or
> >> non-public information under international, federal, or state securities
> >> laws. Unauthorized forwarding, printing, copying, distribution, or use
> of
> >> such information is strictly prohibited and may be unlawful. If you are
> not
> >> the addressee, please promptly delete this message and notify the
> sender of
> >> the delivery error by e-mail or you may call Cerner's corporate offices
> in
> >> Kansas City, Missouri, U.S.A at (+1) (816)221-1024.
> >>
>

[jira] [Commented] (KAFKA-3233) Per topic consumer metrics missing in new consumer's metrics reporter

[Yifan Ying (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183449#comment-15183449
 ] 

Yifan Ying commented on KAFKA-3233:
---

Hi Guozhang, thanks for merging my change. Is there any chance this fix could 
get into the next release? 

> Per topic consumer metrics missing in new consumer's metrics reporter
> -
>
> Key: KAFKA-3233
> URL: https://issues.apache.org/jira/browse/KAFKA-3233
> Project: Kafka
>  Issue Type: Bug
>Reporter: Yifan Ying
> Fix For: 0.10.0.0
>
>
> In version of 0.8.2.1, the old consumer will provide the metrics reporter 
> per-topic consumer metrics under group 'ConsumerTopicMetrics'. For example:
> *.ConsumerTopicMetrics.clientId.[topic name].BytesPerSec.count
> *.ConsumerTopicMetrics.clientId.[topic name].MessagesPerSec.count
> These consumer metrics are useful since it provides consumer rate for each 
> topic. But the new consumer(0.9.0.0) doesn't expose per topic metrics 
> anymore, even though I did find sensor objects in consumer metrics object 
> collecting per-topic metrics. 
> After investigation, I found that these sensors don't register any 
> KafkaMetrics.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-2273: Sticky partition assignment strate...

[vahidhashemian]

GitHub user vahidhashemian opened a pull request:

https://github.com/apache/kafka/pull/1020

KAFKA-2273: Sticky partition assignment strategy

This PR implements a new partition assignment strategy called "sticky", and 
it's purpose is to balance partitions across consumers in a way that minimizes 
moving partitions around, or, in other words, preserves existing partition 
assignments as much as possible.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vahidhashemian/kafka KAFKA-2273

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1020.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1020


commit 1f774ea784e57377752f55dd4fe91212e8bcbda1
Author: Vahid Hashemian 
Date:   2016-02-18T13:44:17Z

KAFKA-2273: Sticky partition assignment strategy

This PR implements a new partition assignment strategy called "sticky", and 
it's purpose is to balance partitions across consumers in a way that minimizes 
moving partitions around, or in other words, preserving partition assignments 
as much as possible.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Ashish Singh]

On Fri, Mar 4, 2016 at 5:46 PM, Jay Kreps  wrote:

> Hey Ashish,
>
> Both good points.
>
> I think the issue with the general metadata request is the same as the
> issue with a version-specific metadata request from the other
> proposal--basically it's a chicken and egg problem, to find out anything
> about the cluster you have to be able to communicate something in a format
> the server can understand without knowing a priori what version it's on. I
> guess the question is how can you continue to evolve the metadata request
> (whether it is the existing metadata or a protocol-version specific
> metadata request) given that you need this information to bootstrap you
> have to be more careful in how that request evolves.
>
You are correct. It's just that protocol version request would be very
specific to retrieve the protocol versions. Changes to protocol version
request itself should be very rare, if at all. However, the general
metadata request carries a lot more information and its format is more
probable to evolve. This boils down to higher probability of change vs a
definite network round-trip for each re/connect. It does sound like, it is
better to avoid a definite penalty than to avoid a probable rare issue.

>
> I think deprecation/removal may be okay. Ultimately clients will always use
> the highest possible version of the protocol the server supports so if
> we've already deprecated and removed your highest version then you are
> screwed and you're going to get an error no matter what, right? Basically
> there is nothing dynamic you can do in that case.
>
Sure, this should be expected. Just wanted to make sure deprecation is
still on the table.

>
> -Jay
>
> On Fri, Mar 4, 2016 at 4:05 PM, Ashish Singh  wrote:
>
> > Hello Jay,
> >
> > The overall approach sounds good. I do realize that this discussion has
> > gotten too lengthy and is starting to shoot tangents. Maybe a KIP call
> will
> > help us getting to a decision faster. I do have a few questions though.
> >
> > On Fri, Mar 4, 2016 at 9:52 AM, Jay Kreps  wrote:
> >
> > > Yeah here is my summary of my take:
> > >
> > > 1. Negotiating a per-connection protocol actually does add a lot of
> > > complexity to clients (many more failure states to get right).
> > >
> > > 2. Having the client configure the protocol version manually is doable
> > now
> > > but probably a worse state. I suspect this will lead to more not less
> > > confusion.
> > >
> > > 3. I don't think the current state is actually that bad. Integrators
> > pick a
> > > conservative version and build against that. There is a tradeoff
> between
> > > getting the new features and being compatible with old Kafka versions.
> > But
> > > a large part of this tradeoff is essential since new features aren't
> > going
> > > to magically appear on old servers, so even if you upgrade your client
> > you
> > > likely aren't going to get the new stuff (since we will end up
> > dynamically
> > > turning it off). Having client features that are there but don't work
> > > because you're on an old cluster may actually be a worse experience if
> > not
> > > handled very carefully..
> > >
> > > 4. The problems Dana brought up are totally orthogonal to the problem
> of
> > > having per-api versions or overall versions. The problem was that we
> > > changed behavior subtly without changing the version. This will be an
> > issue
> > > regardless of whether the version is global or not.
> > >
> > > 5. Using the broker release as the version is strictly worse than
> using a
> > > global protocol version (0, 1, 2, ...) that increments any time any api
> > > changes but doesn't increment just because non-protocol code is
> changed.
> > > The problem with using the broker release version is we want to be able
> > to
> > > keep Kafka releasable from any commit which means there isn't as clear
> a
> > > sequencing of releases as you would think.
> > >
> > > 6. We need to consider the case of mixed version clusters during the
> time
> > > period when you are upgrading Kafka.
> > >
> > > So overall I think this is not a critical thing to do right now, but if
> > we
> > > are going to do it we should do it in a way that actually improves
> > things.
> > >
> > > Here would be one proposal for that:
> > > a. Add a global protocol version that increments with any api version
> > > update. Move the documentation so that the docs are by version. This is
> > > basically just a short-hand for a complete set of supported api
> versions.
> > > b. Include a field in the metadata response for each broker that adds
> the
> > > protocol version.
> > >
> > There might be an issue here where the metadata request version sent by
> > client is not supported by broker, an older broker. However, if we are
> > clearly stating that a client is not guaranteed to work with an older
> > broker then this becomes expected. This will potentially limit us in
> terms
> > of supporting downgrades though, if we ever want to.
> >
> > > c. To ma

[jira] [Commented] (KAFKA-2273) Add rebalance with a minimal number of reassignments to server-defined strategy list

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183462#comment-15183462
 ] 

ASF GitHub Bot commented on KAFKA-2273:
---

GitHub user vahidhashemian opened a pull request:

https://github.com/apache/kafka/pull/1020

KAFKA-2273: Sticky partition assignment strategy

This PR implements a new partition assignment strategy called "sticky", and 
it's purpose is to balance partitions across consumers in a way that minimizes 
moving partitions around, or, in other words, preserves existing partition 
assignments as much as possible.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vahidhashemian/kafka KAFKA-2273

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1020.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1020


commit 1f774ea784e57377752f55dd4fe91212e8bcbda1
Author: Vahid Hashemian 
Date:   2016-02-18T13:44:17Z

KAFKA-2273: Sticky partition assignment strategy

This PR implements a new partition assignment strategy called "sticky", and 
it's purpose is to balance partitions across consumers in a way that minimizes 
moving partitions around, or in other words, preserving partition assignments 
as much as possible.




> Add rebalance with a minimal number of reassignments to server-defined 
> strategy list
> 
>
> Key: KAFKA-2273
> URL: https://issues.apache.org/jira/browse/KAFKA-2273
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Olof Johansson
>Assignee: Vahid Hashemian
>  Labels: newbie++, newbiee
> Fix For: 0.10.1.0
>
>
> Add a new partitions.assignment.strategy to the server-defined list that will 
> do reassignments based on moving as few partitions as possible. This should 
> be a quite common reassignment strategy especially for the cases where the 
> consumer has to maintain state, either in memory, or on disk.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] Deprecating the old Scala producers for 0.10.0.0

[Ewen Cheslack-Postava]

+1

On Mon, Mar 7, 2016 at 10:16 AM, Joel Koshy  wrote:

> +1
>
> On Thu, Mar 3, 2016 at 2:36 PM, Ismael Juma  wrote:
>
> > Hi all,
> >
> > The new Java producer was introduced in 0.8.2.0 (released in February
> > 2015). It has become the default implementation for various tools since
> > 0.9.0.0 (released in October 2015) and it is the only implementation with
> > support for the security features introduced in 0.9.0.0.
> >
> > Given this, I think we should deprecate the old Scala producers for
> > 0.10.0.0 by adding @deprecated annotations in the code and updating the
> the
> > documentation to encourage usage of the new Java producer. This would
> give
> > our users a stronger signal regarding our plans to focus on the new Java
> > producer going forward.
> >
> > Note that this proposal is only about deprecating the old Scala producers
> > as,
> > in my opinion, it is too early to do the same for the old Scala
> consumers.
> > The
> > new Java consumer was only introduced in 0.9.0.0 and it's still marked as
> > beta. It would be good to have a full release cycle where the new
> consumer
> > is no longer in beta before we deprecate the old consumers. We are hoping
> > to remove the beta label for the consumer for 0.10.0.0, but that's a
> > separate discussion.
> >
> > With regards to removal of the deprecated producers, the current thinking
> > is to remove all Scala clients at the same time, so it will take at least
> > two non bug-fix release cycles (it could take longer depending on users'
> > feedback).
> >
> > The feedback was mostly positive in the discuss thread although some
> points
> > were raised about deprecating the old producers before deprecating the
> old
> > consumers:
> >
> >
> >
> http://search-hadoop.com/m/uyzND1KVJJmcbgAf2&subj=+DISCUSS+Deprecating+the+old+Scala+producers+for+the+next+release
> >
> > The JIRA for tracking this is KAFKA-2982.
> >
> > The vote will run for 72 hours.
> >
> > Thanks,
> > Ismael
> >
>



-- 
Thanks,
Ewen

[jira] [Commented] (KAFKA-3233) Per topic consumer metrics missing in new consumer's metrics reporter

[Gwen Shapira (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183502#comment-15183502
 ] 

Gwen Shapira commented on KAFKA-3233:
-

Since the change was merged into trunk, it will be included in 0.10.0 release.

> Per topic consumer metrics missing in new consumer's metrics reporter
> -
>
> Key: KAFKA-3233
> URL: https://issues.apache.org/jira/browse/KAFKA-3233
> Project: Kafka
>  Issue Type: Bug
>Reporter: Yifan Ying
> Fix For: 0.10.0.0
>
>
> In version of 0.8.2.1, the old consumer will provide the metrics reporter 
> per-topic consumer metrics under group 'ConsumerTopicMetrics'. For example:
> *.ConsumerTopicMetrics.clientId.[topic name].BytesPerSec.count
> *.ConsumerTopicMetrics.clientId.[topic name].MessagesPerSec.count
> These consumer metrics are useful since it provides consumer rate for each 
> topic. But the new consumer(0.9.0.0) doesn't expose per topic metrics 
> anymore, even though I did find sensor objects in consumer metrics object 
> collecting per-topic metrics. 
> After investigation, I found that these sensors don't register any 
> KafkaMetrics.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-3339: Fix java docs for seekToEnd and se...

[SinghAsDev]

GitHub user SinghAsDev opened a pull request:

https://github.com/apache/kafka/pull/1021

KAFKA-3339: Fix java docs for seekToEnd and seekToBeginning.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/SinghAsDev/kafka KAFKA-3339

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1021.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1021


commit 56714878b519edae53fb14e8502aed5bd7a50137
Author: Ashish Singh 
Date:   2016-03-07T19:20:22Z

KAFKA-3339: Fix java docs for seekToEnd and seekToBeginning.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3339) org.apache.kafka.clients.consumer.KafkaConsumer javadoc for seekToBeginning, seekToEnd incomplete

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183511#comment-15183511
 ] 

ASF GitHub Bot commented on KAFKA-3339:
---

GitHub user SinghAsDev opened a pull request:

https://github.com/apache/kafka/pull/1021

KAFKA-3339: Fix java docs for seekToEnd and seekToBeginning.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/SinghAsDev/kafka KAFKA-3339

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1021.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1021


commit 56714878b519edae53fb14e8502aed5bd7a50137
Author: Ashish Singh 
Date:   2016-03-07T19:20:22Z

KAFKA-3339: Fix java docs for seekToEnd and seekToBeginning.




> org.apache.kafka.clients.consumer.KafkaConsumer javadoc for seekToBeginning, 
> seekToEnd incomplete
> -
>
> Key: KAFKA-3339
> URL: https://issues.apache.org/jira/browse/KAFKA-3339
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Affects Versions: 0.9.0.1
>Reporter: Harald Kirsch
>
> The api documentation for seekToBeginning and seekToEnd in  
> org.apache.kafka.clients.consumer.KafkaConsumer these methods should remark 
> that all subscribed partitions are seeked if no TopicPartition is provided.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] Release plan - Kafka 0.10.0

[Ewen Cheslack-Postava]

+1

On Mon, Mar 7, 2016 at 9:42 AM, Joe Stein  wrote:

> +1
>
> quick question/definition for the release cut (assuming vote passes) your
> proposing please.
>
> Critical bug fixes for new features and regression or just regression and
> new feature can get pulled if not working right if less impactful to-do so?
> Understandably that is dependent on the feature and/or fix but we have a
> bunch on the plan for
> https://cwiki.apache.org/confluence/display/KAFKA/Future+release+plan and
> is that the hard freeze? I think the producer and consumer interceptors are
> part of streams so maybe just an update on that?
>
> +1 the timeline seems manageable and adjusting for what is released or not
> doesn't affect the approach so g2g lots to get out
>
> for 0.11 I would like to suggest trying to nominate or if volunteer always
> good a release manager along with what is being worked on and collaborate
> around for different KIP so folks know where to contribute and work
> downstream too operational.
>
> ~ Joe Stein
>
> On Mon, Mar 7, 2016 at 12:27 PM, Gwen Shapira  wrote:
>
> > Greetings Kafka Developer Community,
> >
> > As you all know, we have few big features that are almost complete
> > (Timestamps! Interceptors! Streams!). It is time to start planning our
> > next release.
> >
> > I suggest the following:
> > * Cut branches on March 21st
> > * Publish the first release candidate the next day
> > * Start testing, finding important issues, fixing them, rolling out new
> > releases
> > * And eventually get a release candidate that we all agree is awesome
> > enough to release. Hopefully this won't take too many iterations :)
> >
> > Note that this is a 2 weeks heads-up on branch cutting. After we cut
> > branches, we will try to minimize cherrypicks to just critical bugs
> > (because last major release was a bit insane).
> > Therefore,  if you have a feature that you really want to see in
> > 0.10.0 - you'll need to have it committed by March 21st. As a curtesy
> > to the release manager, if you have features that you are not planning
> > on getting in for 0.10.0, please change the "fix version" field in
> > JIRA accordingly.
> >
> > I will send a heads-up few days before cutting branches, to give
> > everyone a chance to get stragglers in.
> >
> > The vote will be open for 72 hours.
> > All in favor, please reply with +1.
> >
> > Gwen Shapira
> >
>



-- 
Thanks,
Ewen

Re: [VOTE] Release plan - Kafka 0.10.0

[Gwen Shapira]

Hi Joe,

The branch cutting means that new features will go into trunk, bug
fixes will go into either trunk and branch or just trunk - depending
on committer decision. Committers should take into consideration the
importance of the fix vs the stability of the planned release. I don't
have specific rules in mind - our committers have excellent judgement
:)

I hope this clarifies?

Gwen



On Mon, Mar 7, 2016 at 9:42 AM, Joe Stein  wrote:
> +1
>
> quick question/definition for the release cut (assuming vote passes) your
> proposing please.
>
> Critical bug fixes for new features and regression or just regression and
> new feature can get pulled if not working right if less impactful to-do so?
> Understandably that is dependent on the feature and/or fix but we have a
> bunch on the plan for
> https://cwiki.apache.org/confluence/display/KAFKA/Future+release+plan and
> is that the hard freeze? I think the producer and consumer interceptors are
> part of streams so maybe just an update on that?
>
> +1 the timeline seems manageable and adjusting for what is released or not
> doesn't affect the approach so g2g lots to get out
>
> for 0.11 I would like to suggest trying to nominate or if volunteer always
> good a release manager along with what is being worked on and collaborate
> around for different KIP so folks know where to contribute and work
> downstream too operational.
>
> ~ Joe Stein
>
> On Mon, Mar 7, 2016 at 12:27 PM, Gwen Shapira  wrote:
>
>> Greetings Kafka Developer Community,
>>
>> As you all know, we have few big features that are almost complete
>> (Timestamps! Interceptors! Streams!). It is time to start planning our
>> next release.
>>
>> I suggest the following:
>> * Cut branches on March 21st
>> * Publish the first release candidate the next day
>> * Start testing, finding important issues, fixing them, rolling out new
>> releases
>> * And eventually get a release candidate that we all agree is awesome
>> enough to release. Hopefully this won't take too many iterations :)
>>
>> Note that this is a 2 weeks heads-up on branch cutting. After we cut
>> branches, we will try to minimize cherrypicks to just critical bugs
>> (because last major release was a bit insane).
>> Therefore,  if you have a feature that you really want to see in
>> 0.10.0 - you'll need to have it committed by March 21st. As a curtesy
>> to the release manager, if you have features that you are not planning
>> on getting in for 0.10.0, please change the "fix version" field in
>> JIRA accordingly.
>>
>> I will send a heads-up few days before cutting branches, to give
>> everyone a chance to get stragglers in.
>>
>> The vote will be open for 72 hours.
>> All in favor, please reply with +1.
>>
>> Gwen Shapira
>>

[jira] [Commented] (KAFKA-3339) org.apache.kafka.clients.consumer.KafkaConsumer javadoc for seekToBeginning, seekToEnd incomplete

[Ashish K Singh (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183517#comment-15183517
 ] 

Ashish K Singh commented on KAFKA-3339:
---

[~haraldk] could you review the PR. Thanks for reporting the issue.

> org.apache.kafka.clients.consumer.KafkaConsumer javadoc for seekToBeginning, 
> seekToEnd incomplete
> -
>
> Key: KAFKA-3339
> URL: https://issues.apache.org/jira/browse/KAFKA-3339
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Affects Versions: 0.9.0.1
>Reporter: Harald Kirsch
>
> The api documentation for seekToBeginning and seekToEnd in  
> org.apache.kafka.clients.consumer.KafkaConsumer these methods should remark 
> that all subscribed partitions are seeked if no TopicPartition is provided.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] Release plan - Kafka 0.10.0

[Neha Narkhede]

+1 (binding)

On Mon, Mar 7, 2016 at 11:26 AM, Gwen Shapira  wrote:

> Hi Joe,
>
> The branch cutting means that new features will go into trunk, bug
> fixes will go into either trunk and branch or just trunk - depending
> on committer decision. Committers should take into consideration the
> importance of the fix vs the stability of the planned release. I don't
> have specific rules in mind - our committers have excellent judgement
> :)
>
> I hope this clarifies?
>
> Gwen
>
>
>
> On Mon, Mar 7, 2016 at 9:42 AM, Joe Stein  wrote:
> > +1
> >
> > quick question/definition for the release cut (assuming vote passes) your
> > proposing please.
> >
> > Critical bug fixes for new features and regression or just regression and
> > new feature can get pulled if not working right if less impactful to-do
> so?
> > Understandably that is dependent on the feature and/or fix but we have a
> > bunch on the plan for
> > https://cwiki.apache.org/confluence/display/KAFKA/Future+release+plan
> and
> > is that the hard freeze? I think the producer and consumer interceptors
> are
> > part of streams so maybe just an update on that?
> >
> > +1 the timeline seems manageable and adjusting for what is released or
> not
> > doesn't affect the approach so g2g lots to get out
> >
> > for 0.11 I would like to suggest trying to nominate or if volunteer
> always
> > good a release manager along with what is being worked on and collaborate
> > around for different KIP so folks know where to contribute and work
> > downstream too operational.
> >
> > ~ Joe Stein
> >
> > On Mon, Mar 7, 2016 at 12:27 PM, Gwen Shapira  wrote:
> >
> >> Greetings Kafka Developer Community,
> >>
> >> As you all know, we have few big features that are almost complete
> >> (Timestamps! Interceptors! Streams!). It is time to start planning our
> >> next release.
> >>
> >> I suggest the following:
> >> * Cut branches on March 21st
> >> * Publish the first release candidate the next day
> >> * Start testing, finding important issues, fixing them, rolling out new
> >> releases
> >> * And eventually get a release candidate that we all agree is awesome
> >> enough to release. Hopefully this won't take too many iterations :)
> >>
> >> Note that this is a 2 weeks heads-up on branch cutting. After we cut
> >> branches, we will try to minimize cherrypicks to just critical bugs
> >> (because last major release was a bit insane).
> >> Therefore,  if you have a feature that you really want to see in
> >> 0.10.0 - you'll need to have it committed by March 21st. As a curtesy
> >> to the release manager, if you have features that you are not planning
> >> on getting in for 0.10.0, please change the "fix version" field in
> >> JIRA accordingly.
> >>
> >> I will send a heads-up few days before cutting branches, to give
> >> everyone a chance to get stragglers in.
> >>
> >> The vote will be open for 72 hours.
> >> All in favor, please reply with +1.
> >>
> >> Gwen Shapira
> >>
>



-- 
Thanks,
Neha

[GitHub] kafka pull request: KAFKA-3339: Fix java docs for seekToEnd and se...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1021


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (KAFKA-3339) org.apache.kafka.clients.consumer.KafkaConsumer javadoc for seekToBeginning, seekToEnd incomplete

[Gwen Shapira (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gwen Shapira resolved KAFKA-3339.
-
   Resolution: Fixed
Fix Version/s: 0.10.0.0

Issue resolved by pull request 1021
[https://github.com/apache/kafka/pull/1021]

> org.apache.kafka.clients.consumer.KafkaConsumer javadoc for seekToBeginning, 
> seekToEnd incomplete
> -
>
> Key: KAFKA-3339
> URL: https://issues.apache.org/jira/browse/KAFKA-3339
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Affects Versions: 0.9.0.1
>Reporter: Harald Kirsch
>Assignee: Ashish K Singh
> Fix For: 0.10.0.0
>
>
> The api documentation for seekToBeginning and seekToEnd in  
> org.apache.kafka.clients.consumer.KafkaConsumer these methods should remark 
> that all subscribed partitions are seeked if no TopicPartition is provided.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (KAFKA-3339) org.apache.kafka.clients.consumer.KafkaConsumer javadoc for seekToBeginning, seekToEnd incomplete

[Ashish K Singh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish K Singh reassigned KAFKA-3339:
-

Assignee: Ashish K Singh

> org.apache.kafka.clients.consumer.KafkaConsumer javadoc for seekToBeginning, 
> seekToEnd incomplete
> -
>
> Key: KAFKA-3339
> URL: https://issues.apache.org/jira/browse/KAFKA-3339
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Affects Versions: 0.9.0.1
>Reporter: Harald Kirsch
>Assignee: Ashish K Singh
> Fix For: 0.10.0.0
>
>
> The api documentation for seekToBeginning and seekToEnd in  
> org.apache.kafka.clients.consumer.KafkaConsumer these methods should remark 
> that all subscribed partitions are seeked if no TopicPartition is provided.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3339) org.apache.kafka.clients.consumer.KafkaConsumer javadoc for seekToBeginning, seekToEnd incomplete

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183523#comment-15183523
 ] 

ASF GitHub Bot commented on KAFKA-3339:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1021


> org.apache.kafka.clients.consumer.KafkaConsumer javadoc for seekToBeginning, 
> seekToEnd incomplete
> -
>
> Key: KAFKA-3339
> URL: https://issues.apache.org/jira/browse/KAFKA-3339
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Affects Versions: 0.9.0.1
>Reporter: Harald Kirsch
>Assignee: Ashish K Singh
> Fix For: 0.10.0.0
>
>
> The api documentation for seekToBeginning and seekToEnd in  
> org.apache.kafka.clients.consumer.KafkaConsumer these methods should remark 
> that all subscribed partitions are seeked if no TopicPartition is provided.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] Release plan - Kafka 0.10.0

[Ismael Juma]

+1 (non-binding)

Ismael

On Mon, Mar 7, 2016 at 7:27 PM, Neha Narkhede  wrote:

> +1 (binding)
>
> On Mon, Mar 7, 2016 at 11:26 AM, Gwen Shapira  wrote:
>
> > Hi Joe,
> >
> > The branch cutting means that new features will go into trunk, bug
> > fixes will go into either trunk and branch or just trunk - depending
> > on committer decision. Committers should take into consideration the
> > importance of the fix vs the stability of the planned release. I don't
> > have specific rules in mind - our committers have excellent judgement
> > :)
> >
> > I hope this clarifies?
> >
> > Gwen
> >
> >
> >
> > On Mon, Mar 7, 2016 at 9:42 AM, Joe Stein  wrote:
> > > +1
> > >
> > > quick question/definition for the release cut (assuming vote passes)
> your
> > > proposing please.
> > >
> > > Critical bug fixes for new features and regression or just regression
> and
> > > new feature can get pulled if not working right if less impactful to-do
> > so?
> > > Understandably that is dependent on the feature and/or fix but we have
> a
> > > bunch on the plan for
> > > https://cwiki.apache.org/confluence/display/KAFKA/Future+release+plan
> > and
> > > is that the hard freeze? I think the producer and consumer interceptors
> > are
> > > part of streams so maybe just an update on that?
> > >
> > > +1 the timeline seems manageable and adjusting for what is released or
> > not
> > > doesn't affect the approach so g2g lots to get out
> > >
> > > for 0.11 I would like to suggest trying to nominate or if volunteer
> > always
> > > good a release manager along with what is being worked on and
> collaborate
> > > around for different KIP so folks know where to contribute and work
> > > downstream too operational.
> > >
> > > ~ Joe Stein
> > >
> > > On Mon, Mar 7, 2016 at 12:27 PM, Gwen Shapira 
> wrote:
> > >
> > >> Greetings Kafka Developer Community,
> > >>
> > >> As you all know, we have few big features that are almost complete
> > >> (Timestamps! Interceptors! Streams!). It is time to start planning our
> > >> next release.
> > >>
> > >> I suggest the following:
> > >> * Cut branches on March 21st
> > >> * Publish the first release candidate the next day
> > >> * Start testing, finding important issues, fixing them, rolling out
> new
> > >> releases
> > >> * And eventually get a release candidate that we all agree is awesome
> > >> enough to release. Hopefully this won't take too many iterations :)
> > >>
> > >> Note that this is a 2 weeks heads-up on branch cutting. After we cut
> > >> branches, we will try to minimize cherrypicks to just critical bugs
> > >> (because last major release was a bit insane).
> > >> Therefore,  if you have a feature that you really want to see in
> > >> 0.10.0 - you'll need to have it committed by March 21st. As a curtesy
> > >> to the release manager, if you have features that you are not planning
> > >> on getting in for 0.10.0, please change the "fix version" field in
> > >> JIRA accordingly.
> > >>
> > >> I will send a heads-up few days before cutting branches, to give
> > >> everyone a chance to get stragglers in.
> > >>
> > >> The vote will be open for 72 hours.
> > >> All in favor, please reply with +1.
> > >>
> > >> Gwen Shapira
> > >>
> >
>
>
>
> --
> Thanks,
> Neha
>

[GitHub] kafka pull request: MINOR: streams javadoc corrections

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1019


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3341) Improve error handling on invalid requests

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183532#comment-15183532
 ] 

ASF GitHub Bot commented on KAFKA-3341:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1017


> Improve error handling on invalid requests
> --
>
> Key: KAFKA-3341
> URL: https://issues.apache.org/jira/browse/KAFKA-3341
> Project: Kafka
>  Issue Type: Bug
>  Components: network
>Affects Versions: 0.9.0.1
>Reporter: Ismael Juma
>Assignee: Ismael Juma
> Fix For: 0.10.0.0
>
>
> We should include the request id in the error message if parsing of 
> `RequestHeader` fails and we should not try to mute the selector after 
> closing the connection due to an error (as it causes a second exception to be 
> thrown).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-3341: Improve error handling on invalid ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1017


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-3341) Improve error handling on invalid requests

[Gwen Shapira (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gwen Shapira updated KAFKA-3341:

Resolution: Fixed
Status: Resolved  (was: Patch Available)

Issue resolved by pull request 1017
[https://github.com/apache/kafka/pull/1017]

> Improve error handling on invalid requests
> --
>
> Key: KAFKA-3341
> URL: https://issues.apache.org/jira/browse/KAFKA-3341
> Project: Kafka
>  Issue Type: Bug
>  Components: network
>Affects Versions: 0.9.0.1
>Reporter: Ismael Juma
>Assignee: Ismael Juma
> Fix For: 0.10.0.0
>
>
> We should include the request id in the error message if parsing of 
> `RequestHeader` fails and we should not try to mute the selector after 
> closing the connection due to an error (as it causes a second exception to be 
> thrown).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] KIP-43: Kafka SASL enhancements

[Rajini Sivaram]

It will be good to get the SASL enhancements into 0.10.0.0 if possible.
There are no outstanding questions related to the KIP and the PR is ready
for review.

Feedback is appreciated.

Thank you...

On Thu, Mar 3, 2016 at 10:37 AM, Rajini Sivaram <
rajinisiva...@googlemail.com> wrote:

>
> I would like to start the voting process for *KIP-43: Kafka SASL
> enhancements*. This KIP extends the SASL implementation in Kafka
> to support new SASL mechanisms to enable Kafka to be integrated with
> different authentication servers.
>
> The KIP is available here for reference:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-43:+Kafka+SASL+enhancements
>
> And here's is a link to the discussion on the mailing list:
>
> http://mail-archives.apache.org/mod_mbox/kafka-dev/201601.mbox/%3CCAOJcB39b9Vy7%3DZEM3tLw2zarCS4A_s-%2BU%2BC%3DuEcWs0712UaYrQ%40mail.gmail.com%3E
>
>
> Thank you...
>
> Regards,
>
> Rajini
>



-- 
Regards,

Rajini

[GitHub] kafka pull request: KAFKA-3334: Add a note on potential message lo...

[SinghAsDev]

GitHub user SinghAsDev opened a pull request:

https://github.com/apache/kafka/pull/1022

KAFKA-3334: Add a note on potential message loss while using auto top…

…ics creation.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/SinghAsDev/kafka KAFKA-3334

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1022.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1022


commit d78df396dfbcc0b5e13765dfff33f54156cb9c5e
Author: Ashish Singh 
Date:   2016-03-07T19:46:58Z

KAFKA-3334: Add a note on potential message loss while using auto topics 
creation.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3334) First message on new topic not actually being sent, no exception thrown

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183559#comment-15183559
 ] 

ASF GitHub Bot commented on KAFKA-3334:
---

GitHub user SinghAsDev opened a pull request:

https://github.com/apache/kafka/pull/1022

KAFKA-3334: Add a note on potential message loss while using auto top…

…ics creation.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/SinghAsDev/kafka KAFKA-3334

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1022.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1022


commit d78df396dfbcc0b5e13765dfff33f54156cb9c5e
Author: Ashish Singh 
Date:   2016-03-07T19:46:58Z

KAFKA-3334: Add a note on potential message loss while using auto topics 
creation.




> First message on new topic not actually being sent, no exception thrown
> ---
>
> Key: KAFKA-3334
> URL: https://issues.apache.org/jira/browse/KAFKA-3334
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.9.0.0
> Environment: Linux, Java
>Reporter: Aleksandar Stojadinovic
>Assignee: Ashish K Singh
>
> Although I've seen this issue pop around the internet in a few forms, I'm not 
> sure it is yet properly fixed. 
> When publishing to a new topic, with auto create-enabled, the java client 
> (0.9.0) shows this WARN message in the log, and the message is not sent 
> obviously:
> org.apache.kafka.clients.NetworkClient - Error while fetching metadata with 
> correlation id 0 : {file.topic=LEADER_NOT_AVAILABLE}
> In the meantime I see in the console the message that a log for partition is 
> created. The next messages are patched through normally, but the first one is 
> never sent. No exception is ever thrown, either by calling get on the future, 
> or with the async usage, like everything is perfect.
> I notice when I leave my application blocked on the get call, in the 
> debugger, then the message may be processed, but with significant delay. This 
> is consistent with another issue I found for the python client. Also, if I 
> call partitionsFor previously, the topic is created and the message is sent. 
> But it seems silly to call it every time, just to mitigate this issue.
> {code}
> Future recordMetadataFuture = producer.send(new 
> ProducerRecord<>(topic, key, file));
> RecordMetadata recordMetadata = recordMetadataFuture.get(30, 
> TimeUnit.SECONDS);
> {code}
> I hope I'm clear enough.
> Related similar (but not same) issues:
> https://issues.apache.org/jira/browse/KAFKA-1124
> https://github.com/dpkp/kafka-python/issues/150
> http://stackoverflow.com/questions/35187933/how-to-resolve-leader-not-available-kafka-error-when-trying-to-consume



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: kafka-trunk-jdk7 #1095

[Apache Jenkins Server]

See 

Changes:

[cshapi] KAFKA-3339: Fix java docs for seekToEnd and seekToBeginning.

[cshapi] MINOR: streams javadoc corrections

[cshapi] KAFKA-3341: Improve error handling on invalid requests

--
[...truncated 48 lines...]
Building project 'core' with Scala version 2.10.6
:kafka-trunk-jdk7:clients:compileJavaNote: 

 uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

:kafka-trunk-jdk7:clients:processResources UP-TO-DATE
:kafka-trunk-jdk7:clients:classes
:kafka-trunk-jdk7:clients:determineCommitId UP-TO-DATE
:kafka-trunk-jdk7:clients:createVersionFile
:kafka-trunk-jdk7:clients:jar
:kafka-trunk-jdk7:core:compileJava UP-TO-DATE
:kafka-trunk-jdk7:core:compileScala
:79:
 value DEFAULT_TIMESTAMP in object OffsetCommitRequest is deprecated: see 
corresponding Javadoc for more information.

org.apache.kafka.common.requests.OffsetCommitRequest.DEFAULT_TIMESTAMP
 ^
:36:
 value DEFAULT_TIMESTAMP in object OffsetCommitRequest is deprecated: see 
corresponding Javadoc for more information.
 commitTimestamp: Long = 
org.apache.kafka.common.requests.OffsetCommitRequest.DEFAULT_TIMESTAMP,

  ^
:37:
 value DEFAULT_TIMESTAMP in object OffsetCommitRequest is deprecated: see 
corresponding Javadoc for more information.
 expireTimestamp: Long = 
org.apache.kafka.common.requests.OffsetCommitRequest.DEFAULT_TIMESTAMP) {

  ^
:400:
 value DEFAULT_TIMESTAMP in object OffsetCommitRequest is deprecated: see 
corresponding Javadoc for more information.
  if (value.expireTimestamp == 
org.apache.kafka.common.requests.OffsetCommitRequest.DEFAULT_TIMESTAMP)

^
:298:
 value DEFAULT_TIMESTAMP in object OffsetCommitRequest is deprecated: see 
corresponding Javadoc for more information.
if (offsetAndMetadata.commitTimestamp == 
org.apache.kafka.common.requests.OffsetCommitRequest.DEFAULT_TIMESTAMP)

  ^
:307:
 a pure expression does nothing in statement position; you may be omitting 
necessary parentheses
ControllerStats.uncleanLeaderElectionRate
^
:308:
 a pure expression does nothing in statement position; you may be omitting 
necessary parentheses
ControllerStats.leaderElectionTimer
^
:391:
 constructor UpdateMetadataRequest in class UpdateMetadataRequest is 
deprecated: see corresponding Javadoc for more information.
new UpdateMetadataRequest(controllerId, controllerEpoch, 
liveBrokers.asJava, partitionStates.asJava)
^
:129:
 method readFromReadableChannel in class NetworkReceive is deprecated: see 
corresponding Javadoc for more information.
  response.readFromReadableChannel(channel)
   ^
there were 15 feature warning(s); re-run with -feature for details
10 warnings found
:kafka-trunk-jdk7:core:processResources UP-TO-DATE
:kafka-trunk-jdk7:core:classes
:kafka-trunk-jdk7:clients:compileTestJavaNote: 

 uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

:kafka-trunk-jdk7:cli

Re: [VOTE] KIP-43: Kafka SASL enhancements

[Ismael Juma]

+1 (non-binding)

On Thu, Mar 3, 2016 at 10:37 AM, Rajini Sivaram <
rajinisiva...@googlemail.com> wrote:

> I would like to start the voting process for *KIP-43: Kafka SASL
> enhancements*. This KIP extends the SASL implementation in Kafka to support
> new SASL mechanisms to enable Kafka to be integrated with different
> authentication servers.
>
> The KIP is available here for reference:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-43:+Kafka+SASL+enhancements
>
> And here's is a link to the discussion on the mailing list:
>
> http://mail-archives.apache.org/mod_mbox/kafka-dev/201601.mbox/%3CCAOJcB39b9Vy7%3DZEM3tLw2zarCS4A_s-%2BU%2BC%3DuEcWs0712UaYrQ%40mail.gmail.com%3E
>
>
> Thank you...
>
> Regards,
>
> Rajini
>

KIP call to discuss some of the outstanding KIPs

[Ashish Singh]

Hello Jun,

Could we have a KIP call tomorrow to discuss some of the outstanding KIPs.
As 0.10 code freeze date, as per current suggestion, is fast approaching,
it will really help out with moving forward with following KIPs.

* KIP-4: Command line and centralized operations
* KIP-35: Retrieve protocol version
* KIP-43: Kafka SASL enhancements
* KIP-48: Support for delegation tokens
* KIP-50: Enhance Authorizer interface to be aware of supported Principal
Types

Please feel free to add any other KIPs which we deem important for 0.10.
Not all the above KIPs need to be discussed, as some of them are already in
voting phase, like KIP-43. However, some KIPs, like KIP-35, KIP-48 and
KIP-50 have a few points that we can talk about and help them in moving
along.
-- 

Regards,
Ashish

[jira] [Commented] (KAFKA-3334) First message on new topic not actually being sent, no exception thrown

[Aleksandar Stojadinovic (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183597#comment-15183597
 ] 

Aleksandar Stojadinovic commented on KAFKA-3334:


The addition sounds very reasonable. Maybe just add what does "proper" mean. 
Larger, shorter? I'm not sure someone without experience would understand.

> First message on new topic not actually being sent, no exception thrown
> ---
>
> Key: KAFKA-3334
> URL: https://issues.apache.org/jira/browse/KAFKA-3334
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.9.0.0
> Environment: Linux, Java
>Reporter: Aleksandar Stojadinovic
>Assignee: Ashish K Singh
>
> Although I've seen this issue pop around the internet in a few forms, I'm not 
> sure it is yet properly fixed. 
> When publishing to a new topic, with auto create-enabled, the java client 
> (0.9.0) shows this WARN message in the log, and the message is not sent 
> obviously:
> org.apache.kafka.clients.NetworkClient - Error while fetching metadata with 
> correlation id 0 : {file.topic=LEADER_NOT_AVAILABLE}
> In the meantime I see in the console the message that a log for partition is 
> created. The next messages are patched through normally, but the first one is 
> never sent. No exception is ever thrown, either by calling get on the future, 
> or with the async usage, like everything is perfect.
> I notice when I leave my application blocked on the get call, in the 
> debugger, then the message may be processed, but with significant delay. This 
> is consistent with another issue I found for the python client. Also, if I 
> call partitionsFor previously, the topic is created and the message is sent. 
> But it seems silly to call it every time, just to mitigate this issue.
> {code}
> Future recordMetadataFuture = producer.send(new 
> ProducerRecord<>(topic, key, file));
> RecordMetadata recordMetadata = recordMetadataFuture.get(30, 
> TimeUnit.SECONDS);
> {code}
> I hope I'm clear enough.
> Related similar (but not same) issues:
> https://issues.apache.org/jira/browse/KAFKA-1124
> https://github.com/dpkp/kafka-python/issues/150
> http://stackoverflow.com/questions/35187933/how-to-resolve-leader-not-available-kafka-error-when-trying-to-consume



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-2273: Sticky partition assignment strate...

[vahidhashemian]

GitHub user vahidhashemian reopened a pull request:

https://github.com/apache/kafka/pull/1020

KAFKA-2273: Sticky partition assignment strategy

This PR implements a new partition assignment strategy called "sticky", and 
it's purpose is to balance partitions across consumers in a way that minimizes 
moving partitions around, or, in other words, preserves existing partition 
assignments as much as possible.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vahidhashemian/kafka KAFKA-2273

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1020.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1020


commit 2a48c9fbf9260d6c8659a97ae2c8673607badd13
Author: Vahid Hashemian 
Date:   2016-02-18T13:44:17Z

KAFKA-2273: Sticky partition assignment strategy

This PR implements a new partition assignment strategy called "sticky", and 
it's purpose is to balance partitions across consumers in a way that minimizes 
moving partitions around, or in other words, preserving partition assignments 
as much as possible.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request: KAFKA-2273: Sticky partition assignment strate...

[vahidhashemian]

Github user vahidhashemian closed the pull request at:

https://github.com/apache/kafka/pull/1020


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-2273) Add rebalance with a minimal number of reassignments to server-defined strategy list

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183600#comment-15183600
 ] 

ASF GitHub Bot commented on KAFKA-2273:
---

GitHub user vahidhashemian reopened a pull request:

https://github.com/apache/kafka/pull/1020

KAFKA-2273: Sticky partition assignment strategy

This PR implements a new partition assignment strategy called "sticky", and 
it's purpose is to balance partitions across consumers in a way that minimizes 
moving partitions around, or, in other words, preserves existing partition 
assignments as much as possible.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vahidhashemian/kafka KAFKA-2273

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1020.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1020


commit 2a48c9fbf9260d6c8659a97ae2c8673607badd13
Author: Vahid Hashemian 
Date:   2016-02-18T13:44:17Z

KAFKA-2273: Sticky partition assignment strategy

This PR implements a new partition assignment strategy called "sticky", and 
it's purpose is to balance partitions across consumers in a way that minimizes 
moving partitions around, or in other words, preserving partition assignments 
as much as possible.




> Add rebalance with a minimal number of reassignments to server-defined 
> strategy list
> 
>
> Key: KAFKA-2273
> URL: https://issues.apache.org/jira/browse/KAFKA-2273
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Olof Johansson
>Assignee: Vahid Hashemian
>  Labels: newbie++, newbiee
> Fix For: 0.10.1.0
>
>
> Add a new partitions.assignment.strategy to the server-defined list that will 
> do reassignments based on moving as few partitions as possible. This should 
> be a quite common reassignment strategy especially for the cases where the 
> consumer has to maintain state, either in memory, or on disk.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2273) Add rebalance with a minimal number of reassignments to server-defined strategy list

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183599#comment-15183599
 ] 

ASF GitHub Bot commented on KAFKA-2273:
---

Github user vahidhashemian closed the pull request at:

https://github.com/apache/kafka/pull/1020


> Add rebalance with a minimal number of reassignments to server-defined 
> strategy list
> 
>
> Key: KAFKA-2273
> URL: https://issues.apache.org/jira/browse/KAFKA-2273
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Olof Johansson
>Assignee: Vahid Hashemian
>  Labels: newbie++, newbiee
> Fix For: 0.10.1.0
>
>
> Add a new partitions.assignment.strategy to the server-defined list that will 
> do reassignments based on moving as few partitions as possible. This should 
> be a quite common reassignment strategy especially for the cases where the 
> consumer has to maintain state, either in memory, or on disk.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3343) GroupMetadataManager should use NoTimestamp for message v0

[Jiangjie Qin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jiangjie Qin updated KAFKA-3343:

Fix Version/s: 0.10.0.0

> GroupMetadataManager should use NoTimestamp for message v0
> --
>
> Key: KAFKA-3343
> URL: https://issues.apache.org/jira/browse/KAFKA-3343
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
> Fix For: 0.10.0.0
>
>
> When message version for __consumer_offsets is v0, the timestamp should be 
> NoTimestamp. Otherwise message instantiation will fail.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-3343) GroupMetadataManager should use NoTimestamp for message v0

[Jiangjie Qin (JIRA)]

Jiangjie Qin created KAFKA-3343:
---

 Summary: GroupMetadataManager should use NoTimestamp for message v0
 Key: KAFKA-3343
 URL: https://issues.apache.org/jira/browse/KAFKA-3343
 Project: Kafka
  Issue Type: Bug
Reporter: Jiangjie Qin
Assignee: Jiangjie Qin


When message version for __consumer_offsets is v0, the timestamp should be 
NoTimestamp. Otherwise message instantiation will fail.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3343) GroupMetadataManager should use NoTimestamp for message v0

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-3343:
---
Reviewer: Jun Rao

> GroupMetadataManager should use NoTimestamp for message v0
> --
>
> Key: KAFKA-3343
> URL: https://issues.apache.org/jira/browse/KAFKA-3343
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
>Priority: Blocker
> Fix For: 0.10.0.0
>
>
> When message version for __consumer_offsets is v0, the timestamp should be 
> NoTimestamp. Otherwise message instantiation will fail.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3343) GroupMetadataManager should use NoTimestamp for message v0

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-3343:
---
Priority: Blocker  (was: Major)

> GroupMetadataManager should use NoTimestamp for message v0
> --
>
> Key: KAFKA-3343
> URL: https://issues.apache.org/jira/browse/KAFKA-3343
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
>Priority: Blocker
> Fix For: 0.10.0.0
>
>
> When message version for __consumer_offsets is v0, the timestamp should be 
> NoTimestamp. Otherwise message instantiation will fail.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-3344) Remove previous system test's leftover test-log4j.properties

[Ashish K Singh (JIRA)]

Ashish K Singh created KAFKA-3344:
-

 Summary: Remove previous system test's leftover 
test-log4j.properties
 Key: KAFKA-3344
 URL: https://issues.apache.org/jira/browse/KAFKA-3344
 Project: Kafka
  Issue Type: Bug
Reporter: Ashish K Singh
Assignee: Ashish K Singh


test-log4j.properties seems to be leftover from previous system tests that got 
removed after moving over to ducktape based system tests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-3343: Use NoTimestamp in GroupMetadataMa...

[becketqin]

GitHub user becketqin opened a pull request:

https://github.com/apache/kafka/pull/1023

KAFKA-3343: Use NoTimestamp in GroupMetadataManager when message v0 i…

…s used.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/becketqin/kafka KAFKA-3343

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1023.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1023


commit 2b0cc6dd5583716eda9cef6c03a837af701a91a5
Author: Jiangjie Qin 
Date:   2016-03-07T21:07:39Z

KAFKA-3343: Use NoTimestamp in GroupMetadataManager when message v0 is used.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3343) GroupMetadataManager should use NoTimestamp for message v0

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183715#comment-15183715
 ] 

ASF GitHub Bot commented on KAFKA-3343:
---

GitHub user becketqin opened a pull request:

https://github.com/apache/kafka/pull/1023

KAFKA-3343: Use NoTimestamp in GroupMetadataManager when message v0 i…

…s used.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/becketqin/kafka KAFKA-3343

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1023.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1023


commit 2b0cc6dd5583716eda9cef6c03a837af701a91a5
Author: Jiangjie Qin 
Date:   2016-03-07T21:07:39Z

KAFKA-3343: Use NoTimestamp in GroupMetadataManager when message v0 is used.




> GroupMetadataManager should use NoTimestamp for message v0
> --
>
> Key: KAFKA-3343
> URL: https://issues.apache.org/jira/browse/KAFKA-3343
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
>Priority: Blocker
> Fix For: 0.10.0.0
>
>
> When message version for __consumer_offsets is v0, the timestamp should be 
> NoTimestamp. Otherwise message instantiation will fail.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3343) GroupMetadataManager should use NoTimestamp for message v0

[Jiangjie Qin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jiangjie Qin updated KAFKA-3343:

Status: Patch Available  (was: Open)

> GroupMetadataManager should use NoTimestamp for message v0
> --
>
> Key: KAFKA-3343
> URL: https://issues.apache.org/jira/browse/KAFKA-3343
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
>Priority: Blocker
> Fix For: 0.10.0.0
>
>
> When message version for __consumer_offsets is v0, the timestamp should be 
> NoTimestamp. Otherwise message instantiation will fail.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3344) Remove previous system test's leftover test-log4j.properties

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183718#comment-15183718
 ] 

ASF GitHub Bot commented on KAFKA-3344:
---

GitHub user SinghAsDev opened a pull request:

https://github.com/apache/kafka/pull/1024

KAFKA-3344: Remove previous system test's leftover test-log4j.properties



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/SinghAsDev/kafka KAFKA-3344

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1024.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1024


commit 88fceb5de4d0065a14c16aa350be3397b68a3d48
Author: Ashish Singh 
Date:   2016-03-07T21:09:02Z

KAFKA-3344: Remove previous system test's leftover test-log4j.properties




> Remove previous system test's leftover test-log4j.properties
> 
>
> Key: KAFKA-3344
> URL: https://issues.apache.org/jira/browse/KAFKA-3344
> Project: Kafka
>  Issue Type: Bug
>Reporter: Ashish K Singh
>Assignee: Ashish K Singh
>
> test-log4j.properties seems to be leftover from previous system tests that 
> got removed after moving over to ducktape based system tests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-3344: Remove previous system test's left...

[SinghAsDev]

GitHub user SinghAsDev opened a pull request:

https://github.com/apache/kafka/pull/1024

KAFKA-3344: Remove previous system test's leftover test-log4j.properties



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/SinghAsDev/kafka KAFKA-3344

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1024.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1024


commit 88fceb5de4d0065a14c16aa350be3397b68a3d48
Author: Ashish Singh 
Date:   2016-03-07T21:09:02Z

KAFKA-3344: Remove previous system test's leftover test-log4j.properties




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-3344) Remove previous system test's leftover test-log4j.properties

[Ashish K Singh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish K Singh updated KAFKA-3344:
--
Status: Patch Available  (was: Open)

> Remove previous system test's leftover test-log4j.properties
> 
>
> Key: KAFKA-3344
> URL: https://issues.apache.org/jira/browse/KAFKA-3344
> Project: Kafka
>  Issue Type: Bug
>Reporter: Ashish K Singh
>Assignee: Ashish K Singh
>
> test-log4j.properties seems to be leftover from previous system tests that 
> got removed after moving over to ducktape based system tests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-3345) ProducerResponse could gracefully handle no throttle time provided

[Bryan Baugher (JIRA)]

Bryan Baugher created KAFKA-3345:


 Summary: ProducerResponse could gracefully handle no throttle time 
provided
 Key: KAFKA-3345
 URL: https://issues.apache.org/jira/browse/KAFKA-3345
 Project: Kafka
  Issue Type: Improvement
Reporter: Bryan Baugher
Priority: Minor


When doing some compatibility testing between kafka 0.8 and 0.9 I found that 
the old producer using 0.9 libraries could write to a cluster running 0.8 if 
'request.required.acks' was set to 0. If it was set to anything else it would 
fail with,

{code}
java.nio.BufferUnderflowException
at java.nio.Buffer.nextGetIndex(Buffer.java:506) 
at java.nio.HeapByteBuffer.getInt(HeapByteBuffer.java:361) 
at kafka.api.ProducerResponse$.readFrom(ProducerResponse.scala:41) 
at kafka.producer.SyncProducer.send(SyncProducer.scala:109) 
{code}

In 0.9 there was a one line change to the response here[1] to look for a 
throttle time value in the response. It seems if the 0.9 code gracefully 
handled throttle time not being provided this would work. Would you be open to 
this change?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3345) ProducerResponse could gracefully handle no throttle time provided

[Bryan Baugher (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bryan Baugher updated KAFKA-3345:
-
Description: 
When doing some compatibility testing between kafka 0.8 and 0.9 I found that 
the old producer using 0.9 libraries could write to a cluster running 0.8 if 
'request.required.acks' was set to 0. If it was set to anything else it would 
fail with,

{code}
java.nio.BufferUnderflowException
at java.nio.Buffer.nextGetIndex(Buffer.java:506) 
at java.nio.HeapByteBuffer.getInt(HeapByteBuffer.java:361) 
at kafka.api.ProducerResponse$.readFrom(ProducerResponse.scala:41) 
at kafka.producer.SyncProducer.send(SyncProducer.scala:109) 
{code}

In 0.9 there was a one line change to the response here[1] to look for a 
throttle time value in the response. It seems if the 0.9 code gracefully 
handled throttle time not being provided this would work. Would you be open to 
this change?

[1] - 
https://github.com/apache/kafka/blob/0.9.0.1/core/src/main/scala/kafka/api/ProducerResponse.scala#L41

  was:
When doing some compatibility testing between kafka 0.8 and 0.9 I found that 
the old producer using 0.9 libraries could write to a cluster running 0.8 if 
'request.required.acks' was set to 0. If it was set to anything else it would 
fail with,

{code}
java.nio.BufferUnderflowException
at java.nio.Buffer.nextGetIndex(Buffer.java:506) 
at java.nio.HeapByteBuffer.getInt(HeapByteBuffer.java:361) 
at kafka.api.ProducerResponse$.readFrom(ProducerResponse.scala:41) 
at kafka.producer.SyncProducer.send(SyncProducer.scala:109) 
{code}

In 0.9 there was a one line change to the response here[1] to look for a 
throttle time value in the response. It seems if the 0.9 code gracefully 
handled throttle time not being provided this would work. Would you be open to 
this change?


> ProducerResponse could gracefully handle no throttle time provided
> --
>
> Key: KAFKA-3345
> URL: https://issues.apache.org/jira/browse/KAFKA-3345
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Bryan Baugher
>Priority: Minor
>
> When doing some compatibility testing between kafka 0.8 and 0.9 I found that 
> the old producer using 0.9 libraries could write to a cluster running 0.8 if 
> 'request.required.acks' was set to 0. If it was set to anything else it would 
> fail with,
> {code}
> java.nio.BufferUnderflowException
>   at java.nio.Buffer.nextGetIndex(Buffer.java:506) 
>   at java.nio.HeapByteBuffer.getInt(HeapByteBuffer.java:361) 
>   at kafka.api.ProducerResponse$.readFrom(ProducerResponse.scala:41) 
>   at kafka.producer.SyncProducer.send(SyncProducer.scala:109) 
> {code}
> In 0.9 there was a one line change to the response here[1] to look for a 
> throttle time value in the response. It seems if the 0.9 code gracefully 
> handled throttle time not being provided this would work. Would you be open 
> to this change?
> [1] - 
> https://github.com/apache/kafka/blob/0.9.0.1/core/src/main/scala/kafka/api/ProducerResponse.scala#L41



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

RE: [VOTE] KIP-43: Kafka SASL enhancements

[Andrew Schofield]

+1 (non-binding)


> From: ism...@juma.me.uk
> Date: Mon, 7 Mar 2016 19:52:11 +
> Subject: Re: [VOTE] KIP-43: Kafka SASL enhancements
> To: dev@kafka.apache.org
>
> +1 (non-binding)
>
> On Thu, Mar 3, 2016 at 10:37 AM, Rajini Sivaram <
> rajinisiva...@googlemail.com> wrote:
>
>> I would like to start the voting process for *KIP-43: Kafka SASL
>> enhancements*. This KIP extends the SASL implementation in Kafka to support
>> new SASL mechanisms to enable Kafka to be integrated with different
>> authentication servers.
>>
>> The KIP is available here for reference:
>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-43:+Kafka+SASL+enhancements
>>
>> And here's is a link to the discussion on the mailing list:
>>
>> http://mail-archives.apache.org/mod_mbox/kafka-dev/201601.mbox/%3CCAOJcB39b9Vy7%3DZEM3tLw2zarCS4A_s-%2BU%2BC%3DuEcWs0712UaYrQ%40mail.gmail.com%3E
>>
>>
>> Thank you...
>>
>> Regards,
>>
>> Rajini
>>
  

[GitHub] kafka pull request: KAFKA-2273: Sticky partition assignment strate...

[vahidhashemian]

GitHub user vahidhashemian reopened a pull request:

https://github.com/apache/kafka/pull/1020

KAFKA-2273: Sticky partition assignment strategy

This PR implements a new partition assignment strategy called "sticky", and 
it's purpose is to balance partitions across consumers in a way that minimizes 
moving partitions around, or, in other words, preserves existing partition 
assignments as much as possible.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vahidhashemian/kafka KAFKA-2273

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1020.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1020


commit 2a48c9fbf9260d6c8659a97ae2c8673607badd13
Author: Vahid Hashemian 
Date:   2016-02-18T13:44:17Z

KAFKA-2273: Sticky partition assignment strategy

This PR implements a new partition assignment strategy called "sticky", and 
it's purpose is to balance partitions across consumers in a way that minimizes 
moving partitions around, or in other words, preserving partition assignments 
as much as possible.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request: KAFKA-2273: Sticky partition assignment strate...

[vahidhashemian]

Github user vahidhashemian closed the pull request at:

https://github.com/apache/kafka/pull/1020


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-2273) Add rebalance with a minimal number of reassignments to server-defined strategy list

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183782#comment-15183782
 ] 

ASF GitHub Bot commented on KAFKA-2273:
---

GitHub user vahidhashemian reopened a pull request:

https://github.com/apache/kafka/pull/1020

KAFKA-2273: Sticky partition assignment strategy

This PR implements a new partition assignment strategy called "sticky", and 
it's purpose is to balance partitions across consumers in a way that minimizes 
moving partitions around, or, in other words, preserves existing partition 
assignments as much as possible.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vahidhashemian/kafka KAFKA-2273

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1020.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1020


commit 2a48c9fbf9260d6c8659a97ae2c8673607badd13
Author: Vahid Hashemian 
Date:   2016-02-18T13:44:17Z

KAFKA-2273: Sticky partition assignment strategy

This PR implements a new partition assignment strategy called "sticky", and 
it's purpose is to balance partitions across consumers in a way that minimizes 
moving partitions around, or in other words, preserving partition assignments 
as much as possible.




> Add rebalance with a minimal number of reassignments to server-defined 
> strategy list
> 
>
> Key: KAFKA-2273
> URL: https://issues.apache.org/jira/browse/KAFKA-2273
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Olof Johansson
>Assignee: Vahid Hashemian
>  Labels: newbie++, newbiee
> Fix For: 0.10.1.0
>
>
> Add a new partitions.assignment.strategy to the server-defined list that will 
> do reassignments based on moving as few partitions as possible. This should 
> be a quite common reassignment strategy especially for the cases where the 
> consumer has to maintain state, either in memory, or on disk.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2273) Add rebalance with a minimal number of reassignments to server-defined strategy list

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183781#comment-15183781
 ] 

ASF GitHub Bot commented on KAFKA-2273:
---

Github user vahidhashemian closed the pull request at:

https://github.com/apache/kafka/pull/1020


> Add rebalance with a minimal number of reassignments to server-defined 
> strategy list
> 
>
> Key: KAFKA-2273
> URL: https://issues.apache.org/jira/browse/KAFKA-2273
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Olof Johansson
>Assignee: Vahid Hashemian
>  Labels: newbie++, newbiee
> Fix For: 0.10.1.0
>
>
> Add a new partitions.assignment.strategy to the server-defined list that will 
> do reassignments based on moving as few partitions as possible. This should 
> be a quite common reassignment strategy especially for the cases where the 
> consumer has to maintain state, either in memory, or on disk.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-3312: Add utility offset methods to ZkUt...

[vahidhashemian]

GitHub user vahidhashemian opened a pull request:

https://github.com/apache/kafka/pull/1025

KAFKA-3312: Add utility offset methods to ZkUtils



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vahidhashemian/kafka KAFKA-3312

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1025.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1025


commit ac6d6926d5e23640b53fa2bca927ae5e423c12f7
Author: Vahid Hashemian 
Date:   2016-03-05T00:47:05Z

KAFKA-3312: Add utility offset methods to ZkUtils




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] KIP-43: Kafka SASL enhancements

[Gwen Shapira]

Hi,

Before I can vote on this KIP, I have two additional questions /
comments on the new configuration:

1. sasl.callback.handler.class - it looks like we want a single class
that implements all mechanisms. I think this will make it difficult to
extend since the only way I can add a mechanism will be by
implementing every single existing mechanism (otherwise customers will
need to choose between new and existing when selecting which class to
use). If Microsoft releases a proprietary "AD Mechanism" and Oracle
releases "OID mechanism", there will be no class that implements both.
Can we make it a list of classes instead? I realize this complicates
the code a bit (some kind of factory will be required to choose the
right class to use), but important IMO.
2. similar for sasl.login.class - if I have a class for Kerberos (with
refresh thread) and a class for "plain", we need to be able to load
both.

Gwen

On Wed, Mar 2, 2016 at 12:30 AM, Rajini Sivaram
 wrote:
> Jun,
>
> Thanks, I have added a note to the KIP. I will add a comment in the
> implementation and also add a unit test to ensure that conflicts are
> avoided when version number is modified.
>
> On Tue, Mar 1, 2016 at 5:43 PM, Jun Rao  wrote:
>
>> Rajini,
>>
>> Thanks for the explanation. For 1, this implies that we have to be careful
>> with changing the 2-byte version in the future to avoid conflict. Could you
>> document this in the KIP and also in the implementation?
>>
>> Jun
>>
>> On Tue, Mar 1, 2016 at 2:47 AM, Rajini Sivaram <
>> rajinisiva...@googlemail.com
>> > wrote:
>>
>> > Jun,
>> >
>> > Thank you for the review.
>> >
>> >
>> >1. With GSSAPI, the first context establishment packet starts with the
>> >byte 0x60 (tag for APPLICATION-0) followed by a variable-length
>> encoded
>> >size, followed by various tags and contents. And the packet also
>> > contains a
>> >checksum. This is completely different from the mechanism packet from
>> > Kafka
>> >clients which start with a two-byte version set to zero currently,
>> > followed
>> >by just a String mechanism.
>> >2. Agreed, I have removed the version from the server response in the
>> >KIP. Thanks.
>> >
>> >
>> > On Tue, Mar 1, 2016 at 2:33 AM, Jun Rao  wrote:
>> >
>> > > Rajini,
>> > >
>> > > Thanks for the updates. Just a couple of minor comments.
>> > >
>> > > 1. With the default GSSAPI, what's the first packet that the client
>> sends
>> > > to the server? Is that completely different from the packet format that
>> > we
>> > > will use for non-GSSAPI mechanisms?
>> > >
>> > > 2. In the server response, it doesn't seem that we need to include the
>> > > version since the client knows the version of the request that it
>> sends.
>> > >
>> > > Jun
>> > >
>> > > On Mon, Feb 29, 2016 at 10:14 AM, Rajini Sivaram <
>> > > rajinisiva...@googlemail.com> wrote:
>> > >
>> > > > Harsha,
>> > > >
>> > > > Thank you for the review. I will wait another day to see if there is
>> > more
>> > > > feedback and then start a voting thread.
>> > > >
>> > > > Rajini
>> > > >
>> > > > On Mon, Feb 29, 2016 at 2:51 PM, Harsha  wrote:
>> > > >
>> > > > > Rajini,
>> > > > >   Thanks for the changes to the KIP. It looks good to
>> > me. I
>> > > > >   think we can move to voting.
>> > > > > Thanks,
>> > > > > Harsha
>> > > > >
>> > > > > On Mon, Feb 29, 2016, at 12:43 AM, Rajini Sivaram wrote:
>> > > > > > I have added some more detail to the KIP based on the discussion
>> in
>> > > the
>> > > > > > last KIP meeting to simplify support for multiple mechanisms.
>> Have
>> > > also
>> > > > > > changed the property names to reflect this.
>> > > > > >
>> > > > > > Also updated the PR in
>> > > > https://issues.apache.org/jira/browse/KAFKA-3149
>> > > > > > to
>> > > > > > reflect the KIP.
>> > > > > >
>> > > > > > Any feedback is appreciated.
>> > > > > >
>> > > > > >
>> > > > > > On Tue, Feb 23, 2016 at 9:36 PM, Rajini Sivaram <
>> > > > > > rajinisiva...@googlemail.com> wrote:
>> > > > > >
>> > > > > > > I have updated the KIP based on the discussion in the KIP
>> meeting
>> > > > > today.
>> > > > > > >
>> > > > > > > Comments and feedback are welcome.
>> > > > > > >
>> > > > > > > On Wed, Feb 3, 2016 at 7:20 PM, Rajini Sivaram <
>> > > > > > > rajinisiva...@googlemail.com> wrote:
>> > > > > > >
>> > > > > > >> Hi Harsha,
>> > > > > > >>
>> > > > > > >> Thank you for the review. Can you clarify - I think you are
>> > saying
>> > > > > that
>> > > > > > >> the client should send its mechanism over the wire to the
>> > server.
>> > > Is
>> > > > > that
>> > > > > > >> correct? The exchange is slightly different in the KIP (the PR
>> > > > > matches the
>> > > > > > >> KIP) from the one you described to enable interoperability
>> with
>> > > > > 0.9.0.0.
>> > > > > > >>
>> > > > > > >>
>> > > > > > >> On Wed, Feb 3, 2016 at 1:56 PM, Harsha 
>> wrote:
>> > > > > > >>
>> > > > > > >>> Rajini,
>> > > > > > >>>I looked at the PR you have. I think its better

[jira] [Commented] (KAFKA-3312) Add a offsets methods to ZkUtils and replace relevant usages

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183797#comment-15183797
 ] 

ASF GitHub Bot commented on KAFKA-3312:
---

GitHub user vahidhashemian opened a pull request:

https://github.com/apache/kafka/pull/1025

KAFKA-3312: Add utility offset methods to ZkUtils



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vahidhashemian/kafka KAFKA-3312

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1025.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1025


commit ac6d6926d5e23640b53fa2bca927ae5e423c12f7
Author: Vahid Hashemian 
Date:   2016-03-05T00:47:05Z

KAFKA-3312: Add utility offset methods to ZkUtils




> Add a offsets methods to ZkUtils and replace relevant usages
> 
>
> Key: KAFKA-3312
> URL: https://issues.apache.org/jira/browse/KAFKA-3312
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.9.0.1
>Reporter: Grant Henke
>Assignee: Vahid Hashemian
>
> There are many places in the code that manually build a zookeeper path and 
> get or update offsets. Moving this logic to a common location in ZkUtils 
> would be nice. 
> Ex:
> {code}
> zkUtils.readDataMaybeNull(s"${topicDirs.consumerOffsetDir}/${topicPartition.partition}")._1
> {code}
> {code}
>  zkUtils.readData(topicDirs.consumerOffsetDir + "/" + 
> topicAndPartition.partition)._1.toLong
> {code}
> {code}
> zkUtils.updatePersistentPath(s"${topicDirs.consumerOffsetDir}/${topicPartition.partition}",
>  partitionData.offset.toString)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2967) Move Kafka documentation to ReStructuredText

[Christian Posta (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183815#comment-15183815
 ] 

Christian Posta commented on KAFKA-2967:


All,

I've got a quick spike of converting docs to gitbook https://www.gitbook.com 
which is basically markdown with some styling and can build websites, ebooks, 
pdf, etc. 

Point of this spike was to 1) get some feedback 2) understand how the current 
docs are folded into the website, and 3) figure out where some of the configs 
(e.g., broker configs) are in the website at apache, but not in the src :) For 
example, not all of these in the table here 
http://kafka.apache.org/documentation.html#brokerconfigs are in here 
https://github.com/apache/kafka/blob/trunk/docs/configuration.html#L147


This branch here as all of the code/integrated with gradle builds, etc 
https://github.com/christian-posta/kafka/tree/ceposta-doco

again, this is currently WIP for discussion purposes, not complete by any means.



> Move Kafka documentation to ReStructuredText
> 
>
> Key: KAFKA-2967
> URL: https://issues.apache.org/jira/browse/KAFKA-2967
> Project: Kafka
>  Issue Type: Bug
>Reporter: Gwen Shapira
>Assignee: Gwen Shapira
>
> Storing documentation as HTML is kind of BS :)
> * Formatting is a pain, and making it look good is even worse
> * Its just HTML, can't generate PDFs
> * Reading and editting is painful
> * Validating changes is hard because our formatting relies on all kinds of 
> Apache Server features.
> I suggest:
> * Move to RST
> * Generate HTML and PDF during build using Sphinx plugin for Gradle.
> Lots of Apache projects are doing this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (KAFKA-2967) Move Kafka documentation to ReStructuredText

[Christian Posta (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183815#comment-15183815
 ] 

Christian Posta edited comment on KAFKA-2967 at 3/7/16 10:05 PM:
-

All,

I've got a quick spike of converting docs to gitbook https://www.gitbook.com 
which is basically markdown with some styling and can build websites, ebooks, 
pdf, etc. 

http://christianposta.com/kafka-docs/_book/index.html

Point of this spike was to 1) get some feedback 2) understand how the current 
docs are folded into the website, and 3) figure out where some of the configs 
(e.g., broker configs) are in the website at apache, but not in the src :) For 
example, not all of these in the table here 
http://kafka.apache.org/documentation.html#brokerconfigs are in here 
https://github.com/apache/kafka/blob/trunk/docs/configuration.html#L147


This branch here as all of the code/integrated with gradle builds, etc 
https://github.com/christian-posta/kafka/tree/ceposta-doco

again, this is currently WIP for discussion purposes, not complete by any means.




was (Author: ceposta):
All,

I've got a quick spike of converting docs to gitbook https://www.gitbook.com 
which is basically markdown with some styling and can build websites, ebooks, 
pdf, etc. 

Point of this spike was to 1) get some feedback 2) understand how the current 
docs are folded into the website, and 3) figure out where some of the configs 
(e.g., broker configs) are in the website at apache, but not in the src :) For 
example, not all of these in the table here 
http://kafka.apache.org/documentation.html#brokerconfigs are in here 
https://github.com/apache/kafka/blob/trunk/docs/configuration.html#L147


This branch here as all of the code/integrated with gradle builds, etc 
https://github.com/christian-posta/kafka/tree/ceposta-doco

again, this is currently WIP for discussion purposes, not complete by any means.



> Move Kafka documentation to ReStructuredText
> 
>
> Key: KAFKA-2967
> URL: https://issues.apache.org/jira/browse/KAFKA-2967
> Project: Kafka
>  Issue Type: Bug
>Reporter: Gwen Shapira
>Assignee: Gwen Shapira
>
> Storing documentation as HTML is kind of BS :)
> * Formatting is a pain, and making it look good is even worse
> * Its just HTML, can't generate PDFs
> * Reading and editting is painful
> * Validating changes is hard because our formatting relies on all kinds of 
> Apache Server features.
> I suggest:
> * Move to RST
> * Generate HTML and PDF during build using Sphinx plugin for Gradle.
> Lots of Apache projects are doing this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3173) Error while moving some partitions to OnlinePartition state

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-3173:
---
Assignee: Flavio Junqueira

> Error while moving some partitions to OnlinePartition state 
> 
>
> Key: KAFKA-3173
> URL: https://issues.apache.org/jira/browse/KAFKA-3173
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.9.0.0
>Reporter: Flavio Junqueira
>Assignee: Flavio Junqueira
>Priority: Critical
> Fix For: 0.10.0.0
>
>
> We observed another instance of the problem reported in KAFKA-2300, but this 
> time the error appeared in the partition state machine. In KAFKA-2300, we 
> haven't cleaned up the state in {{PartitionStateMachine}} and 
> {{ReplicaStateMachine}} as we do in {{KafkaController}}.
> Here is the stack trace:
> {noformat}
> 2016-01-29 15:26:51,393] ERROR [Partition state machine on Controller 0]: 
> Error while moving some partitions to OnlinePartition state 
> (kafka.controller.PartitionStateMachine)java.lang.IllegalStateException: 
> Controller to broker state change requests batch is not empty while creating 
> a new one. 
> Some LeaderAndIsr state changes Map(0 -> Map(foo-0 -> (LeaderAndIsrInfo:
> (Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:1),ReplicationFactor:1),AllReplicas:0)))
>  might be lostat 
> kafka.controller.ControllerBrokerRequestBatch.newBatch(ControllerChannelManager.scala:254)
> at 
> kafka.controller.PartitionStateMachine.handleStateChanges(PartitionStateMachine.scala:144)
> at 
> kafka.controller.KafkaController.onNewPartitionCreation(KafkaController.scala:517)
> at 
> kafka.controller.KafkaController.onNewTopicCreation(KafkaController.scala:504)
> at 
> kafka.controller.PartitionStateMachine$TopicChangeListener$$anonfun$handleChildChange$1.apply$mcV$sp(PartitionStateMachine.scala:437)
> at 
> kafka.controller.PartitionStateMachine$TopicChangeListener$$anonfun$handleChildChange$1.apply(PartitionStateMachine.scala:419)
> at 
> kafka.controller.PartitionStateMachine$TopicChangeListener$$anonfun$handleChildChange$1.apply(PartitionStateMachine.scala:419)
> at 
> kafka.utils.CoreUtils$.inLock(CoreUtils.scala:262)at 
> kafka.controller.PartitionStateMachine$TopicChangeListener.handleChildChange(PartitionStateMachine.scala:418)
> at 
> org.I0Itec.zkclient.ZkClient$10.run(ZkClient.java:842)at 
> org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3173) Error while moving some partitions to OnlinePartition state

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183871#comment-15183871
 ] 

Ismael Juma commented on KAFKA-3173:


Checked with [~fpj] and he said he would take a look.

> Error while moving some partitions to OnlinePartition state 
> 
>
> Key: KAFKA-3173
> URL: https://issues.apache.org/jira/browse/KAFKA-3173
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.9.0.0
>Reporter: Flavio Junqueira
>Assignee: Flavio Junqueira
>Priority: Critical
> Fix For: 0.10.0.0
>
>
> We observed another instance of the problem reported in KAFKA-2300, but this 
> time the error appeared in the partition state machine. In KAFKA-2300, we 
> haven't cleaned up the state in {{PartitionStateMachine}} and 
> {{ReplicaStateMachine}} as we do in {{KafkaController}}.
> Here is the stack trace:
> {noformat}
> 2016-01-29 15:26:51,393] ERROR [Partition state machine on Controller 0]: 
> Error while moving some partitions to OnlinePartition state 
> (kafka.controller.PartitionStateMachine)java.lang.IllegalStateException: 
> Controller to broker state change requests batch is not empty while creating 
> a new one. 
> Some LeaderAndIsr state changes Map(0 -> Map(foo-0 -> (LeaderAndIsrInfo:
> (Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:1),ReplicationFactor:1),AllReplicas:0)))
>  might be lostat 
> kafka.controller.ControllerBrokerRequestBatch.newBatch(ControllerChannelManager.scala:254)
> at 
> kafka.controller.PartitionStateMachine.handleStateChanges(PartitionStateMachine.scala:144)
> at 
> kafka.controller.KafkaController.onNewPartitionCreation(KafkaController.scala:517)
> at 
> kafka.controller.KafkaController.onNewTopicCreation(KafkaController.scala:504)
> at 
> kafka.controller.PartitionStateMachine$TopicChangeListener$$anonfun$handleChildChange$1.apply$mcV$sp(PartitionStateMachine.scala:437)
> at 
> kafka.controller.PartitionStateMachine$TopicChangeListener$$anonfun$handleChildChange$1.apply(PartitionStateMachine.scala:419)
> at 
> kafka.controller.PartitionStateMachine$TopicChangeListener$$anonfun$handleChildChange$1.apply(PartitionStateMachine.scala:419)
> at 
> kafka.utils.CoreUtils$.inLock(CoreUtils.scala:262)at 
> kafka.controller.PartitionStateMachine$TopicChangeListener.handleChildChange(PartitionStateMachine.scala:418)
> at 
> org.I0Itec.zkclient.ZkClient$10.run(ZkClient.java:842)at 
> org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: kafka-trunk-jdk8 #423

[Apache Jenkins Server]

See 

Changes:

[cshapi] KAFKA-3339: Fix java docs for seekToEnd and seekToBeginning.

[cshapi] MINOR: streams javadoc corrections

[cshapi] KAFKA-3341: Improve error handling on invalid requests

--
[...truncated 2241 lines...]

kafka.admin.AdminTest > testTopicCreationWithCollision PASSED

kafka.admin.AdminTest > testTopicCreationInZK PASSED

kafka.admin.DeleteTopicTest > testDeleteTopicWithCleaner PASSED

kafka.admin.DeleteTopicTest > testResumeDeleteTopicOnControllerFailover PASSED

kafka.admin.DeleteTopicTest > testResumeDeleteTopicWithRecoveredFollower PASSED

kafka.admin.DeleteTopicTest > testDeleteTopicAlreadyMarkedAsDeleted PASSED

kafka.admin.DeleteTopicTest > testPartitionReassignmentDuringDeleteTopic PASSED

kafka.admin.DeleteTopicTest > testDeleteNonExistingTopic PASSED

kafka.admin.DeleteTopicTest > testRecreateTopicAfterDeletion PASSED

kafka.admin.DeleteTopicTest > testAddPartitionDuringDeleteTopic PASSED

kafka.admin.DeleteTopicTest > testDeleteTopicWithAllAliveReplicas PASSED

kafka.admin.DeleteTopicTest > testDeleteTopicDuringAddPartition PASSED

kafka.admin.AclCommandTest > testInvalidAuthorizerProperty PASSED

kafka.admin.AclCommandTest > testAclCli PASSED

kafka.admin.AclCommandTest > testProducerConsumerCli PASSED

kafka.api.ApiUtilsTest > testShortStringNonASCII PASSED

kafka.api.ApiUtilsTest > testShortStringASCII PASSED

kafka.api.QuotasTest > testProducerConsumerOverrideUnthrottled PASSED

kafka.api.QuotasTest > testThrottledProducerConsumer PASSED

kafka.api.ProducerBounceTest > testBrokerFailure PASSED

kafka.api.ProducerFailureHandlingTest > testCannotSendToInternalTopic PASSED

kafka.api.ProducerFailureHandlingTest > testTooLargeRecordWithAckOne PASSED

kafka.api.ProducerFailureHandlingTest > testWrongBrokerList PASSED

kafka.api.ProducerFailureHandlingTest > testNotEnoughReplicas PASSED

kafka.api.ProducerFailureHandlingTest > testNonExistentTopic PASSED

kafka.api.ProducerFailureHandlingTest > testInvalidPartition PASSED

kafka.api.ProducerFailureHandlingTest > testSendAfterClosed PASSED

kafka.api.ProducerFailureHandlingTest > testTooLargeRecordWithAckZero PASSED

kafka.api.ProducerFailureHandlingTest > 
testNotEnoughReplicasAfterBrokerShutdown PASSED

kafka.api.PlaintextConsumerTest > testPartitionsForAutoCreate PASSED

kafka.api.PlaintextConsumerTest > testShrinkingTopicSubscriptions PASSED

kafka.api.PlaintextConsumerTest > testMultiConsumerSessionTimeoutOnStopPolling 
PASSED

kafka.api.PlaintextConsumerTest > testPartitionsForInvalidTopic PASSED

kafka.api.PlaintextConsumerTest > testSeek PASSED

kafka.api.PlaintextConsumerTest > testPositionAndCommit PASSED

kafka.api.PlaintextConsumerTest > testMultiConsumerSessionTimeoutOnClose PASSED

kafka.api.PlaintextConsumerTest > testFetchRecordTooLarge PASSED

kafka.api.PlaintextConsumerTest > testMultiConsumerDefaultAssignment PASSED

kafka.api.PlaintextConsumerTest > testAutoCommitOnClose PASSED

kafka.api.PlaintextConsumerTest > testExpandingTopicSubscriptions PASSED

kafka.api.PlaintextConsumerTest > testInterceptors PASSED

kafka.api.PlaintextConsumerTest > testPatternUnsubscription PASSED

kafka.api.PlaintextConsumerTest > testGroupConsumption PASSED

kafka.api.PlaintextConsumerTest > testPartitionsFor PASSED

kafka.api.PlaintextConsumerTest > testInterceptorsWithWrongKeyValue PASSED

kafka.api.PlaintextConsumerTest > testMultiConsumerRoundRobinAssignment PASSED

kafka.api.PlaintextConsumerTest > testPartitionPauseAndResume PASSED

kafka.api.PlaintextConsumerTest > testConsumeMessagesWithLogAppendTime PASSED

kafka.api.PlaintextConsumerTest > testAutoCommitOnCloseAfterWakeup PASSED

kafka.api.PlaintextConsumerTest > testMaxPollRecords PASSED

kafka.api.PlaintextConsumerTest > testAutoOffsetReset PASSED

kafka.api.PlaintextConsumerTest > testFetchInvalidOffset PASSED

kafka.api.PlaintextConsumerTest > testAutoCommitIntercept PASSED

kafka.api.PlaintextConsumerTest > testCommitMetadata PASSED

kafka.api.PlaintextConsumerTest > testRoundRobinAssignment PASSED

kafka.api.PlaintextConsumerTest > testPatternSubscription PASSED

kafka.api.PlaintextConsumerTest > testPauseStateNotPreservedByRebalance PASSED

kafka.api.PlaintextConsumerTest > testUnsubscribeTopic PASSED

kafka.api.PlaintextConsumerTest > testListTopics PASSED

kafka.api.PlaintextConsumerTest > testAutoCommitOnRebalance PASSED

kafka.api.PlaintextConsumerTest > testSimpleConsumption PASSED

kafka.api.PlaintextConsumerTest > testPartitionReassignmentCallback PASSED

kafka.api.PlaintextConsumerTest > testCommitSpecifiedOffsets PASSED

kafka.api.SslProducerSendTest > testSendNonCompressedMessageWithCreateTime 
PASSED

kafka.api.SslProducerSendTest > testSendCompressedMessageWithLogAppendTime 
PASSED

kafka.api.SslProducerSendTest > testClose PASSED

kafka.api.SslProducerSendTest > testFlush PASSED

kafka.api.SslProducerSendTest > testSendToPartition PASSED

kafka.api.

[jira] [Created] (KAFKA-3346) Rename "Mode" to "SslMode"

[Gwen Shapira (JIRA)]

Gwen Shapira created KAFKA-3346:
---

 Summary: Rename "Mode" to "SslMode"
 Key: KAFKA-3346
 URL: https://issues.apache.org/jira/browse/KAFKA-3346
 Project: Kafka
  Issue Type: Bug
Reporter: Gwen Shapira


In the channel builders, the Mode enum is undocumented, so it is unclear that 
it is used to signify whether the connection is for SSL client or SSL server.

I suggest renaming to SslMode (although adding documentation will be ok too, if 
people object to the rename)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[Headsup] Rename "Mode" enum to "SslMode"

[Gwen Shapira]

Hi,

I opened a JIRA to rename the Mode enum, used in channel builders.
It is an internal API, so it doesn't require a KIP.
But - if you are using this internal API - please shout so we can
avoid breaking your code :)

https://issues.apache.org/jira/browse/KAFKA-3346

Gwen

[GitHub] kafka pull request: KAFKA-2273: Sticky partition assignment strate...

[vahidhashemian]

GitHub user vahidhashemian reopened a pull request:

https://github.com/apache/kafka/pull/1020

KAFKA-2273: Sticky partition assignment strategy

This PR implements a new partition assignment strategy called "sticky", and 
it's purpose is to balance partitions across consumers in a way that minimizes 
moving partitions around, or, in other words, preserves existing partition 
assignments as much as possible.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vahidhashemian/kafka KAFKA-2273

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1020.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1020


commit 2a48c9fbf9260d6c8659a97ae2c8673607badd13
Author: Vahid Hashemian 
Date:   2016-02-18T13:44:17Z

KAFKA-2273: Sticky partition assignment strategy

This PR implements a new partition assignment strategy called "sticky", and 
it's purpose is to balance partitions across consumers in a way that minimizes 
moving partitions around, or in other words, preserving partition assignments 
as much as possible.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-2273) Add rebalance with a minimal number of reassignments to server-defined strategy list

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183905#comment-15183905
 ] 

ASF GitHub Bot commented on KAFKA-2273:
---

Github user vahidhashemian closed the pull request at:

https://github.com/apache/kafka/pull/1020


> Add rebalance with a minimal number of reassignments to server-defined 
> strategy list
> 
>
> Key: KAFKA-2273
> URL: https://issues.apache.org/jira/browse/KAFKA-2273
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Olof Johansson
>Assignee: Vahid Hashemian
>  Labels: newbie++, newbiee
> Fix For: 0.10.1.0
>
>
> Add a new partitions.assignment.strategy to the server-defined list that will 
> do reassignments based on moving as few partitions as possible. This should 
> be a quite common reassignment strategy especially for the cases where the 
> consumer has to maintain state, either in memory, or on disk.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-2273: Sticky partition assignment strate...

[vahidhashemian]

Github user vahidhashemian closed the pull request at:

https://github.com/apache/kafka/pull/1020


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-2273) Add rebalance with a minimal number of reassignments to server-defined strategy list

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183906#comment-15183906
 ] 

ASF GitHub Bot commented on KAFKA-2273:
---

GitHub user vahidhashemian reopened a pull request:

https://github.com/apache/kafka/pull/1020

KAFKA-2273: Sticky partition assignment strategy

This PR implements a new partition assignment strategy called "sticky", and 
it's purpose is to balance partitions across consumers in a way that minimizes 
moving partitions around, or, in other words, preserves existing partition 
assignments as much as possible.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vahidhashemian/kafka KAFKA-2273

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1020.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1020


commit 2a48c9fbf9260d6c8659a97ae2c8673607badd13
Author: Vahid Hashemian 
Date:   2016-02-18T13:44:17Z

KAFKA-2273: Sticky partition assignment strategy

This PR implements a new partition assignment strategy called "sticky", and 
it's purpose is to balance partitions across consumers in a way that minimizes 
moving partitions around, or in other words, preserving partition assignments 
as much as possible.




> Add rebalance with a minimal number of reassignments to server-defined 
> strategy list
> 
>
> Key: KAFKA-2273
> URL: https://issues.apache.org/jira/browse/KAFKA-2273
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Olof Johansson
>Assignee: Vahid Hashemian
>  Labels: newbie++, newbiee
> Fix For: 0.10.1.0
>
>
> Add a new partitions.assignment.strategy to the server-defined list that will 
> do reassignments based on moving as few partitions as possible. This should 
> be a quite common reassignment strategy especially for the cases where the 
> consumer has to maintain state, either in memory, or on disk.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] KIP-33 - Add a time based log index to Kafka

[Becket Qin]

Hi Jun,

What do you think about the above solution? I am trying to include KIP-33
into 0.10.0 because the log retention has been a long pending issue.

Thanks,

Jiangjie (Becket) Qin

On Tue, Mar 1, 2016 at 8:18 PM, Becket Qin  wrote:

> Hi Jun,
>
> I see. If we only use index.interval.bytes, the time index entry will be
> inserted when (1) the largest timestamp is in this segment AND (2) at least
> index.interval.bytes have been appended since last time index entry
> insertion.
> In this case (1) becomes implicit instead of having an explicit threshold
> of time.index.interval.ms. This should work fine.
>
> For 1, the current proposal is actually intended to use the offsets of the
> messages instead of the file position. The reason is that
> OffsetRequest(ListOffsetRequest) gives back a list of offsets. Having
> offsets instead of file position is more convenient in that case. So far we
> don't have an interface for consumer to directly consume from a given
> timestamp. Supposedly we will have to first return the offset to the
> consumer then the consumer can issue a fetch request. But this does mean
> that we need to look up the index twice if we want to search to a
> particular message using timestamp.
>
> For 2, that is a good point. It looks we have to persist the max timestamp
> of each segment. I am thinking of reserving the first time index entry in
> each time index file. When broker shuts down or rolls out a new segment, we
> persist the max timestamp in the segment by writing a time index entry to
> the first entry. During timestamp search we will ignore the first time
> index entry. So the log retention will always be able to know for sure if a
> log segment is supposed to be deleted or not by looking at the first entry
> of time index.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
>
>
>
> On Tue, Mar 1, 2016 at 4:30 PM, Jun Rao  wrote:
>
>> Hi, Jiangjie,
>>
>> I was thinking perhaps just reusing index.interval.bytes is enough. Not
>> sure if there is much value in adding an additional
>> time.index.interval.ms.
>>
>> For 1, the timestamp index has entries of timestamp -> file position. So,
>> there is actually no offset in the index, right?
>>
>> For 2, what you said makes sense for time-based retention. Does that apply
>> if the retention is trigged by size? The difference here is that we can't
>> assume all segments with messages of timestamp smaller than the latest
>> timestamp will be deleted after the message with the latest timestamp is
>> deleted.
>>
>> Thanks,
>>
>> Jun
>>
>> On Tue, Mar 1, 2016 at 1:00 PM, Becket Qin  wrote:
>>
>> > Hi Jun,
>> >
>> > Rolling out a new segment when the time index is full sounds good. So
>> both
>> > time index and offset index will be sharing the configuration of max
>> index
>> > size.
>> > If we do that, do you think we still want to reuse
>> index.interval.bytes? If
>> > we don't, the risk is that in some corner cases, we might end up with
>> many
>> > small segments. (e.g. small time.index.interval.ms with small max index
>> > size). But this is probably more of a misconfiguration.
>> >
>> > 2. If the broker is still running when all the segments except the
>> active
>> > segment is deleted, we will have an in memory latest timestamp. So that
>> is
>> > not a problem.
>> >
>> > In another case, if a broker boots up and sees only one segment with an
>> > empty time index file, we can scan the active segment and rebuild the
>> time
>> > index.  i.e. we do not need to care about the previous largest timestamp
>> > but simply start over. (We need to scan the active segment because it is
>> > possible that the last message appended to the log has a timestamp not
>> > expired, but the broker died before inserting the time index entry for
>> > it.). If all the messages in the active segment has expired, we should
>> roll
>> > out a new segment and reset the latest timetamp to -1.
>> > The principal here is that we will try to build the time indices for the
>> > existing segments that have not expired. If the message with previously
>> > latest timestamp has already been deleted, there is no need to remember
>> > that any more.
>> >
>> > That said, I believe this corner case is really because user is not
>> > configuring the acceptable time difference threshold appropriately.
>> >
>> > Thanks,
>> >
>> > Jiangjie (Becket) Qin
>> >
>> >
>> > On Tue, Mar 1, 2016 at 11:55 AM, Jun Rao  wrote:
>> >
>> > > Jiangjie,
>> > >
>> > > Currently, we roll a new log segment if the index is full. We can
>> > probably
>> > > just do the same on the time index. This will bound the index size.
>> > >
>> > > 1. Sounds good.
>> > >
>> > > 2. I was wondering an edge case where the largest timestamp is in the
>> > > oldest segment and the time index is empty is in all newer segments.
>> At
>> > > some point, we delete the oldest segment since it has expired. Then,
>> we
>> > > delete all but the active segment. Now, what should the largest
>> timestamp
>> > > be? Should it be the pre

[jira] [Created] (KAFKA-3347) Configure java to prefer ipv4

[Jeremy Custenborder (JIRA)]

Jeremy Custenborder created KAFKA-3347:
--

 Summary: Configure java to prefer ipv4
 Key: KAFKA-3347
 URL: https://issues.apache.org/jira/browse/KAFKA-3347
 Project: Kafka
  Issue Type: Improvement
  Components: core
Affects Versions: 0.9.0.1
Reporter: Jeremy Custenborder
Priority: Minor


I've noticed that ports are sometimes binding on IPv6 addresses rather than the 
IPv4 address I'm expecting. Can we change this so we bing on the IPv4 address 
rather than the IPv6 address? I'm proposing to add this to 
KAFKA_JVM_PERFORMANCE_OPTS.

{code}
-Djava.net.preferIPv4Stack=true
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-3347 - Configure java to prefer ipv4

[jcustenborder]

GitHub user jcustenborder opened a pull request:

https://github.com/apache/kafka/pull/1026

KAFKA-3347 - Configure java to prefer ipv4

Modified KAFKA_JVM_PERFORMANCE_OPTS to include 
-Djava.net.preferIPv4Stack=true. Added an additional space at the end of the 
string to be consistent with the other variables.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jcustenborder/kafka KAFKA-3347

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1026.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1026


commit e717faa1e1e82ed6fe2146db4663a3f4ac528c3b
Author: Jeremy Custenborder 
Date:   2016-03-07T23:14:33Z

KAFKA-3347 - Modified KAFKA_JVM_PERFORMANCE_OPTS to include 
-Djava.net.preferIPv4Stack=true.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3347) Configure java to prefer ipv4

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183972#comment-15183972
 ] 

ASF GitHub Bot commented on KAFKA-3347:
---

GitHub user jcustenborder opened a pull request:

https://github.com/apache/kafka/pull/1026

KAFKA-3347 - Configure java to prefer ipv4

Modified KAFKA_JVM_PERFORMANCE_OPTS to include 
-Djava.net.preferIPv4Stack=true. Added an additional space at the end of the 
string to be consistent with the other variables.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jcustenborder/kafka KAFKA-3347

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1026.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1026


commit e717faa1e1e82ed6fe2146db4663a3f4ac528c3b
Author: Jeremy Custenborder 
Date:   2016-03-07T23:14:33Z

KAFKA-3347 - Modified KAFKA_JVM_PERFORMANCE_OPTS to include 
-Djava.net.preferIPv4Stack=true.




> Configure java to prefer ipv4
> -
>
> Key: KAFKA-3347
> URL: https://issues.apache.org/jira/browse/KAFKA-3347
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 0.9.0.1
>Reporter: Jeremy Custenborder
>Priority: Minor
>
> I've noticed that ports are sometimes binding on IPv6 addresses rather than 
> the IPv4 address I'm expecting. Can we change this so we bing on the IPv4 
> address rather than the IPv6 address? I'm proposing to add this to 
> KAFKA_JVM_PERFORMANCE_OPTS.
> {code}
> -Djava.net.preferIPv4Stack=true
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] Release plan - Kafka 0.10.0

[Joel Koshy]

+1

On Mon, Mar 7, 2016 at 9:27 AM, Gwen Shapira  wrote:

> Greetings Kafka Developer Community,
>
> As you all know, we have few big features that are almost complete
> (Timestamps! Interceptors! Streams!). It is time to start planning our
> next release.
>
> I suggest the following:
> * Cut branches on March 21st
> * Publish the first release candidate the next day
> * Start testing, finding important issues, fixing them, rolling out new
> releases
> * And eventually get a release candidate that we all agree is awesome
> enough to release. Hopefully this won't take too many iterations :)
>
> Note that this is a 2 weeks heads-up on branch cutting. After we cut
> branches, we will try to minimize cherrypicks to just critical bugs
> (because last major release was a bit insane).
> Therefore,  if you have a feature that you really want to see in
> 0.10.0 - you'll need to have it committed by March 21st. As a curtesy
> to the release manager, if you have features that you are not planning
> on getting in for 0.10.0, please change the "fix version" field in
> JIRA accordingly.
>
> I will send a heads-up few days before cutting branches, to give
> everyone a chance to get stragglers in.
>
> The vote will be open for 72 hours.
> All in favor, please reply with +1.
>
> Gwen Shapira
>

Re: [DISCUSS] KIP-45 Standardize all client sequence interaction on j.u.Collection.

[Ismael Juma]

Coming back to this, see below.

On Wed, Jan 27, 2016 at 9:01 PM, Jason Gustafson  wrote:

>
> 1. For subscribe() and assign(), change the parameter type to collection as
> planned in the KIP. This is at least source-compatible, so as long as users
> compile against the updated release, there shouldn't be any problems.
>

I think this one seems to be the least controversial part of the proposal.
And I agree with this suggestion.

2. Instead of changing the signatures of the current pause/resume/seek
> APIs, maybe we can overload them. This keeps compatibility and supports the
> more convenient collection usage, but the cost is some API bloat.
>

It seems like there is no clear winner, so I am OK with this too.

Given the release plan for 0.10.0.0 that is being voted on, I think we
should make a decision on this one way or another very soon.

Ismael

Re: [DISCUSS] KIP-50 - Enhance Authorizer interface to be aware of supported Principal Types

[Gwen Shapira]

Ashish,

I'm neutral on this (+0), but would be good to have feedback from
Harsha and Parth. If you can get their "sounds good", we can probably
get it through fairly soon and in time for 0.10.0.

Gwen

On Wed, Mar 2, 2016 at 9:47 AM, Ashish Singh  wrote:
> Here is link to the KIP,
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-50+-+Enhance+Authorizer+interface+to+be+aware+of+supported+Principal+Types
>
> On Wed, Mar 2, 2016 at 9:46 AM, Ashish Singh  wrote:
>
>> Hi Guys,
>>
>> I would like to initiate a discuss thread on KIP-50. Kafka authorizer is
>> agnostic of principal types it supports, so are the acls CRUD methods
>> in kafka.security.auth.Authorizer. The intent behind is to keep Kafka
>> authorization pluggable, which is really great. However, this leads to Acls
>> CRUD methods not performing any check on validity of acls, as they are not
>> aware of what principal types Authorizer implementation supports. This
>> opens up space for lots of user errors, KAFKA-3097
>>  for an instance.
>>
>> This KIP proposes adding a getSupportedPrincipalTypes method to authorizer
>> and use that for acls verification during acls CRUD.
>>
>> Feedbacks and comments are welcome.
>>
>> --
>>
>> Regards,
>> Ashish
>>
>
>
>
> --
>
> Regards,
> Ashish

Re: [DISCUSS] KIP-43: Kafka SASL enhancements

[Rajini Sivaram]

Gwen,

In cases where you want completely different callbacks for different
mechanisms, I was thinking that the choice would be between a map of
classes (mechanism -> callbackHandler class) or a delegator class that
chooses the appropriate callback handler class based on mechanism. I chose
the latter since it makes it easier to configure in Kafka. Since we create
a callback handler for each channel and configure it with the
client-selected mechanism, it is straightforward to have one wrapper class
that delegates to the right mechanism-specific class to handle callbacks.
In many cases, a single class may be sufficient (the PR uses a single
callback class for Kerberos and PLAIN). I do see your point about the
flexibility that multiple classes would provide, but since you need to be
able to associate the callback with a mechanism for this to be useful, I am
not sure if just a list would add value.

Login class is slightly different since the proposal is to use a single
login context with multiple login modules to handle multiple mechanisms. In
this case, you want to perform login for all the mechanisms that are
enabled. And you want to call loginContext.login() only once. Again, you
can delegate to multiple classes if you wish to add some complex
mechanism-specific logic, but a single login class makes the mapping to a
single login context and the login cache more obvious (the PR has a test
that includes Kerberos and PLAIN).

Thoughts?

On Mon, Mar 7, 2016 at 9:57 PM, Gwen Shapira  wrote:

> Hi,
>
> Before I can vote on this KIP, I have two additional questions /
> comments on the new configuration:
>
> 1. sasl.callback.handler.class - it looks like we want a single class
> that implements all mechanisms. I think this will make it difficult to
> extend since the only way I can add a mechanism will be by
> implementing every single existing mechanism (otherwise customers will
> need to choose between new and existing when selecting which class to
> use). If Microsoft releases a proprietary "AD Mechanism" and Oracle
> releases "OID mechanism", there will be no class that implements both.
> Can we make it a list of classes instead? I realize this complicates
> the code a bit (some kind of factory will be required to choose the
> right class to use), but important IMO.
> 2. similar for sasl.login.class - if I have a class for Kerberos (with
> refresh thread) and a class for "plain", we need to be able to load
> both.
>
> Gwen
>
> On Wed, Mar 2, 2016 at 12:30 AM, Rajini Sivaram
>  wrote:
> > Jun,
> >
> > Thanks, I have added a note to the KIP. I will add a comment in the
> > implementation and also add a unit test to ensure that conflicts are
> > avoided when version number is modified.
> >
> > On Tue, Mar 1, 2016 at 5:43 PM, Jun Rao  wrote:
> >
> >> Rajini,
> >>
> >> Thanks for the explanation. For 1, this implies that we have to be
> careful
> >> with changing the 2-byte version in the future to avoid conflict. Could
> you
> >> document this in the KIP and also in the implementation?
> >>
> >> Jun
> >>
> >> On Tue, Mar 1, 2016 at 2:47 AM, Rajini Sivaram <
> >> rajinisiva...@googlemail.com
> >> > wrote:
> >>
> >> > Jun,
> >> >
> >> > Thank you for the review.
> >> >
> >> >
> >> >1. With GSSAPI, the first context establishment packet starts with
> the
> >> >byte 0x60 (tag for APPLICATION-0) followed by a variable-length
> >> encoded
> >> >size, followed by various tags and contents. And the packet also
> >> > contains a
> >> >checksum. This is completely different from the mechanism packet
> from
> >> > Kafka
> >> >clients which start with a two-byte version set to zero currently,
> >> > followed
> >> >by just a String mechanism.
> >> >2. Agreed, I have removed the version from the server response in
> the
> >> >KIP. Thanks.
> >> >
> >> >
> >> > On Tue, Mar 1, 2016 at 2:33 AM, Jun Rao  wrote:
> >> >
> >> > > Rajini,
> >> > >
> >> > > Thanks for the updates. Just a couple of minor comments.
> >> > >
> >> > > 1. With the default GSSAPI, what's the first packet that the client
> >> sends
> >> > > to the server? Is that completely different from the packet format
> that
> >> > we
> >> > > will use for non-GSSAPI mechanisms?
> >> > >
> >> > > 2. In the server response, it doesn't seem that we need to include
> the
> >> > > version since the client knows the version of the request that it
> >> sends.
> >> > >
> >> > > Jun
> >> > >
> >> > > On Mon, Feb 29, 2016 at 10:14 AM, Rajini Sivaram <
> >> > > rajinisiva...@googlemail.com> wrote:
> >> > >
> >> > > > Harsha,
> >> > > >
> >> > > > Thank you for the review. I will wait another day to see if there
> is
> >> > more
> >> > > > feedback and then start a voting thread.
> >> > > >
> >> > > > Rajini
> >> > > >
> >> > > > On Mon, Feb 29, 2016 at 2:51 PM, Harsha  wrote:
> >> > > >
> >> > > > > Rajini,
> >> > > > >   Thanks for the changes to the KIP. It looks good
> to
> >> > me. I
> >> > > > >   think we can move to voti

[jira] [Commented] (KAFKA-3346) Rename "Mode" to "SslMode"

[Rajini Sivaram (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184063#comment-15184063
 ] 

Rajini Sivaram commented on KAFKA-3346:
---

Since Mode is used by both SSL and SASL, perhaps SslMode is a bit confusing?

> Rename "Mode" to "SslMode"
> --
>
> Key: KAFKA-3346
> URL: https://issues.apache.org/jira/browse/KAFKA-3346
> Project: Kafka
>  Issue Type: Bug
>Reporter: Gwen Shapira
>
> In the channel builders, the Mode enum is undocumented, so it is unclear that 
> it is used to signify whether the connection is for SSL client or SSL server.
> I suggest renaming to SslMode (although adding documentation will be ok too, 
> if people object to the rename)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-45 Standardize all client sequence interaction on j.u.Collection.

[Jason Gustafson]

Hey Ismael,

Thanks for bringing this up again. Just a quick question: if we do #1, then
there's no way that a user binary could work against both 0.9 and 0.10 of
kafka-clients, right? I'm not sure if that is much of a problem, but may
cause a little pain if a user somehow depends transitively on different
versions. Excluding this change, would we otherwise expect
kafka-clients-0.9 to work with an 0.10 broker? I thought the changes for
KIP-32 continued to support the old message format, but I could be wrong.

-Jason

On Mon, Mar 7, 2016 at 3:29 PM, Ismael Juma  wrote:

> Coming back to this, see below.
>
> On Wed, Jan 27, 2016 at 9:01 PM, Jason Gustafson 
> wrote:
>
> >
> > 1. For subscribe() and assign(), change the parameter type to collection
> as
> > planned in the KIP. This is at least source-compatible, so as long as
> users
> > compile against the updated release, there shouldn't be any problems.
> >
>
> I think this one seems to be the least controversial part of the proposal.
> And I agree with this suggestion.
>
> 2. Instead of changing the signatures of the current pause/resume/seek
> > APIs, maybe we can overload them. This keeps compatibility and supports
> the
> > more convenient collection usage, but the cost is some API bloat.
> >
>
> It seems like there is no clear winner, so I am OK with this too.
>
> Given the release plan for 0.10.0.0 that is being voted on, I think we
> should make a decision on this one way or another very soon.
>
> Ismael
>

Re: [DISCUSS] KIP-45 Standardize all client sequence interaction on j.u.Collection.

[Jason Gustafson]

+users

On Mon, Mar 7, 2016 at 4:09 PM, Jason Gustafson  wrote:

> Hey Ismael,
>
> Thanks for bringing this up again. Just a quick question: if we do #1,
> then there's no way that a user binary could work against both 0.9 and 0.10
> of kafka-clients, right? I'm not sure if that is much of a problem, but may
> cause a little pain if a user somehow depends transitively on different
> versions. Excluding this change, would we otherwise expect
> kafka-clients-0.9 to work with an 0.10 broker? I thought the changes for
> KIP-32 continued to support the old message format, but I could be wrong.
>
> -Jason
>
> On Mon, Mar 7, 2016 at 3:29 PM, Ismael Juma  wrote:
>
>> Coming back to this, see below.
>>
>> On Wed, Jan 27, 2016 at 9:01 PM, Jason Gustafson 
>> wrote:
>>
>> >
>> > 1. For subscribe() and assign(), change the parameter type to
>> collection as
>> > planned in the KIP. This is at least source-compatible, so as long as
>> users
>> > compile against the updated release, there shouldn't be any problems.
>> >
>>
>> I think this one seems to be the least controversial part of the proposal.
>> And I agree with this suggestion.
>>
>> 2. Instead of changing the signatures of the current pause/resume/seek
>> > APIs, maybe we can overload them. This keeps compatibility and supports
>> the
>> > more convenient collection usage, but the cost is some API bloat.
>> >
>>
>> It seems like there is no clear winner, so I am OK with this too.
>>
>> Given the release plan for 0.10.0.0 that is being voted on, I think we
>> should make a decision on this one way or another very soon.
>>
>> Ismael
>>
>
>

[jira] [Commented] (KAFKA-3346) Rename "Mode" to "SslMode"

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184069#comment-15184069
 ] 

Ismael Juma commented on KAFKA-3346:


That's true, it's also used to decide whether to create a 
`SaslClientAuthenticator` or `SaslServerAuthenticator` (and for SASL_SSL, for 
the SSL part as well). `ConnectionMode` may be better, not sure. Whatever we 
do, we should at least document it. :)

> Rename "Mode" to "SslMode"
> --
>
> Key: KAFKA-3346
> URL: https://issues.apache.org/jira/browse/KAFKA-3346
> Project: Kafka
>  Issue Type: Bug
>Reporter: Gwen Shapira
>
> In the channel builders, the Mode enum is undocumented, so it is unclear that 
> it is used to signify whether the connection is for SSL client or SSL server.
> I suggest renaming to SslMode (although adding documentation will be ok too, 
> if people object to the rename)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (KAFKA-3346) Rename "Mode" to "SslMode"

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184069#comment-15184069
 ] 

Ismael Juma edited comment on KAFKA-3346 at 3/8/16 12:13 AM:
-

That's true, it's also used to decide whether to create a 
`SaslClientAuthenticator` or `SaslServerAuthenticator` (and SSL mode for 
SASL_SSL). `ConnectionMode` may be better, not sure. Whatever we do, we should 
at least document it. :)


was (Author: ijuma):
That's true, it's also used to decide whether to create a 
`SaslClientAuthenticator` or `SaslServerAuthenticator` (and for SASL_SSL, for 
the SSL part as well). `ConnectionMode` may be better, not sure. Whatever we 
do, we should at least document it. :)

> Rename "Mode" to "SslMode"
> --
>
> Key: KAFKA-3346
> URL: https://issues.apache.org/jira/browse/KAFKA-3346
> Project: Kafka
>  Issue Type: Bug
>Reporter: Gwen Shapira
>
> In the channel builders, the Mode enum is undocumented, so it is unclear that 
> it is used to signify whether the connection is for SSL client or SSL server.
> I suggest renaming to SslMode (although adding documentation will be ok too, 
> if people object to the rename)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: KIP call to discuss some of the outstanding KIPs

[Gwen Shapira]

Hi,

Jun is out on a secret mission this week, so we will need to postpone
the meeting to next week.

Meanwhile, I created a release plan page with our favorite KIPs, to
make sure everyone is on the same page:
https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+0.10.0

I hope I didn't miss anything. Feel free to edit.

We can continue discussing the on-discussion KIPs in the mailing list
until next week.

Gwen


On Mon, Mar 7, 2016 at 11:58 AM, Ashish Singh  wrote:
> Hello Jun,
>
> Could we have a KIP call tomorrow to discuss some of the outstanding KIPs.
> As 0.10 code freeze date, as per current suggestion, is fast approaching,
> it will really help out with moving forward with following KIPs.
>
> * KIP-4: Command line and centralized operations
> * KIP-35: Retrieve protocol version
> * KIP-43: Kafka SASL enhancements
> * KIP-48: Support for delegation tokens
> * KIP-50: Enhance Authorizer interface to be aware of supported Principal
> Types
>
> Please feel free to add any other KIPs which we deem important for 0.10.
> Not all the above KIPs need to be discussed, as some of them are already in
> voting phase, like KIP-43. However, some KIPs, like KIP-35, KIP-48 and
> KIP-50 have a few points that we can talk about and help them in moving
> along.
> --
>
> Regards,
> Ashish

Re: KIP call to discuss some of the outstanding KIPs

[Ashish Singh]

Gwen, thanks for the info and the wiki!

On Mon, Mar 7, 2016 at 4:23 PM, Gwen Shapira  wrote:

> Hi,
>
> Jun is out on a secret mission this week, so we will need to postpone
> the meeting to next week.
>
> Meanwhile, I created a release plan page with our favorite KIPs, to
> make sure everyone is on the same page:
> https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+0.10.0
>
> I hope I didn't miss anything. Feel free to edit.
>
> We can continue discussing the on-discussion KIPs in the mailing list
> until next week.
>
> Gwen
>
>
> On Mon, Mar 7, 2016 at 11:58 AM, Ashish Singh  wrote:
> > Hello Jun,
> >
> > Could we have a KIP call tomorrow to discuss some of the outstanding
> KIPs.
> > As 0.10 code freeze date, as per current suggestion, is fast approaching,
> > it will really help out with moving forward with following KIPs.
> >
> > * KIP-4: Command line and centralized operations
> > * KIP-35: Retrieve protocol version
> > * KIP-43: Kafka SASL enhancements
> > * KIP-48: Support for delegation tokens
> > * KIP-50: Enhance Authorizer interface to be aware of supported Principal
> > Types
> >
> > Please feel free to add any other KIPs which we deem important for 0.10.
> > Not all the above KIPs need to be discussed, as some of them are already
> in
> > voting phase, like KIP-43. However, some KIPs, like KIP-35, KIP-48 and
> > KIP-50 have a few points that we can talk about and help them in moving
> > along.
> > --
> >
> > Regards,
> > Ashish
>



-- 

Regards,
Ashish

Re: [DISCUSS] KIP-50 - Enhance Authorizer interface to be aware of supported Principal Types

[Ashish Singh]

Thanks Gwen.

@Parth, @Harsha pinging you guys for your feedback. Based on discussion on
JIRA, we have following open questions.

   1.

   How to allow an authorizer implementation to specify supported principal
   types?

   An alternative of providing supported Principal types via interface is
   via a config option. Having a config option will be helpful for certain
   third party implementations that uses SimpleAclAuthorizer but support more
   PrincipalTypes. However, it requires adds one more config.

   2.

   ACLs validation should be done by client or by authorizer?

   Once this method is added we expect the Client of the authorizer to do
   the validation on principal types and the authorizer will still not do any
   validation by it self. As an alternative we can add the validation at
   Authorizer level. Having validation done at client side enables clients to
   fail fast for invalid principal types, whereas implementing it at
   authorization level removes the requirement of having the validation done
   on each client implementation.


On Mon, Mar 7, 2016 at 3:47 PM, Gwen Shapira  wrote:

Ashish,
>
> I'm neutral on this (+0), but would be good to have feedback from
> Harsha and Parth. If you can get their "sounds good", we can probably
> get it through fairly soon and in time for 0.10.0.
>
> Gwen
>
> On Wed, Mar 2, 2016 at 9:47 AM, Ashish Singh  wrote:
> > Here is link to the KIP,
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-50+-+Enhance+Authorizer+interface+to+be+aware+of+supported+Principal+Types
> >
> > On Wed, Mar 2, 2016 at 9:46 AM, Ashish Singh 
> wrote:
> >
> >> Hi Guys,
> >>
> >> I would like to initiate a discuss thread on KIP-50. Kafka authorizer is
> >> agnostic of principal types it supports, so are the acls CRUD methods
> >> in kafka.security.auth.Authorizer. The intent behind is to keep Kafka
> >> authorization pluggable, which is really great. However, this leads to
> Acls
> >> CRUD methods not performing any check on validity of acls, as they are
> not
> >> aware of what principal types Authorizer implementation supports. This
> >> opens up space for lots of user errors, KAFKA-3097
> >>  for an instance.
> >>
> >> This KIP proposes adding a getSupportedPrincipalTypes method to
> authorizer
> >> and use that for acls verification during acls CRUD.
> >>
> >> Feedbacks and comments are welcome.
> >>
> >> --
> >>
> >> Regards,
> >> Ashish
> >>
> >
> >
> >
> > --
> >
> > Regards,
> > Ashish
>
​
-- 

Regards,
Ashish

Re: [DISCUSS] KIP-50 - Enhance Authorizer interface to be aware of supported Principal Types

[Ashish Singh]

+ Parth, Harsha

On Mon, Mar 7, 2016 at 4:32 PM, Ashish Singh  wrote:

> Thanks Gwen.
>
> @Parth, @Harsha pinging you guys for your feedback. Based on discussion on
> JIRA, we have following open questions.
>
>1.
>
>How to allow an authorizer implementation to specify supported
>principal types?
>
>An alternative of providing supported Principal types via interface is
>via a config option. Having a config option will be helpful for certain
>third party implementations that uses SimpleAclAuthorizer but support more
>PrincipalTypes. However, it requires adds one more config.
>
>2.
>
>ACLs validation should be done by client or by authorizer?
>
>Once this method is added we expect the Client of the authorizer to do
>the validation on principal types and the authorizer will still not do any
>validation by it self. As an alternative we can add the validation at
>Authorizer level. Having validation done at client side enables clients to
>fail fast for invalid principal types, whereas implementing it at
>authorization level removes the requirement of having the validation done
>on each client implementation.
>
>
> On Mon, Mar 7, 2016 at 3:47 PM, Gwen Shapira  wrote:
>
> Ashish,
>>
>> I'm neutral on this (+0), but would be good to have feedback from
>> Harsha and Parth. If you can get their "sounds good", we can probably
>> get it through fairly soon and in time for 0.10.0.
>>
>> Gwen
>>
>> On Wed, Mar 2, 2016 at 9:47 AM, Ashish Singh  wrote:
>> > Here is link to the KIP,
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-50+-+Enhance+Authorizer+interface+to+be+aware+of+supported+Principal+Types
>> >
>> > On Wed, Mar 2, 2016 at 9:46 AM, Ashish Singh 
>> wrote:
>> >
>> >> Hi Guys,
>> >>
>> >> I would like to initiate a discuss thread on KIP-50. Kafka authorizer
>> is
>> >> agnostic of principal types it supports, so are the acls CRUD methods
>> >> in kafka.security.auth.Authorizer. The intent behind is to keep Kafka
>> >> authorization pluggable, which is really great. However, this leads to
>> Acls
>> >> CRUD methods not performing any check on validity of acls, as they are
>> not
>> >> aware of what principal types Authorizer implementation supports. This
>> >> opens up space for lots of user errors, KAFKA-3097
>> >>  for an instance.
>> >>
>> >> This KIP proposes adding a getSupportedPrincipalTypes method to
>> authorizer
>> >> and use that for acls verification during acls CRUD.
>> >>
>> >> Feedbacks and comments are welcome.
>> >>
>> >> --
>> >>
>> >> Regards,
>> >> Ashish
>> >>
>> >
>> >
>> >
>> > --
>> >
>> > Regards,
>> > Ashish
>>
> ​
> --
>
> Regards,
> Ashish
>



-- 

Regards,
Ashish

Re: [VOTE] Release plan - Kafka 0.10.0

[Becket Qin]

+1 (non-binding)

BTW, if possible, I would like to have KIP-33 in 0.10.0 to solve the long
pending retention issue.

On Mon, Mar 7, 2016 at 3:26 PM, Joel Koshy  wrote:

> +1
>
> On Mon, Mar 7, 2016 at 9:27 AM, Gwen Shapira  wrote:
>
> > Greetings Kafka Developer Community,
> >
> > As you all know, we have few big features that are almost complete
> > (Timestamps! Interceptors! Streams!). It is time to start planning our
> > next release.
> >
> > I suggest the following:
> > * Cut branches on March 21st
> > * Publish the first release candidate the next day
> > * Start testing, finding important issues, fixing them, rolling out new
> > releases
> > * And eventually get a release candidate that we all agree is awesome
> > enough to release. Hopefully this won't take too many iterations :)
> >
> > Note that this is a 2 weeks heads-up on branch cutting. After we cut
> > branches, we will try to minimize cherrypicks to just critical bugs
> > (because last major release was a bit insane).
> > Therefore,  if you have a feature that you really want to see in
> > 0.10.0 - you'll need to have it committed by March 21st. As a curtesy
> > to the release manager, if you have features that you are not planning
> > on getting in for 0.10.0, please change the "fix version" field in
> > JIRA accordingly.
> >
> > I will send a heads-up few days before cutting branches, to give
> > everyone a chance to get stragglers in.
> >
> > The vote will be open for 72 hours.
> > All in favor, please reply with +1.
> >
> > Gwen Shapira
> >
>

Re: [VOTE] Release plan - Kafka 0.10.0

[Gwen Shapira]

In order to track the many KIPs that we are trying to land in the next
two weeks, I created this page:
https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+0.10.0

There are a lot of KIPs under discussion and not too much time to
finalize all of them. Lets do our best to review, discuss and vote on
the KIPs, so the developers will have a chance of getting a patch in
on time.

Gwen

On Mon, Mar 7, 2016 at 4:46 PM, Becket Qin  wrote:
> +1 (non-binding)
>
> BTW, if possible, I would like to have KIP-33 in 0.10.0 to solve the long
> pending retention issue.
>
> On Mon, Mar 7, 2016 at 3:26 PM, Joel Koshy  wrote:
>
>> +1
>>
>> On Mon, Mar 7, 2016 at 9:27 AM, Gwen Shapira  wrote:
>>
>> > Greetings Kafka Developer Community,
>> >
>> > As you all know, we have few big features that are almost complete
>> > (Timestamps! Interceptors! Streams!). It is time to start planning our
>> > next release.
>> >
>> > I suggest the following:
>> > * Cut branches on March 21st
>> > * Publish the first release candidate the next day
>> > * Start testing, finding important issues, fixing them, rolling out new
>> > releases
>> > * And eventually get a release candidate that we all agree is awesome
>> > enough to release. Hopefully this won't take too many iterations :)
>> >
>> > Note that this is a 2 weeks heads-up on branch cutting. After we cut
>> > branches, we will try to minimize cherrypicks to just critical bugs
>> > (because last major release was a bit insane).
>> > Therefore,  if you have a feature that you really want to see in
>> > 0.10.0 - you'll need to have it committed by March 21st. As a curtesy
>> > to the release manager, if you have features that you are not planning
>> > on getting in for 0.10.0, please change the "fix version" field in
>> > JIRA accordingly.
>> >
>> > I will send a heads-up few days before cutting branches, to give
>> > everyone a chance to get stragglers in.
>> >
>> > The vote will be open for 72 hours.
>> > All in favor, please reply with +1.
>> >
>> > Gwen Shapira
>> >
>>

Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism

[Gwen Shapira]

Hi Team,

Since KIP-48 depends on KIP-43, which is already a bit of a risk for
the next release - any chance we can delay delegation tokens to Kafka
0.10.1?
With the community working on a release every 3 month, this is not a huge delay.

Gwen

On Fri, Feb 26, 2016 at 5:11 PM, Ashish Singh  wrote:
> Parth,
>
> Thanks again for the awesome write up. Following our discussion from the
> JIRA, I think it will be easier to compare various alternatives if they are
> listed together. I am stating below a few alternatives along with a the
> current proposal.
> (Current proposal) Store Delegation Token, DT, on ZK.
>
>1. Client authenticates with a broker.
>2. Once a client is authenticated, it will make a broker side call to
>issue a delegation token.
>3. The broker generates a shared secret based on HMAC-SHA256(a
>Password/Secret shared between all brokers, randomly generated tokenId).
>4. Broker stores this token in its in memory cache. Broker also stores
>the DelegationToken without the hmac in the zookeeper.
>5. All brokers will have a cache backed by zookeeper so they will all
>get notified whenever a new token is generated and they will update their
>local cache whenever token state changes.
>6. Broker returns the token to Client.
>
> Probable issues and fixes
>
>1. Probable race condition, client tries to authenticate with a broker
>that is yet to be updated with the newly generated DT. This can probably be
>dealt with making dtRequest block until all brokers have updated their DT
>cache. Zk barrier or similar mechanism can be used. However, all such
>mechanisms will increase complexity.
>2. Using a static secret key from config file. Will require yet another
>config and uses a static secret key. It is advised to rotate secret keys
>periodically. This can be avoided with controller generating secretKey and
>passing to brokers periodically. However, this will require brokers to
>maintain certain counts of secretKeys.
>
> (Alternative 1) Have controller generate delegation token.
>
>1. Client authenticates with a broker.
>2. Once a client is authenticated, it will make a broker side call to
>issue a delegation token.
>3. Broker forwards the request to controller.
>4. Controller generates a DT and broadcasts to all brokers.
>5. Broker stores this token in its memory cache.
>6. Controller responds to broker’s DT req.
>7. Broker returns the token to Client.
>
> Probable issues and fixes
>
>1. We will have to add new APIs to support controller pushing tokens to
>brokers on top of the minimal APIs that are currently proposed.
>2. We will also have to add APIs to support the bootstrapping case, i.e,
>when a new broker comes up it will have to get all delegation tokens from
>the controller.
>3. In catastrophic failures where all brokers go down, the tokens will
>be lost even if servers are restarted as tokens are not persisted anywhere.
>If this happens, then there are more important things to worry about and
>maybe it is better to re-authenticate.
>
> (Alternative 2) Do not distribute DT to other brokers at all.
>
>1. Client authenticates with a broker.
>2. Once a client is authenticated, it will make a broker side call to
>issue a delegation token.
>3. The broker generates DT of form, [hmac + (owner, renewer,
>maxLifeTime, id, hmac, expirationTime)] and passes back this DT to client.
>hmac is generated via {HMAC-SHA256(owner, renewer, maxLifeTime, id, hmac,
>expirationTime) using SecretKey}. Note that all brokers have this 
> SecretKey.
>4. Client then goes to any broker and to authenticate sends the DT.
>Broker recalculates hmac using (owner, renewer, maxLifeTime, id, hmac,
>expirationTime) info from DT and its SecretKey. If it matches with hmac of
>DT, client is authenticated. Yes, it will do other obvious checks of
>timestamp expiry and such.
>
> Note that secret key will be generated by controller and passed to brokers
> periodically.
> Probable issues and fixes
>
>1. How to delete a DT? Yes, that is a downside here. However, this can
>be handled with brokers maintaining a blacklist of DTs, DTs from this list
>can be removed after expiry.
>2. In catastrophic failures where all brokers go down, the tokens will
>be lost even if servers are restarted as tokens are not persisted anywhere.
>If this happens, then there are more important things to worry about and
>maybe it is better to re-authenticate.
>
>
>
> On Fri, Feb 26, 2016 at 1:58 PM, Parth Brahmbhatt <
> pbrahmbh...@hortonworks.com> wrote:
>
>> Hi,
>>
>> I have filed KIP-48 so we can offer hadoop like delegation tokens in
>> kafka. You can review the design
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka.
>> This KIP depends on KIP-43 and we have also discussed an alternative to
>> propos

Re: [DISCUSS] KIP-49: Fair Partition Assignment Strategy

[Gwen Shapira]

Hi Team,

Since there is a fairly busy release coming up in 2 weeks, and since
partition-assignors are pluggable and don't need to be part of an
Apache Kafka release in order to be useful, can we delay this KIP to
release 0.10.1 or 0.11.0 (whichever is earlier)?

This will give the community a chance to focus on features that are
critical for the next release.

Gwen

On Mon, Mar 7, 2016 at 10:30 AM, Joel Koshy  wrote:
> Hey Andrew,
>
> Thanks for adding the example. No I don't think we have a jira open for
> that issue. I'm not sure if we need to fix it in roundrobin (now that it's
> already out there and some may be using it) vs. just going with your new
> "fair" strategy and maybe add a new explicit roundrobinv2.
>
> Thanks
>
> Joel,
>
> On Mon, Feb 29, 2016 at 7:53 AM, Olson,Andrew  wrote:
>
>> Thanks for the feedback. I have added a concrete example to the document
>> that I think illustrates the benefit relatively well.
>>
>> The observation about scaling the workload of individual consumers is
>> certainly valid. I had not really considered this. Our primary concern is
>> being able to gradually roll out consumption configuration changes in a
>> minimally disruptive fashion, including load-balancing. If the round robin
>> strategy can be enhanced to adequately handle that use case, we would be
>> happy. Is there a Jira open for the "flaw" that you mentioned?
>>
>>
>>
>>
>> On 2/26/16, 7:22 PM, "Joel Koshy"  wrote:
>>
>> >Hi Andrew,
>> >
>> >Thanks for the wiki. Just a couple of comments:
>> >
>> >   - The disruptive config change issue that you mentioned is pretty much
>> a
>> >   non-issue in the new consumer due to central assignment.
>> >   - Optional: but it may be helpful to add a concrete example.
>> >   - More of an orthogonal observation than a comment: with heavily skewed
>> >   subscriptions fairness is sort of moot. i.e., people would generally
>> scale
>> >   up or down subscription counts with the express purpose of
>> >   reducing/increasing load on those instances.
>> >   - WRT roundrobin we later realized a significant flaw in the way we lay
>> >   out partitions: we originally wanted to randomize the partition layout
>> to
>> >   reduce the likelihood of most partitions of the same topic from ending
>> up
>> >   on a given consumer which is important if you have a few very large
>> topics.
>> >   Unfortunately we used hashCode - which does a splendid job of clumping
>> >   partitions from the same topic together :( We can probably just "fix"
>> that
>> >   in the new consumer's roundrobin assignor.
>> >
>> >Thanks,
>> >
>> >Joel
>> >
>> >
>> >On Fri, Feb 26, 2016 at 2:32 PM, Olson,Andrew  wrote:
>> >
>> >> Here is a proposal for a new partition assignment strategy,
>> >>
>> >>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-49+-+Fair+Partition+Assignment+Strategy
>> >>
>> >> This KIP corresponds to these two pending pull requests,
>> >> https://github.com/apache/kafka/pull/146
>> >> https://github.com/apache/kafka/pull/979
>> >>
>> >> thanks,
>> >> Andrew
>> >>
>> >> CONFIDENTIALITY NOTICE This message and any included attachments are
>> from
>> >> Cerner Corporation and are intended only for the addressee. The
>> information
>> >> contained in this message is confidential and may constitute inside or
>> >> non-public information under international, federal, or state securities
>> >> laws. Unauthorized forwarding, printing, copying, distribution, or use
>> of
>> >> such information is strictly prohibited and may be unlawful. If you are
>> not
>> >> the addressee, please promptly delete this message and notify the
>> sender of
>> >> the delivery error by e-mail or you may call Cerner's corporate offices
>> in
>> >> Kansas City, Missouri, U.S.A at (+1) (816)221-1024.
>> >>
>>

Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism

[Ismael Juma]

I agree that it would be good to have more time to review and discuss
KIP-48.

Ismael

On Tue, Mar 8, 2016 at 12:55 AM, Gwen Shapira  wrote:

> Hi Team,
>
> Since KIP-48 depends on KIP-43, which is already a bit of a risk for
> the next release - any chance we can delay delegation tokens to Kafka
> 0.10.1?
> With the community working on a release every 3 month, this is not a huge
> delay.
>
> Gwen
>
> On Fri, Feb 26, 2016 at 5:11 PM, Ashish Singh  wrote:
> > Parth,
> >
> > Thanks again for the awesome write up. Following our discussion from the
> > JIRA, I think it will be easier to compare various alternatives if they
> are
> > listed together. I am stating below a few alternatives along with a the
> > current proposal.
> > (Current proposal) Store Delegation Token, DT, on ZK.
> >
> >1. Client authenticates with a broker.
> >2. Once a client is authenticated, it will make a broker side call to
> >issue a delegation token.
> >3. The broker generates a shared secret based on HMAC-SHA256(a
> >Password/Secret shared between all brokers, randomly generated
> tokenId).
> >4. Broker stores this token in its in memory cache. Broker also stores
> >the DelegationToken without the hmac in the zookeeper.
> >5. All brokers will have a cache backed by zookeeper so they will all
> >get notified whenever a new token is generated and they will update
> their
> >local cache whenever token state changes.
> >6. Broker returns the token to Client.
> >
> > Probable issues and fixes
> >
> >1. Probable race condition, client tries to authenticate with a broker
> >that is yet to be updated with the newly generated DT. This can
> probably be
> >dealt with making dtRequest block until all brokers have updated
> their DT
> >cache. Zk barrier or similar mechanism can be used. However, all such
> >mechanisms will increase complexity.
> >2. Using a static secret key from config file. Will require yet
> another
> >config and uses a static secret key. It is advised to rotate secret
> keys
> >periodically. This can be avoided with controller generating
> secretKey and
> >passing to brokers periodically. However, this will require brokers to
> >maintain certain counts of secretKeys.
> >
> > (Alternative 1) Have controller generate delegation token.
> >
> >1. Client authenticates with a broker.
> >2. Once a client is authenticated, it will make a broker side call to
> >issue a delegation token.
> >3. Broker forwards the request to controller.
> >4. Controller generates a DT and broadcasts to all brokers.
> >5. Broker stores this token in its memory cache.
> >6. Controller responds to broker’s DT req.
> >7. Broker returns the token to Client.
> >
> > Probable issues and fixes
> >
> >1. We will have to add new APIs to support controller pushing tokens
> to
> >brokers on top of the minimal APIs that are currently proposed.
> >2. We will also have to add APIs to support the bootstrapping case,
> i.e,
> >when a new broker comes up it will have to get all delegation tokens
> from
> >the controller.
> >3. In catastrophic failures where all brokers go down, the tokens will
> >be lost even if servers are restarted as tokens are not persisted
> anywhere.
> >If this happens, then there are more important things to worry about
> and
> >maybe it is better to re-authenticate.
> >
> > (Alternative 2) Do not distribute DT to other brokers at all.
> >
> >1. Client authenticates with a broker.
> >2. Once a client is authenticated, it will make a broker side call to
> >issue a delegation token.
> >3. The broker generates DT of form, [hmac + (owner, renewer,
> >maxLifeTime, id, hmac, expirationTime)] and passes back this DT to
> client.
> >hmac is generated via {HMAC-SHA256(owner, renewer, maxLifeTime, id,
> hmac,
> >expirationTime) using SecretKey}. Note that all brokers have this
> SecretKey.
> >4. Client then goes to any broker and to authenticate sends the DT.
> >Broker recalculates hmac using (owner, renewer, maxLifeTime, id, hmac,
> >expirationTime) info from DT and its SecretKey. If it matches with
> hmac of
> >DT, client is authenticated. Yes, it will do other obvious checks of
> >timestamp expiry and such.
> >
> > Note that secret key will be generated by controller and passed to
> brokers
> > periodically.
> > Probable issues and fixes
> >
> >1. How to delete a DT? Yes, that is a downside here. However, this can
> >be handled with brokers maintaining a blacklist of DTs, DTs from this
> list
> >can be removed after expiry.
> >2. In catastrophic failures where all brokers go down, the tokens will
> >be lost even if servers are restarted as tokens are not persisted
> anywhere.
> >If this happens, then there are more important things to worry about
> and
> >maybe it is better to re-authenticate.
> >
> >
> >
> > On Fri, Feb 26, 20