Re: [DISCUSS] KIP-212: Enforce set of legal characters for connector names

[Sönke Liebau]

I've spent some time looking at this and testing various characters and it
would appear that Randall's suspicion was spot on. I think we can support a
fairly large set of characters with very minor changes.

I was put of by the exceptions that were thrown when creating connectors
with certain characters and suspected a larger underlying problem when in
fact the only issue is, that the URL in the rest request used to retrieve
the response for the create connector request needs to be percent encoded
[1].

I've fixed this and done some local testing which worked out quite nicely,
apart from two special cases, I've not been able to find characters that
created issues, even space and slash work.
The mentioned special cases are:
  \  - if the name contains a backslash that is not the beginning of a
valid escape sequence the request fails before we ever get it in
ConnectorsResource, so a backslash would need to be escaped: \\
  "  - Quotation marks need to be escaped as well to keep the json body of
the request legal: \"
In both cases the escape character will be part of the connector name and
need to be specified in the url to retrieve the connector as well, even
though we could URL encode it in a legal way without escaping here. So they
work, not sure if I'd recommend using those characters, but no real reason
to prohibit people from using them that I can see either.


What I'd do going forward is:
- withdraw the KIP, as I don't see a real need for one, since this is not
changing anything, just fixing things.
- add a section to the documentation around legal characters, specify the
ones I tested explicitly (url encoded %20 - %7F) and mention that most
other characters should work as well but no guarantees are given
- update the pull request for KAFKA-4930 to allow all characters but still
prohibit creating a connector with an empty name. I'd propose to keep the
validator though as it'll give us a central location to do any checking
that might turn out to be necessary later on.
- add some integration tests to check connectors with special characters in
their names work
- fix the url encoding line in ConnectorsResource

Does that sound fair to everybody?

Kind regards,
Sönke

[1]
https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/rest/resources/ConnectorsResource.java#L102

On Tue, Oct 24, 2017 at 8:40 PM, Colin McCabe  wrote:

> On Tue, Oct 24, 2017, at 11:28, Sönke Liebau wrote:
> > Hi,
> >
> > after reading your messages I'll grant that I might have picked a
> > somewhat
> > draconic option to solve these issues.
> >
> > In general I believe that properly encoding the URLs after having created
> > the connectors should solve a lot of the issues already. For some
> > characters the rest api returns an error on creating the connector as
> > well,
> > so for that URL encoding won't help. However the connectors do get
> > created
> > even though an error is returned, I've never investigated if they are in
> > a
> > consistent state tbh - I'll give this another look.
> >
> > @colin: Entity encoding would allow us to encode a lot of characters,
> > however I am unsure whether we should prefer it over url encoding in this
> > case, as mostly the end user would have to encode the characters himself.
> > And due to entity encoding ending every character with a ; which causes
> > the
> > embedded jetty server to cut the connector name at that character we'd
> > probably need to encode that character in URL encoding again for that to
> > work out - which might get a bit too complex tbh.
>
> Sorry, I meant to write percent-encoding, not entity refs.
> https://en.wikipedia.org/wiki/Percent-encoding
>
> best,
> Colin
>
>
> > I will further investigate which characters the url decoding that jetty
> > brings to the table will let us use and if all of these are correctly
> > handled during connector creation and report back with a new list of
> > characters that I think we can support fairly easily.
> >
> > Kind regards,
> > Sönke
> >
> >
> > On Tue, Oct 24, 2017 at 6:42 PM, Colin McCabe 
> wrote:
> >
> > > It should be possible to use entity references to encode these
> > > characters in URLs.  See https://dev.w3.org/html5/html-author/charref
> > > Maybe I'm misunderstanding the problem, but can we simply encode the
> > > URLs, rather than restricting the names?
> > >
> > > best,
> > > Colin
> > >
> > >
> > > On Mon, Oct 23, 2017, at 14:12, Randall Hauch wrote:
> > > > Here's the link to KIP-212:
> > > > https://cwiki.apache.org/confluence/pages/viewpage.
> > > action?pageId=74684586
> > > >
> > > > I do think it's worthwhile to define the rules for connector names.
> > > > However, I think it would be better to describe the current
> restrictions
> > > > for names outside of them appearing within URLs. For example, if we
> can
> > > > keep connector names relatively free of constraints but instead
> define
> > > > how
> > > > names should be encoded when used within URLs

Authorize to create a KIP page

[Jan Filipiak]

Hello,

to get KAFKA-3705 moving a little bit more I am thinking of starting a 
KIP to discuss the finaly API design.


My confluence id is: "jfilipiak"

Best Jan

Re: [VOTE] KIP-201: Rationalising policy interfaces

[Tom Bentley]

It's been two weeks since I started the vote on this KIP and although there
are two votes so far there are no binding votes. Any feedback from
committers would be appreciated.

Thanks,

Tom

On 12 October 2017 at 10:09, Edoardo Comar  wrote:

> Thanks Tom with the last additions (changes to the protocol) it now
> supersedes KIP-170
>
> +1 non-binding
> --
>
> Edoardo Comar
>
> IBM Message Hub
>
> IBM UK Ltd, Hursley Park, SO21 2JN
>
>
>
> From:   Tom Bentley 
> To: dev@kafka.apache.org
> Date:   11/10/2017 09:21
> Subject:[VOTE] KIP-201: Rationalising policy interfaces
>
>
>
> I would like to start a vote on KIP-201, which proposes to replace the
> existing policy interfaces with a single new policy interface that also
> extends policy support to cover new and existing APIs in the AdminClient.
>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.
> apache.org_confluence_display_KAFKA_KIP-2D201-253A-
> 2BRationalising-2BPolicy-2Binterfaces&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=
> EzRhmSah4IHsUZVekRUIINhltZK7U0OaeRo7hgW4_tQ&m=tE3xo2lmmoCoFZAX60PBT-
> J8TBDWcv-tarJyAlgwfJY&s=puFqZ3Ny4Xcdil5A5huwA5WZtS3WZpD9517uJkCgrCk&e=
>
>
> Thanks for your time.
>
> Tom
>
>
>
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>

[GitHub] kafka pull request #4131: Remove maven central repository, use only jcenter

[yinonavraham]

GitHub user yinonavraham opened a pull request:

https://github.com/apache/kafka/pull/4131

Remove maven central repository, use only jcenter

jcenter is a super set on top of maven central, so having both of those 
repositories is redundant, and the preferred one should be jcenter.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/yinonavraham/kafka patch-1

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/4131.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4131


commit 11951eaec6e865a1fa32aab4c846f96ae8d08fe2
Author: Yinon Avraham 
Date:   2017-10-25T08:46:07Z

Remove maven central repository, use only jcenter

jcenter is a super set on top of maven central, so having both of those 
repositories is redundant, and the preferred one should be jcenter.




---

Re: Authorize to create a KIP page

[Damian Guy]

Hi Jan,

You should have permissions now.

Thanks,
Damian


On Wed, 25 Oct 2017 at 09:41 Jan Filipiak  wrote:

> Hello,
>
> to get KAFKA-3705 moving a little bit more I am thinking of starting a
> KIP to discuss the finaly API design.
>
> My confluence id is: "jfilipiak"
>
> Best Jan
>

Re: [DISCUSS] KIP-179: Change ReassignPartitionsCommand to use AdminClient

[Tom Bentley]

If there are no further comments, I will start a vote on this next week.

Thanks,

Tom

On 20 October 2017 at 08:33, Tom Bentley  wrote:

> Hi,
>
> I've made a fairly major update to KIP-179 to propose APIs for setting
> throttled rates and throttled replicas with the ability to remove these
> automatically at the end of reassignment.
>
> I'd be grateful for your feedback:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-179+-+Change+
> ReassignPartitionsCommand+to+use+AdminClient
>
> Thanks,
>
> Tom
>
> On 2 October 2017 at 13:15, Tom Bentley  wrote:
>
>> One question I have is about whether/how to scope throttling to a
>> reassignment. Currently throttles are only loosely associated with
>> reassignment: You can start a reassignment without any throttling, add
>> throttling to an in-flight reassignment, and remember/forget to remove
>> throttling after the reassignment is complete. There's is great flexibility
>> in that, but also the risk that you forget the remove the throttle(s).
>>
>> Just adding an API for setting the throttled rate makes this situation
>> worse: While it's nice to be able to auto-remove the throttles rate what
>> about the config for the throttled replicas? Also you might add a throttle
>> thinking a reassignment is in-flight, but it has in fact just finished:
>> Those throttles will now hang around until reset or the end of the next
>> reassignment. For these reasons it would be good if the throttle were more
>> directly scoped to the reassignment.
>>
>> On the other hand, taking LinkedIn's Cruise Control as an example, there
>> they seem to modify the reassignment znode directly and incrementally and
>> so there is no notion of "the reassignment". Reassignments will be running
>> continuously, with partitions added before all of the current partitions
>> have completed. If there is no meaningful cluster-wide "reassignment" then
>> it would be better to remove remove the throttle by changing the list of
>> replicas as each replica catches up.
>>
>> I'm interested in any use cases people can share on this, as I'd like the
>> throttle API to be useful for a broad range of use cases, rather than being
>> too narrowly focussed on what's needed by the existing CLI tools.
>>
>> Thanks,
>>
>> Tom
>>
>>
>>
>>
>> On 28 September 2017 at 17:22, Tom Bentley  wrote:
>>
>>> I'm starting to think about KIP-179 again. In order to have more
>>> manageably-scoped KIPs and PRs I think it might be worth factoring-out the
>>> throttling part into a separate KIP. Wdyt?
>>>
>>> Keeping the throttling discussion in this thread for the moment...
>>>
>>> The throttling behaviour is currently spread across the
>>> `(leader|follower).replication.throttled.replicas` topic config and the
>>> `(leader|follower).replication.throttled.rate` dynamic broker config.
>>> It's not really clear to me exactly what "removing the throttle" is
>>> supposed to mean. I mean we could reset the rate to Long.MAV_VALUE or we
>>> could change the list of replicas to an empty list. The
>>> ReassignPartitionsCommand does both, but there is some small utility in
>>> leaving the rate, but clearing the list, if you've discovered the "right"
>>> rate for your cluster/workload and to want it to be sticky for next time.
>>> Does any one do this in practice?
>>>
>>> With regards to throttling, it would be
> worth thinking about a way where the throttling configs can be
> automatically removed without the user having to re-run the tool.
>

 Isn't that just a matter of updating the topic configs for
 (leader|follower).replication.throttled.replicas at the same time we
 remove the reassignment znode? That leaves open the question about whether
 to reset the rates at the same time.

>>>
>>> Thinking some more about my "update the configs at the same time we
>>> remove the reassignment znode" suggestion. The reassignment znode is
>>> persistent, so the reassignment will survive a zookeeper restart. If there
>>> was a flag for the auto-removal of the throttle it would likewise need to
>>> be persistent. Otherwise a ZK restart would remember the reassignment, but
>>> forget about the preference for auto removal of throttles. So, we would use
>>> a persistent znode (a child of the reassignment path, perhaps) to store a
>>> flag for throttle removal.
>>>
>>> Thoughts?
>>>
>>> Cheers,
>>>
>>> Tom
>>>
>>
>>
>

Use self contained tokens instead of ACL

[Postmann, P. (Peter)]

Hi everyone,

I´m working on a concept to use Kafka with self-contained tokens (instead of 
ACL).

The idea:

-  A client requests access to a certain topic (in some kind of portal)

-  The owner of the topic approves the request (in some kind of portal)

-  The client receives a signed tokens which contains the topic (in 
some kind of portal)

-  The client sends the token when he connects to Kafka

-  Kafka validates the token and grants access

Token Format:

-  List of Topics and methods

o   E.g. read /topic1

-  Expire Date

-  Signature

Implementation Idea:

-  Create a custom Authorization Class which checks the signature

-  Implement the possibility to send arbitrary data (key->value) along 
with the request when the client connects to the cluster

I´m looking forward for feedback on this approach and would be happy if you 
could give me a starting where to start with the implementation (or if there 
already is a way to send arbitrary data to a custom Authorizer).

Kind Regards,
Peter

-
ATTENTION:
The information in this e-mail is confidential and only meant for the intended 
recipient. If you are not the intended recipient, don't use or disclose it in 
any way. Please let the sender know and delete the message immediately.
-

[jira] [Created] (KAFKA-6117) One Broker down can't rejoin the cluster

[GangGu (JIRA)]

GangGu created KAFKA-6117:
-

 Summary: One Broker down can't rejoin the cluster
 Key: KAFKA-6117
 URL: https://issues.apache.org/jira/browse/KAFKA-6117
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.8.2.2
Reporter: GangGu
 Attachments: server.log

One broker shutdown, I restart it, but it can't rejoin in the cluster.
My broker's port is 6667. When i restart the broker,
I used 'netstat -apn | grep '
tcp0  0 10.221.157.109:5648210.221.157.109:6667 
TIME_WAIT   -   
tcp0  0 :::10.221.157.109:6667  :::*
LISTEN  7349/java 

these is no client connect.
log list:
-rw-r--r--. 1 kafka hadoop215 Oct 25 18:03 controller.log
-rw-r--r--. 1 kafka hadoop  0 Oct 25 18:03 kafka.err
-rw-r--r--. 1 kafka hadoop 161010 Oct 25 18:22 kafka.out
-rw-r--r--. 1 kafka hadoop  0 Oct 25 18:03 kafka-request.log
-rw-r--r--. 1 kafka hadoop   2144 Oct 25 18:13 kafkaServer-gc.log
-rw-r--r--. 1 kafka hadoop  0 Oct 25 18:03 log-cleaner.log
-rw-r--r--. 1 kafka hadoop  97055 Oct 25 18:22 server.log
-rw-r--r--. 1 kafka hadoop  0 Oct 25 18:03 state-change.log



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Use self contained tokens instead of ACL

[Sönke Liebau]

The concept you describe sounds similar to what Microsoft calls "claims
based authorization".

At a high level I should think that using Kerberos as a vehicle to
transport the information would be the way to go, as it is established and
already supported by Kafka. I believe tickets have a field that can be used
for authorization information, so if information about the topics that a
user has access to were to be encoded in this field you could probably
extend Kafka to extract that information and use it instead of ACLs.

I am not well versed in what exactly Microsoft does and how you can control
the granting side of things, but I do believe that AD server has support
for something along those lines already.

The upside of this would be that you don't have to implement anything
around security, trust, encryption, etc. because everything is provided by
Kerberos.

Not much information in here I am afraid, but maybe a useful direction for
future research.

Kind regards,
Sönke

On Wed, Oct 25, 2017 at 11:55 AM, Postmann, P. (Peter) <
peter.postm...@ing.com.invalid> wrote:

> Hi everyone,
>
> I´m working on a concept to use Kafka with self-contained tokens (instead
> of ACL).
>
> The idea:
>
> -  A client requests access to a certain topic (in some kind of
> portal)
>
> -  The owner of the topic approves the request (in some kind of
> portal)
>
> -  The client receives a signed tokens which contains the topic
> (in some kind of portal)
>
> -  The client sends the token when he connects to Kafka
>
> -  Kafka validates the token and grants access
>
> Token Format:
>
> -  List of Topics and methods
>
> o   E.g. read /topic1
>
> -  Expire Date
>
> -  Signature
>
> Implementation Idea:
>
> -  Create a custom Authorization Class which checks the signature
>
> -  Implement the possibility to send arbitrary data (key->value)
> along with the request when the client connects to the cluster
>
> I´m looking forward for feedback on this approach and would be happy if
> you could give me a starting where to start with the implementation (or if
> there already is a way to send arbitrary data to a custom Authorizer).
>
> Kind Regards,
> Peter
>
> -
> ATTENTION:
> The information in this e-mail is confidential and only meant for the
> intended recipient. If you are not the intended recipient, don't use or
> disclose it in any way. Please let the sender know and delete the message
> immediately.
> -
>



-- 
Sönke Liebau
Partner
Tel. +49 179 7940878
OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany

[GitHub] kafka pull request #4132: KAFKA-5925: Adding records deletion operation to t...

[ppatierno]

GitHub user ppatierno opened a pull request:

https://github.com/apache/kafka/pull/4132

KAFKA-5925: Adding records deletion operation to the new Admin Client API

This is the PR related to the 
[KIP-204](https://cwiki.apache.org/confluence/display/KAFKA/KIP-204+%3A+Adding+records+deletion+operation+to+the+new+Admin+Client+API)
 in order to add the `deleteRecords` operation to the new Admin Client (it's 
already available in the "legacy" one).
Other than that, unit test and integration tests are added as well (such 
integration tests come from the "legacy" integration tests in order to test the 
new addition in the same way as the "legacy" one).

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ppatierno/kafka kafka-5925

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/4132.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4132


commit a2f25eb7337ddea8d421b9e7cc8b3b9adf0349c6
Author: Paolo Patierno 
Date:   2017-09-28T16:33:56Z

Started to add skeleton about delete records operation on Admin Client

commit 1781192e062f50e720bf44f4c2265ea4ea45577e
Author: Paolo Patierno 
Date:   2017-10-03T13:54:41Z

Added first skeleton for deleteRecords method in the Admin Client

commit 4bd2c2112dc310bacb963e4f28bb14472c009a15
Author: Paolo Patierno 
Date:   2017-10-04T08:41:37Z

Completed the DeleteRecordsResult
Added handle response logic for the deleteRecords operation

commit d4ad3fd6a5e4d9f89ffb320de6de5701f6127d44
Author: Paolo Patierno 
Date:   2017-10-04T10:09:04Z

Fixed wrong checking on no error condition

commit e45292004f98c35c93e4d0c4ca6b890e5285044f
Author: Paolo Patierno 
Date:   2017-10-04T10:54:20Z

Added a unit test on Admin Client delete records operation

commit 016fb425ad3b27e2d2b22c87539a1fff34842753
Author: Paolo Patierno 
Date:   2017-10-05T07:40:53Z

Updated inheritance of AdminClientIntegrationTest from 
IntegrationTestHarness for consumer/producer support (delete records tests to 
add)

commit af8162ab3f57b2f60c08c47f1d644bba62d93552
Author: Paolo Patierno 
Date:   2017-10-20T09:22:48Z

Added DeleteRecords class for wrapping long offset

commit 1e716f3c57382f8a8f66e81974d7d597dc076f8e
Author: Paolo Patierno 
Date:   2017-10-20T09:32:50Z

Updated AdminClient interface and implementation with new DeleteRecords 
class

commit 7c83dbce7b8b23d0ba51ec99f51180442f5e7e80
Author: Paolo Patierno 
Date:   2017-10-23T07:53:13Z

Renamed DeleteRecords to DeleteRecordsTarget

commit fbbccea767afa302e821652fe2b0c07e8e91a395
Author: Paolo Patierno 
Date:   2017-10-23T15:51:02Z

Modified getting metadata strategy using a nested Call approach (metadata 
then delete records request)
Added integration tests as "legacy" admin client tests
Added a unit test

commit 9e1051620f2f34cbb2a1dfa03c209ab49de9ebd8
Author: Paolo Patierno 
Date:   2017-10-23T16:18:07Z

Minor fix on creating Map instances

commit db60a16bc6e21d12318ba52f64cbc7effb32f469
Author: Paolo Patierno 
Date:   2017-10-24T09:03:54Z

Renamed DeleteRecordsTarget to RecordsToDelete with related methods
Fixed not working unit test for records deletion

commit 749b4076261f24aac582ed8f059c41a7c25cc5b2
Author: Paolo Patierno 
Date:   2017-10-25T09:00:04Z

Minor fixes on unit and integration tests

commit 520441183c202f24ff86b375189e25f51142bb32
Author: Paolo Patierno 
Date:   2017-10-25T09:13:32Z

Fixed some style errors after checking

commit 599e604d5bb774e5bd90b354a16cf31883b10ae5
Author: Paolo Patierno 
Date:   2017-10-25T10:13:27Z

Fixed conflict with trunk

commit ab718b0e6bd5c1afe82803a11ca4bc2e0979e345
Author: Paolo Patierno 
Date:   2017-10-25T10:19:31Z

Merge branch 'trunk' into kafka-5925

commit 80895bf4adfba517c376a0685efe8ec5de7f17c6
Author: Paolo Patierno 
Date:   2017-10-25T10:35:42Z

Fixed conflict on generateConfigs method




---

Re: [DISCUSS] KIP-204 : adding records deletion operation to the new Admin Client API

[Paolo Patierno]

Thanks for all your feedback guys. I have updated my current code as well.

I know that the vote for this KIP is not started yet (actually I opened it due 
to no feedback on this KIP after a while but then the discussion started and it 
was really useful !) but I have already opened a PR for that.

Maybe feedback could be useful on that as well :


https://github.com/apache/kafka/pull/4132


Thanks


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Azure & IoT
Microsoft Azure Advisor

Twitter : @ppatierno
Linkedin : paolopatierno
Blog : DevExperience



From: Colin McCabe 
Sent: Monday, October 23, 2017 4:34 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-204 : adding records deletion operation to the new 
Admin Client API

On Mon, Oct 23, 2017, at 01:37, Tom Bentley wrote:
> At the risk of muddying the waters further, have you considered
> "RecordsToDelete" as the name of the class? It's both shorter and more
> descriptive imho.

+1 for RecordsToDelete

>
> Also "deleteBefore()" as the factory method name isn't very future proof
> if
> we came to support time-based deletion. Something like "beforeOffset()"
> would be clearer, imho.

Great idea.

best,
Colin

>
> Putting these together: RecordsToDelete.beforeOffset() seems much clearer
> to me than DeleteRecordsTarget.deleteBefore()
>
>
> On 23 October 2017 at 08:45, Paolo Patierno  wrote:
>
> > About the name I just started to have a doubt about DeletetionTarget
> > because it could be bounded to any deletion operation (i.e. delete topic,
> > ...) and not just what we want now, so records deletion.
> >
> > I have updated the KIP-204 using DeleteRecordsTarget so it's clear that
> > it's related to the delete records operation and what it means, so the
> > target for such operation.
> >
> >
> > Paolo Patierno
> > Senior Software Engineer (IoT) @ Red Hat
> > Microsoft MVP on Azure & IoT
> > Microsoft Azure Advisor
> >
> > Twitter : @ppatierno
> > Linkedin : paolopatierno
> > Blog : DevExperience
> >
> >
> > 
> > From: Paolo Patierno 
> > Sent: Monday, October 23, 2017 7:38 AM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-204 : adding records deletion operation to the
> > new Admin Client API
> >
> > Hi Colin,
> >
> > I was using the long primitive in the code but not updated the KIP yet,
> > sorry ... now it's updated !
> >
> > At same time I agree on using DeletionTarget ... KIP updated !
> >
> >
> > Regarding the deleteBefore factory method, it's a pattern already used
> > witn NewPartitions.increaseTo which I think it's really clear and give us
> > more possibility to evolve this DeletionTarget class if we'll add different
> > ways to specify such target not only offset based.
> >
> >
> > Thanks,
> >
> >
> > Paolo Patierno
> > Senior Software Engineer (IoT) @ Red Hat
> > Microsoft MVP on Azure & IoT
> > Microsoft Azure Advisor
> >
> > Twitter : @ppatierno
> > Linkedin : paolopatierno
> > Blog : DevExperience
> >
> >
> > 
> > From: Colin McCabe 
> > Sent: Friday, October 20, 2017 8:18 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-204 : adding records deletion operation to the
> > new Admin Client API
> >
> > > /** Describe records to delete */
> >  > public class DeleteRecords {
> >  > private Long offset;
> >
> > "DeleteRecords" doesn't really explain what the class is, though.  How
> > about "DeletionTarget"?  Also, why do we need a Long object rather than
> > a long primitive?
> >
> >  >
> >  > /**
> >  > * Delete all the records before the given {@code offset}
> >  > */
> >  > public static DeleteRecords deleteBefore(Long offset) { ... }
> >
> > This seems confusing to me.  What's wrong with a regular constructor for
> > DeletionTarget?
> >
> > best,
> > Colin
> >
> >
> > On Fri, Oct 20, 2017, at 01:28, Paolo Patierno wrote:
> > > Hi all,
> > >
> > >
> > > I have just updated the KIP with your suggestion.
> > >
> > > I'm going to continue implementation and tests with these changes,
> > > waiting for further discussion.
> > >
> > >
> > > Thanks,
> > >
> > >
> > > Paolo Patierno
> > > Senior Software Engineer (IoT) @ Red Hat
> > > Microsoft MVP on Azure & IoT
> > > Microsoft Azure Advisor
> > >
> > > Twitter : @ppatierno
> > > Linkedin : paolopatierno
> > > Blog : DevExperience
> > >
> > >
> > > 
> > > From: Paolo Patierno 
> > > Sent: Thursday, October 19, 2017 1:37 PM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [DIS

RE: Use self contained tokens instead of ACL

[Postmann, P. (Peter)]

Hi Sönke,

Thanks for the fast replay. We don’t want to use Kerberos since we want to do 
the authorization on Application level and without involvement of a 3rd party 
during runtime.

-Original Message-
From: Sönke Liebau [mailto:soenke.lie...@opencore.com.INVALID] 
Sent: Mittwoch, 25. Oktober 2017 12:37
To: dev@kafka.apache.org
Subject: Re: Use self contained tokens instead of ACL

The concept you describe sounds similar to what Microsoft calls "claims based 
authorization".

At a high level I should think that using Kerberos as a vehicle to transport 
the information would be the way to go, as it is established and already 
supported by Kafka. I believe tickets have a field that can be used for 
authorization information, so if information about the topics that a user has 
access to were to be encoded in this field you could probably extend Kafka to 
extract that information and use it instead of ACLs.

I am not well versed in what exactly Microsoft does and how you can control the 
granting side of things, but I do believe that AD server has support for 
something along those lines already.

The upside of this would be that you don't have to implement anything around 
security, trust, encryption, etc. because everything is provided by Kerberos.

Not much information in here I am afraid, but maybe a useful direction for 
future research.

Kind regards,
Sönke

On Wed, Oct 25, 2017 at 11:55 AM, Postmann, P. (Peter) < 
peter.postm...@ing.com.invalid> wrote:

> Hi everyone,
>
> I´m working on a concept to use Kafka with self-contained tokens 
> (instead of ACL).
>
> The idea:
>
> -  A client requests access to a certain topic (in some kind of
> portal)
>
> -  The owner of the topic approves the request (in some kind of
> portal)
>
> -  The client receives a signed tokens which contains the topic
> (in some kind of portal)
>
> -  The client sends the token when he connects to Kafka
>
> -  Kafka validates the token and grants access
>
> Token Format:
>
> -  List of Topics and methods
>
> o   E.g. read /topic1
>
> -  Expire Date
>
> -  Signature
>
> Implementation Idea:
>
> -  Create a custom Authorization Class which checks the signature
>
> -  Implement the possibility to send arbitrary data (key->value)
> along with the request when the client connects to the cluster
>
> I´m looking forward for feedback on this approach and would be happy 
> if you could give me a starting where to start with the implementation 
> (or if there already is a way to send arbitrary data to a custom Authorizer).
>
> Kind Regards,
> Peter
>
> -
> ATTENTION:
> The information in this e-mail is confidential and only meant for the 
> intended recipient. If you are not the intended recipient, don't use 
> or disclose it in any way. Please let the sender know and delete the 
> message immediately.
> -
>



--
Sönke Liebau
Partner
Tel. +49 179 7940878
OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany

-
ATTENTION:
The information in this e-mail is confidential and only meant for the intended 
recipient. If you are not the intended recipient, don't use or disclose it in 
any way. Please let the sender know and delete the message immediately.
-

Re: [DISCUSS] KIP-209 Connection String Support

[Clebert Suconic]

+1... I will update the KIP by the weekend...

(I am taking this on my spare time.. although I can rush it if anyone
needs it sooner).

On Tue, Oct 24, 2017 at 12:27 PM, Colin McCabe  wrote:
> Hi Clebert,
>
> As some other people mentioned, a comma is probably not a great choice
> for the entry separator.  We have a lot of configuration values that
> already include commas.  How about using a semicolon instead?
>
> You also need an escaping system in case someone needs a semicolon (or
> whatever) that is part of a configuration key or configuration value.
> How about a simple backslash?  And then if you want a literal backslash,
> you put in two backslashes.
>
> On Thu, Oct 19, 2017, at 18:10, Michael André Pearce wrote:
>> Just another point to why I’d propose the below change to the string
>> format I propose , is an ability to encode the strings easily.
>>
>> We should note that it’s quite typical for serializers to user a
>> schematic registry where one of their properties they will need to set
>> would be in some form like:
>>
>> schema.registry.url=http://schema1:80,schema2:80/api
>>
>> So being able to safely encode this is important.
>>
>> Sent from my iPhone
>>
>> > On 20 Oct 2017, at 01:47, Michael André Pearce 
>> >  wrote:
>> >
>> > Hi Clebert
>> >
>> > Great kip!
>> >
>> > Instead of ‘;’ to separate the host sections with the params section could 
>> > it be a ‘?’
>> >
>> > And like wise ‘,’ param separator could this be ‘&’ (keep the ‘,’ for host 
>> > separator just makes easier to distinguish)
>> >
>> > Also this was it makes it easier to encode params etc as can just re use 
>> > url encoders.
>
> Please, no.  URL encoders will mangle a lot of things horribly (like
> less than signs, greater than signs, etc.)  We should not make this a
> URL or pseudo-URL (see the discussion above).  We should make it clear
> that this is not a URL.
>
>> Invalid conversions would throw InvalidArgumentException (with a description 
>> of the invalid conversion)
>> Invalid parameters would throw InvalidArgumentException (with the name of 
>> the invalid parameter).
>
> This will cause a lot of compatibility problems, right?  If I switch
> back and forth between two Kafka versions, they will support slightly
> different sets of configuration parameters.  It seems saner to simply
> ignore configuration parameters that we don't understand, like we do
> now.
>
> best,
> Colin
>
>
>> >
>> > Also as like many systems it typical to note what the connection string is 
>> > for with a prefix eg ‘kafka://‘
>> >
>> > Just makes it obvious when an app has a list of connection strings in 
>> > their runtime properties which is for which technology.
>> >
>> > Eg example connection string would be:
>> >
>> > kafka://host1:port1,host2:port2?param1=value1&parm2=value2
>> >
>> > Cheers
>> > Mike
>> >
>> > Sent from my iPhone
>> >
>> >> On 19 Oct 2017, at 19:29, Clebert Suconic  
>> >> wrote:
>> >>
>> >> Do I have to do anything here?
>> >>
>> >> I wonder how long I need to wait before proposing the vote.
>> >>
>> >> On Tue, Oct 17, 2017 at 1:17 PM, Clebert Suconic
>> >>  wrote:
>> >>> I had these updates in already... you just changed the names at the
>> >>> string.. but it was pretty much the same thing I think... I had taken
>> >>> you suggestions though.
>> >>>
>> >>>
>> >>> The Exceptions.. these would be implementation details... all I wanted
>> >>> to make sure is that users would get the name of the invalid parameter
>> >>> as part of a string on a message.
>> >>>
>> >>> On Tue, Oct 17, 2017 at 3:15 AM, Satish Duggana
>> >>>  wrote:
>>  You may need to update KIP with the details discussed in this thread in
>>  proposed changes section.
>> 
>> >> My proposed format for the connection string would be:
>> >> IP1:host1,IP2:host2,...IPN:hostn;parameterName=value1;parameterName2=value2;...
>>  parameterNameN=valueN
>>  Format should be
>>  host1:port1,host2:port2,…host:portn;param-name1=param-val1,..
>> 
>> >> Invalid conversions would throw InvalidArgumentException (with a
>>  description of the invalid conversion)
>> >> Invalid parameters would throw InvalidArgumentException (with the 
>> >> name of
>>  the invalid parameter).
>> 
>>  Should throw IllegalArgumentException with respective message.
>> 
>>  Thanks,
>>  Satish.
>> 
>>  On Tue, Oct 17, 2017 at 4:46 AM, Clebert Suconic 
>>  
>>  wrote:
>> 
>> > That works.
>> >
>> >> On Mon, Oct 16, 2017 at 6:59 PM Ted Yu  wrote:
>> >>
>> >> Can't you use IllegalArgumentException ?
>> >>
>> >> Some example in current code base:
>> >>
>> >> clients/src/main/java/org/apache/kafka/clients/Metadata.java:
>> >> throw new IllegalArgumentException("Max time to wait for metadata
>> > updates
>> >> should not be < 0 milliseconds");
>> >>
>> >> On Mon, Oct 16, 2017 at 3:06 PM, Clebert Suconic <
>> >> clebert.suco...@gmail.com>
>> 

[jira] [Resolved] (KAFKA-6114) kafka Java API Consumer and producer Offset value comparison?

[JIRA]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sönke Liebau resolved KAFKA-6114.
-
Resolution: Invalid
  Assignee: Sönke Liebau

> kafka Java API Consumer and producer Offset value comparison?
> -
>
> Key: KAFKA-6114
> URL: https://issues.apache.org/jira/browse/KAFKA-6114
> Project: Kafka
>  Issue Type: Wish
>  Components: consumer, offset manager, producer 
>Affects Versions: 0.11.0.0
> Environment: Linux 
>Reporter: veerendra nath jasthi
>Assignee: Sönke Liebau
>
> I have a requirement to match Kafka producer offset value to consumer offset 
> by using Java API?
> I am new to KAFKA,Could anyone suggest how to proceed with this ?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [DISCUSS] KIP-209 Connection String Support

[Clebert Suconic]

>
> This will cause a lot of compatibility problems, right?  If I switch
> back and forth between two Kafka versions, they will support slightly
> different sets of configuration parameters.  It seems saner to simply
> ignore configuration parameters that we don't understand, like we do
> now.



I am fine either way. The issue I'm thinking is how to capture and
treat typos. a log.warn would be ok?

Re: [VOTE] KIP-205: Add all() and range() API to ReadOnlyWindowStore

[Damian Guy]

+1

On Tue, 24 Oct 2017 at 16:46 Guozhang Wang  wrote:

> +1. Thanks.
>
> On Mon, Oct 23, 2017 at 8:11 PM, Richard Yu 
> wrote:
>
> > Hi all,
> >
> > I want to propose KIP-205 for the addition of new API. It is about adding
> > methods similar to those found in ReadOnlyKeyValueStore to the
> > ReadOnlyWindowStore class. As it appears the discussion has reached a
> > conclusion, I would like to start the voting process.
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 205%3A+Add+all%28%29+and+range%28%29+API+to+ReadOnlyWindowStore
> >
> > Thanks for your patience!
> >
>
>
>
> --
> -- Guozhang
>

[GitHub] kafka pull request #4133: Fix typo dev guide title

[joel-hamill]

GitHub user joel-hamill opened a pull request:

https://github.com/apache/kafka/pull/4133

Fix typo dev guide title



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/joel-hamill/kafka dev-guide-title

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/4133.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4133


commit 97d4d1e09bc24d6cd1b19d809a28e9f88edfb1dd
Author: Joel Hamill 
Date:   2017-10-25T14:50:07Z

Fix typo dev guide title




---

[GitHub] kafka-site issue #103: MINOR: Fix typo in title

[joel-hamill]

Github user joel-hamill commented on the issue:

https://github.com/apache/kafka-site/pull/103
  
@guozhangwang done https://github.com/apache/kafka/pull/4133


---

[GitHub] kafka pull request #4134: KAFKA-6075 Kafka cannot recover after an unclean s...

[tedyu]

GitHub user tedyu opened a pull request:

https://github.com/apache/kafka/pull/4134

KAFKA-6075 Kafka cannot recover after an unclean shutdown on Windows

As Vahid commented, Files.deleteIfExists(file.toPath) seems to destabilize 
Windows environment.

This PR reverts to calling delete() directly.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tedyu/kafka trunk

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/4134.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4134


commit c734471f496b2bace8359f9c899cca73e636aa8d
Author: tedyu 
Date:   2017-10-25T16:28:18Z

KAFKA-6075 Kafka cannot recover after an unclean shutdown on Windows

commit 7355e8c282b1cb7d70f2c290702b5b216f28d3cd
Author: tedyu 
Date:   2017-10-25T16:36:50Z

Use delete()




---

Build failed in Jenkins: kafka-0.11.0-jdk7 #326

[Apache Jenkins Server]

See 


Changes:

[wangguoz] HOTFIX: Remove sysout logging

--
[...truncated 977.65 KB...]
kafka.utils.UtilsTest > testReadInt PASSED

kafka.utils.UtilsTest > testUrlSafeBase64EncodeUUID STARTED

kafka.utils.UtilsTest > testUrlSafeBase64EncodeUUID PASSED

kafka.utils.UtilsTest > testCsvMap STARTED

kafka.utils.UtilsTest > testCsvMap PASSED

kafka.utils.UtilsTest > testInLock STARTED

kafka.utils.UtilsTest > testInLock PASSED

kafka.utils.UtilsTest > testSwallow STARTED

kafka.utils.UtilsTest > testSwallow PASSED

kafka.utils.timer.TimerTest > testAlreadyExpiredTask STARTED

kafka.utils.timer.TimerTest > testAlreadyExpiredTask PASSED

kafka.utils.timer.TimerTest > testTaskExpiration STARTED

kafka.utils.timer.TimerTest > testTaskExpiration PASSED

kafka.utils.timer.TimerTaskListTest > testAll STARTED

kafka.utils.timer.TimerTaskListTest > testAll PASSED

kafka.utils.IteratorTemplateTest > testIterator STARTED

kafka.utils.IteratorTemplateTest > testIterator PASSED

kafka.controller.ControllerIntegrationTest > 
testPartitionReassignmentWithOfflineReplicaHaltingProgress STARTED

kafka.controller.ControllerIntegrationTest > 
testPartitionReassignmentWithOfflineReplicaHaltingProgress PASSED

kafka.controller.ControllerIntegrationTest > 
testControllerEpochPersistsWhenAllBrokersDown STARTED

kafka.controller.ControllerIntegrationTest > 
testControllerEpochPersistsWhenAllBrokersDown PASSED

kafka.controller.ControllerIntegrationTest > 
testTopicCreationWithOfflineReplica STARTED

kafka.controller.ControllerIntegrationTest > 
testTopicCreationWithOfflineReplica PASSED

kafka.controller.ControllerIntegrationTest > 
testPartitionReassignmentResumesAfterReplicaComesOnline STARTED

kafka.controller.ControllerIntegrationTest > 
testPartitionReassignmentResumesAfterReplicaComesOnline PASSED

kafka.controller.ControllerIntegrationTest > 
testLeaderAndIsrWhenEntireIsrOfflineAndUncleanLeaderElectionDisabled STARTED

kafka.controller.ControllerIntegrationTest > 
testLeaderAndIsrWhenEntireIsrOfflineAndUncleanLeaderElectionDisabled PASSED

kafka.controller.ControllerIntegrationTest > 
testTopicPartitionExpansionWithOfflineReplica STARTED

kafka.controller.ControllerIntegrationTest > 
testTopicPartitionExpansionWithOfflineReplica PASSED

kafka.controller.ControllerIntegrationTest > 
testPreferredReplicaLeaderElectionWithOfflinePreferredReplica STARTED

kafka.controller.ControllerIntegrationTest > 
testPreferredReplicaLeaderElectionWithOfflinePreferredReplica PASSED

kafka.controller.ControllerIntegrationTest > 
testAutoPreferredReplicaLeaderElection STARTED

kafka.controller.ControllerIntegrationTest > 
testAutoPreferredReplicaLeaderElection PASSED

kafka.controller.ControllerIntegrationTest > testTopicCreation STARTED

kafka.controller.ControllerIntegrationTest > testTopicCreation PASSED

kafka.controller.ControllerIntegrationTest > testPartitionReassignment STARTED

kafka.controller.ControllerIntegrationTest > testPartitionReassignment PASSED

kafka.controller.ControllerIntegrationTest > testTopicPartitionExpansion STARTED

kafka.controller.ControllerIntegrationTest > testTopicPartitionExpansion PASSED

kafka.controller.ControllerIntegrationTest > 
testControllerMoveIncrementsControllerEpoch STARTED

kafka.controller.ControllerIntegrationTest > 
testControllerMoveIncrementsControllerEpoch PASSED

kafka.controller.ControllerIntegrationTest > 
testLeaderAndIsrWhenEntireIsrOfflineAndUncleanLeaderElectionEnabled STARTED

kafka.controller.ControllerIntegrationTest > 
testLeaderAndIsrWhenEntireIsrOfflineAndUncleanLeaderElectionEnabled PASSED

kafka.controller.ControllerIntegrationTest > testEmptyCluster STARTED

kafka.controller.ControllerIntegrationTest > testEmptyCluster PASSED

kafka.controller.ControllerIntegrationTest > testPreferredReplicaLeaderElection 
STARTED

kafka.controller.ControllerIntegrationTest > testPreferredReplicaLeaderElection 
PASSED

kafka.controller.ControllerEventManagerTest > testEventThatThrowsException 
STARTED

kafka.controller.ControllerEventManagerTest > testEventThatThrowsException 
PASSED

kafka.controller.ControllerEventManagerTest > testSuccessfulEvent STARTED

kafka.controller.ControllerEventManagerTest > testSuccessfulEvent PASSED

kafka.controller.ControllerFailoverTest > testHandleIllegalStateException 
STARTED

kafka.controller.ControllerFailoverTest > testHandleIllegalStateException PASSED

kafka.network.SocketServerTest > testGracefulClose STARTED

kafka.network.SocketServerTest > testGracefulClose PASSED

kafka.network.SocketServerTest > testClientDisconnectionUpdatesRequestMetrics 
STARTED

kafka.network.SocketServerTest > testClientDisconnectionUpdatesRequestMetrics 
PASSED

kafka.network.SocketServerTest > testProcessorMetricsTags STARTED

kafka.network.SocketServerTest > testProcessorMetricsTags PASSED

kafka.network.SocketServerTest > testMaxConnectionsPerIp START

[GitHub] kafka pull request #4133: Fix typo dev guide title

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/4133


---

[jira] [Created] (KAFKA-6118) Transient failure in kafka.api.SaslScramSslEndToEndAuthorizationTest.testTwoConsumersWithDifferentSaslCredentials

[Guozhang Wang (JIRA)]

Guozhang Wang created KAFKA-6118:


 Summary: Transient failure in 
kafka.api.SaslScramSslEndToEndAuthorizationTest.testTwoConsumersWithDifferentSaslCredentials
 Key: KAFKA-6118
 URL: https://issues.apache.org/jira/browse/KAFKA-6118
 Project: Kafka
  Issue Type: Sub-task
  Components: core, unit tests
Affects Versions: 1.0.0
Reporter: Guozhang Wang


Saw this failure on trunk jenkins job:

https://builds.apache.org/job/kafka-pr-jdk9-scala2.12/2274/testReport/junit/kafka.api/SaslScramSslEndToEndAuthorizationTest/testTwoConsumersWithDifferentSaslCredentials/

{code}
Stacktrace

org.apache.kafka.common.errors.GroupAuthorizationException: Not authorized to 
access group: group
Standard Output

[2017-10-25 15:09:49,986] ERROR ZKShutdownHandler is not registered, so 
ZooKeeper server won't take any action on ERROR or SHUTDOWN server state 
changes (org.apache.zookeeper.server.ZooKeeperServer:472)
Adding ACLs for resource `Cluster:kafka-cluster`: 
User:scram-admin has Allow permission for operations: ClusterAction 
from hosts: * 

Current ACLs for resource `Cluster:kafka-cluster`: 
User:scram-admin has Allow permission for operations: ClusterAction 
from hosts: * 

Completed Updating config for entity: user-principal 'scram-admin'.
[2017-10-25 15:09:50,654] ERROR [ReplicaFetcher replicaId=0, leaderId=2, 
fetcherId=0] Error for partition __consumer_offsets-0 from broker 2 
(kafka.server.ReplicaFetcherThread:107)
org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to 
access topics: [Topic authorization failed.]
[2017-10-25 15:09:50,654] ERROR [ReplicaFetcher replicaId=1, leaderId=2, 
fetcherId=0] Error for partition __consumer_offsets-0 from broker 2 
(kafka.server.ReplicaFetcherThread:107)
org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to 
access topics: [Topic authorization failed.]
Adding ACLs for resource `Topic:*`: 
User:scram-admin has Allow permission for operations: Read from hosts: 
* 

Current ACLs for resource `Topic:*`: 
User:scram-admin has Allow permission for operations: Read from hosts: 
* 

Completed Updating config for entity: user-principal 'scram-user'.
Completed Updating config for entity: user-principal 'scram-user2'.
Adding ACLs for resource `Topic:e2etopic`: 
User:scram-user has Allow permission for operations: Write from hosts: *
User:scram-user has Allow permission for operations: Describe from 
hosts: * 

Adding ACLs for resource `Cluster:kafka-cluster`: 
User:scram-user has Allow permission for operations: Create from hosts: 
* 

Current ACLs for resource `Topic:e2etopic`: 
User:scram-user has Allow permission for operations: Write from hosts: *
User:scram-user has Allow permission for operations: Describe from 
hosts: * 

Adding ACLs for resource `Topic:e2etopic`: 
User:scram-user has Allow permission for operations: Read from hosts: *
User:scram-user has Allow permission for operations: Describe from 
hosts: * 

Adding ACLs for resource `Group:group`: 
User:scram-user has Allow permission for operations: Read from hosts: * 

Current ACLs for resource `Topic:e2etopic`: 
User:scram-user has Allow permission for operations: Write from hosts: *
User:scram-user has Allow permission for operations: Describe from 
hosts: *
User:scram-user has Allow permission for operations: Read from hosts: * 

Current ACLs for resource `Group:group`: 
User:scram-user has Allow permission for operations: Read from hosts: * 

[2017-10-25 15:09:52,788] ERROR Error while creating ephemeral at /controller 
with return code: OK 
(kafka.controller.KafkaControllerZkUtils$CheckedEphemeral:101)
[2017-10-25 15:09:54,078] ERROR ZKShutdownHandler is not registered, so 
ZooKeeper server won't take any action on ERROR or SHUTDOWN server state 
changes (org.apache.zookeeper.server.ZooKeeperServer:472)
[2017-10-25 15:09:54,112] ERROR ZKShutdownHandler is not registered, so 
ZooKeeper server won't take any action on ERROR or SHUTDOWN server state 
changes (org.apache.zookeeper.server.ZooKeeperServer:472)
Adding ACLs for resource `Cluster:kafka-cluster`: 
User:scram-admin has Allow permission for operations: ClusterAction 
from hosts: * 

Current ACLs for resource `Cluster:kafka-cluster`: 
User:scram-admin has Allow permission for operations: ClusterAction 
from hosts: * 

Completed Updating config for entity: user-principal 'scram-admin'.
[2017-10-25 15:09:54,739] ERROR [ReplicaFetcher replicaId=0, leaderId=2, 
fetcherId=0] Error for partition __consumer_offsets-0 from broker 2 
(kafka.server.ReplicaFetcherThread:107)
org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to 
access topics: [Topic authorization failed.]
[2017-10-25 15:09:54,739] ERROR [ReplicaFetcher replicaId=1, leaderId=2,

Re: [VOTE] 1.0.0 RC3

[Guozhang Wang]

Ted:

Thanks for the reminder. Yes it is a typo. In fact this is the "forth"
candidate of the release, not the "third" one :)


Jaikiran:

That's a fair point. Though I do not know how to achieve that with the
maven central staging repository mechanism today [1]. If anyone has ideas
how to do that I'm all ears.


All:

The passed Jenkins builders for this RC can now be found here:

System test: https://jenkins.confluent.io/job/system-test-kafka-1.0/14/
Unit test: https://builds.apache.org/job/kafka-1.0-jdk7/55/


Please help verify the quickstarts / tutorials / binary signatures /
anything you can and cast your vote before the voting deadline.

Guozhang


[1] repository.apache.org/#stagingRepositories


On Mon, Oct 23, 2017 at 6:06 PM, Ted Yu  wrote:

> bq. Tag to be voted upon (off 1.0 branch) is the 1.0.0-rc2 tag:
>
> There seems to be a typo above: 1.0.0-rc3 tag
>
> FYI
>
> On Mon, Oct 23, 2017 at 6:00 PM, Guozhang Wang  wrote:
>
> > Hello Kafka users, developers and client-developers,
> >
> > This is the third candidate for release of Apache Kafka 1.0.0. The main
> PRs
> > that gets merged in after RC1 are the following:
> >
> > https://github.com/apache/kafka/commit/dc6bfa553e73ffccd1e604963e076c
> > 78d8ddcd69
> >
> > It's worth noting that starting in this version we are using a different
> > version protocol with three digits: *major.minor.bug-fix*
> >
> > Any and all testing is welcome, but the following areas are worth
> > highlighting:
> >
> > 1. Client developers should verify that their clients can produce/consume
> > to/from 1.0.0 brokers (ideally with compressed and uncompressed data).
> > 2. Performance and stress testing. Heroku and LinkedIn have helped with
> > this in the past (and issues have been found and fixed).
> > 3. End users can verify that their apps work correctly with the new
> > release.
> >
> > This is a major version release of Apache Kafka. It includes 29 new KIPs.
> > See the release notes and release plan
> > (*https://cwiki.apache.org/confluence/pages/viewpage.
> > action?pageId=71764913
> >  action?pageId=71764913
> > >*)
> > for more details. A few feature highlights:
> >
> > * Java 9 support with significantly faster TLS and CRC32C implementations
> > * JBOD improvements: disk failure only disables failed disk but not the
> > broker (KIP-112/KIP-113 part I)
> > * Controller improvements: reduced logging change to greatly accelerate
> > admin request handling.
> > * Newly added metrics across all the modules (KIP-164, KIP-168, KIP-187,
> > KIP-188, KIP-196)
> > * Kafka Streams API improvements (KIP-120 / 130 / 138 / 150 / 160 / 161),
> > and drop compatibility "Evolving" annotations
> >
> > Release notes for the 1.0.0 release:
> > *http://home.apache.org/~guozhang/kafka-1.0.0-rc3/RELEASE_NOTES.html
> > *
> >
> >
> >
> > *** Please download, test and vote by Friday, October 20, 8pm PT
> >
> > Kafka's KEYS file containing PGP keys we use to sign the release:
> > http://kafka.apache.org/KEYS
> >
> > * Release artifacts to be voted upon (source and binary):
> > *http://home.apache.org/~guozhang/kafka-1.0.0-rc3/
> > *
> >
> > * Maven artifacts to be voted upon:
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> >
> > * Javadoc:
> > *http://home.apache.org/~guozhang/kafka-1.0.0-rc3/javadoc/
> > *
> >
> > * Tag to be voted upon (off 1.0 branch) is the 1.0.0-rc2 tag:
> >
> > https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=
> > 7774b0da8ead0d9edd1d4b2f7e1cd743af694112
> >
> >
> > * Documentation:
> > Note the documentation can't be pushed live due to changes that will not
> go
> > live until the release. You can manually verify by downloading
> > http://home.apache.org/~guozhang/kafka-1.0.0-rc3/
> > kafka_2.11-1.0.0-site-docs.tgz
> >
> > I will update this thread with up coming Jenkins builds for this RC
> later,
> > they are currently being executed and will be done tomorrow.
> >
> >
> > /**
> >
> >
> > Thanks,
> > -- Guozhang
> >
>



-- 
-- Guozhang

[GitHub] kafka pull request #4135: KAFKA-5848: Perform a complete topic name validati...

[vahidhashemian]

GitHub user vahidhashemian opened a pull request:

https://github.com/apache/kafka/pull/4135

KAFKA-5848: Perform a complete topic name validation in KafkaConsumer's 
assign/subscribe



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vahidhashemian/kafka KAFKA-5848

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/4135.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4135


commit 572e13c2ec9e88f154574a0500c64f403bf0657e
Author: Vahid Hashemian 
Date:   2017-10-05T21:44:15Z

KAFKA-5848: Perform a complete topic name validation in KafkaConsumer's 
assign/subscribe




---

Re: [VOTE] 1.0.0 RC3

[Dana Powers]

Does the voting deadline also need an update?

> *** Please download, test and vote by Friday, October 20, 8pm PT

On Wed, Oct 25, 2017 at 10:37 AM, Guozhang Wang  wrote:

> Ted:
>
> Thanks for the reminder. Yes it is a typo. In fact this is the "forth"
> candidate of the release, not the "third" one :)
>
>
> Jaikiran:
>
> That's a fair point. Though I do not know how to achieve that with the
> maven central staging repository mechanism today [1]. If anyone has ideas
> how to do that I'm all ears.
>
>
> All:
>
> The passed Jenkins builders for this RC can now be found here:
>
> System test: https://jenkins.confluent.io/job/system-test-kafka-1.0/14/
> Unit test: https://builds.apache.org/job/kafka-1.0-jdk7/55/
>
>
> Please help verify the quickstarts / tutorials / binary signatures /
> anything you can and cast your vote before the voting deadline.
>
> Guozhang
>
>
> [1] repository.apache.org/#stagingRepositories
>
>
> On Mon, Oct 23, 2017 at 6:06 PM, Ted Yu  wrote:
>
> > bq. Tag to be voted upon (off 1.0 branch) is the 1.0.0-rc2 tag:
> >
> > There seems to be a typo above: 1.0.0-rc3 tag
> >
> > FYI
> >
> > On Mon, Oct 23, 2017 at 6:00 PM, Guozhang Wang 
> wrote:
> >
> > > Hello Kafka users, developers and client-developers,
> > >
> > > This is the third candidate for release of Apache Kafka 1.0.0. The main
> > PRs
> > > that gets merged in after RC1 are the following:
> > >
> > > https://github.com/apache/kafka/commit/dc6bfa553e73ffccd1e604963e076c
> > > 78d8ddcd69
> > >
> > > It's worth noting that starting in this version we are using a
> different
> > > version protocol with three digits: *major.minor.bug-fix*
> > >
> > > Any and all testing is welcome, but the following areas are worth
> > > highlighting:
> > >
> > > 1. Client developers should verify that their clients can
> produce/consume
> > > to/from 1.0.0 brokers (ideally with compressed and uncompressed data).
> > > 2. Performance and stress testing. Heroku and LinkedIn have helped with
> > > this in the past (and issues have been found and fixed).
> > > 3. End users can verify that their apps work correctly with the new
> > > release.
> > >
> > > This is a major version release of Apache Kafka. It includes 29 new
> KIPs.
> > > See the release notes and release plan
> > > (*https://cwiki.apache.org/confluence/pages/viewpage.
> > > action?pageId=71764913
> > >  > action?pageId=71764913
> > > >*)
> > > for more details. A few feature highlights:
> > >
> > > * Java 9 support with significantly faster TLS and CRC32C
> implementations
> > > * JBOD improvements: disk failure only disables failed disk but not the
> > > broker (KIP-112/KIP-113 part I)
> > > * Controller improvements: reduced logging change to greatly accelerate
> > > admin request handling.
> > > * Newly added metrics across all the modules (KIP-164, KIP-168,
> KIP-187,
> > > KIP-188, KIP-196)
> > > * Kafka Streams API improvements (KIP-120 / 130 / 138 / 150 / 160 /
> 161),
> > > and drop compatibility "Evolving" annotations
> > >
> > > Release notes for the 1.0.0 release:
> > > *http://home.apache.org/~guozhang/kafka-1.0.0-rc3/RELEASE_NOTES.html
> > > *
> > >
> > >
> > >
> > > *** Please download, test and vote by Friday, October 20, 8pm PT
> > >
> > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > http://kafka.apache.org/KEYS
> > >
> > > * Release artifacts to be voted upon (source and binary):
> > > *http://home.apache.org/~guozhang/kafka-1.0.0-rc3/
> > > *
> > >
> > > * Maven artifacts to be voted upon:
> > > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > >
> > > * Javadoc:
> > > *http://home.apache.org/~guozhang/kafka-1.0.0-rc3/javadoc/
> > > *
> > >
> > > * Tag to be voted upon (off 1.0 branch) is the 1.0.0-rc2 tag:
> > >
> > > https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=
> > > 7774b0da8ead0d9edd1d4b2f7e1cd743af694112
> > >
> > >
> > > * Documentation:
> > > Note the documentation can't be pushed live due to changes that will
> not
> > go
> > > live until the release. You can manually verify by downloading
> > > http://home.apache.org/~guozhang/kafka-1.0.0-rc3/
> > > kafka_2.11-1.0.0-site-docs.tgz
> > >
> > > I will update this thread with up coming Jenkins builds for this RC
> > later,
> > > they are currently being executed and will be done tomorrow.
> > >
> > >
> > > /**
> > >
> > >
> > > Thanks,
> > > -- Guozhang
> > >
> >
>
>
>
> --
> -- Guozhang
>

Re: [VOTE] 1.0.0 RC3

[Guozhang Wang]

The deadline would be:

"Please download, test and vote by Friday, October 27, 8pm PT"


Guozhang

On Wed, Oct 25, 2017 at 12:16 PM, Dana Powers  wrote:

> Does the voting deadline also need an update?
>
> > *** Please download, test and vote by Friday, October 20, 8pm PT
>
> On Wed, Oct 25, 2017 at 10:37 AM, Guozhang Wang 
> wrote:
>
> > Ted:
> >
> > Thanks for the reminder. Yes it is a typo. In fact this is the "forth"
> > candidate of the release, not the "third" one :)
> >
> >
> > Jaikiran:
> >
> > That's a fair point. Though I do not know how to achieve that with the
> > maven central staging repository mechanism today [1]. If anyone has ideas
> > how to do that I'm all ears.
> >
> >
> > All:
> >
> > The passed Jenkins builders for this RC can now be found here:
> >
> > System test: https://jenkins.confluent.io/job/system-test-kafka-1.0/14/
> > Unit test: https://builds.apache.org/job/kafka-1.0-jdk7/55/
> >
> >
> > Please help verify the quickstarts / tutorials / binary signatures /
> > anything you can and cast your vote before the voting deadline.
> >
> > Guozhang
> >
> >
> > [1] repository.apache.org/#stagingRepositories
> >
> >
> > On Mon, Oct 23, 2017 at 6:06 PM, Ted Yu  wrote:
> >
> > > bq. Tag to be voted upon (off 1.0 branch) is the 1.0.0-rc2 tag:
> > >
> > > There seems to be a typo above: 1.0.0-rc3 tag
> > >
> > > FYI
> > >
> > > On Mon, Oct 23, 2017 at 6:00 PM, Guozhang Wang 
> > wrote:
> > >
> > > > Hello Kafka users, developers and client-developers,
> > > >
> > > > This is the third candidate for release of Apache Kafka 1.0.0. The
> main
> > > PRs
> > > > that gets merged in after RC1 are the following:
> > > >
> > > > https://github.com/apache/kafka/commit/
> dc6bfa553e73ffccd1e604963e076c
> > > > 78d8ddcd69
> > > >
> > > > It's worth noting that starting in this version we are using a
> > different
> > > > version protocol with three digits: *major.minor.bug-fix*
> > > >
> > > > Any and all testing is welcome, but the following areas are worth
> > > > highlighting:
> > > >
> > > > 1. Client developers should verify that their clients can
> > produce/consume
> > > > to/from 1.0.0 brokers (ideally with compressed and uncompressed
> data).
> > > > 2. Performance and stress testing. Heroku and LinkedIn have helped
> with
> > > > this in the past (and issues have been found and fixed).
> > > > 3. End users can verify that their apps work correctly with the new
> > > > release.
> > > >
> > > > This is a major version release of Apache Kafka. It includes 29 new
> > KIPs.
> > > > See the release notes and release plan
> > > > (*https://cwiki.apache.org/confluence/pages/viewpage.
> > > > action?pageId=71764913
> > > >  > > action?pageId=71764913
> > > > >*)
> > > > for more details. A few feature highlights:
> > > >
> > > > * Java 9 support with significantly faster TLS and CRC32C
> > implementations
> > > > * JBOD improvements: disk failure only disables failed disk but not
> the
> > > > broker (KIP-112/KIP-113 part I)
> > > > * Controller improvements: reduced logging change to greatly
> accelerate
> > > > admin request handling.
> > > > * Newly added metrics across all the modules (KIP-164, KIP-168,
> > KIP-187,
> > > > KIP-188, KIP-196)
> > > > * Kafka Streams API improvements (KIP-120 / 130 / 138 / 150 / 160 /
> > 161),
> > > > and drop compatibility "Evolving" annotations
> > > >
> > > > Release notes for the 1.0.0 release:
> > > > *http://home.apache.org/~guozhang/kafka-1.0.0-rc3/RELEASE_NOTES.html
> > > >  >*
> > > >
> > > >
> > > >
> > > > *** Please download, test and vote by Friday, October 20, 8pm PT
> > > >
> > > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > > http://kafka.apache.org/KEYS
> > > >
> > > > * Release artifacts to be voted upon (source and binary):
> > > > *http://home.apache.org/~guozhang/kafka-1.0.0-rc3/
> > > > *
> > > >
> > > > * Maven artifacts to be voted upon:
> > > > https://repository.apache.org/content/groups/staging/org/
> apache/kafka/
> > > >
> > > > * Javadoc:
> > > > *http://home.apache.org/~guozhang/kafka-1.0.0-rc3/javadoc/
> > > > *
> > > >
> > > > * Tag to be voted upon (off 1.0 branch) is the 1.0.0-rc2 tag:
> > > >
> > > > https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=
> > > > 7774b0da8ead0d9edd1d4b2f7e1cd743af694112
> > > >
> > > >
> > > > * Documentation:
> > > > Note the documentation can't be pushed live due to changes that will
> > not
> > > go
> > > > live until the release. You can manually verify by downloading
> > > > http://home.apache.org/~guozhang/kafka-1.0.0-rc3/
> > > > kafka_2.11-1.0.0-site-docs.tgz
> > > >
> > > > I will update this thread with up coming Jenkins builds for this RC
> > > later,
> > > > they are currently being executed and will be done tomorrow.
> 

Jenkins build is back to normal : kafka-trunk-jdk8 #2166

[Apache Jenkins Server]

See 

[GitHub] kafka pull request #4123: MINOR: reset state in cleanup, fixes jmx mixin fla...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/4123


---

[jira] [Created] (KAFKA-6119) Silent Data Loss in Kafka011 Transactional Producer

[Gary Y. (JIRA)]

Gary Y. created KAFKA-6119:
--

 Summary: Silent Data Loss in Kafka011 Transactional Producer
 Key: KAFKA-6119
 URL: https://issues.apache.org/jira/browse/KAFKA-6119
 Project: Kafka
  Issue Type: Bug
  Components: core, producer 
Affects Versions: 0.11.0.0, 0.11.0.1
 Environment: openjdk version "1.8.0_144"
OpenJDK Runtime Environment (Zulu 8.23.0.3-macosx) (build 1.8.0_144-b01)
OpenJDK 64-Bit Server VM (Zulu 8.23.0.3-macosx) (build 25.144-b01, mixed mode)
Reporter: Gary Y.
Priority: Blocker


Kafka can lose data published by a transactional {{KafkaProducer}} under some 
circumstances, i.e., data that should be committed atomically may not be fully 
visible from a consumer with {{read_committed}} isolation level.
 
*Steps to reproduce:*
# Set {{transaction.timeout.ms}} to a low value such as {{100}}
# Publish two messages in one transaction to different partitions of a topic 
with a sufficiently long time in-between the messages (e.g., 70 s).
# Only the second message is visible with {{read_committed}} isolation level.

See 
https://github.com/GJL/kafka011-transactional-producer-bug-demo/blob/master/src/main/java/com/garyyao/App.java
 for a full example. Detailed instructions can be found in the {{README.md}}: 
https://github.com/GJL/kafka011-transactional-producer-bug-demo

*Why is this possible?*
Because the transaction timeout is set to a low value, the transaction will be 
rolled back quickly after sending the first message. Indeed, in the broker the 
following logs could be found:
{code}
[2017-10-25 22:54:58,224] INFO [Transaction Coordinator 0]: Initialized 
transactionalId test-producer-1508964897483 with producerId 5 and producer 
epoch 0 on partition __transaction_state-10 
(kafka.coordinator.transaction.TransactionCoordinator)
[2017-10-25 22:55:24,260] INFO [Transaction Coordinator 0]: Completed rollback 
ongoing transaction of transactionalId: test-producer-1508964897483 due to 
timeout (kafka.coordinator.transaction.TransactionCoordinator)
{code}

After rollback the second message is sent to a different partition than the 
first message. 
Upon, transaction commit, 
{{org.apache.kafka.clients.producer.internals.TransactionManager}} may enqueue 
the request {{addPartitionsToTransactionHandler}}:
{code}
private TransactionalRequestResult beginCompletingTransaction(TransactionResult 
transactionResult) {
if (!newPartitionsInTransaction.isEmpty())
enqueueRequest(addPartitionsToTransactionHandler());
EndTxnRequest.Builder builder = new 
EndTxnRequest.Builder(transactionalId, producerIdAndEpoch.producerId,
producerIdAndEpoch.epoch, transactionResult);
EndTxnHandler handler = new EndTxnHandler(builder);
enqueueRequest(handler);
return handler.result;
}
{code}

As can be seen, the condition is fulfilled if {{newPartitionsInTransaction}} is 
non-empty. I suspect because the second message goes to a different partition, 
this condition is satisfied.

In {{KafkaApis.scala}}, I can see that {{handleAddPartitionToTxnRequest}} 
eventually may call {{prepareAddPartitions}}:
{code}
 def prepareAddPartitions(addedTopicPartitions: immutable.Set[TopicPartition], 
updateTimestamp: Long): TxnTransitMetadata = {
val newTxnStartTimestamp = state match {
  case Empty | CompleteAbort | CompleteCommit => updateTimestamp
  case _ => txnStartTimestamp
}

prepareTransitionTo(Ongoing, producerId, producerEpoch, txnTimeoutMs, 
(topicPartitions ++ addedTopicPartitions).toSet,
  newTxnStartTimestamp, updateTimestamp)
  }
{code}

Note that the method's first argument {{newState}} of is always *Ongoing* here. 
I suspect that this puts the transaction, which should be aborted, to _Ongoing_ 
again.





--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (KAFKA-6120) RecordCollectorImpl should not retry sending

[Matthias J. Sax (JIRA)]

Matthias J. Sax created KAFKA-6120:
--

 Summary: RecordCollectorImpl should not retry sending
 Key: KAFKA-6120
 URL: https://issues.apache.org/jira/browse/KAFKA-6120
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 1.0.0
Reporter: Matthias J. Sax
Assignee: Matthias J. Sax
 Fix For: 1.1.0, 1.0.1


Currently, RecordCollectorImpl implements an internal retry loop for sending 
data with a hard coded retry maximum. This raises the problem, that data might 
be send out-of-order while at the same time, does not improve the overall 
resilience much, as the number of retires is hardcoded.

Thus, we should remove this loop and only rely an producer configuration 
parameter {{retires}} that uses can configure accordingly.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (KAFKA-6121) Restore and global consumer should not use auto.offset.reset

[Matthias J. Sax (JIRA)]

Matthias J. Sax created KAFKA-6121:
--

 Summary: Restore and global consumer should not use 
auto.offset.reset
 Key: KAFKA-6121
 URL: https://issues.apache.org/jira/browse/KAFKA-6121
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 1.0.0
Reporter: Matthias J. Sax
Assignee: Matthias J. Sax
 Fix For: 1.1.0, 1.0.1


Streams uses three different consumers internally. The main consumer, as well 
as one consumer for state restore (restore consumer, also used by StandbyTasks) 
and a consumer for global state (used by GlobalThreadThread). While main 
consumer handles InvalidOffsetException correctly, restore and global consumer 
don't. Currently, they rely on auto.offset.reset with default value "latest" -- 
thus, if there is an InvalidOffsetException we just jump to the end of the 
changelog topic instead of proper handler this case.

An InvalidOffsetException can occur for two cases:
# An Kafka Streams application is offline for some time and on restart it reads 
it local offset file. This offset file might contain offsets that are not valid 
anymore as the log got compacted in between.
# Even if we have valid offset and we do a seek, log compaction can actually 
tick an in the background at any point and could make our offset invalid -- 
this is a rather rare race conditions but we need to handle it anyway

For both cases, we can apply the same strategy: wipe out the local RocksDB, 
seekToBeginning, and recreate the store from scratch.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [DISCUSS] KIP-205: Add getAllKeys() API to ReadOnlyWindowStore

[Richard Yu]

Xavier: There has been two pluses on the voting thread. Are you fine with
the current formation?

On Tue, Oct 24, 2017 at 4:26 PM, Richard Yu 
wrote:

> I think we can come up with this compromise: range(long timeFrom, long
> timeTo) will be changed to getKeys(long timeFrom, long timeTo). Sounds fair?
>
>
> On Tue, Oct 24, 2017 at 10:44 AM, Xavier Léauté 
> wrote:
>
>> >
>> > Generally I think having `all / range` is better in terms of consistency
>> > with key-value windows. I.e. queries with key are named as `get / fetch`
>> > for kv / window stores, and queries without key are named as `range /
>> all`.
>> >
>>
>> For kv stores, range takes a range of keys, and with this proposal range
>> on
>> window stores would take a range of time, that does not sound consistent
>> to
>> me at all.
>>
>> We also already have fetch which take both a range of time and keys.
>>
>
>

[jira] [Created] (KAFKA-6122) Global Consumer should handle TimeoutException

[Matthias J. Sax (JIRA)]

Matthias J. Sax created KAFKA-6122:
--

 Summary: Global Consumer should handle TimeoutException
 Key: KAFKA-6122
 URL: https://issues.apache.org/jira/browse/KAFKA-6122
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 1.0.0
Reporter: Matthias J. Sax
Assignee: Matthias J. Sax
 Fix For: 1.1.0, 1.0.1


Global consumer does not handle {{TimeoutException}} that might be throw by 
{{partitonsFor()}} and {{endOffsets()}}. This affect the bootstrapping phase 
only.

We need to allow for a proper retry strategy, with configurable max number of 
retries and back off time between retries. We might need to introduce new 
configuration parameters -- for this case, we need to do a KIP.





--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (KAFKA-5301) Improve exception handling on consumer path

[Matthias J. Sax (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax resolved KAFKA-5301.

Resolution: Duplicate

This issue is contained by KAFKA-6121 and KAFKA-6122, thus I am closing this 
one.

> Improve exception handling on consumer path
> ---
>
> Key: KAFKA-5301
> URL: https://issues.apache.org/jira/browse/KAFKA-5301
> Project: Kafka
>  Issue Type: Sub-task
>  Components: streams
>Affects Versions: 0.11.0.0
>Reporter: Eno Thereska
>Assignee: Matthias J. Sax
>
> Used in StreamsThread.java, mostly to .poll() but also to restore data.
> Used in StreamsTask.java, mostly to .pause(), .resume()
> All exceptions here are currently caught all the way up to the main running 
> loop in a broad catch(Exception e) statement in StreamThread.run().
> One main concern on the consumer path is handling deserialization errors that 
> happen before streams has even had a chance to look at the data: 
> https://issues.apache.org/jira/browse/KAFKA-5157  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (KAFKA-5217) Improve Streams internal exception handling

[Matthias J. Sax (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax resolved KAFKA-5217.

Resolution: Duplicate

This JIRA should be covered by multiple different once. Thus closing for now. 
For details compare KAFKA-5152, KAFKA-4593 and KAFKA-4857

> Improve Streams internal exception handling
> ---
>
> Key: KAFKA-5217
> URL: https://issues.apache.org/jira/browse/KAFKA-5217
> Project: Kafka
>  Issue Type: Sub-task
>  Components: streams
>Affects Versions: 0.10.2.1
>Reporter: Matthias J. Sax
>Assignee: Matthias J. Sax
>
> Streams does not handle all exceptions gracefully atm, but tend to throw 
> exceptions to the user, even if we could handle them internally and recover 
> automatically. We want to revisit this exception handling to be more 
> resilient.
> For example, for any kind of rebalance exception, we should just log it, and 
> rejoin the consumer group.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[DISCUSS] KIP-213: Add zookeeper.max.in.flight.requests config to the broker

[Onur Karaman]

Hey everyone.

I made a config kip, KIP-213: Add zookeeper.max.in.flight.requests config
to the broker:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-213%3A+Add+zookeeper.max.in.flight.requests+config+to+the+broker

Comments are welcome.

- Onur

[jira] [Resolved] (KAFKA-5302) Improve exception handling on streams client (communication with brokers)

[Matthias J. Sax (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax resolved KAFKA-5302.

Resolution: Duplicate

After we replaced our internal client with new {{KafkaAdminClient}} we don't 
need to worry about runtime exceptions anymore as we don't use 
{{StreamsKafkaClient}} at runtime anymore

> Improve exception handling on streams client (communication with brokers)
> -
>
> Key: KAFKA-5302
> URL: https://issues.apache.org/jira/browse/KAFKA-5302
> Project: Kafka
>  Issue Type: Sub-task
>  Components: streams
>Affects Versions: 0.11.0.0
>Reporter: Eno Thereska
>Assignee: Matthias J. Sax
>
> These are exceptions in StreamsKafkaClient.java.
> Currently throws either StreamsException or BrokerNotFoundException.
> Used by InternalTopicManager to create topics and get their metadata.
> Used by StreamPartitionAssignor. 
> Currently InternalTopicManager retries a few times after catching an 
> exception. 
> A failure here is sent all the way up to the stream thread and will stop the 
> streams pipeline. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (KAFKA-5313) Improve exception handling on coordinator interactions

[Matthias J. Sax (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax resolved KAFKA-5313.

Resolution: Duplicate

This is contained by all other exception handling tasks of this umbrella, thus 
I am closing as duplicate.

> Improve exception handling on coordinator interactions
> --
>
> Key: KAFKA-5313
> URL: https://issues.apache.org/jira/browse/KAFKA-5313
> Project: Kafka
>  Issue Type: Sub-task
>  Components: streams
>Affects Versions: 0.11.0.0
>Reporter: Eno Thereska
>Assignee: Matthias J. Sax
> Fix For: 1.1.0
>
>
> Exceptions during assignment of tasks are caught in ConsumerCoordinator.java 
> and streams becomes aware of them during the 
> StreamThread.onPartitionsAssigned() and StreamThread.onPartitionsRevoked() 
> methods. Eventually these exceptions go through StreamThread.pollRequests() 
> all the way up to StreamThread.runLoop() and will halt the stream thread that 
> is processing these exceptions. Other stream threads may continue processing, 
> however it is likely they will experience problems too soon after.
> Exceptions here include LockExceptions that are thrown if tasks cannot use a 
> particular directory due to previous tasks not releasing locks on them during 
> reassignment. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (KAFKA-6047) Allow retries configuration for InternalTopicManager

[Matthias J. Sax (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax resolved KAFKA-6047.

Resolution: Duplicate

The new {{KafkaAdminClient}} allows to configure number of retries. After we 
have replace out internal {{StreamsKafkaClient}} with the new client, this 
issue is resolved.

> Allow retries configuration for InternalTopicManager
> 
>
> Key: KAFKA-6047
> URL: https://issues.apache.org/jira/browse/KAFKA-6047
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Affects Versions: 0.11.0.0
>Reporter: Dmitry Vsekhvalnov
>Assignee: Matthias J. Sax
>Priority: Minor
>
> There is hardcoded number of retries when kafka-streams attempts to create 
> internal topics.
> *InternalTopicManager.MAX_TOPIC_READY_TRY*
> Which is not resilient in some scenarios. Consider setup where replication 
> factor for internal streams topics == number of brokers. (RF=3 and x3 kafka 
> brokers). When any of brokers dies kafka-streams can shutdown before broker 
> is resurrected with approximate log:
> {code}
> [WARN ] [org.apache.kafka.streams.processor.internals.InternalTopicManager] 
> [Could not create internal topics: Found only 2 brokers,  but replication 
> factor is 3. Decrease replication factor for internal topics via 
> StreamsConfig parameter "replication.factor" or add more brokers to your 
> cluster. Retry #2]
> [WARN ] [org.apache.kafka.streams.processor.internals.InternalTopicManager] 
> [Could not create internal topics: Found only 2 brokers,  but replication 
> factor is 3. Decrease replication factor for internal topics via 
> StreamsConfig parameter "replication.factor" or add more brokers to your 
> cluster. Retry #3]
> [WARN ] [org.apache.kafka.streams.processor.internals.InternalTopicManager] 
> [Could not create internal topics: Found only 2 brokers,  but replication 
> factor is 3. Decrease replication factor for internal topics via 
> StreamsConfig parameter "replication.factor" or add more brokers to your 
> cluster. Retry #4]
> [INFO ] [org.apache.kafka.streams.processor.internals.StreamThread] 
> [stream-thread [Shutting down]
> {code}
> Would be nice if kafka-streams provides configuration for 
> InternalTopicManager retries and ideally for retry backoff strategy. To have 
> possibility for resilience tuning. Possibly can re-use corresponding producer 
> configuration. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [DISCUSS] KIP-213: Add zookeeper.max.in.flight.requests config to the broker

[Ted Yu]

This is for KAFKA-5894, right ?
Please fill out the JIRA link.

+1 on this proposal.

On Wed, Oct 25, 2017 at 4:11 PM, Onur Karaman 
wrote:

> Hey everyone.
>
> I made a config kip, KIP-213: Add zookeeper.max.in.flight.requests config
> to the broker:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 213%3A+Add+zookeeper.max.in.flight.requests+config+to+the+broker
>
> Comments are welcome.
>
> - Onur
>

[jira] [Created] (KAFKA-6123) MetricsReporter does not get auto-generated client.id

[Kevin Lu (JIRA)]

Kevin Lu created KAFKA-6123:
---

 Summary: MetricsReporter does not get auto-generated client.id
 Key: KAFKA-6123
 URL: https://issues.apache.org/jira/browse/KAFKA-6123
 Project: Kafka
  Issue Type: Improvement
  Components: clients, metrics
Affects Versions: 0.11.0.0
Reporter: Kevin Lu
Priority: Minor


When a {{MetricsReporter}} is configured for a client, it will receive the 
user-specified configurations via {{Configurable.configure(Map 
configs)}}. Likewise, {{ProducerInterceptor}} and {{ConsumerInterceptor}} 
receive user-specified configurations in their configure methods. 

The difference is when a user does not specify the {{client.id}} field, Kafka 
will auto-generate client ids (producer-1, producer-2, consumer-1, consumer-2, 
etc). This auto-generated {{client.id}} will be passed into the interceptors' 
configure method, but it is not passed to the {{MetricsReporter}} configure 
method.

This makes it harder to directly map {{MetricsReporter}} with the interceptors 
for the client when users do not specify the {{client.id}} field. The 
{{client.id}} can be determined from identifying a metric with the 
{{client.id}} tag, but this is hacky and requires traversal. 

It would be useful to have auto-generated {{client.id}} field also passed to 
the {{MetricsReporter}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (KAFKA-6124) Revisit default config for internal client with regard to resilience

[Matthias J. Sax (JIRA)]

Matthias J. Sax created KAFKA-6124:
--

 Summary: Revisit default config for internal client with regard to 
resilience
 Key: KAFKA-6124
 URL: https://issues.apache.org/jira/browse/KAFKA-6124
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 1.0.0
Reporter: Matthias J. Sax
Assignee: Matthias J. Sax
 Fix For: 1.1.0


We should reevaluate the default config of our internally used clients, to 
update them to make Streams more resilient out-of-the-box.

For example:
 - increase producer "retries"
 - increase producer "max.block.ms"
 - consider impact on max.poll.internal.ms (should we keep it at 
Integer.MAX_VALUE -- note, that KAFKA-5152 resolve the issue why we did set it 
to infinity)
 - double check all other defaults including {{KafkaAdmintClient}}

We should also document all finding in the docs and explain how users can 
configure their application to be more resilient if they want to.

This Jira requires a KIP.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (KAFKA-6125) Avoid third party exception to flow through streams code base

[Matthias J. Sax (JIRA)]

Matthias J. Sax created KAFKA-6125:
--

 Summary: Avoid third party exception to flow through streams code 
base
 Key: KAFKA-6125
 URL: https://issues.apache.org/jira/browse/KAFKA-6125
 Project: Kafka
  Issue Type: Bug
  Components: streams
Reporter: Matthias J. Sax


Streams uses multiple internal client that might throw fatal exceptions (some 
should actually never occur, and if, this would indicate a bug).

We should wrap all calls to the used clients with a {{try-catch}}, and log 
those exceptions as ERRORs immediately. For exceptions that can only occur due 
to a bug (e.g., IllegalStateException, IllegalArgumentException, 
WakeupException, InterruptException) we should ask users in the log message to 
report this as a bug.

Last, we rethrow all those exceptions as {{StreamsException}} (to avoid that a 
standard library exception might be caught by accident somewhere else in our 
code base).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (KAFKA-6126) Reduce rebalance time by not checking if created topics are available

[Matthias J. Sax (JIRA)]

Matthias J. Sax created KAFKA-6126:
--

 Summary: Reduce rebalance time by not checking if created topics 
are available
 Key: KAFKA-6126
 URL: https://issues.apache.org/jira/browse/KAFKA-6126
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 1.0.0
Reporter: Matthias J. Sax
 Fix For: 1.1.0


Within {{StreamPartitionAssignor#assign}} we create new topics and afterwards 
wait in an "infinite loop" until topic metadata propagated throughout the 
cluster. We do this, to make sure topics are available when we start processing.

However, with this approach we "extend" the time in the rebalance phase and 
thus are not responsive (no calls to `poll` for liveness check and 
{{KafkaStreams#close}} suffers). Thus, we might want to remove this check and 
handle potential "topic not found" exceptions in the main thread gracefully.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (KAFKA-6127) Streams should never block infinitely

[Matthias J. Sax (JIRA)]

Matthias J. Sax created KAFKA-6127:
--

 Summary: Streams should never block infinitely
 Key: KAFKA-6127
 URL: https://issues.apache.org/jira/browse/KAFKA-6127
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 1.0.0
Reporter: Matthias J. Sax


Streams uses three consumer APIs that can block infinite: {{commitSync()}}, 
{{committed()}}, and {{position()}}.

If we block within one operation, the whole {{StreamThread}} would block, and 
the instance does not make any progress, becomes unresponsive (for example, 
{{KafkaStreams#close()}} suffers), and we also might drop out of the consumer 
group.

We might consider to use {{wakeup()}} calls to unblock those operations to keep 
{{StreamThread}} in a responsive state.

Note: there are discussion to add timeout to those calls, and thus, we could 
get {{TimeoutExceptions}}. This would be easier to handle than using 
{{wakeup()}}. Thus, we should keep an eye on those discussions. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [DISCUSS] KIP-213 Support non-key joining in KTable

[Matthias J. Sax]

Thanks a lot for the KIP. Can we please move the discussion to the dev list?

Thus, after fixing the KIP collision, just start a new DISCUSS thread.

Thx.


-Matthias

On 10/25/17 4:20 PM, Ted Yu wrote:
> Have you seen the email a moment ago from Onur which uses the same KIP
> number ?
> 
> Looks like there was race condition in modifying wiki.
> 
> Please consider bumping the KIP number.
> 
> Thanks
> 
> On Wed, Oct 25, 2017 at 4:14 PM, Jan Filipiak 
> wrote:
> 
>> Hello Kafka-users,
>>
>> I want to continue with the development of KAFKA-3705, which allows the
>> Streams DSL to perform KTableKTable-Joins when the KTables have a
>> one-to-many relationship.
>> To make sure we cover the requirements of as many users as possible and
>> have a good solution afterwards I invite everyone to read through the KIP I
>> put together and
>> discuss it here in this Thread.
>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-213+
>> Support+non-key+joining+in+KTable
>> https://issues.apache.org/jira/browse/KAFKA-3705
>> https://github.com/apache/kafka/pull/3720
>>
>> I think a public discussion and vote on a solution is exactly what is
>> needed to bring this feauture into kafka-streams. I am looking forward to
>> everyones opinion!
>>
>> Please keep the discussion on the mailing list rather than commenting on
>> the wiki (wiki discussions get unwieldy fast).
>>
>> Best
>> Jan
>>
>>
>>
>>
>>
>>
>>
> 



signature.asc
Description: OpenPGP digital signature

[GitHub] kafka pull request #4136: KAFKA-6100: Down-grade RocksDB to 5.7.3

[guozhangwang]

GitHub user guozhangwang opened a pull request:

https://github.com/apache/kafka/pull/4136

KAFKA-6100: Down-grade RocksDB to 5.7.3



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/guozhangwang/kafka 
K6100-rocksdb-580-regression

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/4136.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4136


commit 92b5797bbbe0cec7faddd000a8bfee715a3f0df7
Author: Guozhang Wang 
Date:   2017-10-26T00:16:08Z

Downgrade rocksDB 5.8.0 to 5.7.3




---

Re: [VOTE] KIP-207:The Offsets which ListOffsetsResponse returns should monotonically increase even during a partition leader change

[Jason Gustafson]

+1. Thanks for the KIP.

On Mon, Oct 23, 2017 at 11:30 AM, Colin McCabe  wrote:

> On Mon, Oct 23, 2017, at 10:29, Jason Gustafson wrote:
> > Thanks for the KIP. I'm assuming the new behavior only affects
> > ListOffsets requests from the consumer.
>
> That's a very good point.  I will add a caveat that we only apply the
> KIP-207 behavior to requests from clients, not requests from other
> brokers (such as the ones made by ReplicaFetcherThread).
>
> > Might be worth mentioning that in the KIP.
> > Also, does it affect all ListOffsets requests, or only those that specify
> > the latest offset?
>
> I don't feel great about allowing someone to ask for the offset at time
> T, get back X, and then ask again for the offset at T the next second
> and get back InvalidOffsetException.  So it's probably best just to
> apply the KIP-207 behavior to all ListOffsets requests from consumers.
>
> Thinking about it a bit more, we should disable the KIP-207 behavior
> when unclean leader elections are enabled on the broker.  When unclean
> leader elections are enabled, data loss is possible.  So we cannot
> guarantee that offsets will always go forwards, even in theory, in this
> mode.
>
> I update the kip-- check it out.
>
> best,
> Colin
>
>
> >
> > -Jason
> >
> > On Wed, Oct 18, 2017 at 9:15 AM, Colin McCabe 
> wrote:
> >
> > > On Wed, Oct 18, 2017, at 04:09, Ismael Juma wrote:
> > > > Thanks for the KIP, +1 (binding). A few comments:
> > > >
> > > > 1. I agree with Jun about LEADER_NOT_AVAILABLE for the error code for
> > > > older
> > > > versions.
> > > > 2. OffsetNotAvailableException seems clear enough (i.e. we don't
> need the
> > > > "ForPartition" part)
> > >
> > > Yeah, that is shorter and probably clearer.  Changed.
> > >
> > > > 3. The KIP seems to be missing the compatibility section.
> > >
> > > Added.
> > >
> > > > 4. It would be good to mention that it's now possible for a fetch to
> > > > succeed while list offsets will not for a period of time. And for
> older
> > > > versions, the latter will return LeaderNotAvailable while the former
> > > > would
> > > > work fine, which is a bit unexpected. Not much we can do about it,
> but
> > > > worth mentioning it in my opinion.
> > >
> > > Fair enough
> > >
> > > cheers,
> > > Colin
> > >
> > > >
> > > > Ismael
> > > >
> > > > On Tue, Oct 17, 2017 at 9:26 PM, Jun Rao  wrote:
> > > >
> > > > > Hi, Colin,
> > > > >
> > > > > Thanks for the KIP. +1. Just a minor comment. For the old client
> > > requests,
> > > > > would it be better to return a LEADER_NOT_AVAILABLE error instead?
> > > > >
> > > > > Jun
> > > > >
> > > > > On Tue, Oct 17, 2017 at 11:11 AM, Colin McCabe  >
> > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I'd like to start the voting process for KIP-207:The  Offsets
> which
> > > > > > ListOffsetsResponse returns should monotonically increase even
> > > during a
> > > > > > partition leader change.
> > > > > >
> > > > > > See
> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > > 207%3A+Offsets+returned+by+ListOffsetsResponse+should+be+
> > > > > > monotonically+increasing+even+during+a+partition+leader+change
> > > > > > for details.
> > > > > >
> > > > > > The voting process will run for at least 72 hours.
> > > > > >
> > > > > > regards,
> > > > > > Colin
> > > > > >
> > > > >
> > >
>

[GitHub] kafka pull request #4137: KAFKA-6119: Bump epoch when expiring transactions ...

[apurvam]

GitHub user apurvam opened a pull request:

https://github.com/apache/kafka/pull/4137

KAFKA-6119: Bump epoch when expiring transactions in the 
TransactionCoordinator

A description of the problem is in the JIRA. I have added an integration 
test which reproduces the original scenario, and also added unit test cases.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/apurvam/kafka 
KAFKA-6119-bump-epoch-when-expiring-transactions

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/4137.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4137


commit 4405e4f9c30e82864417fdbafe3817ee4acee661
Author: Apurva Mehta 
Date:   2017-10-26T00:58:44Z

Bump the epoch when we abort a transaction on the coordinator

commit 9945b4f8315dc8c82aaeb003c07458a6231ee96c
Author: Apurva Mehta 
Date:   2017-10-26T01:06:04Z

Lock the transaction metadata before fencing the epoch. Make the case 
matching exhaustive




---

[GitHub] kafka pull request #4112: MINOR: Rename and change package of async ZooKeepe...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/4112


---

Re: [DISCUSS] KIP-213 Support non-key joining in KTable

[Onur Karaman]

Looks like Jan technically made his KIP wiki page first so I'll just change
my KIP number.

On Wed, Oct 25, 2017 at 4:59 PM, Matthias J. Sax 
wrote:

> Thanks a lot for the KIP. Can we please move the discussion to the dev
> list?
>
> Thus, after fixing the KIP collision, just start a new DISCUSS thread.
>
> Thx.
>
>
> -Matthias
>
> On 10/25/17 4:20 PM, Ted Yu wrote:
> > Have you seen the email a moment ago from Onur which uses the same KIP
> > number ?
> >
> > Looks like there was race condition in modifying wiki.
> >
> > Please consider bumping the KIP number.
> >
> > Thanks
> >
> > On Wed, Oct 25, 2017 at 4:14 PM, Jan Filipiak 
> > wrote:
> >
> >> Hello Kafka-users,
> >>
> >> I want to continue with the development of KAFKA-3705, which allows the
> >> Streams DSL to perform KTableKTable-Joins when the KTables have a
> >> one-to-many relationship.
> >> To make sure we cover the requirements of as many users as possible and
> >> have a good solution afterwards I invite everyone to read through the
> KIP I
> >> put together and
> >> discuss it here in this Thread.
> >>
> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-213+
> >> Support+non-key+joining+in+KTable
> >> https://issues.apache.org/jira/browse/KAFKA-3705
> >> https://github.com/apache/kafka/pull/3720
> >>
> >> I think a public discussion and vote on a solution is exactly what is
> >> needed to bring this feauture into kafka-streams. I am looking forward
> to
> >> everyones opinion!
> >>
> >> Please keep the discussion on the mailing list rather than commenting on
> >> the wiki (wiki discussions get unwieldy fast).
> >>
> >> Best
> >> Jan
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >
>
>

[DISCUSS] KIP-214: Add zookeeper.max.in.flight.requests config to the broker

[Onur Karaman]

Hey everyone.

Giving this another shot since it looks like there was a KIP number
collision on the wiki page.

I made a config kip, KIP-214: Add zookeeper.max.in.flight.requests config
to the broker:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-214%3A+Add+zookeeper.max.in.flight.requests+config+to+the+broker

Comments are welcome.

- Onur

Re: [DISCUSS] KIP-213: Add zookeeper.max.in.flight.requests config to the broker

[Onur Karaman]

It looks like I hit a KIP number collision on the wiki page. Let's move the
discussion over to the thread with subject "[DISCUSS] KIP-214: Add
zookeeper.max.in.flight.requests config to the broker".

https://www.mail-archive.com/dev@kafka.apache.org/msg81900.html

On Wed, Oct 25, 2017 at 4:17 PM, Ted Yu  wrote:

> This is for KAFKA-5894, right ?
> Please fill out the JIRA link.
>
> +1 on this proposal.
>
> On Wed, Oct 25, 2017 at 4:11 PM, Onur Karaman <
> onurkaraman.apa...@gmail.com>
> wrote:
>
> > Hey everyone.
> >
> > I made a config kip, KIP-213: Add zookeeper.max.in.flight.requests
> config
> > to the broker:
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 213%3A+Add+zookeeper.max.in.flight.requests+config+to+the+broker
> >
> > Comments are welcome.
> >
> > - Onur
> >
>

Re: [DISCUSS] KIP-213 Support non-key joining in KTable

[Onur Karaman]

Done. I changed my KIP to KIP-214. So this KIP doesn't need to change.

On Wed, Oct 25, 2017 at 10:33 PM, Onur Karaman  wrote:

> Looks like Jan technically made his KIP wiki page first so I'll just
> change my KIP number.
>
> On Wed, Oct 25, 2017 at 4:59 PM, Matthias J. Sax 
> wrote:
>
>> Thanks a lot for the KIP. Can we please move the discussion to the dev
>> list?
>>
>> Thus, after fixing the KIP collision, just start a new DISCUSS thread.
>>
>> Thx.
>>
>>
>> -Matthias
>>
>> On 10/25/17 4:20 PM, Ted Yu wrote:
>> > Have you seen the email a moment ago from Onur which uses the same KIP
>> > number ?
>> >
>> > Looks like there was race condition in modifying wiki.
>> >
>> > Please consider bumping the KIP number.
>> >
>> > Thanks
>> >
>> > On Wed, Oct 25, 2017 at 4:14 PM, Jan Filipiak > >
>> > wrote:
>> >
>> >> Hello Kafka-users,
>> >>
>> >> I want to continue with the development of KAFKA-3705, which allows the
>> >> Streams DSL to perform KTableKTable-Joins when the KTables have a
>> >> one-to-many relationship.
>> >> To make sure we cover the requirements of as many users as possible and
>> >> have a good solution afterwards I invite everyone to read through the
>> KIP I
>> >> put together and
>> >> discuss it here in this Thread.
>> >>
>> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-213+
>> >> Support+non-key+joining+in+KTable
>> >> https://issues.apache.org/jira/browse/KAFKA-3705
>> >> https://github.com/apache/kafka/pull/3720
>> >>
>> >> I think a public discussion and vote on a solution is exactly what is
>> >> needed to bring this feauture into kafka-streams. I am looking forward
>> to
>> >> everyones opinion!
>> >>
>> >> Please keep the discussion on the mailing list rather than commenting
>> on
>> >> the wiki (wiki discussions get unwieldy fast).
>> >>
>> >> Best
>> >> Jan
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >
>>
>>
>