Re: [DISCUSS] KIP-252: Extend ACLs to allow filtering based on ip ranges and subnets

[Sönke Liebau]

Hi Colin,

I agree, we need to have a way of failing incorrect ranges server-side,
I'll amend the KIP and look into that. I think INVALID_REQUEST should fit
fine, afaik we can send a message along with that code, so that could
explain the actual reason.

Regarding prohibiting these ACLs from being created before the inter-broker
protocol is updated, I am a bit hesitant about this for two reasons.

1. I can't help but feel that we are mixing things in with each other here
that don't really belong together. The broker protocol and ACL versions and
potential ACL versioning are to my mind only loosely linked, because broker
and authorizer are usually updated at the same time. But if we look at
Sentry, Ranger or the Confluent LDAP authorizer I am not sure that this
will hold true forever.
Also, this doesn't address the issue of potential future ACL changes that
might actually be incompatible with existing ACLs.  If we go down this road
I think we should go all the way and come up with a full ACL
versioning-scheme. I've written down a few thoughts/suggestions/questions
on that in this thread.

2. I agree that not preventing this creates a security risk, however in a
very limited scenario.
The same scenario applied when we introduced prefixed-ACLs in version 2 and
back then it was not addressed as far as I can tell. The only reference I
can find is one sentence in the KIP that states the risk.
For people using the Confluent LDAP authorizer (or any other authorizer
that is based on the SimpleACLAuthorizer) there is not much we can do, as
once the cluster is updated these ACLs could be created, but the
Authorizers would only honor them after they themselves are upgraded (which
will probably be a rolling restart as well, hence have the same issue).

I am really not trying to duck the work here, but just addressing this
specific change feels like it misses the larger issue to me.

Best regards,
Sönke

On Wed, Apr 3, 2019 at 4:36 PM Colin McCabe  wrote:

> Hi Sönke,
>
> Maybe a reasonable design here would be to not allow creating ACLs based
> on ip ranges and subnets unless the inter-broker protocol setting has been
> upgraded.  If an upgrade is done correctly, the IBP should not be upgraded
> until all the brokers have been upgraded, so there shouldn't be older
> brokers in the cluster erroneously giving access to things they shouldn't.
> In that case, perhaps we can hold off on introducing an ACL versioning
> scheme for now.
>
> Another thing that is important here is having some way of rejecting
> malformed ip address ranges in the CreateAcls call.  This is probably not
> too difficult, but it should be spelled out.  We could use INVALID_REQUEST
> as the error code for this situation, or maybe create a new one to be more
> specific.
>
> best,
> Colin
>
>
> On Wed, Apr 3, 2019, at 04:58, Sönke Liebau wrote:
> > All,
> >
> > as this thread has now been dormant for about three months again I'll am
> > willing to consider the attempt at looking at a larger versioning scheme
> > for ACLs as failed.
> >
> > I am away for a long weekend tomorrow and will start a [VOTE] thread on
> > implementing this as is on Monday, as I personally consider the security
> > implications of these ACLs in a mixed version cluster quite minimal and
> > addressable via the release notes.
> >
> > Best,
> > Sönke
> >
> > On Sat, Mar 16, 2019 at 1:32 PM Sönke Liebau  >
> > wrote:
> >
> > > Just a quick bump, as this has been quiet for a while again.
> > >
> > > On Tue, Jan 8, 2019 at 12:44 PM Sönke Liebau <
> soenke.lie...@opencore.com>
> > > wrote:
> > >
> > >> Hi Colin,
> > >>
> > >> thanks for your response!
> > >>
> > >> in theory we could get away without any additional path changes I
> > >> think.. I am still somewhat unsure about the best way of addressing
> > >> this. I'll outline my current idea and concerns that I still have,
> > >> maybe you have some thoughts on it.
> > >>
> > >> ACLs are currently stored in two places in ZK: /kafka-acl and
> > >> /kafka-acl-extended based on whether they make use of prefixes or not.
> > >> The reasoning[1] for this is not fundamentally changed by anything we
> > >> are discussing here, so I think that split will need to remain.
> > >>
> > >> ACLs are then stored in the form of a json array:
> > >> [zk: 127.0.0.1:2181(CONNECTED) 9] get /kafka-acl/Topic/*
> > >>
> > >>
> {"version":1,"acls":[{"principal":"User:sliebau","permissionType":"Allow","operation":"Read","host":"*"},{"principal":"User:sliebau","permissionType":"Allow","operation":"Describe","host":"*"},{"principal":"User:sliebau2","permissionType":"Allow","operation":"Describe","host":"*"},{"principal":"User:sliebau2","permissionType":"Allow","operation":"Read","host":"*"}]}
> > >>
> > >> What we could do is add a version property to the individual ACL
> > >> elements like so:
> > >> [
> > >>   {
> > >> "principal": "User:sliebau",
> > >> "permissionType": "Allow",
> > >> "operation": "Read",
> > >> "host": "*",
> > >> "ac

Kafka Broker Config (logs.dir) on Kubernetes

[M. Manna]

 Hello,

The question might trigger people to reply with "Confluent" - but it's not
related to confluent as the kubernetes offering is not for publi/community
edition. So, discussing Helm charts and intro to Confluent isn't our
objective here.

What I am trying to understand is how does the log files (kafka message
logs, consumer offsets) are managed in Kubernetes (e.g. persistent volume,
statefulsets, etc). I have a 3 node cluster running over 3 physical Linux
VMs, and would like to move this setup to Kubernetes.

The only part where we are strugging is with the following:

1) how does logs.dir configuration work per Pod?
2) Assuming I have 3 PODs (3 brokers), and one of the Pod Goes down - how
do I manage the message log and offset files ? If a POD goes does, that
means it will delete everything in the logs.dir location, won't it?
3) I am assuming broker.id will be supplied using some form of configMap,
but if there is anything worth knowing here, please do share.

We have a reliable service on bare metal cloud, so we don't want to disrupt
it unless we are sure about the changes.

Does anyone have any experience with this? If so, it would be great if you
can share any gists or configs.

Much appreciated.

Regards,

[jira] [Resolved] (KAFKA-6399) Consider reducing "max.poll.interval.ms" default for Kafka Streams

[Bill Bejeck (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bill Bejeck resolved KAFKA-6399.

Resolution: Fixed

> Consider reducing "max.poll.interval.ms" default for Kafka Streams
> --
>
> Key: KAFKA-6399
> URL: https://issues.apache.org/jira/browse/KAFKA-6399
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Affects Versions: 1.0.0
>Reporter: Matthias J. Sax
>Assignee: John Roesler
>Priority: Minor
>
> In Kafka {{0.10.2.1}} we change the default value of 
> {{max.poll.intervall.ms}} for Kafka Streams to {{Integer.MAX_VALUE}}. The 
> reason was that long state restore phases during rebalance could yield 
> "rebalance storms" as consumers drop out of a consumer group even if they are 
> healthy as they didn't call {{poll()}} during state restore phase.
> In version {{0.11}} and {{1.0}} the state restore logic was improved a lot 
> and thus, now Kafka Streams does call {{poll()}} even during restore phase. 
> Therefore, we might consider setting a smaller timeout for 
> {{max.poll.intervall.ms}} to detect bad behaving Kafka Streams applications 
> (ie, targeting user code) that don't make progress any more during regular 
> operations.
> The open question would be, what a good default might be. Maybe the actual 
> consumer default of 30 seconds might be sufficient. During one {{poll()}} 
> roundtrip, we would only call {{restoreConsumer.poll()}} once and restore a 
> single batch of records. This should take way less time than 30 seconds.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] KIP-446: Add changelog topic configuration to KTable suppress

[John Roesler]

Hi Maarten,

Thanks for the KIP!

It looks good to me. It seems appropriate to stick close to the same API
presented by Materialized.

I did notice one mis-statement in the proposed Javadoc:
> * Indicates that a changelog should be created for the suppressed KTable.
Actually, the changelog is for the buffer only, not the KTable. This may be
an important distinction for some configurations that folks might set. When
the suppress operator gets data from upstream, it adds it to the buffer
(and also sends that data to the changelog). When it emits records from the
buffer, it also sends a tombstone to the changelog topic. Thus, data in the
buffer is *much* shorter lived than data in other KTables, and the buffer
changelog also may be much more compactable than KTable changelogs. I only
bring this up because it might lead people to configure the changelog topic
differently.

One question that I have is whether we should place any restriction on the
use of these methods. Since there is currently only an in-memory
implementation of the suppression buffer, disabling the changelog is almost
guaranteed to lead to data loss during rebalances or restarts. Then again,
the same thing is true if you use an in-memory state store with changelog
disabled. Thoughts?

Thanks again for the KIP. It'll be great to get this in.

-John

On Tue, Apr 2, 2019 at 4:39 PM Maarten Duijn  wrote:

> Kafka Streams currently does not allow configuring the internal changelog
> topic created by KTable.suppress. This KIP introduces a design for adding
> topic configurations to the suppress API.
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-446%3A+Add+changelog+topic+configuration+to+KTable+suppress
>
>

Need help in setting up security in Kafka systems

[Suman B N]

Team,

Can anyone help me share the configs to be set to achieve the below
security in Kafka systems?

   - Broker-Broker should be PLAINTEXT(No Authentication and Authorization
   between brokers)
   - Zookeeper-Broker should be PLAINTEXT(No Authentication and
   Authorization between brokers and zookeeper)
   - Client-Broker should be SASL_PLAINTEXT(Using JAAS).

Any help to finalize broker configs and client configs will be very helpful.

I am still trying out some configs. I will update the configs with
respective issues observed very soon.

Thanks,
Suman
-- 
*Suman*
*OlaCabs*

Re: [DISCUSS] KIP-411: Add option to make Kafka Connect task client ID values unique

[Randall Hauch]

Sounds great.

I did make a few minor grammatical edits to the "Proposed Changes" section
to avoid the notion that the sink and source tasks create the consumers and
producers, respectively. I think it's important to accurately denote that
the framework creates the producers and consumers for the tasks. (This in
no way changes the proposal at all, and feel free to roll back if you
disagree with the changes. I felt it was easier to change than to explain.)

Looking forward to a vote.

Best regards,

Randall

On Wed, Apr 3, 2019 at 6:49 PM pdavidson 
wrote:

> Thanks Randall, I updated the proposal as suggested. Let me know if any
> other changes need to be made, otherwise I think the KIP-411 proposal is
> ready to finalize.  I will aim to call a vote on Friday.
>
> On Mon, Mar 25, 2019 at 7:12 AM Ryanne Dolan 
> wrote:
>
> > Randall, Paul, the proposal looks great, thanks.
> >
> > Ryanne
> >
> > On Mon, Mar 25, 2019, 9:03 AM Randall Hauch  wrote:
> >
> > > Paul,
> > >
> > > Thanks for updating the KIP with the proposal. I do think the KIP
> should
> > at
> > > least mention that the prior behavior is to allow the worker to
> override
> > > the `producer.client.id` or `consumer.client.id`, which is entirely
> > > possible (though unlikely since there would be an MBean conflict, as
> > > pointed out in the discussion). It might be sufficient to just add a
> > > sentence to the "Compatibility, Deprecation, and Migration Plan"
> section,
> > > like "Any client IDs specified in the worker configuration via `
> > > producer.client.id` or `consumer.client.id` properties will be
> > unchanged,
> > > as those will take precedence." Thoughts?
> > >
> > > Ryanne,
> > >
> > > IIUC your last message, I think the latest KIP proposal will align
> pretty
> > > closely with your suggestion. Can you review and confirm?
> > >
> > > Best regards,
> > >
> > > Randall
> > >
> > > On Fri, Mar 1, 2019 at 3:04 PM Ryanne Dolan 
> > wrote:
> > >
> > > > Paul, Randall, I don't think most people will care to exercise so
> much
> > > > control over the client IDs, so long as they are filled in
> > automatically
> > > in
> > > > a way that eliminates duplicate metrics and remains somewhat legible.
> > If
> > > we
> > > > let the user specify a pattern or something, we're really just making
> > the
> > > > user worry about these requirements.
> > > >
> > > > For example, if they specify "foo" as the client.id, they'll get a
> > bunch
> > > > of
> > > > exceptions about that MBean already existing. So they'll try
> > > > "${connectorName}-foo", which won't work because connectors that get
> > > > restarted will re-use the same client ID and the same MBean again.
> And
> > so
> > > > on, until they end up solving the same problem we are trying to solve
> > > here.
> > > >
> > > > I think you at least need something like
> > "connect--producer-dlq"
> > > to
> > > > avoid MBeans being re-registered within the same JVM. I believe the
> > task
> > > ID
> > > > is based on the connector name, so you'd get e.g.
> > > > "connect-myconnector-1-producer".
> > > >
> > > > Ryanne
> > > >
> > > >
> > > > On Fri, Mar 1, 2019 at 12:44 PM Paul Davidson
> > > >  wrote:
> > > >
> > > > > Thanks Randall.  I like your suggestion: as you say, this would
> make
> > it
> > > > > possible to usefully override the default client id properties.
> > > > >
> > > > > I'm not sure how we would handle the dead-letter queue case though
> -
> > > > maybe
> > > > > we could automatically add a "dlq-" prefix to the producer client
> id?
> > > > >
> > > > > If there is agreement on this change I will update the KIP and the
> PR
> > > > (when
> > > > > I find some time).
> > > > >
> > > > >
> > > > > On Thu, Feb 21, 2019 at 8:12 AM Randall Hauch 
> > > wrote:
> > > > >
> > > > > > Hi, Paul. Thanks for the update to KIP-411 to reflect adding
> > > defaults,
> > > > > and
> > > > > > creating/updating https://github.com/apache/kafka/pull/6097 to
> > > reflect
> > > > > > this
> > > > > > approach.
> > > > > >
> > > > > > Now that we've avoided adding a new config and have changed the
> > > > default `
> > > > > > client.id` to include some context, the connector name, and task
> > > > > number, I
> > > > > > think it makes overriding the client ID via worker config `
> > > > > > producer.client.id` or `consumer.client.id` properties less
> > valuable
> > > > > > because those overridden client IDs will be exactly the same for
> > all
> > > > > > connectors and tasks.
> > > > > >
> > > > > > One one hand, we can leave this as-is, and any users that
> include `
> > > > > > producer.client.id` and `consumer.client.id` in their worker
> > configs
> > > > > keep
> > > > > > the same (sort of useless) behavior. In fact, most users would
> > > probably
> > > > > be
> > > > > > better off by removing these worker config properties and instead
> > > > relying
> > > > > > upon the defaults.
> > > > > >
> > > > > > On the other, similar to what Ewen suggested earlier (in a
> > different
>

Re: [DISCUSS] KIP-411: Add option to make Kafka Connect task client ID values unique

[pdavidson]

Thanks Randall.  You're absolutely right that Worker creates the clients
before passing them to the tasks, so I'm very happy with your changes.

Paul

On Thu, Apr 4, 2019 at 8:02 AM Randall Hauch  wrote:

> Sounds great.
>
> I did make a few minor grammatical edits to the "Proposed Changes" section
> to avoid the notion that the sink and source tasks create the consumers and
> producers, respectively. I think it's important to accurately denote that
> the framework creates the producers and consumers for the tasks. (This in
> no way changes the proposal at all, and feel free to roll back if you
> disagree with the changes. I felt it was easier to change than to explain.)
>
> Looking forward to a vote.
>
> Best regards,
>
> Randall
>
> On Wed, Apr 3, 2019 at 6:49 PM pdavidson  .invalid>
> wrote:
>
> > Thanks Randall, I updated the proposal as suggested. Let me know if any
> > other changes need to be made, otherwise I think the KIP-411 proposal is
> > ready to finalize.  I will aim to call a vote on Friday.
> >
> > On Mon, Mar 25, 2019 at 7:12 AM Ryanne Dolan 
> > wrote:
> >
> > > Randall, Paul, the proposal looks great, thanks.
> > >
> > > Ryanne
> > >
> > > On Mon, Mar 25, 2019, 9:03 AM Randall Hauch  wrote:
> > >
> > > > Paul,
> > > >
> > > > Thanks for updating the KIP with the proposal. I do think the KIP
> > should
> > > at
> > > > least mention that the prior behavior is to allow the worker to
> > override
> > > > the `producer.client.id` or `consumer.client.id`, which is entirely
> > > > possible (though unlikely since there would be an MBean conflict, as
> > > > pointed out in the discussion). It might be sufficient to just add a
> > > > sentence to the "Compatibility, Deprecation, and Migration Plan"
> > section,
> > > > like "Any client IDs specified in the worker configuration via `
> > > > producer.client.id` or `consumer.client.id` properties will be
> > > unchanged,
> > > > as those will take precedence." Thoughts?
> > > >
> > > > Ryanne,
> > > >
> > > > IIUC your last message, I think the latest KIP proposal will align
> > pretty
> > > > closely with your suggestion. Can you review and confirm?
> > > >
> > > > Best regards,
> > > >
> > > > Randall
> > > >
> > > > On Fri, Mar 1, 2019 at 3:04 PM Ryanne Dolan 
> > > wrote:
> > > >
> > > > > Paul, Randall, I don't think most people will care to exercise so
> > much
> > > > > control over the client IDs, so long as they are filled in
> > > automatically
> > > > in
> > > > > a way that eliminates duplicate metrics and remains somewhat
> legible.
> > > If
> > > > we
> > > > > let the user specify a pattern or something, we're really just
> making
> > > the
> > > > > user worry about these requirements.
> > > > >
> > > > > For example, if they specify "foo" as the client.id, they'll get a
> > > bunch
> > > > > of
> > > > > exceptions about that MBean already existing. So they'll try
> > > > > "${connectorName}-foo", which won't work because connectors that
> get
> > > > > restarted will re-use the same client ID and the same MBean again.
> > And
> > > so
> > > > > on, until they end up solving the same problem we are trying to
> solve
> > > > here.
> > > > >
> > > > > I think you at least need something like
> > > "connect--producer-dlq"
> > > > to
> > > > > avoid MBeans being re-registered within the same JVM. I believe the
> > > task
> > > > ID
> > > > > is based on the connector name, so you'd get e.g.
> > > > > "connect-myconnector-1-producer".
> > > > >
> > > > > Ryanne
> > > > >
> > > > >
> > > > > On Fri, Mar 1, 2019 at 12:44 PM Paul Davidson
> > > > >  wrote:
> > > > >
> > > > > > Thanks Randall.  I like your suggestion: as you say, this would
> > make
> > > it
> > > > > > possible to usefully override the default client id properties.
> > > > > >
> > > > > > I'm not sure how we would handle the dead-letter queue case
> though
> > -
> > > > > maybe
> > > > > > we could automatically add a "dlq-" prefix to the producer client
> > id?
> > > > > >
> > > > > > If there is agreement on this change I will update the KIP and
> the
> > PR
> > > > > (when
> > > > > > I find some time).
> > > > > >
> > > > > >
> > > > > > On Thu, Feb 21, 2019 at 8:12 AM Randall Hauch 
> > > > wrote:
> > > > > >
> > > > > > > Hi, Paul. Thanks for the update to KIP-411 to reflect adding
> > > > defaults,
> > > > > > and
> > > > > > > creating/updating https://github.com/apache/kafka/pull/6097 to
> > > > reflect
> > > > > > > this
> > > > > > > approach.
> > > > > > >
> > > > > > > Now that we've avoided adding a new config and have changed the
> > > > > default `
> > > > > > > client.id` to include some context, the connector name, and
> task
> > > > > > number, I
> > > > > > > think it makes overriding the client ID via worker config `
> > > > > > > producer.client.id` or `consumer.client.id` properties less
> > > valuable
> > > > > > > because those overridden client IDs will be exactly the same
> for
> > > all
> > > > > > > connectors and tasks.
> > > > > > >
> > > >

Re: Need help in setting up security in Kafka systems

[M. Manna]

Hi,

Have you checked the section on security here? It's got a comprehensive
guide

https://kafka.apache.org/documentation/#security_sasl

To answer your questions briefly

- Broker to Broker should be plainTEXT (or SSL if inter-broker security is
enabled then broker2brorker works as a client coms)
- zookeeper to broker should be plaintext
- client to broker should be plaintext

But then, I would recommend you read the guidlines above. We setup our
security nicely with the above guidelines, however they were not using SASL
but only SSL.

Thanks,

On Thu, 4 Apr 2019 at 15:53, Suman B N  wrote:

> Team,
>
> Can anyone help me share the configs to be set to achieve the below
> security in Kafka systems?
>
>- Broker-Broker should be PLAINTEXT(No Authentication and Authorization
>between brokers)
>- Zookeeper-Broker should be PLAINTEXT(No Authentication and
>Authorization between brokers and zookeeper)
>- Client-Broker should be SASL_PLAINTEXT(Using JAAS).
>
> Any help to finalize broker configs and client configs will be very
> helpful.
>
> I am still trying out some configs. I will update the configs with
> respective issues observed very soon.
>
> Thanks,
> Suman
> --
> *Suman*
> *OlaCabs*
>

[jira] [Resolved] (KAFKA-8090) Replace ControlledShutdown request/response with automated protocol

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-8090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-8090.
--
   Resolution: Fixed
Fix Version/s: 2.3.0

Issue resolved by pull request 6423
[https://github.com/apache/kafka/pull/6423]

> Replace ControlledShutdown request/response with automated protocol
> ---
>
> Key: KAFKA-8090
> URL: https://issues.apache.org/jira/browse/KAFKA-8090
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Mickael Maison
>Assignee: Mickael Maison
>Priority: Major
> Fix For: 2.3.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : kafka-trunk-jdk11 #419

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-8189) Streams should have an option to require names for stateful components

[John Roesler (JIRA)]

John Roesler created KAFKA-8189:
---

 Summary: Streams should have an option to require names for 
stateful components
 Key: KAFKA-8189
 URL: https://issues.apache.org/jira/browse/KAFKA-8189
 Project: Kafka
  Issue Type: Improvement
  Components: streams
Reporter: John Roesler


With the introduction of 
[https://cwiki.apache.org/confluence/display/KAFKA/KIP-372%3A+Naming+Repartition+Topics+for+Joins+and+Grouping]
 , in conjunction with naming state via Materialized and Suppressed, Streams 
users have the ability to alter their topologies and restart without losing 
track of operator state or repartition topics.

It would be a robust pattern to always name stateful components, but this 
pattern is vulnerable to simple coding and configuration mistakes. If 
developers lose vigilence even once and deploy a topology with *any* state not 
named, the only way to correct it is with an application reset.

Streams can support topology compatibility by offering a config option to 
require names on all stateful components. Then, if someone accidentally adds an 
anonymous stateful operator, Streams would throw an exception instead of 
generating a name, preserving the integrity of the application.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : kafka-trunk-jdk8 #3511

[Apache Jenkins Server]

See 

Re: [VOTE] KIP-369 Alternative Partitioner to Support "Always Round-Robin" Selection

[M. Manna]

Hello,

Trying to revive this thread again. Would anyone be interested in having
this KiP through


Thanks,

On Fri, 25 Jan 2019 at 16:44, M. Manna  wrote:

> Hello,
>
> I am trying to revive this thread. I only got 1 binding vote so far.
>
> Please feel free to revisit and comment here.
>
> Thanks,
>
> On Thu, 25 Oct 2018 at 00:15, M. Manna  wrote:
>
>> Hey IJ,
>>
>> Thanks for your interest in the KIP.
>>
>> My point was simply that the round-robin should happen even if the key is
>> not null. As for the importance of key in our case, we treat the key as
>> metadata. Each key is composed of certain info which are parsed by our
>> consumer thread. We will then determine whether it's an actionable message
>> (e.g. process it), or a loopback(ignore it). You could argue, "Why not
>> append this metadata with the record and parse it there?". But that means
>> the following:
>>
>> 1) I'm always passing null key to achieve this - I would like to pass
>> Null/Not-Null/Other key i.e. flexibility
>> 2) Suppose the message size is 99 KB and and max message bytes allowed is
>> 100K. Now prefixing metadata with message results into the actual message
>> being 101K. This will fail at producer level and cause a retry/log this in
>> our DB for future pickup.
>>
>> To avoid all these, we are simply proposing this new partitioner class.
>> but all Kafka new releases will still have DefaultPartitioner as default,
>> unless they change the prop file to use our new class.
>>
>> Regards,
>>
>> On Sun, 21 Oct 2018 at 04:05, Ismael Juma  wrote:
>>
>>> Thanks for the KIP. Can you please elaborate on the need for the key in
>>> this case? The KIP simply states that the key is needed for metadata, but
>>> doesn't give any more details.
>>>
>>> Ismael
>>>
>>> On Tue, Sep 4, 2018 at 3:39 AM M. Manna  wrote:
>>>
>>> > Hello,
>>> >
>>> > I have made necessary changes as per the original discussion thread,
>>> and
>>> > would like to put it for votes.
>>> >
>>> > Thank you very much for your suggestion and guidance so far.
>>> >
>>> > Regards,
>>> >
>>>
>>

Re: [DISCUSS] KIP-446: Add changelog topic configuration to KTable suppress

[Maarten Duijn]

Thank you for the explanation regarding the internals, I have edited the KIP 
accordingly and updated the Javadoc. About the possible data loss when altering 
changelog config, I think we can improve by doing (one of) the following.

1) Add a warning in the comments that clearly states what might happen when 
change logging is disabled and adjust it when persistent stores are added.

2) Change `withLoggingDisabled` to `minimizeLogging`. Instead of disabling 
logging, a call to this method minimizes the topic size by aggressively 
removing the records emitted downstream by the suppress operator. I believe 
this can be achieved by setting `delete.retention.ms=0` in the topic config.

3) Remove `withLoggingDisabled` from the proposal.

4) Leave both methods as-proposed, as you indicated, this is in line with the 
other parts of the Streams API

A user might want to disable logging when downstream is not a Kafka topic but 
some other service that does not benefit from atleast-once-delivery of the 
suppressed records in case of failover or rebalance.
Seeing as it might cause data loss, the methods should not be used lightly and 
I think some comments are warranted. Personally, I rely purely on Kafka to 
prevent data loss even when a store persisted locally, so when support is added 
for persistent suppression, I feel the comments may stay.

Maarten

Build failed in Jenkins: kafka-trunk-jdk11 #420

[Apache Jenkins Server]

See 


Changes:

[manikumar] KAFKA-8090: Use automatic RPC generation in ControlledShutdown

--
[...truncated 2.35 MB...]
org.apache.kafka.connect.converters.ByteArrayConverterTest > 
testFromConnectSchemaless PASSED

org.apache.kafka.connect.converters.ByteArrayConverterTest > testToConnectNull 
STARTED

org.apache.kafka.connect.converters.ByteArrayConverterTest > testToConnectNull 
PASSED

org.apache.kafka.connect.converters.ByteArrayConverterTest > 
testFromConnectBadSchema STARTED

org.apache.kafka.connect.converters.ByteArrayConverterTest > 
testFromConnectBadSchema PASSED

org.apache.kafka.connect.converters.ByteArrayConverterTest > 
testFromConnectNull STARTED

org.apache.kafka.connect.converters.ByteArrayConverterTest > 
testFromConnectNull PASSED

org.apache.kafka.connect.converters.ByteArrayConverterTest > testToConnect 
STARTED

org.apache.kafka.connect.converters.ByteArrayConverterTest > testToConnect 
PASSED

org.apache.kafka.connect.converters.ByteArrayConverterTest > testFromConnect 
STARTED

org.apache.kafka.connect.converters.ByteArrayConverterTest > testFromConnect 
PASSED

org.apache.kafka.connect.converters.ByteArrayConverterTest > 
testFromConnectInvalidValue STARTED

org.apache.kafka.connect.converters.ByteArrayConverterTest > 
testFromConnectInvalidValue PASSED

org.apache.kafka.connect.converters.ShortConverterTest > testBytesNullToNumber 
STARTED

org.apache.kafka.connect.converters.ShortConverterTest > testBytesNullToNumber 
PASSED

org.apache.kafka.connect.converters.ShortConverterTest > 
testSerializingIncorrectType STARTED

org.apache.kafka.connect.converters.ShortConverterTest > 
testSerializingIncorrectType PASSED

org.apache.kafka.connect.converters.ShortConverterTest > 
testDeserializingHeaderWithTooManyBytes STARTED

org.apache.kafka.connect.converters.ShortConverterTest > 
testDeserializingHeaderWithTooManyBytes PASSED

org.apache.kafka.connect.converters.ShortConverterTest > testNullToBytes STARTED

org.apache.kafka.connect.converters.ShortConverterTest > testNullToBytes PASSED

org.apache.kafka.connect.converters.ShortConverterTest > 
testSerializingIncorrectHeader STARTED

org.apache.kafka.connect.converters.ShortConverterTest > 
testSerializingIncorrectHeader PASSED

org.apache.kafka.connect.converters.ShortConverterTest > 
testDeserializingDataWithTooManyBytes STARTED

org.apache.kafka.connect.converters.ShortConverterTest > 
testDeserializingDataWithTooManyBytes PASSED

org.apache.kafka.connect.converters.ShortConverterTest > 
testConvertingSamplesToAndFromBytes STARTED

org.apache.kafka.connect.converters.ShortConverterTest > 
testConvertingSamplesToAndFromBytes PASSED

org.apache.kafka.connect.converters.IntegerConverterTest > 
testBytesNullToNumber STARTED

org.apache.kafka.connect.converters.IntegerConverterTest > 
testBytesNullToNumber PASSED

org.apache.kafka.connect.converters.IntegerConverterTest > 
testSerializingIncorrectType STARTED

org.apache.kafka.connect.converters.IntegerConverterTest > 
testSerializingIncorrectType PASSED

org.apache.kafka.connect.converters.IntegerConverterTest > 
testDeserializingHeaderWithTooManyBytes STARTED

org.apache.kafka.connect.converters.IntegerConverterTest > 
testDeserializingHeaderWithTooManyBytes PASSED

org.apache.kafka.connect.converters.IntegerConverterTest > testNullToBytes 
STARTED

org.apache.kafka.connect.converters.IntegerConverterTest > testNullToBytes 
PASSED

org.apache.kafka.connect.converters.IntegerConverterTest > 
testSerializingIncorrectHeader STARTED

org.apache.kafka.connect.converters.IntegerConverterTest > 
testSerializingIncorrectHeader PASSED

org.apache.kafka.connect.converters.IntegerConverterTest > 
testDeserializingDataWithTooManyBytes STARTED

org.apache.kafka.connect.converters.IntegerConverterTest > 
testDeserializingDataWithTooManyBytes PASSED

org.apache.kafka.connect.converters.IntegerConverterTest > 
testConvertingSamplesToAndFromBytes STARTED

org.apache.kafka.connect.converters.IntegerConverterTest > 
testConvertingSamplesToAndFromBytes PASSED

> Task :streams:streams-scala:test

org.apache.kafka.streams.scala.kstream.KStreamTest > filter a KStream should 
filter records satisfying the predicate STARTED

org.apache.kafka.streams.scala.kstream.KStreamTest > filter a KStream should 
filter records satisfying the predicate PASSED

org.apache.kafka.streams.scala.kstream.KStreamTest > filterNot a KStream should 
filter records not satisfying the predicate STARTED

org.apache.kafka.streams.scala.kstream.KStreamTest > filterNot a KStream should 
filter records not satisfying the predicate PASSED

org.apache.kafka.streams.scala.kstream.KStreamTest > foreach a KStream should 
run foreach actions on records STARTED

org.apache.kafka.streams.scala.kstream.KStreamTest > foreach a KStream should 
run foreach actions on records PASSED

org.apache.kafka.streams.scala.kstream.KS

Re: [VOTE] KIP-369 Alternative Partitioner to Support "Always Round-Robin" Selection

[Harsha]

Looks like the KIP is passed with 3 binding votes.  From Matthias, Bill Bejeck 
and myself you got 3 binding votes.
You can do the full tally of the votes and send out a close of vote thread.

Thanks,
Harsha

On Thu, Apr 4, 2019, at 12:24 PM, M. Manna wrote:
> Hello,
> 
> Trying to revive this thread again. Would anyone be interested in having
> this KiP through
> 
> 
> Thanks,
> 
> On Fri, 25 Jan 2019 at 16:44, M. Manna  wrote:
> 
> > Hello,
> >
> > I am trying to revive this thread. I only got 1 binding vote so far.
> >
> > Please feel free to revisit and comment here.
> >
> > Thanks,
> >
> > On Thu, 25 Oct 2018 at 00:15, M. Manna  wrote:
> >
> >> Hey IJ,
> >>
> >> Thanks for your interest in the KIP.
> >>
> >> My point was simply that the round-robin should happen even if the key is
> >> not null. As for the importance of key in our case, we treat the key as
> >> metadata. Each key is composed of certain info which are parsed by our
> >> consumer thread. We will then determine whether it's an actionable message
> >> (e.g. process it), or a loopback(ignore it). You could argue, "Why not
> >> append this metadata with the record and parse it there?". But that means
> >> the following:
> >>
> >> 1) I'm always passing null key to achieve this - I would like to pass
> >> Null/Not-Null/Other key i.e. flexibility
> >> 2) Suppose the message size is 99 KB and and max message bytes allowed is
> >> 100K. Now prefixing metadata with message results into the actual message
> >> being 101K. This will fail at producer level and cause a retry/log this in
> >> our DB for future pickup.
> >>
> >> To avoid all these, we are simply proposing this new partitioner class.
> >> but all Kafka new releases will still have DefaultPartitioner as default,
> >> unless they change the prop file to use our new class.
> >>
> >> Regards,
> >>
> >> On Sun, 21 Oct 2018 at 04:05, Ismael Juma  wrote:
> >>
> >>> Thanks for the KIP. Can you please elaborate on the need for the key in
> >>> this case? The KIP simply states that the key is needed for metadata, but
> >>> doesn't give any more details.
> >>>
> >>> Ismael
> >>>
> >>> On Tue, Sep 4, 2018 at 3:39 AM M. Manna  wrote:
> >>>
> >>> > Hello,
> >>> >
> >>> > I have made necessary changes as per the original discussion thread,
> >>> and
> >>> > would like to put it for votes.
> >>> >
> >>> > Thank you very much for your suggestion and guidance so far.
> >>> >
> >>> > Regards,
> >>> >
> >>>
> >>
>

Re: [VOTE] KIP-392: Allow consumers to fetch from the closest replica

[Jason Gustafson]

Hi Jun,

I have updated the KIP to remove `replica.selection.policy` from the
consumer configuration. Thanks for the suggestion.

Best,
Jason

On Wed, Mar 27, 2019 at 9:46 AM Jason Gustafson  wrote:

> @Jun
>
> Re; 200: It's a fair point that it is useful to minimize the client
> changes that are needed to get a benefit from affinity. I think the high
> level argument that this is mostly the concern of operators and should be
> under their control. Since there is a protocol bump here, users will have
> to upgrade clients at a minimum. An alternative would be to make
> "preferred" the default option for `replica.selection.policy`. But I agree
> that the value of the configuration becomes less clear in this case.
> Overall this suggestion sounds good to me, but let me see if there is any
> additional feedback before I update the KIP.
>
> Re; 201: Ack.
>
> @Guozhang
>
> I think rack.id is still an easier and more reliable way for many users
> to determine local affinity. This lets us provide the simple rack-aware
> implementation which is probably sufficient for a fair number of use cases
> and wouldn't require users to write any custom code.
>
> Thanks,
> Jason
>
>
> On Wed, Mar 27, 2019 at 9:05 AM Guozhang Wang  wrote:
>
>> Hello Jun,
>>
>> Regarding 200: if we assume that most client would not bother setting
>> rack.id at all and affinity can be determined w/o rack.id via TCP header,
>> plus rack.id may not be "future-proof" additional information is needed
>> as
>> well, then do we still need to change the protocol of metadata request to
>> add `rack.id`?
>>
>>
>> Guozhang
>>
>> On Tue, Mar 26, 2019 at 6:23 PM Jun Rao  wrote:
>>
>> > Hi, Jason,
>> >
>> > Thanks for the KIP. Just a couple of more comments.
>> >
>> > 200. I am wondering if we really need the replica.selection.policy
>> config
>> > in the consumer. A slight variant is that we (1) let the consumer always
>> > fetch from the PreferredReplica and (2) provide a default
>> implementation of
>> > ReplicaSelector that always returns the leader replica in select() for
>> > backward compatibility. Then, we can get rid of
>> replica.selection.policy in
>> > the consumer. The benefits are that (1) fewer configs, (2) affinity
>> > optimization can potentially be turned on with just a broker side change
>> > (assuming affinity can be determined w/o client rack.id).
>> >
>> > 201. I am wondering if PreferredReplica in the protocol should be named
>> > PreferredReadReplica since it's intended for reads?
>> >
>> > Jun
>> >
>> > On Mon, Mar 25, 2019 at 9:07 AM Jason Gustafson 
>> > wrote:
>> >
>> > > Hi All, discussion on the KIP seems to have died down, so I'd like to
>> go
>> > > ahead and start a vote. Here is a link to the KIP:
>> > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-392%3A+Allow+consumers+to+fetch+from+closest+replica
>> > > .
>> > >
>> > > +1 from me (duh)
>> > >
>> > > -Jason
>> > >
>> >
>>
>>
>> --
>> -- Guozhang
>>
>

Re: [VOTE] KIP-392: Allow consumers to fetch from the closest replica

[Gwen Shapira]

+1

On Thu, Apr 4, 2019 at 2:26 PM Jason Gustafson  wrote:

> Hi Jun,
>
> I have updated the KIP to remove `replica.selection.policy` from the
> consumer configuration. Thanks for the suggestion.
>
> Best,
> Jason
>
> On Wed, Mar 27, 2019 at 9:46 AM Jason Gustafson 
> wrote:
>
> > @Jun
> >
> > Re; 200: It's a fair point that it is useful to minimize the client
> > changes that are needed to get a benefit from affinity. I think the high
> > level argument that this is mostly the concern of operators and should be
> > under their control. Since there is a protocol bump here, users will have
> > to upgrade clients at a minimum. An alternative would be to make
> > "preferred" the default option for `replica.selection.policy`. But I
> agree
> > that the value of the configuration becomes less clear in this case.
> > Overall this suggestion sounds good to me, but let me see if there is any
> > additional feedback before I update the KIP.
> >
> > Re; 201: Ack.
> >
> > @Guozhang
> >
> > I think rack.id is still an easier and more reliable way for many users
> > to determine local affinity. This lets us provide the simple rack-aware
> > implementation which is probably sufficient for a fair number of use
> cases
> > and wouldn't require users to write any custom code.
> >
> > Thanks,
> > Jason
> >
> >
> > On Wed, Mar 27, 2019 at 9:05 AM Guozhang Wang 
> wrote:
> >
> >> Hello Jun,
> >>
> >> Regarding 200: if we assume that most client would not bother setting
> >> rack.id at all and affinity can be determined w/o rack.id via TCP
> header,
> >> plus rack.id may not be "future-proof" additional information is needed
> >> as
> >> well, then do we still need to change the protocol of metadata request
> to
> >> add `rack.id`?
> >>
> >>
> >> Guozhang
> >>
> >> On Tue, Mar 26, 2019 at 6:23 PM Jun Rao  wrote:
> >>
> >> > Hi, Jason,
> >> >
> >> > Thanks for the KIP. Just a couple of more comments.
> >> >
> >> > 200. I am wondering if we really need the replica.selection.policy
> >> config
> >> > in the consumer. A slight variant is that we (1) let the consumer
> always
> >> > fetch from the PreferredReplica and (2) provide a default
> >> implementation of
> >> > ReplicaSelector that always returns the leader replica in select() for
> >> > backward compatibility. Then, we can get rid of
> >> replica.selection.policy in
> >> > the consumer. The benefits are that (1) fewer configs, (2) affinity
> >> > optimization can potentially be turned on with just a broker side
> change
> >> > (assuming affinity can be determined w/o client rack.id).
> >> >
> >> > 201. I am wondering if PreferredReplica in the protocol should be
> named
> >> > PreferredReadReplica since it's intended for reads?
> >> >
> >> > Jun
> >> >
> >> > On Mon, Mar 25, 2019 at 9:07 AM Jason Gustafson 
> >> > wrote:
> >> >
> >> > > Hi All, discussion on the KIP seems to have died down, so I'd like
> to
> >> go
> >> > > ahead and start a vote. Here is a link to the KIP:
> >> > >
> >> > >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-392%3A+Allow+consumers+to+fetch+from+closest+replica
> >> > > .
> >> > >
> >> > > +1 from me (duh)
> >> > >
> >> > > -Jason
> >> > >
> >> >
> >>
> >>
> >> --
> >> -- Guozhang
> >>
> >
>


-- 
*Gwen Shapira*
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter  | blog

Build failed in Jenkins: kafka-trunk-jdk8 #3512

[Apache Jenkins Server]

See 


Changes:

[manikumar] KAFKA-8090: Use automatic RPC generation in ControlledShutdown

--
[...truncated 2.36 MB...]

org.apache.kafka.connect.data.DateTest > testToLogicalInvalidSchema PASSED

org.apache.kafka.connect.util.ConnectorUtilsTest > testGroupPartitions STARTED

org.apache.kafka.connect.util.ConnectorUtilsTest > testGroupPartitions PASSED

org.apache.kafka.connect.util.ConnectorUtilsTest > 
testGroupPartitionsInvalidCount STARTED

org.apache.kafka.connect.util.ConnectorUtilsTest > 
testGroupPartitionsInvalidCount PASSED

org.apache.kafka.connect.header.ConnectHeaderTest > shouldGetSchemaFromStruct 
STARTED

org.apache.kafka.connect.header.ConnectHeaderTest > shouldGetSchemaFromStruct 
PASSED

org.apache.kafka.connect.header.ConnectHeaderTest > shouldAllowNonNullValue 
STARTED

org.apache.kafka.connect.header.ConnectHeaderTest > shouldAllowNonNullValue 
PASSED

org.apache.kafka.connect.header.ConnectHeaderTest > shouldSatisfyEquals STARTED

org.apache.kafka.connect.header.ConnectHeaderTest > shouldSatisfyEquals PASSED

org.apache.kafka.connect.header.ConnectHeaderTest > shouldAllowNullSchema 
STARTED

org.apache.kafka.connect.header.ConnectHeaderTest > shouldAllowNullSchema PASSED

org.apache.kafka.connect.header.ConnectHeaderTest > shouldAllowNullValues 
STARTED

org.apache.kafka.connect.header.ConnectHeaderTest > shouldAllowNullValues PASSED

org.apache.kafka.connect.header.ConnectHeadersTest > shouldNotAllowNullKey 
STARTED

org.apache.kafka.connect.header.ConnectHeadersTest > shouldNotAllowNullKey 
PASSED

org.apache.kafka.connect.header.ConnectHeadersTest > shouldValidateLogicalTypes 
STARTED

org.apache.kafka.connect.header.ConnectHeadersTest > shouldValidateLogicalTypes 
PASSED

org.apache.kafka.connect.header.ConnectHeadersTest > shouldHaveToString STARTED

org.apache.kafka.connect.header.ConnectHeadersTest > shouldHaveToString PASSED

org.apache.kafka.connect.header.ConnectHeadersTest > 
shouldTransformHeadersWithKey STARTED

org.apache.kafka.connect.header.ConnectHeadersTest > 
shouldTransformHeadersWithKey PASSED

org.apache.kafka.connect.header.ConnectHeadersTest > shouldAddDate STARTED

org.apache.kafka.connect.header.ConnectHeadersTest > shouldAddDate PASSED

org.apache.kafka.connect.header.ConnectHeadersTest > shouldAddTime STARTED

org.apache.kafka.connect.header.ConnectHeadersTest > shouldAddTime PASSED

org.apache.kafka.connect.header.ConnectHeadersTest > 
shouldNotAddHeadersWithObjectValuesAndMismatchedSchema STARTED

org.apache.kafka.connect.header.ConnectHeadersTest > 
shouldNotAddHeadersWithObjectValuesAndMismatchedSchema PASSED

org.apache.kafka.connect.header.ConnectHeadersTest > 
shouldNotValidateMismatchedValuesWithBuiltInTypes STARTED

org.apache.kafka.connect.header.ConnectHeadersTest > 
shouldNotValidateMismatchedValuesWithBuiltInTypes PASSED

org.apache.kafka.connect.header.ConnectHeadersTest > 
shouldRemoveAllHeadersWithSameKey STARTED

org.apache.kafka.connect.header.ConnectHeadersTest > 
shouldRemoveAllHeadersWithSameKey PASSED

org.apache.kafka.connect.header.ConnectHeadersTest > 
shouldTransformAndRemoveHeaders STARTED

org.apache.kafka.connect.header.ConnectHeadersTest > 
shouldTransformAndRemoveHeaders PASSED

org.apache.kafka.connect.header.ConnectHeadersTest > shouldBeEquals STARTED

org.apache.kafka.connect.header.ConnectHeadersTest > shouldBeEquals PASSED

org.apache.kafka.connect.header.ConnectHeadersTest > shouldValidateBuildInTypes 
STARTED

org.apache.kafka.connect.header.ConnectHeadersTest > shouldValidateBuildInTypes 
PASSED

org.apache.kafka.connect.header.ConnectHeadersTest > shouldRemoveAllHeaders 
STARTED

org.apache.kafka.connect.header.ConnectHeadersTest > shouldRemoveAllHeaders 
PASSED

org.apache.kafka.connect.header.ConnectHeadersTest > 
shouldNotValidateNullValuesWithBuiltInTypes STARTED

org.apache.kafka.connect.header.ConnectHeadersTest > 
shouldNotValidateNullValuesWithBuiltInTypes PASSED

org.apache.kafka.connect.header.ConnectHeadersTest > shouldTransformHeaders 
STARTED

org.apache.kafka.connect.header.ConnectHeadersTest > shouldTransformHeaders 
PASSED

org.apache.kafka.connect.header.ConnectHeadersTest > 
shouldAddMultipleHeadersWithSameKeyAndRetainLatest STARTED

org.apache.kafka.connect.header.ConnectHeadersTest > 
shouldAddMultipleHeadersWithSameKeyAndRetainLatest PASSED

org.apache.kafka.connect.header.ConnectHeadersTest > 
shouldAddHeadersWithPrimitiveValues STARTED

org.apache.kafka.connect.header.ConnectHeadersTest > 
shouldAddHeadersWithPrimitiveValues PASSED

org.apache.kafka.connect.header.ConnectHeadersTest > 
shouldNotAddHeadersWithNullObjectValuesWithNonOptionalSchema STARTED

org.apache.kafka.connect.header.ConnectHeadersTest > 
shouldNotAddHeadersWithNullObjectValuesWithNonOptionalSchema PASSED

org.apache.kafka.connect.header.ConnectHeadersTest > 
shouldDuplicateAndAlwaysReturnEquivalentButD

[jira] [Created] (KAFKA-8190) Keystore update without file change doesn't update SSLContext

[Rajini Sivaram (JIRA)]

Rajini Sivaram created KAFKA-8190:
-

 Summary: Keystore update without file change doesn't update 
SSLContext
 Key: KAFKA-8190
 URL: https://issues.apache.org/jira/browse/KAFKA-8190
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 2.1.1, 2.2.0, 2.1.0
Reporter: Rajini Sivaram
Assignee: Rajini Sivaram
 Fix For: 2.1.2, 2.2.1


We reload SSL keystores and truststores on a dynamic config update on the 
broker if one of the configs has changed or if the file modification time has 
changed. For update without config change, we compare the modification time at 
the last load with the current modification time of the file. But we load the 
file for validation of dynamic configs and when reconfiguring we compare the 
time updated during validation with file modification time. As a result 
SSLContext is not updated.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-8191) Add pluggability of KeyManager to generate the broker Private Keys and Certificates

[Sai Sandeep (JIRA)]

Sai Sandeep created KAFKA-8191:
--

 Summary: Add pluggability of KeyManager to generate the broker 
Private Keys and Certificates
 Key: KAFKA-8191
 URL: https://issues.apache.org/jira/browse/KAFKA-8191
 Project: Kafka
  Issue Type: Bug
  Components: security
Affects Versions: 1.1.1, 1.1.0
Reporter: Sai Sandeep
 Fix For: 1.1.1, 1.1.0


 

*Context:* Currently, in SslFactory.java, if the keystore is created null 
(caused by passing an empty config value to ssl.keystore.location), the default 
Sun KeyManager is used ignoring the 'ssl.keymanager.algorithm' provided.

We need changes to fetch KeyManager from the KeyManagerFactory based on the 
provided keymanager algorithm, populated by 'ssl.keymanager.algorithm' if the 
keystore is found empty

 

*Background and Use Case:* Kafka allows users to configure truststore and 
keystore to enable TLS connections from clients to brokers. Often this means 
during deployment, one needs to pre-provision keystores to enable clients to 
communicate with brokers on TLS port. Most of the time users end up configuring 
a long-lived certificate which is not good for security. Although KAFKA-4701 
introduced the reload of keystores it still a cumbersome to distribute these 
files onto compute system for clients. 
There are several projects that allows one to distribute the certificates 
through a local agent, example [Spiffe|[https://spiffe.io/]]. To take advantage 
of such systems we need changes to consider 'ssl.keymanager.algorithm' for 
KeyManagerFactory creation

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] KIP-360: Improve handling of unknown producer

[Jason Gustafson]

Hi Everyone,

Sorry for the long delay on this KIP. I have updated it to include the
handling of INVALID_PRODUCER_ID_MAPPING as suggested above. If there are no
further comments, I will plan to start a vote early next week.

Thanks!
Jason

On Mon, Mar 25, 2019 at 2:33 PM Adam Bellemare 
wrote:

> Ach - Sorry. I meant Jason. I had just read a John Roesler email.
>
> On Mon, Mar 25, 2019 at 5:21 PM Adam Bellemare 
> wrote:
>
> > Hi John
> >
> > What is the status of this KIP?
> >
> > My teammates and I are running into the "UNKNOWN_PRODUCER_ID" error on
> > 2.1.1 for a multitude of our internal topics, and I suspect that a proper
> > fix is needed.
> >
> > Adam
> >
> > On Mon, Jan 7, 2019 at 7:42 PM Guozhang Wang  wrote:
> >
> >> Thanks Jason. The proposed solution sounds good to me.
> >>
> >>
> >> Guozhang
> >>
> >> On Mon, Jan 7, 2019 at 3:52 PM Jason Gustafson 
> >> wrote:
> >>
> >> > Hey Guozhang,
> >> >
> >> > Thanks for sharing the article. The INVALID_PRODUCER_ID_MAPPING error
> >> > occurs following expiration of the producerId. It's possible that
> >> another
> >> > producerId has been installed in its place following expiration (if
> >> another
> >> > producer instance has become active), or the mapping is empty. We can
> >> > safely retry the InitProducerId with the logic in this KIP in order to
> >> > detect which case it is. So I'd suggest something like this:
> >> >
> >> > 1. After receiving INVALID_PRODUCER_ID_MAPPING, the producer can send
> >> > InitProducerId using the current producerId and epoch.
> >> > 2. If no mapping exists, the coordinator can generate a new producerId
> >> and
> >> > return it. If a transaction is in progress on the client, it will have
> >> to
> >> > be aborted, but the producer can continue afterwards.
> >> > 3. Otherwise if a different producerId has been assigned, then we can
> >> > return INVALID_PRODUCER_ID_MAPPING. To simplify error handling, we can
> >> > probably raise this as ProducerFencedException since that is
> effectively
> >> > what has happened. Ideally this is the only fatal case that users have
> >> to
> >> > handle.
> >> >
> >> > I'll give it a little more thought and update the KIP.
> >> >
> >> > Thanks,
> >> > Jason
> >> >
> >> > On Thu, Jan 3, 2019 at 1:38 PM Guozhang Wang 
> >> wrote:
> >> >
> >> > > You're right about the dangling txn since it will actually block
> >> > > read-committed consumers from proceeding at all. I'd agree that
> since
> >> > this
> >> > > is a very rare case, we can consider fixing it not via broker-side
> >> logic
> >> > > but via tooling in a future work.
> >> > >
> >> > > I've also discovered some related error handling logic inside
> producer
> >> > that
> >> > > may be addressed together with this KIP (since it is mostly for
> >> internal
> >> > > implementations the wiki itself does not need to be modified):
> >> > >
> >> > >
> >> > >
> >> >
> >>
> https://stackoverflow.com/questions/53976117/why-did-the-kafka-stream-fail-to-produce-data-after-a-long-time/54029181#54029181
> >> > >
> >> > > Guozhang
> >> > >
> >> > >
> >> > >
> >> > > On Thu, Nov 29, 2018 at 2:25 PM Jason Gustafson  >
> >> > > wrote:
> >> > >
> >> > > > Hey Guozhang,
> >> > > >
> >> > > > To clarify, the broker does not actually use the ApiVersion API
> for
> >> > > > inter-broker communications. The use of an API and its
> corresponding
> >> > > > version is controlled by `inter.broker.protocol.version`.
> >> > > >
> >> > > > Nevertheless, it sounds like we're on the same page about removing
> >> > > > DescribeTransactionState. The impact of a dangling transaction is
> a
> >> > > little
> >> > > > worse than what you describe though. Consumers with the
> >> read_committed
> >> > > > isolation level will be stuck. Still, I think we agree that this
> >> case
> >> > > > should be rare and we can reconsider for future work. Rather than
> >> > > > preventing dangling transactions, perhaps we should consider
> options
> >> > > which
> >> > > > allows us to detect them and recover. Anyway, this needs more
> >> thought.
> >> > I
> >> > > > will update the KIP.
> >> > > >
> >> > > > Best,
> >> > > > Jason
> >> > > >
> >> > > > On Tue, Nov 27, 2018 at 6:51 PM Guozhang Wang  >
> >> > > wrote:
> >> > > >
> >> > > > > 0. My original question is about the implementation details
> >> > primarily,
> >> > > > > since current the handling logic of the APIVersionResponse is
> >> simply
> >> > > "use
> >> > > > > the highest supported version of the corresponding request", but
> >> if
> >> > the
> >> > > > > returned response from APIVersionRequest says "I don't even know
> >> > about
> >> > > > the
> >> > > > > DescribeTransactionStateRequest at all", then we need additional
> >> > logic
> >> > > > for
> >> > > > > the falling back logic. Currently this logic is embedded in
> >> > > NetworkClient
> >> > > > > which is shared by all clients, so I'd like to avoid making this
> >> > logic
> >> > > > more
> >> > > > > complicated.
> >> > > > >
> >> > > > > As for the

Re: [VOTE] KIP-369 Alternative Partitioner to Support "Always Round-Robin" Selection

[M. Manna]

Thanks Harsha.

As per your comments, I have counted 3 binding votes so far.

Thanks everyone for your comments and support. I’ll update the kip next
morning and do  the needful.

Regards,

On Thu, 4 Apr 2019 at 22:10, Harsha  wrote:

> Looks like the KIP is passed with 3 binding votes.  From Matthias, Bill
> Bejeck and myself you got 3 binding votes.
> You can do the full tally of the votes and send out a close of vote thread.
>
> Thanks,
> Harsha
>
> On Thu, Apr 4, 2019, at 12:24 PM, M. Manna wrote:
> > Hello,
> >
> > Trying to revive this thread again. Would anyone be interested in having
> > this KiP through
> >
> >
> > Thanks,
> >
> > On Fri, 25 Jan 2019 at 16:44, M. Manna  wrote:
> >
> > > Hello,
> > >
> > > I am trying to revive this thread. I only got 1 binding vote so far.
> > >
> > > Please feel free to revisit and comment here.
> > >
> > > Thanks,
> > >
> > > On Thu, 25 Oct 2018 at 00:15, M. Manna  wrote:
> > >
> > >> Hey IJ,
> > >>
> > >> Thanks for your interest in the KIP.
> > >>
> > >> My point was simply that the round-robin should happen even if the
> key is
> > >> not null. As for the importance of key in our case, we treat the key
> as
> > >> metadata. Each key is composed of certain info which are parsed by our
> > >> consumer thread. We will then determine whether it's an actionable
> message
> > >> (e.g. process it), or a loopback(ignore it). You could argue, "Why not
> > >> append this metadata with the record and parse it there?". But that
> means
> > >> the following:
> > >>
> > >> 1) I'm always passing null key to achieve this - I would like to pass
> > >> Null/Not-Null/Other key i.e. flexibility
> > >> 2) Suppose the message size is 99 KB and and max message bytes
> allowed is
> > >> 100K. Now prefixing metadata with message results into the actual
> message
> > >> being 101K. This will fail at producer level and cause a retry/log
> this in
> > >> our DB for future pickup.
> > >>
> > >> To avoid all these, we are simply proposing this new partitioner
> class.
> > >> but all Kafka new releases will still have DefaultPartitioner as
> default,
> > >> unless they change the prop file to use our new class.
> > >>
> > >> Regards,
> > >>
> > >> On Sun, 21 Oct 2018 at 04:05, Ismael Juma  wrote:
> > >>
> > >>> Thanks for the KIP. Can you please elaborate on the need for the key
> in
> > >>> this case? The KIP simply states that the key is needed for
> metadata, but
> > >>> doesn't give any more details.
> > >>>
> > >>> Ismael
> > >>>
> > >>> On Tue, Sep 4, 2018 at 3:39 AM M. Manna  wrote:
> > >>>
> > >>> > Hello,
> > >>> >
> > >>> > I have made necessary changes as per the original discussion
> thread,
> > >>> and
> > >>> > would like to put it for votes.
> > >>> >
> > >>> > Thank you very much for your suggestion and guidance so far.
> > >>> >
> > >>> > Regards,
> > >>> >
> > >>>
> > >>
> >
>

Re: Add Jira permission and wiki permission

[Bill Bejeck]

My apologies for the delay, you are all set now on the wiki permissions.

Thanks,
Bill

On Tue, Apr 2, 2019 at 6:23 AM slim ouertani  wrote:

> Hi,
>
> The account details are as follows:
> Full Name Slim Ouertani
> Email ouert...@gmail.com
>
>
> Thanks,
> Slim
>
>
> On Mon, Apr 1, 2019 at 4:49 PM Bill Bejeck  wrote:
>
> > Hi,
> >
> > You're already in Jira as a contributor, but I can't seem to find you in
> > the Apache Confluence (https://cwiki.apache.org/confluence) can you
> > confirm
> > your account there?
> >
> > Thanks,
> > Bill
> >
> > On Mon, Apr 1, 2019 at 1:42 AM slim ouertani  wrote:
> >
> > > Hello,
> > >
> > > user id: ouertani
> > >
> > > Thanks in advance.
> > >
> >
>

Re: [DISCUSS] KIP-236 Interruptible Partition Reassignment

[Jun Rao]

Hi, George,

Thanks for the KIP. Sorry for the late reply. A couple of comments below.

40. I agree that it's better to issue an RPC request to the controller for
reassignment cancellation. If we do that, it would be useful to decide
whether that call blocks on cancellation completion or not.

41. Is it necessary to add the new "original replicas" field in
/admin/reassign_partitions?
The original replicas are already in the topic path in ZK.

Jun

On Tue, Mar 26, 2019 at 5:24 PM George Li 
wrote:

>  Hi Ismael,
>
> Thanks,  I understand your points. I will add the RPC mechanism for
> reassignments to KIP-236.  I can think of a few Requests/Responses
> corresponding to the old Scala AdminClient using ZK:
>
> SubmitReassignments (--execute)
> StatusReassignments (--verify)
> ChangeOrRemoveReassignmentsThrottle (--verify)
> GenerateReassignments (--generate)
> CancelPendingReassignments (new in KIP-236)
>
> To clarify the "structure" change of KIP-236, the ZK public interface
> remains the same:  {topic, partition, new_replicas},  the user client
> generate/submit the reassignment plan the same way as before.  the new
> "original_replicas" in the ZK node is added by admin client before writing
> to the ZK node.  A bit similar to the "log_dirs".  User's direct
> modification of ZK /admin/reassign_partitions is strongly discouraged.
>
> The original_replicas info is essential for cancellation/rollback of the
> reassignments still pending.
>
> Thanks,
> George
>
> On Monday, March 25, 2019, 4:43:30 PM PDT, Ismael Juma <
> ism...@juma.me.uk> wrote:
>
>  Hi George,
>
> The goal is not to prevent people from updating ZK directly. The goal is to
> offer a solution where people don't have to. If people then decide to avoid
> the recommended path, they can deal with the consequences. However, if we
> add another structure in ZK and no RPC mechanism, then there is no
> recommended path apart from updating ZK (implicitly making it an API for
> users).
>
> Ismael
>
> On Mon, Mar 25, 2019 at 3:57 PM George Li  .invalid>
> wrote:
>
> >  Thanks Ismael.  One question, even switch to submitting reassignments
> via
> > RPC instead of Zookeeper.  The reassignment data will still persist in
> > ZooKeeper node /admin/reassign_partitions (e.g. when Controller failover
> it
> > can resume reassignments)?  If yes, how this can keep someone from
> > modifying ZK (/admin/reassign_partitions) directly ?
> >
> >
> > Thanks,
> > George
> >
> >On Saturday, March 23, 2019, 1:07:11 PM PDT, Ismael Juma <
> > isma...@gmail.com> wrote:
> >
> >  Thanks for the KIP, making reassignment more flexible is definitely
> > welcome. As others have mentioned, I think we need to do it via the Kafka
> > protocol and not via ZK. The latter introduces an implicit API that other
> > tools will depend on causing migration challenges. This has already
> > happened with the existing ZK based interface and we should avoid
> > introducing more tech debt here.
> >
> > Ismael
> >
> > On Sat, Mar 23, 2019, 12:09 PM Colin McCabe  wrote:
> >
> > > On Thu, Mar 21, 2019, at 20:51, George Li wrote:
> > > >  Hi Colin,
> > > >
> > > > I agree with your proposal of having administrative APIs through RPC
> > > > instead of ZooKeeper. But seems like it will incur significant
> changes
> > > > to both submitting reassignments and this KIP's cancelling pending
> > > > reassignments.
> > > >
> > > > To make this KIP simple and moving along, I will be happy to do
> another
> > > > follow-up KIP to change all reassignment related operations via RP
> > >
> > > Thanks, George.  I think doing it as a two-step process is fine, but I
> > > suspect it would be much easier and quicker to do the RPC conversion
> > first,
> > > and the interruptible part later.  The reason is because a lot of the
> > > things that people have brought up as concerns with this KIP are really
> > > issues with the API (how will people interact with ZK, how does access
> > > control work, what does the format look like in ZK) that will just go
> > away
> > > once we have an RPC.
> > >
> > > > Just curious,  KIP-4 includes Topics/ACL related operations. In
> > > > addition to Reassignments,  any other operations should be done via
> > > > RPC?
> > >
> > > I think all of the administrative shell scripts have been converted
> > except
> > > kafka-configs.sh.  I believe there is a KIP for that conversion.
> > > Reassigning partitions is probably the biggest KIP-4 gap we have right
> > now.
> > >
> > > best,
> > > Colin
> > >
> > > >
> > > > Thanks,
> > > > George
> > > >
> > > >
> > > >On Wednesday, March 20, 2019, 5:28:59 PM PDT, Colin McCabe
> > > >  wrote:
> > > >
> > > >  Hi George,
> > > >
> > > > One big problem here is that administrative APIs should be done
> through
> > > > RPCs, not through ZooKeeper.  KIP-4 (Command line and centralized
> > > > administrative operations) describes the rationale for this.  We want
> > > > public and stable APIs that don't depend on the internal
> representati

Re: [DISCUSS] KIP-435: Incremental Partition Reassignment

[Jun Rao]

Hi, Viktor,

Thanks for the KIP. A couple of comments below.

1. Another potential thing to do reassignment incrementally is to move a
batch of partitions at a time, instead of all partitions. This may lead to
less data replication since by the time the first batch of partitions have
been completely moved, some data of the next batch may have been deleted
due to retention and doesn't need to be replicated.

2. "Update CR in Zookeeper with TR for the given partition". Which ZK path
is this for?

Jun

On Sat, Feb 23, 2019 at 2:12 AM Viktor Somogyi-Vass 
wrote:

> Hi Harsha,
>
> As far as I understand KIP-236 it's about enabling reassignment
> cancellation and as a future plan providing a queue of replica reassignment
> steps to allow manual reassignment chains. While I agree that the
> reassignment chain has a specific use case that allows fine grain control
> over reassignment process, My proposal on the other hand doesn't talk about
> cancellation but it only provides an automatic way to incrementalize an
> arbitrary reassignment which I think fits the general use case where users
> don't want that level of control but still would like a balanced way of
> reassignments. Therefore I think it's still relevant as an improvement of
> the current algorithm.
> Nevertheless I'm happy to add my ideas to KIP-236 as I think it would be a
> great improvement to Kafka.
>
> Cheers,
> Viktor
>
> On Fri, Feb 22, 2019 at 5:05 PM Harsha  wrote:
>
> > Hi Viktor,
> > There is already KIP-236 for the same feature and George made
> > a PR for this as well.
> > Lets consolidate these two discussions. If you have any cases that are
> not
> > being solved by KIP-236 can you please mention them in that thread. We
> can
> > address as part of KIP-236.
> >
> > Thanks,
> > Harsha
> >
> > On Fri, Feb 22, 2019, at 5:44 AM, Viktor Somogyi-Vass wrote:
> > > Hi Folks,
> > >
> > > I've created a KIP about an improvement of the reassignment algorithm
> we
> > > have. It aims to enable partition-wise incremental reassignment. The
> > > motivation for this is to avoid excess load that the current
> replication
> > > algorithm implicitly carries as in that case there are points in the
> > > algorithm where both the new and old replica set could be online and
> > > replicating which puts double (or almost double) pressure on the
> brokers
> > > which could cause problems.
> > > Instead my proposal would slice this up into several steps where each
> > step
> > > is calculated based on the final target replicas and the current
> replica
> > > assignment taking into account scenarios where brokers could be offline
> > and
> > > when there are not enough replicas to fulfil the min.insync.replica
> > > requirement.
> > >
> > > The link to the KIP:
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-435%3A+Incremental+Partition+Reassignment
> > >
> > > I'd be happy to receive any feedback.
> > >
> > > An important note is that this KIP and another one, KIP-236 that is
> > > about
> > > interruptible reassignment (
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-236%3A+Interruptible+Partition+Reassignment
> > )
> > > should be compatible.
> > >
> > > Thanks,
> > > Viktor
> > >
> >
>

Re: [VOTE] KIP-392: Allow consumers to fetch from the closest replica

[Jun Rao]

Hi, Jason,

Thanks for the updated KIP. +1 from me.

Jun

On Thu, Apr 4, 2019 at 2:26 PM Jason Gustafson  wrote:

> Hi Jun,
>
> I have updated the KIP to remove `replica.selection.policy` from the
> consumer configuration. Thanks for the suggestion.
>
> Best,
> Jason
>
> On Wed, Mar 27, 2019 at 9:46 AM Jason Gustafson 
> wrote:
>
> > @Jun
> >
> > Re; 200: It's a fair point that it is useful to minimize the client
> > changes that are needed to get a benefit from affinity. I think the high
> > level argument that this is mostly the concern of operators and should be
> > under their control. Since there is a protocol bump here, users will have
> > to upgrade clients at a minimum. An alternative would be to make
> > "preferred" the default option for `replica.selection.policy`. But I
> agree
> > that the value of the configuration becomes less clear in this case.
> > Overall this suggestion sounds good to me, but let me see if there is any
> > additional feedback before I update the KIP.
> >
> > Re; 201: Ack.
> >
> > @Guozhang
> >
> > I think rack.id is still an easier and more reliable way for many users
> > to determine local affinity. This lets us provide the simple rack-aware
> > implementation which is probably sufficient for a fair number of use
> cases
> > and wouldn't require users to write any custom code.
> >
> > Thanks,
> > Jason
> >
> >
> > On Wed, Mar 27, 2019 at 9:05 AM Guozhang Wang 
> wrote:
> >
> >> Hello Jun,
> >>
> >> Regarding 200: if we assume that most client would not bother setting
> >> rack.id at all and affinity can be determined w/o rack.id via TCP
> header,
> >> plus rack.id may not be "future-proof" additional information is needed
> >> as
> >> well, then do we still need to change the protocol of metadata request
> to
> >> add `rack.id`?
> >>
> >>
> >> Guozhang
> >>
> >> On Tue, Mar 26, 2019 at 6:23 PM Jun Rao  wrote:
> >>
> >> > Hi, Jason,
> >> >
> >> > Thanks for the KIP. Just a couple of more comments.
> >> >
> >> > 200. I am wondering if we really need the replica.selection.policy
> >> config
> >> > in the consumer. A slight variant is that we (1) let the consumer
> always
> >> > fetch from the PreferredReplica and (2) provide a default
> >> implementation of
> >> > ReplicaSelector that always returns the leader replica in select() for
> >> > backward compatibility. Then, we can get rid of
> >> replica.selection.policy in
> >> > the consumer. The benefits are that (1) fewer configs, (2) affinity
> >> > optimization can potentially be turned on with just a broker side
> change
> >> > (assuming affinity can be determined w/o client rack.id).
> >> >
> >> > 201. I am wondering if PreferredReplica in the protocol should be
> named
> >> > PreferredReadReplica since it's intended for reads?
> >> >
> >> > Jun
> >> >
> >> > On Mon, Mar 25, 2019 at 9:07 AM Jason Gustafson 
> >> > wrote:
> >> >
> >> > > Hi All, discussion on the KIP seems to have died down, so I'd like
> to
> >> go
> >> > > ahead and start a vote. Here is a link to the KIP:
> >> > >
> >> > >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-392%3A+Allow+consumers+to+fetch+from+closest+replica
> >> > > .
> >> > >
> >> > > +1 from me (duh)
> >> > >
> >> > > -Jason
> >> > >
> >> >
> >>
> >>
> >> --
> >> -- Guozhang
> >>
> >
>

Jenkins build is back to normal : kafka-trunk-jdk11 #421

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-8192) Using Failify for e2e testing of Kafka

[Armin Balalaie (JIRA)]

Armin Balalaie created KAFKA-8192:
-

 Summary: Using Failify for e2e testing of Kafka
 Key: KAFKA-8192
 URL: https://issues.apache.org/jira/browse/KAFKA-8192
 Project: Kafka
  Issue Type: Test
Reporter: Armin Balalaie


Hi,

 

I am the author of Failify, a test framework for end-to-end testing of 
distributed systems. Failify can be used to deterministically inject failures 
during a normal test case execution. Currently, node failure, network 
partition, network delay, network packet loss, and clock drift is supported. 
For a few supported languages (right now, Java and Scala), it is possible to 
enforce a specific order between nodes in order to reproduce a specific 
time-sensitive scenario and inject failures before or after a specific method 
is called when a specific stack trace is present. You can find more information 
in [https://failify.io|https://failify.io/].

 

My reasons for Failify being useful to Kafka are:
 * It is Docker-based and less messy and you can run the test cases in a single 
node and in parallel (there are plans for implementing the ability of deploying 
the same test case on a K8S or a Swarm cluster).
 * It is Docker-based so you can easily have test cases that run on different 
OSes. Also, you can define the services you depend on e.g. ZK as another node 
in your deployment definition.
 * The failure kinds supported are a superset of what is being supported now by 
Trogdor (in particular, Network delay and loss, clock drift and a more 
sophisticated network partitioning)
 * There will be more control over when a failure should be introduced in a 
test case.
 * You can write your test cases in Java or Scala or any other language that 
can be run on JVM and can use Java libraries.
 * It can be easily integrated into your build pipeline as you will be writing 
your regular JUnit test cases.
 * The API is compact and intuitive and there is a good documentation for the 
tool

 

Please let me know if you want to give it a try. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: kafka-trunk-jdk8 #3513

[Apache Jenkins Server]

See 


Changes:

[cshapi] MINOR: fix throttling and status in ConnectionStressWorker

--
[...truncated 2.35 MB...]
org.apache.kafka.connect.runtime.ConnectMetricsTest > testRecreateWithoutClose 
STARTED

org.apache.kafka.connect.runtime.ConnectMetricsTest > testRecreateWithoutClose 
PASSED

org.apache.kafka.connect.runtime.ConnectMetricsTest > testGettingGroupWithTags 
STARTED

org.apache.kafka.connect.runtime.ConnectMetricsTest > testGettingGroupWithTags 
PASSED

org.apache.kafka.connect.runtime.ConnectMetricsTest > testRecreateWithClose 
STARTED

org.apache.kafka.connect.runtime.ConnectMetricsTest > testRecreateWithClose 
PASSED

org.apache.kafka.connect.runtime.ConnectMetricsTest > testKafkaMetricsNotNull 
STARTED

org.apache.kafka.connect.runtime.ConnectMetricsTest > testKafkaMetricsNotNull 
PASSED

org.apache.kafka.connect.converters.ShortConverterTest > testBytesNullToNumber 
STARTED

org.apache.kafka.connect.converters.ShortConverterTest > testBytesNullToNumber 
PASSED

org.apache.kafka.connect.converters.ShortConverterTest > 
testSerializingIncorrectType STARTED

org.apache.kafka.connect.converters.ShortConverterTest > 
testSerializingIncorrectType PASSED

org.apache.kafka.connect.converters.ShortConverterTest > 
testDeserializingHeaderWithTooManyBytes STARTED

org.apache.kafka.connect.converters.ShortConverterTest > 
testDeserializingHeaderWithTooManyBytes PASSED

org.apache.kafka.connect.converters.ShortConverterTest > testNullToBytes STARTED

org.apache.kafka.connect.converters.ShortConverterTest > testNullToBytes PASSED

org.apache.kafka.connect.converters.ShortConverterTest > 
testSerializingIncorrectHeader STARTED

org.apache.kafka.connect.converters.ShortConverterTest > 
testSerializingIncorrectHeader PASSED

org.apache.kafka.connect.converters.ShortConverterTest > 
testDeserializingDataWithTooManyBytes STARTED

org.apache.kafka.connect.converters.ShortConverterTest > 
testDeserializingDataWithTooManyBytes PASSED

org.apache.kafka.connect.converters.ShortConverterTest > 
testConvertingSamplesToAndFromBytes STARTED

org.apache.kafka.connect.converters.ShortConverterTest > 
testConvertingSamplesToAndFromBytes PASSED

org.apache.kafka.connect.converters.FloatConverterTest > testBytesNullToNumber 
STARTED

org.apache.kafka.connect.converters.FloatConverterTest > testBytesNullToNumber 
PASSED

org.apache.kafka.connect.converters.FloatConverterTest > 
testSerializingIncorrectType STARTED

org.apache.kafka.connect.converters.FloatConverterTest > 
testSerializingIncorrectType PASSED

org.apache.kafka.connect.converters.FloatConverterTest > 
testDeserializingHeaderWithTooManyBytes STARTED

org.apache.kafka.connect.converters.FloatConverterTest > 
testDeserializingHeaderWithTooManyBytes PASSED

org.apache.kafka.connect.converters.FloatConverterTest > testNullToBytes STARTED

org.apache.kafka.connect.converters.FloatConverterTest > testNullToBytes PASSED

org.apache.kafka.connect.converters.FloatConverterTest > 
testSerializingIncorrectHeader STARTED

org.apache.kafka.connect.converters.FloatConverterTest > 
testSerializingIncorrectHeader PASSED

org.apache.kafka.connect.converters.FloatConverterTest > 
testDeserializingDataWithTooManyBytes STARTED

org.apache.kafka.connect.converters.FloatConverterTest > 
testDeserializingDataWithTooManyBytes PASSED

org.apache.kafka.connect.converters.FloatConverterTest > 
testConvertingSamplesToAndFromBytes STARTED

org.apache.kafka.connect.converters.FloatConverterTest > 
testConvertingSamplesToAndFromBytes PASSED

org.apache.kafka.connect.converters.ByteArrayConverterTest > 
testFromConnectSchemaless STARTED

org.apache.kafka.connect.converters.ByteArrayConverterTest > 
testFromConnectSchemaless PASSED

org.apache.kafka.connect.converters.ByteArrayConverterTest > testToConnectNull 
STARTED

org.apache.kafka.connect.converters.ByteArrayConverterTest > testToConnectNull 
PASSED

org.apache.kafka.connect.converters.ByteArrayConverterTest > 
testFromConnectBadSchema STARTED

org.apache.kafka.connect.converters.ByteArrayConverterTest > 
testFromConnectBadSchema PASSED

org.apache.kafka.connect.converters.ByteArrayConverterTest > 
testFromConnectNull STARTED

org.apache.kafka.connect.converters.ByteArrayConverterTest > 
testFromConnectNull PASSED

org.apache.kafka.connect.converters.ByteArrayConverterTest > testToConnect 
STARTED

org.apache.kafka.connect.converters.ByteArrayConverterTest > testToConnect 
PASSED

org.apache.kafka.connect.converters.ByteArrayConverterTest > testFromConnect 
STARTED

org.apache.kafka.connect.converters.ByteArrayConverterTest > testFromConnect 
PASSED

org.apache.kafka.connect.converters.ByteArrayConverterTest > 
testFromConnectInvalidValue STARTED

org.apache.kafka.connect.converters.ByteArrayConverterTest > 
testFromConnectInvalidValue PASSED

org.apache.kafka.connect.converters.LongConverterTest > testBytesNullToNum

[jira] [Created] (KAFKA-8193) Flaky Test MetricsIntegrationTest#testStreamMetricOfWindowStore

[Konstantine Karantasis (JIRA)]

Konstantine Karantasis created KAFKA-8193:
-

 Summary: Flaky Test 
MetricsIntegrationTest#testStreamMetricOfWindowStore
 Key: KAFKA-8193
 URL: https://issues.apache.org/jira/browse/KAFKA-8193
 Project: Kafka
  Issue Type: Bug
  Components: admin, unit tests
Affects Versions: 2.3.0
Reporter: Konstantine Karantasis
 Fix For: 2.3.0


[https://builds.apache.org/blue/organizations/jenkins/kafka-trunk-jdk8/detail/kafka-trunk-jdk8/3450/tests]
{quote}java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:87)
at org.junit.Assert.assertTrue(Assert.java:42)
at org.junit.Assert.assertTrue(Assert.java:53)
at 
kafka.admin.TopicCommandWithAdminClientTest.testDescribeUnderMinIsrPartitions(TopicCommandWithAdminClientTest.scala:560){quote}
STDOUT
{quote}[2019-03-09 00:13:23,129] ERROR [ReplicaFetcher replicaId=2, leaderId=0, 
fetcherId=0] Error for partition testAlterPartitionCount-yrX0KHVdgf-1 at offset 
0 (kafka.server.ReplicaFetcherThread:76)
org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server 
does not host this topic-partition.
[2019-03-09 00:13:23,130] ERROR [ReplicaFetcher replicaId=3, leaderId=5, 
fetcherId=0] Error for partition testAlterPartitionCount-yrX0KHVdgf-0 at offset 
0 (kafka.server.ReplicaFetcherThread:76)
org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server 
does not host this topic-partition.
[2019-03-09 00:13:33,184] ERROR [ReplicaFetcher replicaId=4, leaderId=1, 
fetcherId=0] Error for partition testAlterPartitionCount-yrX0KHVdgf-2 at offset 
0 (kafka.server.ReplicaFetcherThread:76)
org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server 
does not host this topic-partition.
[2019-03-09 00:13:36,319] WARN Unable to read additional data from client 
sessionid 0x10433c934dc0006, likely client has closed socket 
(org.apache.zookeeper.server.NIOServerCnxn:376)
[2019-03-09 00:13:46,311] WARN Unable to read additional data from client 
sessionid 0x10433c98cb10003, likely client has closed socket 
(org.apache.zookeeper.server.NIOServerCnxn:376)
[2019-03-09 00:13:46,356] WARN Unable to read additional data from client 
sessionid 0x10433c98cb10004, likely client has closed socket 
(org.apache.zookeeper.server.NIOServerCnxn:376)
[2019-03-09 00:14:06,066] WARN Unable to read additional data from client 
sessionid 0x10433c9b17d0005, likely client has closed socket 
(org.apache.zookeeper.server.NIOServerCnxn:376)
[2019-03-09 00:14:06,457] WARN Unable to read additional data from client 
sessionid 0x10433c9b17d0001, likely client has closed socket 
(org.apache.zookeeper.server.NIOServerCnxn:376)
[2019-03-09 00:14:11,206] ERROR [ReplicaFetcher replicaId=2, leaderId=3, 
fetcherId=0] Error for partition kafka.testTopic1-1 at offset 0 
(kafka.server.ReplicaFetcherThread:76)
org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server 
does not host this topic-partition.
[2019-03-09 00:14:11,218] ERROR [ReplicaFetcher replicaId=4, leaderId=1, 
fetcherId=0] Error for partition __consumer_offsets-1 at offset 0 
(kafka.server.ReplicaFetcherThread:76)
org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server 
does not host this topic-partition.
[2019-03-09 00:14:22,096] WARN Unable to read additional data from client 
sessionid 0x10433c9f1210004, likely client has closed socket 
(org.apache.zookeeper.server.NIOServerCnxn:376)
[2019-03-09 00:14:28,290] WARN Unable to read additional data from client 
sessionid 0x10433ca28de0005, likely client has closed socket 
(org.apache.zookeeper.server.NIOServerCnxn:376)
[2019-03-09 00:14:28,733] WARN Unable to read additional data from client 
sessionid 0x10433ca28de0006, likely client has closed socket 
(org.apache.zookeeper.server.NIOServerCnxn:376)
[2019-03-09 00:14:29,529] WARN Unable to read additional data from client 
sessionid 0x10433ca28de, likely client has closed socket 
(org.apache.zookeeper.server.NIOServerCnxn:376)
[2019-03-09 00:14:31,841] WARN Unable to read additional data from client 
sessionid 0x10433ca39ed0002, likely client has closed socket 
(org.apache.zookeeper.server.NIOServerCnxn:376)
[2019-03-09 00:14:40,221] ERROR [ReplicaFetcher replicaId=0, leaderId=1, 
fetcherId=0] Error for partition 
testCreateAlterTopicWithRackAware-IQe98agDrW-16 at offset 0 
(kafka.server.ReplicaFetcherThread:76)
org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server 
does not host this topic-partition.
[2019-03-09 00:14:40,222] ERROR [ReplicaFetcher replicaId=0, leaderId=1, 
fetcherId=0] Error for partition 
testCreateAlterTopicWithRackAware-IQe98agDrW-10 at offset 0 
(kafka.server.ReplicaFetcherThread:76)
org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server 
does not host this topic-partition.
[2019-03-09 00:14:40,227] ERROR [ReplicaFetcher replicaId=0, leaderId=2, 
fetcherId=0] Erro

Re: [DISCUSS] KIP-252: Extend ACLs to allow filtering based on ip ranges and subnets

[Colin McCabe]

On Thu, Apr 4, 2019, at 01:52, Sönke Liebau wrote:
> Hi Colin,
> 
> I agree, we need to have a way of failing incorrect ranges server-side,
> I'll amend the KIP and look into that. I think INVALID_REQUEST should fit
> fine, afaik we can send a message along with that code, so that could
> explain the actual reason.

Hi Sonke,

Sounds good.

> 
> Regarding prohibiting these ACLs from being created before the inter-broker
> protocol is updated, I am a bit hesitant about this for two reasons.
> 
> 1. I can't help but feel that we are mixing things in with each other here
> that don't really belong together. The broker protocol and ACL versions and
> potential ACL versioning are to my mind only loosely linked, because broker
> and authorizer are usually updated at the same time. But if we look at
> Sentry, Ranger or the Confluent LDAP authorizer I am not sure that this
> will hold true forever.

I strongly doubt that anyone actually wants to update the authorizer plugins on 
a separate schedule than the brokers. I worked on Hadoop for a while, so I have 
some perspective on this. Even in an upgrade scenario, you have to bounce the 
broker process at some point, and at the point, it's not clear that you gain 
anything except a headache by bringing it up again with the old authorizer code 
and the new broker code.

> Also, this doesn't address the issue of potential future ACL changes that
> might actually be incompatible with existing ACLs. If we go down this road
> I think we should go all the way and come up with a full ACL
> versioning-scheme. I've written down a few thoughts/suggestions/questions
> on that in this thread.

I think an ACL versioning scheme is a good idea, but mainly just so that we can 
provide clean error messages when someone with old software tries to access 
ACLs in the new format. Right now, we have no way to evolve the format of the 
data in ZK at all, which is why we used the hack of creating a new path under 
ZK to implement prefix ACLs.

I think it makes sense to "gate" using the new ACL versions on having a 
specific value of the inter-broker-protocol configured. The thing to keep in 
mind is that the IBP should not be bumped until all brokers have been upgraded. 
Once all brokers have been upgraded, using the new ACL version should be OK.

> 2. I agree that not preventing this creates a security risk, however in a
> very limited scenario.
> The same scenario applied when we introduced prefixed-ACLs in version 2 and
> back then it was not addressed as far as I can tell. The only reference I
> can find is one sentence in the KIP that states the risk.
> For people using the Confluent LDAP authorizer (or any other authorizer
> that is based on the SimpleACLAuthorizer) there is not much we can do, as
> once the cluster is updated these ACLs could be created, but the
> Authorizers would only honor them after they themselves are upgraded (which
> will probably be a rolling restart as well, hence have the same issue).

That is a fair point-- in both scenarios, DENY ACLs could cause problems.

However, it seems much more likely that people will set up DENY ACLs on a range 
of IPs than on a topic prefix. Prefix-based security is mainly used to grant 
permissions, not to revoke them. For example, you might grant someone 
permission to create topics under "foo_" so that their KSQL jobs can run 
smoothly, etc. On the other hand, it is natural to think in terms of excluding 
certain IPs -- iptables is often used this way.

Prefix ACLs also exist under a separate ZK path, which at least avoids brokers 
crashing or getting exceptions if someone creates a prefix ACL and the cluster 
is partially upgraded.

Finally, there was a lot of discussion during the prefix ACLs KIP about the 
need to fix the versioning issue.

> 
> I am really not trying to duck the work here, but just addressing this
> specific change feels like it misses the larger issue to me.

I think you make some reasonable points. But we really should not keep kicking 
the can down the road.

Let's just create a JSON structure for ACLs in ZK, similar to the other JSON 
structures we have there. Put it under a new ZK root if we must. Call what 
exists today version 0, and make it so that you can't use version 1 (the first 
JSON version) until upgrading the IBP. What do you think?

best,
Colin

> 
> Best regards,
> Sönke
> 
> On Wed, Apr 3, 2019 at 4:36 PM Colin McCabe  wrote:
> 
> > Hi Sönke,
> >
> > Maybe a reasonable design here would be to not allow creating ACLs based
> > on ip ranges and subnets unless the inter-broker protocol setting has been
> > upgraded. If an upgrade is done correctly, the IBP should not be upgraded
> > until all the brokers have been upgraded, so there shouldn't be older
> > brokers in the cluster erroneously giving access to things they shouldn't.
> > In that case, perhaps we can hold off on introducing an ACL versioning
> > scheme for now.
> >
> > Another thing that is important here is having some way of