Re: [ANNOUNCE] New Kafka PMC member: Dong Lin

[Dong Lin]

Thank you everyone!!

It has been my great pleasure to contribute to and be part of the Apache
Kafka community!

On Mon, Aug 20, 2018 at 3:50 PM, Mayuresh Gharat  wrote:

> Congrats Dong !!!
>
> Thanks,
>
> Mayuresh
>
> On Mon, Aug 20, 2018 at 1:36 PM Gwen Shapira  wrote:
>
> > Congrats Dong Lin! Well deserved!
> >
> > On Mon, Aug 20, 2018, 3:55 AM Ismael Juma  wrote:
> >
> > > Hi everyone,
> > >
> > > Dong Lin became a committer in March 2018. Since then, he has remained
> > > active in the community and contributed a number of patches, reviewed
> > > several pull requests and participated in numerous KIP discussions. I
> am
> > > happy to announce that Dong is now a member of the
> > > Apache Kafka PMC.
> > >
> > > Congratulation Dong! Looking forward to your future contributions.
> > >
> > > Ismael, on behalf of the Apache Kafka PMC
> > >
> >
>
>
> --
> -Regards,
> Mayuresh R. Gharat
> (862) 250-7125
>

Re: [ANNOUNCE] New Kafka PMC member: Dong Lin

[Jan Filipiak]

Congrats Dong!




On 20.08.2018 16:35, Attila Sasvári wrote:

Congratulations Dong! Well deserved.

Regards,
Attila
Paolo Patierno  ezt írta (időpont: 2018. aug. 20., H
15:09):


Congratulations Dong!

Paolo Patierno
Principal Software Engineer (IoT) @ Red Hat
Microsoft MVP on Azure & IoT
Microsoft Azure Advisor

Twitter : @ppatierno
Linkedin : paolopatierno
Blog : DevExperience


From: Dongjin Lee 
Sent: Monday, August 20, 2018 1:00 PM
To: dev@kafka.apache.org
Subject: Re: [ANNOUNCE] New Kafka PMC member: Dong Lin

Congratulations!!

On Mon, Aug 20, 2018 at 9:22 PM Mickael Maison 
wrote:


Congratulations Dong!
On Mon, Aug 20, 2018 at 12:46 PM Manikumar 
wrote:

Congrats,  Dong Lin!

On Mon, Aug 20, 2018 at 4:25 PM Ismael Juma  wrote:


Hi everyone,

Dong Lin became a committer in March 2018. Since then, he has

remained

active in the community and contributed a number of patches, reviewed
several pull requests and participated in numerous KIP discussions. I

am

happy to announce that Dong is now a member of the
Apache Kafka PMC.

Congratulation Dong! Looking forward to your future contributions.

Ismael, on behalf of the Apache Kafka PMC


--
*Dongjin Lee*

*A hitchhiker in the mathematical world.*

*github:  github.com/dongjinleekr
linkedin:

kr.linkedin.com/in/dongjinleekr

slideshare:

www.slideshare.net/dongjinleekr

*

[jira] [Resolved] (KAFKA-6188) Broker fails with FATAL Shutdown - log dirs have failed

[Dong Lin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dong Lin resolved KAFKA-6188.
-
Resolution: Fixed

Likely fixed in https://issues.apache.org/jira/browse/KAFKA-7278.

> Broker fails with FATAL Shutdown - log dirs have failed
> ---
>
> Key: KAFKA-6188
> URL: https://issues.apache.org/jira/browse/KAFKA-6188
> Project: Kafka
>  Issue Type: Bug
>  Components: clients, log
>Affects Versions: 1.0.0, 1.0.1
> Environment: Windows 10
>Reporter: Valentina Baljak
>Assignee: Dong Lin
>Priority: Blocker
>  Labels: windows
> Attachments: Segments are opened before deletion, 
> kafka_2.10-0.10.2.1.zip, output.txt
>
>
> Just started with version 1.0.0 after a 4-5 months of using 0.10.2.1. The 
> test environment is very simple, with only one producer and one consumer. 
> Initially, everything started fine, stand alone tests worked as expected. 
> However, running my code, Kafka clients fail after approximately 10 minutes. 
> Kafka won't start after that and it fails with the same error. 
> Deleting logs helps to start again, and the same problem occurs.
> Here is the error traceback:
> [2017-11-08 08:21:57,532] INFO Starting log cleanup with a period of 30 
> ms. (kafka.log.LogManager)
> [2017-11-08 08:21:57,548] INFO Starting log flusher with a default period of 
> 9223372036854775807 ms. (kafka.log.LogManager)
> [2017-11-08 08:21:57,798] INFO Awaiting socket connections on 0.0.0.0:9092. 
> (kafka.network.Acceptor)
> [2017-11-08 08:21:57,813] INFO [SocketServer brokerId=0] Started 1 acceptor 
> threads (kafka.network.SocketServer)
> [2017-11-08 08:21:57,829] INFO [ExpirationReaper-0-Produce]: Starting 
> (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
> [2017-11-08 08:21:57,845] INFO [ExpirationReaper-0-DeleteRecords]: Starting 
> (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
> [2017-11-08 08:21:57,845] INFO [ExpirationReaper-0-Fetch]: Starting 
> (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
> [2017-11-08 08:21:57,845] INFO [LogDirFailureHandler]: Starting 
> (kafka.server.ReplicaManager$LogDirFailureHandler)
> [2017-11-08 08:21:57,860] INFO [ReplicaManager broker=0] Stopping serving 
> replicas in dir C:\Kafka\kafka_2.12-1.0.0\kafka-logs 
> (kafka.server.ReplicaManager)
> [2017-11-08 08:21:57,860] INFO [ReplicaManager broker=0] Partitions  are 
> offline due to failure on log directory C:\Kafka\kafka_2.12-1.0.0\kafka-logs 
> (kafka.server.ReplicaManager)
> [2017-11-08 08:21:57,860] INFO [ReplicaFetcherManager on broker 0] Removed 
> fetcher for partitions  (kafka.server.ReplicaFetcherManager)
> [2017-11-08 08:21:57,892] INFO [ReplicaManager broker=0] Broker 0 stopped 
> fetcher for partitions  because they are in the failed log dir 
> C:\Kafka\kafka_2.12-1.0.0\kafka-logs (kafka.server.ReplicaManager)
> [2017-11-08 08:21:57,892] INFO Stopping serving logs in dir 
> C:\Kafka\kafka_2.12-1.0.0\kafka-logs (kafka.log.LogManager)
> [2017-11-08 08:21:57,892] FATAL Shutdown broker because all log dirs in 
> C:\Kafka\kafka_2.12-1.0.0\kafka-logs have failed (kafka.log.LogManager)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [ANNOUNCE] New Kafka PMC member: Dong Lin

[James Cheng]

Congrats Dong!

-James

> On Aug 20, 2018, at 3:54 AM, Ismael Juma  wrote:
> 
> Hi everyone,
> 
> Dong Lin became a committer in March 2018. Since then, he has remained
> active in the community and contributed a number of patches, reviewed
> several pull requests and participated in numerous KIP discussions. I am
> happy to announce that Dong is now a member of the
> Apache Kafka PMC.
> 
> Congratulation Dong! Looking forward to your future contributions.
> 
> Ismael, on behalf of the Apache Kafka PMC

Re: [ANNOUNCE] New Kafka PMC member: Dong Lin

[Viktor Somogyi-Vass]

Congrats Dong! :)

On Tue, Aug 21, 2018 at 10:09 AM James Cheng  wrote:

> Congrats Dong!
>
> -James
>
> > On Aug 20, 2018, at 3:54 AM, Ismael Juma  wrote:
> >
> > Hi everyone,
> >
> > Dong Lin became a committer in March 2018. Since then, he has remained
> > active in the community and contributed a number of patches, reviewed
> > several pull requests and participated in numerous KIP discussions. I am
> > happy to announce that Dong is now a member of the
> > Apache Kafka PMC.
> >
> > Congratulation Dong! Looking forward to your future contributions.
> >
> > Ismael, on behalf of the Apache Kafka PMC
>
>

Re: [ANNOUNCE] New Kafka PMC member: Dong Lin

[Patrick Williams]

How do I unsubscribe from this list?

 
 
Best,
 
Patrick Williams
 
Enterprise New Business Manager
+44 (0)7549 676279
patrick.willi...@storageos.com
 
20 Midtown
20 Proctor Street
Holborn
London WC1V 6NX
 
Twitter: @patch37
LinkedIn: linkedin.com/in/patrickwilliams4 

https://slack.storageos.com/
 
 

On 21/08/2018, 09:59, "Viktor Somogyi-Vass"  wrote:

Apache Kafka PMC

Build failed in Jenkins: kafka-1.1-jdk7 #185

[Apache Jenkins Server]

See 


Changes:

[lindong28] Cherry-pick KAFKA-7278; replaceSegments() should not call

--
[...truncated 1.93 MB...]
org.apache.kafka.streams.KafkaStreamsTest > shouldReturnThreadMetadata STARTED

org.apache.kafka.streams.KafkaStreamsTest > shouldReturnThreadMetadata PASSED

org.apache.kafka.streams.KafkaStreamsTest > testCloseIsIdempotent STARTED

org.apache.kafka.streams.KafkaStreamsTest > testCloseIsIdempotent PASSED

org.apache.kafka.streams.KafkaStreamsTest > testCannotCleanupWhileRunning 
STARTED

org.apache.kafka.streams.KafkaStreamsTest > testCannotCleanupWhileRunning PASSED

org.apache.kafka.streams.KafkaStreamsTest > testStateThreadClose STARTED

org.apache.kafka.streams.KafkaStreamsTest > testStateThreadClose PASSED

org.apache.kafka.streams.KafkaStreamsTest > testStateChanges STARTED

org.apache.kafka.streams.KafkaStreamsTest > testStateChanges PASSED

org.apache.kafka.streams.KafkaStreamsTest > testCannotStartTwice STARTED

org.apache.kafka.streams.KafkaStreamsTest > testCannotStartTwice PASSED

org.apache.kafka.streams.TopologyTest > 
shouldNotAllowNullTopicsWhenAddingSourceWithPattern STARTED

org.apache.kafka.streams.TopologyTest > 
shouldNotAllowNullTopicsWhenAddingSourceWithPattern PASSED

org.apache.kafka.streams.TopologyTest > 
shouldNotAllowZeroTopicsWhenAddingSource STARTED

org.apache.kafka.streams.TopologyTest > 
shouldNotAllowZeroTopicsWhenAddingSource PASSED

org.apache.kafka.streams.TopologyTest > shouldFailOnUnknownSource STARTED

org.apache.kafka.streams.TopologyTest > shouldFailOnUnknownSource PASSED

org.apache.kafka.streams.TopologyTest > shouldNotAllowNullNameWhenAddingSink 
STARTED

org.apache.kafka.streams.TopologyTest > shouldNotAllowNullNameWhenAddingSink 
PASSED

org.apache.kafka.streams.TopologyTest > 
multipleSourcesShouldHaveDistinctSubtopologies STARTED

org.apache.kafka.streams.TopologyTest > 
multipleSourcesShouldHaveDistinctSubtopologies PASSED

org.apache.kafka.streams.TopologyTest > 
testNamedTopicMatchesAlreadyProvidedPattern STARTED

org.apache.kafka.streams.TopologyTest > 
testNamedTopicMatchesAlreadyProvidedPattern PASSED

org.apache.kafka.streams.TopologyTest > 
processorsWithSharedStateShouldHaveSameSubtopology STARTED

org.apache.kafka.streams.TopologyTest > 
processorsWithSharedStateShouldHaveSameSubtopology PASSED

org.apache.kafka.streams.TopologyTest > 
shouldNotAllowToAddProcessorWithSameName STARTED

org.apache.kafka.streams.TopologyTest > 
shouldNotAllowToAddProcessorWithSameName PASSED

org.apache.kafka.streams.TopologyTest > shouldNotAllowToAddTopicTwice STARTED

org.apache.kafka.streams.TopologyTest > shouldNotAllowToAddTopicTwice PASSED

org.apache.kafka.streams.TopologyTest > 
processorWithMultipleSourcesShouldHaveSingleSubtopology STARTED

org.apache.kafka.streams.TopologyTest > 
processorWithMultipleSourcesShouldHaveSingleSubtopology PASSED

org.apache.kafka.streams.TopologyTest > shouldNotAllowToAddStateStoreToSink 
STARTED

org.apache.kafka.streams.TopologyTest > shouldNotAllowToAddStateStoreToSink 
PASSED

org.apache.kafka.streams.TopologyTest > shouldNotAddNullStateStoreSupplier 
STARTED

org.apache.kafka.streams.TopologyTest > shouldNotAddNullStateStoreSupplier 
PASSED

org.apache.kafka.streams.TopologyTest > 
shouldNotAllowToAddStateStoreToNonExistingProcessor STARTED

org.apache.kafka.streams.TopologyTest > 
shouldNotAllowToAddStateStoreToNonExistingProcessor PASSED

org.apache.kafka.streams.TopologyTest > 
sourceAndProcessorShouldHaveSingleSubtopology STARTED

org.apache.kafka.streams.TopologyTest > 
sourceAndProcessorShouldHaveSingleSubtopology PASSED

org.apache.kafka.streams.TopologyTest > 
shouldNotAllowNullStoreNameWhenConnectingProcessorAndStateStores STARTED

org.apache.kafka.streams.TopologyTest > 
shouldNotAllowNullStoreNameWhenConnectingProcessorAndStateStores PASSED

org.apache.kafka.streams.TopologyTest > 
shouldNotAllowNullNameWhenAddingSourceWithTopic STARTED

org.apache.kafka.streams.TopologyTest > 
shouldNotAllowNullNameWhenAddingSourceWithTopic PASSED

org.apache.kafka.streams.TopologyTest > 
sourceAndProcessorWithStateShouldHaveSingleSubtopology STARTED

org.apache.kafka.streams.TopologyTest > 
sourceAndProcessorWithStateShouldHaveSingleSubtopology PASSED

org.apache.kafka.streams.TopologyTest > shouldNotAllowNullTopicWhenAddingSink 
STARTED

org.apache.kafka.streams.TopologyTest > shouldNotAllowNullTopicWhenAddingSink 
PASSED

org.apache.kafka.streams.TopologyTest > 
shouldNotAllowToAddGlobalStoreWithSourceNameEqualsProcessorName STARTED

org.apache.kafka.streams.TopologyTest > 
shouldNotAllowToAddGlobalStoreWithSourceNameEqualsProcessorName PASSED

org.apache.kafka.streams.TopologyTest > 
singleSourceWithListOfTopicsShouldHaveSingleSubtopology STARTED

org.apache.kafka.streams.TopologyTest > 
singleSourceWithListOfTopicsShouldHaveSingleSubtopology PASSED

org.apache.kafka.streams.TopologyTest >

Re: [ANNOUNCE] New Kafka PMC member: Dong Lin

[Ted Yu]

Congratulation Dong!

On Tue, Aug 21, 2018 at 1:59 AM Viktor Somogyi-Vass 
wrote:

> Congrats Dong! :)
>
> On Tue, Aug 21, 2018 at 10:09 AM James Cheng  wrote:
>
> > Congrats Dong!
> >
> > -James
> >
> > > On Aug 20, 2018, at 3:54 AM, Ismael Juma  wrote:
> > >
> > > Hi everyone,
> > >
> > > Dong Lin became a committer in March 2018. Since then, he has remained
> > > active in the community and contributed a number of patches, reviewed
> > > several pull requests and participated in numerous KIP discussions. I
> am
> > > happy to announce that Dong is now a member of the
> > > Apache Kafka PMC.
> > >
> > > Congratulation Dong! Looking forward to your future contributions.
> > >
> > > Ismael, on behalf of the Apache Kafka PMC
> >
> >
>

Re: KIP-213 - Scalable/Usable Foreign-Key KTable joins - Rebooted.

[Jan Filipiak]

Still havent completly grabbed it.
sorry will read more

On 17.08.2018 21:23, Jan Filipiak wrote:

Cool stuff.

I made some random remarks. Did not touch the core of the algorithm yet.

Will do Monday 100%

I don't see Interactive Queries :) like that!




On 17.08.2018 20:28, Adam Bellemare wrote:

I have submitted a PR with my code against trunk:
https://github.com/apache/kafka/pull/5527

Do I continue on this thread or do we begin a new one for discussion?

On Thu, Aug 16, 2018 at 1:40 AM, Jan Filipiak 
wrote:

even before message headers, the option for me always existed to 
just wrap

the messages into my own custom envelop.
So I of course thought this through. One sentence in your last email
triggered all the thought process I put in the back then
again to design it in the, what i think is the "kafka-way". It ended up
ranting a little about what happened in the past.

I see plenty of colleagues of mine falling into traps in the API, 
that I

did warn about in the 1.0 DSL rewrite. I have the same
feeling again. So I hope it gives you some insights into my though
process. I am aware that since i never ported 213 to higher
streams version, I don't really have a steak here and initially I 
didn't

feel like actually sending it. But maybe you can pull
something good from it.

  Best jan



On 15.08.2018 04:44, Adam Bellemare wrote:


@Jan
Thanks Jan. I take it you mean "key-widening" somehow includes 
information
about which record is processed first? I understand about a 
CombinedKey

with both the Foreign and Primary key, but I don't see how you track
ordering metadata in there unless you actually included a metadata 
field

in
the key type as well.

@Guozhang
As Jan mentioned earlier, is Record Headers mean to strictly be 
used in
just the user-space? It seems that it is possible that a collision 
on the

(key,value) tuple I wish to add to it could occur. For instance, if I
wanted to add a ("foreignKeyOffset",10) to the Headers but the user
already
specified their own header with the same key name, then it appears 
there
would be a collision. (This is one of the issues I brought up in 
the KIP).




I will be posting a prototype PR against trunk within the next day 
or two.
One thing I need to point out is that my design very strictly wraps 
the
entire foreignKeyJoin process entirely within the DSL function. 
There is

no
exposure of CombinedKeys or widened keys, nothing to resolve with 
regards

to out-of-order processing and no need for the DSL user to even know
what's
going on inside of the function. The code simply returns the 
results of

the
join, keyed by the original key. Currently my API mirrors 
identically the
format of the data returned by the regular join function, and I 
believe
that this is very useful to many users of the DSL. It is my 
understanding
that one of the main design goals of the DSL is to provide higher 
level
functionality without requiring the users to know exactly what's 
going on
under the hood. With this in mind, I thought it best to solve 
ordering and
partitioning problems within the function and eliminate the 
requirement

for
users to do additional work after the fact to resolve the results 
of their
join. Basically, I am assuming that most users of the DSL just 
"want it to
work" and want it to be easy. I did this operating under the 
assumption

that if a user truly wants to optimize their own workflow down to the
finest details then they will break from strictly using the DSL and 
move

down to the processors API.


I think. The abstraction is not powerful enough
to not have kafka specifics leak up The leak I currently think this 
has is

that you can not reliable prevent the delete coming out first,
before you emit the correct new record. As it is an abstraction 
entirely

around kafka.
I can only recommend to not to. Honesty and simplicity should always be
first prio
trying to hide this just makes it more complex, less understandable and
will lead to mistakes
in usage.

Exactly why I am also in big disfavour of GraphNodes and later
optimization stages.
Can someone give me an example of an optimisation that really can't be
handled by the user
constructing his topology differently?
Having reusable Processor API components accessible by the DSL and
composable as
one likes is exactly where DSL should max out and KSQL should do the 
next

step.
I find it very unprofessional from a software engineering approach 
to run

software where
you can not at least senseful reason about the inner workings of the
libraries used.
Gives this people have to read and understand in anyway, why try to 
hide

it?

It really miss the beauty of 0.10 version DSL.
Apparently not a thing I can influence but just warn about.

@gouzhang
you can't imagine how many extra IQ-Statestores I constantly prune from
stream app's
because people just keep passing Materialized's into all the 
operations.

:D :'-(
I regret that I couldn't convince you guys back then. Plus

Re: [ANNOUNCE] New Kafka PMC member: Dong Lin

[Eno Thereska]

Congrats Dong!

Eno

On Tue, Aug 21, 2018 at 7:05 AM, Ted Yu  wrote:

> Congratulation Dong!
>
> On Tue, Aug 21, 2018 at 1:59 AM Viktor Somogyi-Vass <
> viktorsomo...@gmail.com>
> wrote:
>
> > Congrats Dong! :)
> >
> > On Tue, Aug 21, 2018 at 10:09 AM James Cheng 
> wrote:
> >
> > > Congrats Dong!
> > >
> > > -James
> > >
> > > > On Aug 20, 2018, at 3:54 AM, Ismael Juma  wrote:
> > > >
> > > > Hi everyone,
> > > >
> > > > Dong Lin became a committer in March 2018. Since then, he has
> remained
> > > > active in the community and contributed a number of patches, reviewed
> > > > several pull requests and participated in numerous KIP discussions. I
> > am
> > > > happy to announce that Dong is now a member of the
> > > > Apache Kafka PMC.
> > > >
> > > > Congratulation Dong! Looking forward to your future contributions.
> > > >
> > > > Ismael, on behalf of the Apache Kafka PMC
> > >
> > >
> >
>

Re: [VOTE] KIP-346 - Improve LogCleaner behavior on error

[Jason Gustafson]

+1 Thanks for the KIP! I'd suggest mentioning the configurations that were
previously proposed in the rejected alternatives section. We may reconsider
them in the future.

On Mon, Aug 13, 2018 at 9:48 AM, Dhruvil Shah  wrote:

> Thanks for the KIP, Stanislav! +1 (non-binding)
>
> - Dhruvil
>
> On Mon, Aug 13, 2018 at 9:39 AM Colin McCabe  wrote:
>
> > +1 (non-binding)
> >
> > best,
> > Colin
> >
> > On Tue, Aug 7, 2018, at 04:19, Stanislav Kozlovski wrote:
> > > Hey everybody,
> > > I'm starting a vote on KIP-346
> > > <
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 346+-+Improve+LogCleaner+behavior+on+error
> > >
> > >
> > > --
> > > Best,
> > > Stanislav
> >
>

Re: [DISCUSS] KIP-110: Add Codec for ZStandard Compression (Updated)

[Dongjin Lee]

Ismael, Jason and all,

I rewrote the backward compatibility strategy & its alternatives like
following, based on Ismael & Jason's comments. Since it is not updated to
the wiki yet, don't hesitate to give me a message if you have any opinion
on it.

```
*Backward Compatibility*

We need to establish some backward-compatibility strategy for the case an
old client subscribes a topic using ZStandard implicitly (i.e.,
'compression.type' configuration of given topic is 'producer' and the
producer compressed the records with ZStandard). We have the following
options for this situation:

*A. Support ZStandard to the old clients which can understand v0, v1
messages only.*

This strategy necessarily requires the down-conversion of v2 message
compressed with Zstandard into v0 or v1 messages, which means a
considerable performance degradation. So we rejected this strategy.

*B. Bump the API version and support only v2-available clients*

With this approach, we can message the old clients that they are old and
should be upgraded. However, there are still several options for the Error
code.

*B.1. INVALID_REQUEST (42)*

This option gives the client so little information; the user can be
confused about why the client worked correctly in the past suddenly
encounters a problem. So we rejected this strategy.

*B.2. CORRUPT_MESSAGE (2)*

This option gives inaccurate information; the user can be surprised and
misunderstand that the log files are broken in some way. So we rejected
this strategy.

*B.3 UNSUPPORTED_FOR_MESSAGE_FORMAT (43)*

The advantage of this approach is we don't need to define a new error code;
we can reuse it and that's all.

The disadvantage of this approach is that it is also a little bit vague;
This error code is defined as a work for KIP-98[^1] and now returned in the
transaction error.

*B.4. UNSUPPORTED_COMPRESSION_TYPE (new)*

The advantage of this approach is that it is clear and provides an exact
description. The disadvantage is we need to add a new error code.
```

*It seems like what we need to choose is now so clear:
UNSUPPORTED_FOR_MESSAGE_FORMAT (B.3) or UNSUPPORTED_COMPRESSION_TYPE (B.4).*
The first one doesn't need a new error message but the latter is more
explicit. Which one do you prefer? Since all of you have much more
experience and knowledge than me, I will follow your decision. The wiki
page will be updated following the decision also.

Best,
Dongjin

[^1]: https://issues.apache.org/jira/browse/KAFKA-4990

On Sun, Aug 19, 2018 at 4:58 AM Ismael Juma  wrote:

> Sounds reasonable to me.
>
> Ismael
>
> On Sat, 18 Aug 2018, 12:20 Jason Gustafson,  wrote:
>
> > Hey Ismael,
> >
> > Your summary looks good to me. I think it might also be a good idea to
> add
> > a new UNSUPPORTED_COMPRESSION_TYPE error code to go along with the
> version
> > bumps. We won't be able to use it for old api versions since the clients
> > will not understand it, but we can use it going forward so that we're not
> > stuck in a similar situation with a new message format and a new codec to
> > support. Another option is to use UNSUPPORTED_FOR_MESSAGE_FORMAT, but it
> is
> > not as explicit.
> >
> > -Jason
> >
> > On Fri, Aug 17, 2018 at 5:19 PM, Ismael Juma  wrote:
> >
> > > Hi Dongjin and Jason,
> > >
> > > I would agree. My summary:
> > >
> > > 1. Support zstd with message format 2 only.
> > > 2. Bump produce and fetch request versions.
> > > 3. Provide broker errors whenever possible based on the request version
> > and
> > > rely on clients for the cases where the broker can't validate
> efficiently
> > > (example message format 2 consumer that supports the latest fetch
> version
> > > but doesn't support zstd).
> > >
> > > If there's general agreement on this, I suggest we update the KIP to
> > state
> > > the proposal and to move the rejected options to its own section. And
> > then
> > > start a vote!
> > >
> > > Ismael
> > >
> > > On Fri, Aug 17, 2018 at 4:00 PM Jason Gustafson 
> > > wrote:
> > >
> > > > Hi Dongjin,
> > > >
> > > > Yes, that's a good summary. For clients which support v2, the client
> > can
> > > > parse the message format and hopefully raise a useful error message
> > > > indicating the unsupported compression type. For older clients, our
> > > options
> > > > are probably (1) to down-convert to the old format using no
> compression
> > > > type, or (2) to return an error code. I'm leaning toward the latter
> as
> > > the
> > > > simpler solution, but the challenge is finding a good error code. Two
> > > > possibilities might be INVALID_REQUEST or CORRUPT_MESSAGE. The
> downside
> > > is
> > > > that old clients probably won't get a helpful message. However, at
> > least
> > > > the behavior will be consistent in the sense that all clients will
> fail
> > > if
> > > > they do not support zstandard.
> > > >
> > > > What do you think?
> > > >
> > > > Thanks,
> > > > Jason
> > > >
> > > > On Fri, Aug 17, 2018 at 8:08 AM, Dongjin Lee 
> > wrote:
> > > >
> > > > > Thanks Jason, I reviewed the

Re: [DISCUSS] KIP-358: Migrate Streams API to Duration instead of long ms times

[John Roesler]

I'll solicit more reviews. Let's get at least one committer to chime in
before we start a vote (since we need their approval anyway).
-John

On Mon, Aug 20, 2018 at 12:39 PM Nikolay Izhikov 
wrote:

> Hello, Ted.
>
> Thanks for the comment.
>
> I've edit KIP and change proposal to `windowSize`.
>
> Guys, any other comments?
>
>
> В Вс, 19/08/2018 в 14:57 -0700, Ted Yu пишет:
> > bq. // or just Duration windowSize();
> >
> > +1 to the above choice.
> > The duration is obvious from the return type. For getter methods, we
> don't
> > use get as prefix (as least for new code).
> >
> > Cheers
> >
> > On Sun, Aug 19, 2018 at 8:03 AM Nikolay Izhikov 
> wrote:
> >
> > > Hello, John.
> > >
> > > Thank you very much for your feedback!
> > > I've addressed all your comments.
> > > Please, see my answers and let my know is anything in KIP [1] needs to
> be
> > > improved.
> > >
> > > > The correct choice is actually "Instant", not> "LocalDateTime"
> > >
> > > I've changed the methods proposed in KIP [1] to use Instant.
> > >
> > > > I noticed some recent APIs are> missing (see KIP-328)
> > > > those APIs were just added and have never been released... you can
> just
> > >
> > > replace them.
> > >
> > > I've added new methods to KIP [1].
> > > Not released methods marked for remove.
> > >
> > > > any existing method that's already deprecated, don't bother
> > >
> > > transitioning it to Duration.
> > >
> > > Fixed.
> > >
> > > > IllegalArgumentException... we should plan to mention this in the
> > >
> > > javadoc for those methods.
> > >
> > > Got it.
> > >
> > > > In Stores, windowSize and segmentInterval should also be durations.
> > >
> > > Fixed.
> > >
> > > > StreamsMetrics, recordLatency ... this one is better left alone.
> > >
> > > OK. I removed this method from KIP [1].
> > >
> > > Two more questions question about implementation:
> > >
> > > 1. We have serveral methods without parameters.
> > > In java we can't have two methods with parameters with the same name.
> > > It wouldn't compile.
> > > So we have to rename new methods. Please, see suggested names and share
> > > your thoughts:
> > >
> > > Windows {
> > > long size() -> Duration windowSize();
> > > }
> > >
> > > Window {
> > > long start() -> Instant startTime();
> > > long end() -> Instant endTime();
> > > }
> > >
> > > SessionWindows {
> > > long inactivityGap() -> Duration inactivityGapDuration();
> > > }
> > >
> > > TimeWindowedDeserializer {
> > > Long getWindowSize() -> Duration getWindowSizeDuration(); // or
> just
> > > Duration windowSize();
> > > }
> > >
> > > SessionBytesStoreSupplier {
> > > long retentionPeriod() -> Duration retentionPeriodDuration();
> > > }
> > >
> > > WindowBytesStoreSupplier {
> > > long windowSize() -> Duration windowSizeDuration();
> > > long retentionPeriod() -> Duration retentionPeriodDuration();
> > > }
> > >
> > > 2. Do we want to use Duration and Instant inside API implementations?
> > >
> > > IGNITE-7277: "Durations potentially worsen memory pressure and gc
> > > performance, so internally, we will still use longMs as the
> representation."
> > > IGNITE-7222: Duration used to store retention.
> > >
> > > [1]
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-358%3A+Migrate+Streams+API+to+Duration+instead+of+long+ms+times
> > > [2]
> > >
> https://github.com/apache/kafka/commit/b3771ba22acad7870e38ff7f58820c5b50946787#diff-47289575d3e3e2449f27b3a7b6788e1aR64
> > >
> > > В Пт, 17/08/2018 в 14:46 -0500, John Roesler пишет:
> > > > Hi Nikolay,
> > > >
> > > > Thanks for this very nice KIP!
> > > >
> > > > To answer your questions:
> > > > 1. Correct, we should not delete existing methods that have been
> > >
> > > released,
> > > > but ...
> > > >
> > > > 2. Yes, we should deprecate the 'long' variants so that we can drop
> them
> > > > later on. Personally, I like to mention which version deprecated the
> > >
> > > method
> > > > so everyone can see later on how long it's been deprecated, but this
> may
> > >
> > > be
> > > > controversial, so let's let other weigh in.
> > > >
> > > > 3. I think you're asking whether it's appropriate to drop the "Ms"
> > >
> > > suffix,
> > > > and I think yes. So "long inactivityGapMs" would become "Duration
> > > > inactivityGap".
> > > > In the places where the parameter's name is just "duration", I think
> we
> > >
> > > can
> > > > pick something more descriptive (I realize it was already
> "durationMs";
> > > > this is just a good time to improve it).
> > > > Also, you're correct that we shouldn't use a Duration to represent a
> > >
> > > moment
> > > > in time, like "startTime". The correct choice is actually "Instant",
> not
> > > > "LocalDateTime", though.
> > > >
> > >
> > >
> https://stackoverflow.com/questions/32437550/whats-the-difference-between-instant-and-localdatetime
> > > > explains why.
> > > >
> > > > I also had a few notes on the KIP itself:
> > > > 4. You might want to pull trunk again. I noticed some recent APIs

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[John Roesler]

This sounds good to me!

Thanks for the time you've spent on it,
-John

On Tue, Aug 21, 2018 at 12:13 AM Boyang Chen  wrote:

> Thanks Matthias for the input. Sorry I was busy recently and haven't got
> time to update this thread. To summarize what we come up so far, here is a
> draft updated plan:
>
>
> Introduce a new config called `member.name` which is supposed to be
> provided uniquely by the consumer client. The broker will maintain a cache
> with [key:member.name, value:member.id]. A join group request with
> member.name set will be treated as `static-membership` strategy, and will
> reject any join group request without member.name. So this coordination
> change will be differentiated from the `dynamic-membership` protocol we
> currently have.
>
>
> When handling static join group request:
>
>   1.   The broker will check the membership to see whether this is a new
> member. If new, broker allocate a unique member id, cache the mapping and
> move to rebalance stage.
>   2.   Following 1, if this is an existing member, broker will not change
> group state, and return its cached member.id and current assignment.
> (unless this is leader, we shall trigger rebalance)
>   3.   Although Guozhang has mentioned we could rejoin with pair member
> name and id, I think for join group request it is ok to leave member id
> blank as member name is the unique identifier. In commit offset request we
> *must* have both.
>
>
> When handling commit offset request, if enabled with static membership,
> each time the commit request must have both member.name and member.id to
> be identified as a `certificated member`. If not, this means there are
> duplicate consumer members with same member name and the request will be
> rejected to guarantee consumption uniqueness.
>
>
> When rolling restart/shutting down gracefully, the client will send a
> leave group request (static membership mode). In static membership, we will
> also define `change-group-timeout` to hold on rebalance provided by leader.
> So we will wait for all the members to rejoin the group and do exactly one
> rebalance since all members are expected to rejoin within timeout. If
> consumer crashes, the join group request from the restarted consumer will
> be recognized as an existing member and be handled as above condition 1;
> However, if the consumer takes longer than session timeout to return, we
> shall still trigger rebalance but it could still try to catch
> `change-group-timeout`. If it failed to catch second timeout, its cached
> state on broker will be garbage collected and trigger a new rebalance when
> it finally joins.
>
>
> And consider the switch between dynamic to static membership.
>
>   1.  Dynamic to static: the first joiner shall revise the membership to
> static and wait for all the current members to restart, since their
> membership is still dynamic. Here our assumption is that the restart
> process shouldn't take a long time, as long restart is breaking the
> `rebalance timeout` in whatever membership protocol we are using. Before
> restart, all dynamic member join requests will be rejected.
>   2.  Static to dynamic: this is more like a downgrade which should be
> smooth: just erase the cached mapping, and wait for session timeout to
> trigger rebalance should be sufficient. (Fallback to current behavior)
>   3.  Halfway switch: a corner case is like some clients keep dynamic
> membership while some keep static membership. This will cause the group
> rebalance forever without progress because dynamic/static states are
> bouncing each other. This could guarantee that we will not make the
> consumer group work in a wrong state by having half static and half dynamic.
>
> To guarantee correctness, we will also push the member name/id pair to
> _consumed_offsets topic (as Matthias pointed out) and upgrade the API
> version, these details will be further discussed back in the KIP.
>
>
> Are there any concern for this high level proposal? Just want to reiterate
> on the core idea of the KIP: "If the broker recognize this consumer as an
> existing member, it shouldn't trigger rebalance".
>
> Thanks a lot for everyone's input! I feel this proposal is much more
> robust than previous one!
>
>
> Best,
>
> Boyang
>
> 
> From: Matthias J. Sax 
> Sent: Friday, August 10, 2018 2:24 AM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
> specifying member id
>
> Hi,
>
> thanks for the detailed discussion. I learned a lot about internals again
> :)
>
> I like the idea or a user config `member.name` and to keep `member.id`
> internal. Also agree with Guozhang, that reusing `client.id` might not
> be a good idea.
>
> To clarify the algorithm, each time we generate a new `member.id`, we
> also need to update the "group membership" information (ie, mapping
> [member.id, Assignment]), right? Ie, the new `member.id` replaces the
> old entry in the cache.
>
> I also think, we n

Re: [ANNOUNCE] New Kafka PMC member: Dong Lin

[Becket Qin]

Congrats, Dong!

> On Aug 21, 2018, at 11:03 PM, Eno Thereska  wrote:
> 
> Congrats Dong!
> 
> Eno
> 
> On Tue, Aug 21, 2018 at 7:05 AM, Ted Yu  wrote:
> 
>> Congratulation Dong!
>> 
>> On Tue, Aug 21, 2018 at 1:59 AM Viktor Somogyi-Vass <
>> viktorsomo...@gmail.com>
>> wrote:
>> 
>>> Congrats Dong! :)
>>> 
>>> On Tue, Aug 21, 2018 at 10:09 AM James Cheng 
>> wrote:
>>> 
 Congrats Dong!
 
 -James
 
> On Aug 20, 2018, at 3:54 AM, Ismael Juma  wrote:
> 
> Hi everyone,
> 
> Dong Lin became a committer in March 2018. Since then, he has
>> remained
> active in the community and contributed a number of patches, reviewed
> several pull requests and participated in numerous KIP discussions. I
>>> am
> happy to announce that Dong is now a member of the
> Apache Kafka PMC.
> 
> Congratulation Dong! Looking forward to your future contributions.
> 
> Ismael, on behalf of the Apache Kafka PMC
 
 
>>> 
>> 

Re: [DISCUSS] KIP-291: Have separate queues for control requests and data requests

[Becket Qin]

Hi Eno,

Thanks for the comments. This KIP is not really about improving the performance 
in general. It is about ensuring the cluster state can still be updated quickly 
even if the brokers are under heavy load.

We have seen quite often that it took dozens of seconds for a broker to process 
the requests sent by the controller when the cluster is under heavy load. This 
leads to the issues Lucas mentioned in the motivation part.

Thanks,

Jiangjie (Becket) Qin

> On Aug 20, 2018, at 11:33 PM, Eno Thereska  wrote:
> 
> Hi folks,
> 
> I looked at the previous numbers that Lucas provided (thanks!) but it's
> still not clear to me whether the performance benefits justify the added
> complexity. I'm looking for some intuition here (a graph would be great but
> not required): for a small/medium/large cluster, what are the expected
> percentage of control requests today that will benefit from the change?
> It's a bit hard to go through this level of detail without knowing the
> expected end-to-end benefit. The best folks to answer this might be ones
> running such clusters, and ideally should pitch in with some data.
> 
> Thanks
> Eno
> 
> On Mon, Aug 20, 2018 at 7:29 AM, Becket Qin  wrote:
> 
>> Hi Lucas,
>> 
>> In KIP-103, we introduced a convention to define and look up the listeners.
>> So it would be good if the later KIPs can follow the same convention.
>> 
>> From what I understand, the advertised.listeners is actually designed for
>> our purpose, i.e. providing a list of listeners that can be used in
>> different cases. In KIP-103 it was used to separate internal traffic from
>> the external traffic. It is not just for the user traffic or data
>> only. So adding
>> a controller listener is not repurposing the config. Also, ZK structure is
>> only visible to brokers, the clients will still only see the listeners they
>> are seeing today.
>> 
>> For this KIP, we are essentially trying to separate the controller traffic
>> from the inter-broker data traffic. So adding a new
>> listener.name.for.controller config seems reasonable. The behavior would
>> be:
>> 1. If the listener.name.for.controller is set, the broker-controller
>> communication will go through that listener.
>> 2. Otherwise, the controller traffic falls back to use
>> inter.broker.listener.name or inter.broker.security.protocol, which is the
>> current behavior.
>> 
>> Regarding updating the security protocol with one line change v.s two-lines
>> change, I am a little confused, can you elaborate?
>> 
>> Regarding the possibility of hurry and misreading. It is the system admin's
>> responsibility to configure the right listener to ensure that different
>> kinds of traffic are using the correct endpoints. So I think it is better
>> that we always follow the same of convention instead of doing it in
>> different ways.
>> 
>> Thanks,
>> 
>> Jiangjie (Becket) Qin
>> 
>> 
>> 
>> On Fri, Aug 17, 2018 at 4:34 AM, Lucas Wang  wrote:
>> 
>>> Thanks for the review, Becket.
>>> 
>>> (1) After comparing the two approaches, I still feel the current writeup
>> is
>>> a little better.
>>> a. The current writeup asks for an explicit endpoint while reusing the
>>> existing "inter.broker.listener.name" with the exactly same semantic,
>>> and your proposed change asks for a new listener name for controller
>> while
>>> reusing the existing "advertised.listeners" config with a slight semantic
>>> change since a new controller endpoint needs to be added to it.
>>> Hence conceptually the current writeup requires one config change instead
>>> of two.
>>> Also with one listener name, e.g. INTERNAL, for inter broker traffic,
>>> instead of two, e.g. "INTERNAL" and "CONTROLLER",
>>> if an operator decides to switch from PLAINTEXT to SSL for internal
>>> traffic, chances are that she wants to upgrade
>>> both controller connections and data connections, she only needs to
>> update
>>> one line in
>>> the "listener.security.protocol.map" config, and avoids possible
>> mistakes.
>>> 
>>> 
>>> b. When this KIP is picked up by an operator who is in a hurry without
>>> reading the docs, if she sees a
>>> new listener name for controller is required, and chances are there is
>>> already a list of listeners,
>>> it's possible for her to simply choose an existing listener name, without
>>> explicitly creating
>>> the new CONTROLLER listener and endpoints. If this is done, Kafka will be
>>> run with the existing
>>> behavior, defeating the purpose of this KIP.
>>> In comparison, if she sees a separate endpoint is being asked, I feel
>> it's
>>> unlikely for her to
>>> copy and paste an existing endpoint.
>>> 
>>> Please let me know your comments.
>>> 
>>> (2) Good catch, it's a typo, and it's been fixed.
>>> 
>>> Thanks,
>>> Lucas
>>> 
>> 

Re: [ANNOUNCE] New Kafka PMC member: Dong Lin

[Ray Chiang]

Congrats Dong!

-Ray

On 8/21/18 9:33 AM, Becket Qin wrote:

Congrats, Dong!


On Aug 21, 2018, at 11:03 PM, Eno Thereska  wrote:

Congrats Dong!

Eno

On Tue, Aug 21, 2018 at 7:05 AM, Ted Yu  wrote:


Congratulation Dong!

On Tue, Aug 21, 2018 at 1:59 AM Viktor Somogyi-Vass <
viktorsomo...@gmail.com>
wrote:


Congrats Dong! :)

On Tue, Aug 21, 2018 at 10:09 AM James Cheng 

wrote:

Congrats Dong!

-James


On Aug 20, 2018, at 3:54 AM, Ismael Juma  wrote:

Hi everyone,

Dong Lin became a committer in March 2018. Since then, he has

remained

active in the community and contributed a number of patches, reviewed
several pull requests and participated in numerous KIP discussions. I

am

happy to announce that Dong is now a member of the
Apache Kafka PMC.

Congratulation Dong! Looking forward to your future contributions.

Ismael, on behalf of the Apache Kafka PMC

Contribution request

[Flavien Raynaud]

Hi there,

Would it be possible to add me to the contributor list?
My JIRA username is: flavr

Cheers,
Flavien

Re: Contribution request

[Jun Rao]

Hi, Flavien,

Thanks for your interest. Just added you to the contributor list.

Jun

On Tue, Aug 21, 2018 at 10:19 AM, Flavien Raynaud  wrote:

> Hi there,
>
> Would it be possible to add me to the contributor list?
> My JIRA username is: flavr
>
> Cheers,
> Flavien
>

Add to contributor list

[Lionel Montrieux]

Hi there,

Could you please add me to the contributors list?
My jira id is lmontrieux

Cheers,
Lionel

-- 
Lionel Montrieux

Re: KIP-213 - Scalable/Usable Foreign-Key KTable joins - Rebooted.

[John Roesler]

Just a quick thought regarding headers:
> I think there is no absolute-safe ways to avoid conflicts, but we can
still
> consider using some name patterns to reduce the likelihood as much as
> possible.. e.g. consider sth. like the internal topics naming: e.g.
> "__internal_[name]"?

I think there is a safe way to avoid conflicts, since these headers are
only needed in internal topics (I think):
For internal and changelog topics, we can namespace all headers:
* user-defined headers are namespaced as "external." + headerKey
* internal headers are namespaced as "internal." + headerKey

This is a lot of characters, so we could use a sigil instead (e.g., "_" for
internal, "~" for external)

We simply apply the namespacing when we read user headers from external
topics into the topology and then de-namespace them before we emit them to
an external topic (via "to" or "through").
Now, it is not possible to collide with user-defined headers.

That said, I'd also be fine with just reserving "__" as a header prefix and
not worrying about collisions.

Thanks for the KIP,
-John

On Tue, Aug 21, 2018 at 9:48 AM Jan Filipiak 
wrote:

> Still havent completly grabbed it.
> sorry will read more
>
> On 17.08.2018 21:23, Jan Filipiak wrote:
> > Cool stuff.
> >
> > I made some random remarks. Did not touch the core of the algorithm yet.
> >
> > Will do Monday 100%
> >
> > I don't see Interactive Queries :) like that!
> >
> >
> >
> >
> > On 17.08.2018 20:28, Adam Bellemare wrote:
> >> I have submitted a PR with my code against trunk:
> >> https://github.com/apache/kafka/pull/5527
> >>
> >> Do I continue on this thread or do we begin a new one for discussion?
> >>
> >> On Thu, Aug 16, 2018 at 1:40 AM, Jan Filipiak  >
> >> wrote:
> >>
> >>> even before message headers, the option for me always existed to
> >>> just wrap
> >>> the messages into my own custom envelop.
> >>> So I of course thought this through. One sentence in your last email
> >>> triggered all the thought process I put in the back then
> >>> again to design it in the, what i think is the "kafka-way". It ended up
> >>> ranting a little about what happened in the past.
> >>>
> >>> I see plenty of colleagues of mine falling into traps in the API,
> >>> that I
> >>> did warn about in the 1.0 DSL rewrite. I have the same
> >>> feeling again. So I hope it gives you some insights into my though
> >>> process. I am aware that since i never ported 213 to higher
> >>> streams version, I don't really have a steak here and initially I
> >>> didn't
> >>> feel like actually sending it. But maybe you can pull
> >>> something good from it.
> >>>
> >>>   Best jan
> >>>
> >>>
> >>>
> >>> On 15.08.2018 04:44, Adam Bellemare wrote:
> >>>
>  @Jan
>  Thanks Jan. I take it you mean "key-widening" somehow includes
>  information
>  about which record is processed first? I understand about a
>  CombinedKey
>  with both the Foreign and Primary key, but I don't see how you track
>  ordering metadata in there unless you actually included a metadata
>  field
>  in
>  the key type as well.
> 
>  @Guozhang
>  As Jan mentioned earlier, is Record Headers mean to strictly be
>  used in
>  just the user-space? It seems that it is possible that a collision
>  on the
>  (key,value) tuple I wish to add to it could occur. For instance, if I
>  wanted to add a ("foreignKeyOffset",10) to the Headers but the user
>  already
>  specified their own header with the same key name, then it appears
>  there
>  would be a collision. (This is one of the issues I brought up in
>  the KIP).
> 
>  
> 
>  I will be posting a prototype PR against trunk within the next day
>  or two.
>  One thing I need to point out is that my design very strictly wraps
>  the
>  entire foreignKeyJoin process entirely within the DSL function.
>  There is
>  no
>  exposure of CombinedKeys or widened keys, nothing to resolve with
>  regards
>  to out-of-order processing and no need for the DSL user to even know
>  what's
>  going on inside of the function. The code simply returns the
>  results of
>  the
>  join, keyed by the original key. Currently my API mirrors
>  identically the
>  format of the data returned by the regular join function, and I
>  believe
>  that this is very useful to many users of the DSL. It is my
>  understanding
>  that one of the main design goals of the DSL is to provide higher
>  level
>  functionality without requiring the users to know exactly what's
>  going on
>  under the hood. With this in mind, I thought it best to solve
>  ordering and
>  partitioning problems within the function and eliminate the
>  requirement
>  for
>  users to do additional work after the fact to resolve the results
>  of their
>  join. Basically, I am assuming that most

Build failed in Jenkins: kafka-trunk-jdk10 #421

[Apache Jenkins Server]

See 


Changes:

[jason] MINOR: Additional testing of logical type handling in Cast transform

--
[...truncated 1.55 MB...]

kafka.zk.KafkaZkClientTest > testCreateAndGetTopicPartitionStatesRaw PASSED

kafka.zk.KafkaZkClientTest > testLogDirGetters STARTED

kafka.zk.KafkaZkClientTest > testLogDirGetters PASSED

kafka.zk.KafkaZkClientTest > testSetGetAndDeletePartitionReassignment STARTED

kafka.zk.KafkaZkClientTest > testSetGetAndDeletePartitionReassignment PASSED

kafka.zk.KafkaZkClientTest > testIsrChangeNotificationsDeletion STARTED

kafka.zk.KafkaZkClientTest > testIsrChangeNotificationsDeletion PASSED

kafka.zk.KafkaZkClientTest > testGetDataAndVersion STARTED

kafka.zk.KafkaZkClientTest > testGetDataAndVersion PASSED

kafka.zk.KafkaZkClientTest > testGetChildren STARTED

kafka.zk.KafkaZkClientTest > testGetChildren PASSED

kafka.zk.KafkaZkClientTest > testSetAndGetConsumerOffset STARTED

kafka.zk.KafkaZkClientTest > testSetAndGetConsumerOffset PASSED

kafka.zk.KafkaZkClientTest > testClusterIdMethods STARTED

kafka.zk.KafkaZkClientTest > testClusterIdMethods PASSED

kafka.zk.KafkaZkClientTest > testEntityConfigManagementMethods STARTED

kafka.zk.KafkaZkClientTest > testEntityConfigManagementMethods PASSED

kafka.zk.KafkaZkClientTest > testUpdateLeaderAndIsr STARTED

kafka.zk.KafkaZkClientTest > testUpdateLeaderAndIsr PASSED

kafka.zk.KafkaZkClientTest > testUpdateBrokerInfo STARTED

kafka.zk.KafkaZkClientTest > testUpdateBrokerInfo PASSED

kafka.zk.KafkaZkClientTest > testCreateRecursive STARTED

kafka.zk.KafkaZkClientTest > testCreateRecursive PASSED

kafka.zk.KafkaZkClientTest > testGetConsumerOffsetNoData STARTED

kafka.zk.KafkaZkClientTest > testGetConsumerOffsetNoData PASSED

kafka.zk.KafkaZkClientTest > testDeleteTopicPathMethods STARTED

kafka.zk.KafkaZkClientTest > testDeleteTopicPathMethods PASSED

kafka.zk.KafkaZkClientTest > testSetTopicPartitionStatesRaw STARTED

kafka.zk.KafkaZkClientTest > testSetTopicPartitionStatesRaw PASSED

kafka.zk.KafkaZkClientTest > testAclManagementMethods STARTED

kafka.zk.KafkaZkClientTest > testAclManagementMethods PASSED

kafka.zk.KafkaZkClientTest > testPreferredReplicaElectionMethods STARTED

kafka.zk.KafkaZkClientTest > testPreferredReplicaElectionMethods PASSED

kafka.zk.KafkaZkClientTest > testPropagateLogDir STARTED

kafka.zk.KafkaZkClientTest > testPropagateLogDir PASSED

kafka.zk.KafkaZkClientTest > testGetDataAndStat STARTED

kafka.zk.KafkaZkClientTest > testGetDataAndStat PASSED

kafka.zk.KafkaZkClientTest > testReassignPartitionsInProgress STARTED

kafka.zk.KafkaZkClientTest > testReassignPartitionsInProgress PASSED

kafka.zk.KafkaZkClientTest > testCreateTopLevelPaths STARTED

kafka.zk.KafkaZkClientTest > testCreateTopLevelPaths PASSED

kafka.zk.KafkaZkClientTest > testIsrChangeNotificationGetters STARTED

kafka.zk.KafkaZkClientTest > testIsrChangeNotificationGetters PASSED

kafka.zk.KafkaZkClientTest > testLogDirEventNotificationsDeletion STARTED

kafka.zk.KafkaZkClientTest > testLogDirEventNotificationsDeletion PASSED

kafka.zk.KafkaZkClientTest > testGetLogConfigs STARTED

kafka.zk.KafkaZkClientTest > testGetLogConfigs PASSED

kafka.zk.KafkaZkClientTest > testBrokerSequenceIdMethods STARTED

kafka.zk.KafkaZkClientTest > testBrokerSequenceIdMethods PASSED

kafka.zk.KafkaZkClientTest > testCreateSequentialPersistentPath STARTED

kafka.zk.KafkaZkClientTest > testCreateSequentialPersistentPath PASSED

kafka.zk.KafkaZkClientTest > testConditionalUpdatePath STARTED

kafka.zk.KafkaZkClientTest > testConditionalUpdatePath PASSED

kafka.zk.KafkaZkClientTest > testDeleteTopicZNode STARTED

kafka.zk.KafkaZkClientTest > testDeleteTopicZNode PASSED

kafka.zk.KafkaZkClientTest > testDeletePath STARTED

kafka.zk.KafkaZkClientTest > testDeletePath PASSED

kafka.zk.KafkaZkClientTest > testGetBrokerMethods STARTED

kafka.zk.KafkaZkClientTest > testGetBrokerMethods PASSED

kafka.zk.KafkaZkClientTest > testCreateTokenChangeNotification STARTED

kafka.zk.KafkaZkClientTest > testCreateTokenChangeNotification PASSED

kafka.zk.KafkaZkClientTest > testGetTopicsAndPartitions STARTED

kafka.zk.KafkaZkClientTest > testGetTopicsAndPartitions PASSED

kafka.zk.KafkaZkClientTest > testRegisterBrokerInfo STARTED

kafka.zk.KafkaZkClientTest > testRegisterBrokerInfo PASSED

kafka.zk.KafkaZkClientTest > testConsumerOffsetPath STARTED

kafka.zk.KafkaZkClientTest > testConsumerOffsetPath PASSED

kafka.zk.KafkaZkClientTest > testControllerManagementMethods STARTED

kafka.zk.KafkaZkClientTest > testControllerManagementMethods PASSED

kafka.zk.KafkaZkClientTest > testTopicAssignmentMethods STARTED

kafka.zk.KafkaZkClientTest > testTopicAssignmentMethods PASSED

kafka.zk.KafkaZkClientTest > testPropagateIsrChanges STARTED

kafka.zk.KafkaZkClientTest > testPropagateIsrChanges PASSED

kafka.zk.KafkaZkClientTest > testControllerEpochMethods STARTED

kafka.zk.KafkaZkC

[jira] [Resolved] (KAFKA-6753) Speed up event processing on the controller

[Jun Rao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao resolved KAFKA-6753.

   Resolution: Fixed
Fix Version/s: 2.1.0

Merged the PR to trunk.

> Speed up event processing on the controller 
> 
>
> Key: KAFKA-6753
> URL: https://issues.apache.org/jira/browse/KAFKA-6753
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Lucas Wang
>Assignee: Lucas Wang
>Priority: Minor
> Fix For: 2.1.0
>
> Attachments: Screen Shot 2018-04-04 at 7.08.55 PM.png
>
>
> The existing controller code updates metrics after processing every event. 
> This can slow down event processing on the controller tremendously. In one 
> profiling we see that updating metrics takes nearly 100% of the CPU for the 
> controller event processing thread. Specifically the slowness can be 
> attributed to two factors:
> 1. Each invocation to update the metrics is expensive. Specifically trying to 
> calculate the offline partitions count requires iterating through all the 
> partitions in the cluster to check if the partition is offline; and 
> calculating the preferred replica imbalance count requires iterating through 
> all the partitions in the cluster to check if a partition has a leader other 
> than the preferred leader. In a large cluster, the number of partitions can 
> be quite large, all seen by the controller. Even if the time spent to check a 
> single partition is small, the accumulation effect of so many partitions in 
> the cluster can make the invocation to update metrics quite expensive. One 
> might argue that maybe the logic for processing each single partition is not 
> optimized, we checked the CPU percentage of leaf nodes in the profiling 
> result, and found that inside the loops of collection objects, e.g. the set 
> of all partitions, no single function dominates the processing. Hence the 
> large number of the partitions in a cluster is the main contributor to the 
> slowness of one invocation to update the metrics.
> 2. The invocation to update metrics is called many times when the is a high 
> number of events to be processed by the controller, one invocation after 
> processing any event.
> The patch that will be submitted tries to fix bullet 2 above, i.e. reducing 
> the number of invocations to update metrics. Instead of updating the metrics 
> after processing any event, we only periodically check if the metrics needs 
> to be updated, i.e. once every second. 
> * If after the previous invocation to update metrics, there are other types 
> of events that changed the controller’s state, then one second later the 
> metrics will be updated. 
> * If after the previous invocation, there has been no other types of events, 
> then the call to update metrics can be bypassed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: KIP-213 - Scalable/Usable Foreign-Key KTable joins - Rebooted.

[Adam Bellemare]

Hi John

That is an excellent idea. The header usage I propose would be limited
entirely to internal topics, and this could very well be the solution to
potential conflicts. If we do not officially reserve a prefix "__" then I
think this would be the safest idea, as it would entirely avoid any
accidents (perhaps if a company is using its own "__" prefix for other
reasons).

Thanks

Adam


On Tue, Aug 21, 2018 at 2:24 PM, John Roesler  wrote:

> Just a quick thought regarding headers:
> > I think there is no absolute-safe ways to avoid conflicts, but we can
> still
> > consider using some name patterns to reduce the likelihood as much as
> > possible.. e.g. consider sth. like the internal topics naming: e.g.
> > "__internal_[name]"?
>
> I think there is a safe way to avoid conflicts, since these headers are
> only needed in internal topics (I think):
> For internal and changelog topics, we can namespace all headers:
> * user-defined headers are namespaced as "external." + headerKey
> * internal headers are namespaced as "internal." + headerKey
>
> This is a lot of characters, so we could use a sigil instead (e.g., "_" for
> internal, "~" for external)
>
> We simply apply the namespacing when we read user headers from external
> topics into the topology and then de-namespace them before we emit them to
> an external topic (via "to" or "through").
> Now, it is not possible to collide with user-defined headers.
>
> That said, I'd also be fine with just reserving "__" as a header prefix and
> not worrying about collisions.
>
> Thanks for the KIP,
> -John
>
> On Tue, Aug 21, 2018 at 9:48 AM Jan Filipiak 
> wrote:
>
> > Still havent completly grabbed it.
> > sorry will read more
> >
> > On 17.08.2018 21:23, Jan Filipiak wrote:
> > > Cool stuff.
> > >
> > > I made some random remarks. Did not touch the core of the algorithm
> yet.
> > >
> > > Will do Monday 100%
> > >
> > > I don't see Interactive Queries :) like that!
> > >
> > >
> > >
> > >
> > > On 17.08.2018 20:28, Adam Bellemare wrote:
> > >> I have submitted a PR with my code against trunk:
> > >> https://github.com/apache/kafka/pull/5527
> > >>
> > >> Do I continue on this thread or do we begin a new one for discussion?
> > >>
> > >> On Thu, Aug 16, 2018 at 1:40 AM, Jan Filipiak <
> jan.filip...@trivago.com
> > >
> > >> wrote:
> > >>
> > >>> even before message headers, the option for me always existed to
> > >>> just wrap
> > >>> the messages into my own custom envelop.
> > >>> So I of course thought this through. One sentence in your last email
> > >>> triggered all the thought process I put in the back then
> > >>> again to design it in the, what i think is the "kafka-way". It ended
> up
> > >>> ranting a little about what happened in the past.
> > >>>
> > >>> I see plenty of colleagues of mine falling into traps in the API,
> > >>> that I
> > >>> did warn about in the 1.0 DSL rewrite. I have the same
> > >>> feeling again. So I hope it gives you some insights into my though
> > >>> process. I am aware that since i never ported 213 to higher
> > >>> streams version, I don't really have a steak here and initially I
> > >>> didn't
> > >>> feel like actually sending it. But maybe you can pull
> > >>> something good from it.
> > >>>
> > >>>   Best jan
> > >>>
> > >>>
> > >>>
> > >>> On 15.08.2018 04:44, Adam Bellemare wrote:
> > >>>
> >  @Jan
> >  Thanks Jan. I take it you mean "key-widening" somehow includes
> >  information
> >  about which record is processed first? I understand about a
> >  CombinedKey
> >  with both the Foreign and Primary key, but I don't see how you track
> >  ordering metadata in there unless you actually included a metadata
> >  field
> >  in
> >  the key type as well.
> > 
> >  @Guozhang
> >  As Jan mentioned earlier, is Record Headers mean to strictly be
> >  used in
> >  just the user-space? It seems that it is possible that a collision
> >  on the
> >  (key,value) tuple I wish to add to it could occur. For instance, if
> I
> >  wanted to add a ("foreignKeyOffset",10) to the Headers but the user
> >  already
> >  specified their own header with the same key name, then it appears
> >  there
> >  would be a collision. (This is one of the issues I brought up in
> >  the KIP).
> > 
> >  
> > 
> >  I will be posting a prototype PR against trunk within the next day
> >  or two.
> >  One thing I need to point out is that my design very strictly wraps
> >  the
> >  entire foreignKeyJoin process entirely within the DSL function.
> >  There is
> >  no
> >  exposure of CombinedKeys or widened keys, nothing to resolve with
> >  regards
> >  to out-of-order processing and no need for the DSL user to even know
> >  what's
> >  going on inside of the function. The code simply returns the
> >  results of
> >  the
> >  join, keyed by the original key. Cu

[jira] [Created] (KAFKA-7319) Add documentation for consumer group management

[Manikumar (JIRA)]

Manikumar created KAFKA-7319:


 Summary: Add documentation for consumer group management
 Key: KAFKA-7319
 URL: https://issues.apache.org/jira/browse/KAFKA-7319
 Project: Kafka
  Issue Type: Task
  Components: documentation
Reporter: Manikumar


This Jira is to add documentation for consumer group management internals.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: kafka-1.0-jdk7 #235

[Apache Jenkins Server]

See 


Changes:

[mjsax] KAFKA-7284: streams should unwrap fenced exception (#5520)

--
[...truncated 376.64 KB...]
kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldRetainLatestEpochOnClearAllEarliest PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldPersistEpochsBetweenInstances STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldPersistEpochsBetweenInstances PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotClearAnythingIfOffsetToFirstOffset STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotClearAnythingIfOffsetToFirstOffset PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotLetOffsetsGoBackwardsEvenIfEpochsProgress STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotLetOffsetsGoBackwardsEvenIfEpochsProgress PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldGetFirstOffsetOfSubsequentEpochWhenOffsetRequestedForPreviousEpoch STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldGetFirstOffsetOfSubsequentEpochWhenOffsetRequestedForPreviousEpoch PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldUpdateOffsetBetweenEpochBoundariesOnClearEarliest2 STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldUpdateOffsetBetweenEpochBoundariesOnClearEarliest2 PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearEarliestOnEmptyCache 
STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearEarliestOnEmptyCache 
PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldPreserveResetOffsetOnClearEarliestIfOneExists STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldPreserveResetOffsetOnClearEarliestIfOneExists PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldUpdateOffsetBetweenEpochBoundariesOnClearEarliest STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldUpdateOffsetBetweenEpochBoundariesOnClearEarliest PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldReturnInvalidOffsetIfEpochIsRequestedWhichIsNotCurrentlyTracked STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldReturnInvalidOffsetIfEpochIsRequestedWhichIsNotCurrentlyTracked PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldFetchEndOffsetOfEmptyCache 
STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldFetchEndOffsetOfEmptyCache 
PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldRetainLatestEpochOnClearAllEarliestAndUpdateItsOffset STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldRetainLatestEpochOnClearAllEarliestAndUpdateItsOffset PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearAllEntries STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearAllEntries PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearLatestOnEmptyCache 
STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearLatestOnEmptyCache 
PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotResetEpochHistoryHeadIfUndefinedPassed STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotResetEpochHistoryHeadIfUndefinedPassed PASSED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldIncreaseLeaderEpochBetweenLeaderRestarts STARTED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldIncreaseLeaderEpochBetweenLeaderRestarts PASSED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldAddCurrentLeaderEpochToMessagesAsTheyAreWrittenToLeader STARTED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldAddCurrentLeaderEpochToMessagesAsTheyAreWrittenToLeader PASSED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldSendLeaderEpochRequestAndGetAResponse STARTED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldSendLeaderEpochRequestAndGetAResponse PASSED

kafka.server.epoch.OffsetsForLeaderEpochTest > shouldGetEpochsFromReplica 
STARTED

kafka.server.epoch.OffsetsForLeaderEpochTest > shouldGetEpochsFromReplica PASSED

kafka.server.epoch.OffsetsForLeaderEpochTest > 
shouldReturnUnknownTopicOrPartitionIfThrown STARTED

kafka.server.epoch.OffsetsForLeaderEpochTest > 
shouldReturnUnknownTopicOrPartitionIfThrown PASSED

kafka.server.epoch.OffsetsForLeaderEpochTest > 
shouldReturnNoLeaderForPartitionIfThrown STARTED

kafka.server.epoch.OffsetsForLeaderEpochTest > 
shouldReturnNoLeaderForPartitionIfThrown PASSED

kafka.server.epoch.EpochDrivenReplicationProtocolAcceptanceTest > 
shouldSurviveFastLeaderChange STARTED

kafka.server.epoch.EpochDrivenReplicationProtocolAcceptanceTest > 
shouldSurviveFastLeaderChange PASSED

kafka.server.epoch.EpochDrivenReplicationProtocolAcceptanceTest > 
offsetsShouldNotGoBackwards STARTED

kafka.server.epoch.EpochDrivenReplicationProtocolAcceptanceTest > 
offsetsShouldNotGoBackwards PASSED

kafka.server.epoch.EpochDrivenReplicationProtocolAcceptanceTest > 
shouldFollowLeaderEpochBasicWorkflow STARTED

kafka.server.epoch.EpochDrivenReplicationP

Build failed in Jenkins: kafka-trunk-jdk10 #422

[Apache Jenkins Server]

See 


Changes:

[junrao] KAFKA-6753: Updating the OfflinePartitions count only when necessary

[jason] MINOR: Improved configuration formatting in documentation (#5532)

--
[...truncated 1.54 MB...]

kafka.security.auth.ZkAuthorizationTest > testZkAntiMigration PASSED

kafka.security.auth.ZkAuthorizationTest > testZkMigration STARTED

kafka.security.auth.ZkAuthorizationTest > testZkMigration PASSED

kafka.security.auth.ZkAuthorizationTest > testChroot STARTED

kafka.security.auth.ZkAuthorizationTest > testChroot PASSED

kafka.security.auth.ZkAuthorizationTest > testDelete STARTED

kafka.security.auth.ZkAuthorizationTest > testDelete PASSED

kafka.security.auth.ZkAuthorizationTest > testDeleteRecursive STARTED

kafka.security.auth.ZkAuthorizationTest > testDeleteRecursive PASSED

kafka.security.auth.AclTest > testAclJsonConversion STARTED

kafka.security.auth.AclTest > testAclJsonConversion PASSED

kafka.security.auth.PermissionTypeTest > testJavaConversions STARTED

kafka.security.auth.PermissionTypeTest > testJavaConversions PASSED

kafka.security.auth.PermissionTypeTest > testFromString STARTED

kafka.security.auth.PermissionTypeTest > testFromString PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testAuthorizeWithPrefixedResource 
STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testAuthorizeWithPrefixedResource 
PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testAllowAllAccess STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testAllowAllAccess PASSED

kafka.security.auth.SimpleAclAuthorizerTest > 
testLocalConcurrentModificationOfResourceAcls STARTED

kafka.security.auth.SimpleAclAuthorizerTest > 
testLocalConcurrentModificationOfResourceAcls PASSED

kafka.security.auth.SimpleAclAuthorizerTest > 
testDeleteAllAclOnWildcardResource STARTED

kafka.security.auth.SimpleAclAuthorizerTest > 
testDeleteAllAclOnWildcardResource PASSED

kafka.security.auth.SimpleAclAuthorizerTest > 
testHighConcurrencyDeletionOfResourceAcls STARTED

kafka.security.auth.SimpleAclAuthorizerTest > 
testHighConcurrencyDeletionOfResourceAcls PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testNoAclFound STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testNoAclFound PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testAclInheritance STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testAclInheritance PASSED

kafka.security.auth.SimpleAclAuthorizerTest > 
testDistributedConcurrentModificationOfResourceAcls STARTED

kafka.security.auth.SimpleAclAuthorizerTest > 
testDistributedConcurrentModificationOfResourceAcls PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testAddAclsOnWildcardResource 
STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testAddAclsOnWildcardResource 
PASSED

kafka.security.auth.SimpleAclAuthorizerTest > 
testWritesExtendedAclChangeEventWhenInterBrokerProtocolAtLeastKafkaV2 STARTED

kafka.security.auth.SimpleAclAuthorizerTest > 
testWritesExtendedAclChangeEventWhenInterBrokerProtocolAtLeastKafkaV2 PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testAclManagementAPIs STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testAclManagementAPIs PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testWildCardAcls STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testWildCardAcls PASSED

kafka.security.auth.SimpleAclAuthorizerTest > 
testWritesLiteralAclChangeEventWhenInterBrokerProtocolIsKafkaV2 STARTED

kafka.security.auth.SimpleAclAuthorizerTest > 
testWritesLiteralAclChangeEventWhenInterBrokerProtocolIsKafkaV2 PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testTopicAcl STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testTopicAcl PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testSuperUserHasAccess STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testSuperUserHasAccess PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testDeleteAclOnPrefixedResource 
STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testDeleteAclOnPrefixedResource 
PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testDenyTakesPrecedence STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testDenyTakesPrecedence PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testNoAclFoundOverride STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testNoAclFoundOverride PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testEmptyAclThrowsException 
STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testEmptyAclThrowsException PASSED

kafka.security.auth.SimpleAclAuthorizerTest > 
testSuperUserWithCustomPrincipalHasAccess STARTED

kafka.security.auth.SimpleAclAuthorizerTest > 
testSuperUserWithCustomPrincipalHasAccess PASSED

kafka.security.auth.SimpleAclAuthorizerTest > 
testAllowAccessWithCustomPrincipal STARTED

kafka.security.auth.SimpleAclAuthorizerTest > 
testAllowAccessWithCustomPrincipal PASSED

kafka.security

[jira] [Resolved] (KAFKA-5407) Mirrormaker dont start after upgrade

[Fernando Vega (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fernando Vega resolved KAFKA-5407.
--
Resolution: Fixed

> Mirrormaker dont start after upgrade
> 
>
> Key: KAFKA-5407
> URL: https://issues.apache.org/jira/browse/KAFKA-5407
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Affects Versions: 0.10.2.1
> Environment: Operating system
> CentOS 6.8
> HW
> Board Mfg : HP
>  Board Product : ProLiant DL380p Gen8
> CPU's x2
> Product Manufacturer  : Intel
>  Product Name  :  Intel(R) Xeon(R) CPU E5-2660 v2 @ 2.20GHz
>  Memory Type   : DDR3 SDRAM
>  SDRAM Capacity: 2048 MB
>  Total Memory: : 64GB
> Hardrives size and layout:
> 9 drives using jbod
> drive size 3.6TB each
>Reporter: Fernando Vega
>Priority: Critical
> Attachments: broker.hkg1.new, debug.hkg1.new, 
> mirrormaker-repl-sjc2-to-hkg1.log.8
>
>
> Currently Im upgrading the cluster from 0.8.2-beta to 0.10.2.1
> So I followed the rolling procedure:
> Here the config files:
> Consumer
> {noformat}
> #
> # Cluster: repl
> # Topic list(goes into command line): 
> REPL-ams1-global,REPL-atl1-global,REPL-sjc2-global,REPL-ams1-global-PN_HXIDMAP_.*,REPL-atl1-global-PN_HXIDMAP_.*,REPL-sjc2-global-PN_HXIDMAP_.*,REPL-ams1-global-PN_HXCONTEXTUALV2_.*,REPL-atl1-global-PN_HXCONTEXTUALV2_.*,REPL-sjc2-global-PN_HXCONTEXTUALV2_.*
> bootstrap.servers=app001:9092,app002:9092,app003:9092,app004:9092
> group.id=hkg1_cluster
> auto.commit.interval.ms=6
> partition.assignment.strategy=org.apache.kafka.clients.consumer.RoundRobinAssignor
> {noformat}
> Producer
> {noformat}
>  hkg1
> # # Producer
> # # hkg1
> bootstrap.servers=app001:9092,app002:9092,app003:9092,app004:9092
> compression.type=gzip
> acks=0
> {noformat}
> Broker
> {noformat}
> auto.leader.rebalance.enable=true
> delete.topic.enable=true
> socket.receive.buffer.bytes=1048576
> socket.send.buffer.bytes=1048576
> default.replication.factor=2
> auto.create.topics.enable=true
> num.partitions=1
> num.network.threads=8
> num.io.threads=40
> log.retention.hours=1
> log.roll.hours=1
> num.replica.fetchers=8
> zookeeper.connection.timeout.ms=3
> zookeeper.session.timeout.ms=3
> inter.broker.protocol.version=0.10.2
> log.message.format.version=0.8.2
> {noformat}
> I tried also using stock configuraiton with no luck.
> The error that I get is this:
> {noformat}
> 2017-06-07 12:24:45,476] INFO ConsumerConfig values:
>   auto.commit.interval.ms = 6
>   auto.offset.reset = latest
>   bootstrap.servers = [app454.sjc2.mytest.com:9092, 
> app455.sjc2.mytest.com:9092, app456.sjc2.mytest.com:9092, 
> app457.sjc2.mytest.com:9092, app458.sjc2.mytest.com:9092, 
> app459.sjc2.mytest.com:9092]
>   check.crcs = true
>   client.id = MirrorMaker_hkg1-1
>   connections.max.idle.ms = 54
>   enable.auto.commit = false
>   exclude.internal.topics = true
>   fetch.max.bytes = 52428800
>   fetch.max.wait.ms = 500
>   fetch.min.bytes = 1
>   group.id = MirrorMaker_hkg1
>   heartbeat.interval.ms = 3000
>   interceptor.classes = null
>   key.deserializer = class 
> org.apache.kafka.common.serialization.ByteArrayDeserializer
>   max.partition.fetch.bytes = 1048576
>   max.poll.interval.ms = 30
>   max.poll.records = 500
>   metadata.max.age.ms = 30
>   metric.reporters = []
>   metrics.num.samples = 2
>   metrics.recording.level = INFO
>   metrics.sample.window.ms = 3
>   partition.assignment.strategy = 
> [org.apache.kafka.clients.consumer.RoundRobinAssignor]
>   receive.buffer.bytes = 65536
>   reconnect.backoff.ms = 50
>   request.timeout.ms = 305000
>   retry.backoff.ms = 100
>   sasl.jaas.config = null
>   sasl.kerberos.kinit.cmd = /usr/bin/kinit
>   sasl.kerberos.min.time.before.relogin = 6
>   sasl.kerberos.service.name = null
>   sasl.kerberos.ticket.renew.jitter = 0.05
>   sasl.kerberos.ticket.renew.window.factor = 0.8
>   sasl.mechanism = GSSAPI
>   security.protocol = PLAINTEXT
>   send.buffer.bytes = 131072
>   session.timeout.ms = 1
>   ssl.cipher.suites = null
>   ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
>   ssl.endpoint.identification.algorithm = null
>   ssl.key.password = null
>   ssl.keymanager.algorithm = SunX509
>   ssl.keystore.location = null
>   ssl.keystore.password = null
>   ssl.keystore.type = JKS
>   ssl.protocol = TLS
>   ssl.provider = null
>   ssl.secure.random.implementation = null
>   ssl.trustmanager.algorithm = PKIX
>   ssl.truststore.location = null
>   ssl.truststore.password = null
>   ssl.truststore.type = JKS
>   value.deserializer = cla

Build failed in Jenkins: kafka-0.11.0-jdk7 #396

[Apache Jenkins Server]

See 


Changes:

[matthias] KAFKA-7284: streams should unwrap fenced exception (#5520)

--
[...truncated 192.02 KB...]
kafka.api.TransactionsTest > testMultipleMarkersOneLeader STARTED

kafka.api.TransactionsTest > testMultipleMarkersOneLeader PASSED

kafka.api.TransactionsTest > testSendOffsets STARTED

kafka.api.TransactionsTest > testSendOffsets PASSED

kafka.api.ApiUtilsTest > testShortStringNonASCII STARTED

kafka.api.ApiUtilsTest > testShortStringNonASCII PASSED

kafka.api.ApiUtilsTest > testShortStringASCII STARTED

kafka.api.ApiUtilsTest > testShortStringASCII PASSED

kafka.api.SslConsumerTest > testCoordinatorFailover STARTED

kafka.api.SslConsumerTest > testCoordinatorFailover PASSED

kafka.api.SslConsumerTest > testSimpleConsumption STARTED

kafka.api.SslConsumerTest > testSimpleConsumption PASSED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testTwoConsumersWithDifferentSaslCredentials STARTED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testTwoConsumersWithDifferentSaslCredentials PASSED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaSubscribe STARTED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaSubscribe PASSED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > testProduceConsumeViaAssign 
STARTED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > testProduceConsumeViaAssign 
PASSED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testNoConsumeWithDescribeAclViaAssign STARTED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testNoConsumeWithDescribeAclViaAssign PASSED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testNoConsumeWithDescribeAclViaSubscribe STARTED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testNoConsumeWithDescribeAclViaSubscribe PASSED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaAssign STARTED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaAssign PASSED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > testNoGroupAcl STARTED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > testNoGroupAcl PASSED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > testNoProduceWithDescribeAcl 
STARTED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > testNoProduceWithDescribeAcl 
PASSED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testProduceConsumeViaSubscribe STARTED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testProduceConsumeViaSubscribe PASSED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testNoProduceWithoutDescribeAcl STARTED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testNoProduceWithoutDescribeAcl PASSED

kafka.api.LegacyAdminClientTest > testSeekToBeginningAfterDeleteRecords STARTED

kafka.api.LegacyAdminClientTest > testSeekToBeginningAfterDeleteRecords PASSED

kafka.api.LegacyAdminClientTest > testConsumeAfterDeleteRecords STARTED

kafka.api.LegacyAdminClientTest > testConsumeAfterDeleteRecords PASSED

kafka.api.LegacyAdminClientTest > testDescribeConsumerGroup STARTED

kafka.api.LegacyAdminClientTest > testDescribeConsumerGroup PASSED

kafka.api.LegacyAdminClientTest > testListGroups STARTED

kafka.api.LegacyAdminClientTest > testListGroups PASSED

kafka.api.LegacyAdminClientTest > testOffsetsForTimesAfterDeleteRecords STARTED

kafka.api.LegacyAdminClientTest > testOffsetsForTimesAfterDeleteRecords PASSED

kafka.api.LegacyAdminClientTest > testDeleteRecordsWithException STARTED

kafka.api.LegacyAdminClientTest > testDeleteRecordsWithException PASSED

kafka.api.LegacyAdminClientTest > testLogStartOffsetCheckpoint STARTED

kafka.api.LegacyAdminClientTest > testLogStartOffsetCheckpoint PASSED

kafka.api.LegacyAdminClientTest > testListAllBrokerVersionInfo STARTED

kafka.api.LegacyAdminClientTest > testListAllBrokerVersionInfo PASSED

kafka.api.LegacyAdminClientTest > testDescribeConsumerGroupForNonExistentGroup 
STARTED

kafka.api.LegacyAdminClientTest > testDescribeConsumerGroupForNonExistentGroup 
PASSED

kafka.api.LegacyAdminClientTest > testLogStartOffsetAfterDeleteRecords STARTED

kafka.api.LegacyAdminClientTest > testLogStartOffsetAfterDeleteRecords PASSED

kafka.api.LegacyAdminClientTest > testGetConsumerGroupSummary STARTED

kafka.api.LegacyAdminClientTest > testGetConsumerGroupSummary PASSED

kafka.api.SaslMultiMechanismConsumerTest > testMultipleBrokerMechanisms STARTED

kafka.api.SaslMultiMechanismConsumerTest > testMultipleBrokerMechanisms PASSED

kafka.api.SaslMultiMechanismConsumerTest > testCoordinatorFailover STARTED

kafka.api.SaslMultiMechanismConsumerTest > testCoordinatorFailover PASSED

kafka.api.SaslMultiMechanismConsumerTest > testSimpleConsumption STARTED

kafka.api.SaslMultiMechanismConsumerTest > testSimpleConsumption PASSED

kafka.api.GroupCoordinatorIntegrationTest > 
testGroupCoordinatorPropag

Re: [VOTE] KIP-328: Ability to suppress updates for KTables

[John Roesler]

Hello again, all,

I belatedly had a better idea for adding grace period to the Windows class
hierarchy (TimeWindows, UnlimitedWindows, JoinWindows). Instead of
providing the grace-setter in the abstract class and having to retract it
in UnlimitedWindows, I've made the getter abstract method in Windows and
only added setters to Time and Join windows.

This should not only improve the ergonomics of grace period, but make the
whole class hierarchy more maintainable.

See the PR for more details: https://github.com/apache/kafka/pull/5536

I've updated the KIP accordingly. Here's the diff:
https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=87295409&selectedPageVersions=11&selectedPageVersions=9

Please let me know if this changes your vote.

Thanks,
-John

On Mon, Aug 13, 2018 at 5:20 PM John Roesler  wrote:

> Hey all,
>
> I just wanted to let you know that a few small issues surfaced during
> implementation and review. I've updated the KIP. Here's the diff:
> https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=87295409&selectedPageVersions=9&selectedPageVersions=8
>
> Basically:
> * the metrics named "*-event-*" are inconsistent with existing
> nomenclature, and will be "*-record-*" instead (late records instead of
> late events, for example)
> * the apis taking and returning Duration will use long millis instead. We
> do want to transition to Duration in the future, but we shouldn't do it
> piecemeal.
>
> Thanks,
> -John
>
> On Tue, Aug 7, 2018 at 12:07 PM John Roesler  wrote:
>
>> Thanks everyone, KIP-328 has passed with 3 binding votes (Guozhang,
>> Damian, and Matthias) and 3 non-binding (Ted, Bill, and me).
>>
>> Thanks for your time,
>> -John
>>
>> On Mon, Aug 6, 2018 at 6:35 PM Matthias J. Sax 
>> wrote:
>>
>>> +1 (binding)
>>>
>>> Thanks for the KIP.
>>>
>>>
>>> -Matthias
>>>
>>> On 8/3/18 12:52 AM, Damian Guy wrote:
>>> > Thanks John! +1
>>> >
>>> > On Mon, 30 Jul 2018 at 23:58 Guozhang Wang  wrote:
>>> >
>>> >> Yes, the addendum lgtm as well. Thanks!
>>> >>
>>> >> On Mon, Jul 30, 2018 at 3:34 PM, John Roesler 
>>> wrote:
>>> >>
>>> >>> Another thing that came up after I started working on an
>>> implementation
>>> >> is
>>> >>> that in addition to deprecating "retention" from the Windows
>>> interface,
>>> >> we
>>> >>> also need to deprecate "segmentInterval", for the same reasons. I
>>> simply
>>> >>> overlooked it previously. I've updated the KIP accordingly.
>>> >>>
>>> >>> Hopefully, this doesn't change anyone's vote.
>>> >>>
>>> >>> Thanks,
>>> >>> -John
>>> >>>
>>> >>> On Mon, Jul 30, 2018 at 5:31 PM John Roesler 
>>> wrote:
>>> >>>
>>>  Thanks Guozhang,
>>> 
>>>  Thanks for that catch. to clarify, currently, events are "late" only
>>> >> when
>>>  they are older than the retention period. Currently, we detect this
>>> in
>>> >>> the
>>>  processor and record it as a "skipped-record". We then do not
>>> attempt
>>> >> to
>>>  store the event in the window store. If a user provided a
>>> >> pre-configured
>>>  window store with a retention period smaller than the one they
>>> specify
>>> >>> via
>>>  Windows#until, the segmented store will drop the update with no
>>> metric
>>> >>> and
>>>  record a debug-level log.
>>> 
>>>  With KIP-328, with the introduction of "grace period" and moving
>>> >>> retention
>>>  fully into the state store, we need to have metrics for both "late
>>> >>> events"
>>>  (new records older than the grace period) and "expired window
>>> events"
>>> >>> (new
>>>  records for windows that are no longer retained in the state
>>> store). I
>>>  already proposed metrics for the late events, and I've just updated
>>> the
>>> >>> KIP
>>>  with metrics for the expired window events. I also updated the KIP
>>> to
>>> >>> make
>>>  it clear that neither late nor expired events will count as
>>>  "skipped-records" any more.
>>> 
>>>  -John
>>> 
>>>  On Mon, Jul 30, 2018 at 4:22 PM Guozhang Wang 
>>> >>> wrote:
>>> 
>>> > Hi John,
>>> >
>>> > Thanks for the updated KIP, +1 from me, and one minor suggestion:
>>> >
>>> > Following your suggestion of the differentiation of
>>> `skipped-records`
>>> >>> v.s.
>>> > `late-event-drop`, we should probably consider moving the scenarios
>>> >>> where
>>> > records got ignored due the window not being available any more in
>>> > windowed
>>> > aggregation operators from the `skipped-records` metrics recording
>>> to
>>> >>> the
>>> > `late-event-drop` metrics recording.
>>> >
>>> >
>>> >
>>> > Guozhang
>>> >
>>> >
>>> > On Mon, Jul 30, 2018 at 1:36 PM, Bill Bejeck 
>>> >> wrote:
>>> >
>>> >> Thanks for the KIP!
>>> >>
>>> >> +1
>>> >>
>>> >> -Bill
>>> >>
>>> >> On Mon, Jul 30, 2018 at 3:42 PM Ted Yu 
>>> wrote:
>>> >>
>>> >>> +1
>>> >>>
>>> >>> On Mon, Jul 30, 2018 at 11:46 AM John Roesle

Re: Add to contributor list

[Matthias J. Sax]

Done.

On 8/21/18 11:14 AM, Lionel Montrieux wrote:
> Hi there,
> 
> Could you please add me to the contributors list?
> My jira id is lmontrieux
> 
> Cheers,
> Lionel
> 



signature.asc
Description: OpenPGP digital signature

[jira] [Created] (KAFKA-7320) Provide ability to disable auto topic creation in KafkaConsumer

[Dhruvil Shah (JIRA)]

Dhruvil Shah created KAFKA-7320:
---

 Summary: Provide ability to disable auto topic creation in 
KafkaConsumer
 Key: KAFKA-7320
 URL: https://issues.apache.org/jira/browse/KAFKA-7320
 Project: Kafka
  Issue Type: Improvement
  Components: consumer
Reporter: Dhruvil Shah
Assignee: Dhruvil Shah


Consumers should have a configuration to control whether subscribing to 
non-existent topics should automatically create the topic or not.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-7301) KTable to KTable join invocation does not resolve in Scala DSL

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-7301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang resolved KAFKA-7301.
--
   Resolution: Fixed
Fix Version/s: 2.1.0

> KTable to KTable join invocation does not resolve in Scala DSL
> --
>
> Key: KAFKA-7301
> URL: https://issues.apache.org/jira/browse/KAFKA-7301
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 2.0.0
>Reporter: Michal
>Priority: Major
>  Labels: scala
> Fix For: 2.1.0
>
>
> I found a peculiar problem while doing KTable to KTable join using Scala DSL. 
> The following code:
>  
> {code:java}
> val t1: KTable[String, Int] = ...
> val t2: KTable[String, Int] = ...
> val result = t1.join(t2)((x: Int, y: Int) => x + y) 
> {code}
>  
> does not compile with "ambiguous reference to overloaded function". 
> A quick look at the code shows the join functions are defined as follows:
>  
> {code:java}
> def join[VO, VR](other: KTable[K, VO])(
>  joiner: (V, VO) => VR,
>  materialized: Materialized[K, VR, ByteArrayKeyValueStore]
> )
> def join[VO, VR](other: KTable[K, VO])(joiner: (V, VO) => VR)
> {code}
>  
> the reason it does not compile is the fact that the first parameter list is 
> identical. For some peculiar reason the KTable class actually compiles...
> The same problem exists for KTable to KTable leftJoin. Other joins 
> (stream-stream, stream-table) do not seem to be affected as there are no 
> overloaded versions of the functions.
> This can be reproduced in smaller scale by some simple scala code:
>  
> {code:java}
> object F {
>  //def x(a: Int): Int = 5
>  //def x(a: Int): Int = 6 //obviously does not compile
>  def f(x: Int)(y: Int): Int = x
>  def f(x: Int)(y: Int, z: Int): Int = x
> }
> val r = F.f(5)(4) //Cannot resolve
> val r2 = F.f(5)(4, 6) //cannot resolve
> val partial = F.f(5) _ //cannot resolve
> /* you get following error:
> Error: ambiguous reference to overloaded definition,
> both method f in object F of type (x: Int)(y: Int, z: Int)Int
> and method f in object F of type (x: Int)(y: Int)Int
> match argument types (Int)
> */{code}
>  
> The solution: get rid of the multiple parameter lists. I fail to see what 
> practical purpose they serve anyways. I am happy to supply appropriate PR if 
> there is agreement.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[DISCUSS] KIP-361: Add Consumer Configuration to Disable Auto Topic Creation

[Dhruvil Shah]

Hi,

I would like to start discussion on KIP-361 that proposes we add a consumer
configuration to disable auto topic creation.

Link to the KIP:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-361%3A+Add+Consumer+Configuration+to+Disable+Auto+Topic+Creation

Suggestions and feedback are welcome!

Thanks,
Dhruvil

[DISCUSS] KIP-360: Improve handling of unknown producer

[Jason Gustafson]

Hi All,

I have a proposal to improve the transactional/idempotent producer's
handling of the UNKNOWN_PRODUCER error, which is the result of losing
producer state following segment removal. The current behavior is both
complex and limited. Please take a look and let me know what you think.

Thanks in advance to Matthias Sax for feedback on the initial draft.

-Jason

Re: [ANNOUNCE] New Kafka PMC member: Dong Lin

[Jason Gustafson]

Congrats!

On Tue, Aug 21, 2018 at 10:03 AM, Ray Chiang  wrote:

> Congrats Dong!
>
> -Ray
>
>
> On 8/21/18 9:33 AM, Becket Qin wrote:
>
>> Congrats, Dong!
>>
>> On Aug 21, 2018, at 11:03 PM, Eno Thereska 
>>> wrote:
>>>
>>> Congrats Dong!
>>>
>>> Eno
>>>
>>> On Tue, Aug 21, 2018 at 7:05 AM, Ted Yu  wrote:
>>>
>>> Congratulation Dong!

 On Tue, Aug 21, 2018 at 1:59 AM Viktor Somogyi-Vass <
 viktorsomo...@gmail.com>
 wrote:

 Congrats Dong! :)
>
> On Tue, Aug 21, 2018 at 10:09 AM James Cheng 
>
 wrote:

> Congrats Dong!
>>
>> -James
>>
>> On Aug 20, 2018, at 3:54 AM, Ismael Juma  wrote:
>>>
>>> Hi everyone,
>>>
>>> Dong Lin became a committer in March 2018. Since then, he has
>>>
>> remained

> active in the community and contributed a number of patches, reviewed
>>> several pull requests and participated in numerous KIP discussions. I
>>>
>> am
>
>> happy to announce that Dong is now a member of the
>>> Apache Kafka PMC.
>>>
>>> Congratulation Dong! Looking forward to your future contributions.
>>>
>>> Ismael, on behalf of the Apache Kafka PMC
>>>
>>
>>
>

[jira] [Created] (KAFKA-7321) ensure timely processing of deletion requests in Kafka topic (Time-based log compaction)

[xiongqi wu (JIRA)]

xiongqi wu created KAFKA-7321:
-

 Summary: ensure timely processing of deletion requests in Kafka 
topic (Time-based log compaction)
 Key: KAFKA-7321
 URL: https://issues.apache.org/jira/browse/KAFKA-7321
 Project: Kafka
  Issue Type: Improvement
  Components: log
Reporter: xiongqi wu


_Compaction enables Kafka to remove old messages that are flagged for deletion 
while other messages can be retained for a relatively longer time.  Today, a 
log segment may remain un-compacted for a long time since the eligibility for 
log compaction is determined based on compaction ratio 
(“min.cleanable.dirty.ratio”) and min compaction lag ("min.compaction.lag.ms") 
setting.  Ability to delete a log message through compaction in a timely manner 
has become an important requirement in some use cases (e.g., GDPR).  For 
example,  one use case is to delete PII (Personal Identifiable information) 
data within 7 days while keeping non-PII indefinitely in compacted format.  The 
goal of this change is to provide a time-based compaction policy that ensures 
the cleanable section is compacted after the specified time interval regardless 
of dirty ratio and “min compaction lag”.  However, dirty ratio and “min 
compaction lag” are still honored if the time based compaction rule is not 
violated. In other words, if Kafka receives a deletion request on a key (e..g, 
a key with null value), the corresponding log segment will be picked up for 
compaction after the configured time interval to remove the key._

 

_This is to track effort in KIP 354:_

_https://cwiki.apache.org/confluence/display/KAFKA/KIP-354%3A+Time-based+log+compaction+policy_



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] KIP-361: Add Consumer Configuration to Disable Auto Topic Creation

[Jason Gustafson]

Hey Dhruvil,

I would suggest using the verb "allow" rather than "enable. The consumer
cannot enable auto topic creation because it is configured on the broker.
All it can do is prevent it from happening if it is enabled.

-Jason

On Tue, Aug 21, 2018 at 3:56 PM, Dhruvil Shah  wrote:

> Hi,
>
> I would like to start discussion on KIP-361 that proposes we add a consumer
> configuration to disable auto topic creation.
>
> Link to the KIP:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-361%3A+Add+Consumer+
> Configuration+to+Disable+Auto+Topic+Creation
>
> Suggestions and feedback are welcome!
>
> Thanks,
> Dhruvil
>

Jenkins build is back to normal : kafka-trunk-jdk10 #423

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-7322) race between compaction thread and retention thread when changing topic cleanup policy

[xiongqi wu (JIRA)]

xiongqi wu created KAFKA-7322:
-

 Summary: race between compaction thread and retention thread when 
changing topic cleanup policy
 Key: KAFKA-7322
 URL: https://issues.apache.org/jira/browse/KAFKA-7322
 Project: Kafka
  Issue Type: Bug
  Components: log
Reporter: xiongqi wu
Assignee: xiongqi wu


The deletion thread will grab the log.lock when it tries to rename log segment 
and schedule for actual deletion.

The compaction thread only grabs the log.lock when it tries to replace the 
original segments with the cleaned segment. The compaction thread doesn't grab 
the log when it reads records from the original segments to build offsetmap and 
new segments. As a result, if both deletion and compaction threads work on the 
same log partition. We have a race condition. 

This race happens when the topic cleanup policy is updated on the fly.  

One case to hit this race condition:

1: topic clean up policy is "compact" initially 

2: log cleaner (compaction) thread picks up the partition for compaction and 
still in progress

3: the topic clean up policy has been updated to "deletion"

4: retention thread pick up the topic partition and delete some old segments.

5: log cleaner thread reads from the deleted log and raise an IO exception. 

 

The proposed solution is to use "inprogress" map that cleaner manager has to 
protect such a race.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] KIP-291: Have separate queues for control requests and data requests

[Lucas Wang]

Thanks Becket. Following the convention of KIP-103 makes sense.
I've updated the KIP with your proposed changes. Please take another look.

Lucas

On Mon, Aug 20, 2018 at 7:29 AM Becket Qin  wrote:

> Hi Lucas,
>
> In KIP-103, we introduced a convention to define and look up the listeners.
> So it would be good if the later KIPs can follow the same convention.
>
> From what I understand, the advertised.listeners is actually designed for
> our purpose, i.e. providing a list of listeners that can be used in
> different cases. In KIP-103 it was used to separate internal traffic from
> the external traffic. It is not just for the user traffic or data
> only. So adding
> a controller listener is not repurposing the config. Also, ZK structure is
> only visible to brokers, the clients will still only see the listeners they
> are seeing today.
>
> For this KIP, we are essentially trying to separate the controller traffic
> from the inter-broker data traffic. So adding a new
> listener.name.for.controller config seems reasonable. The behavior would
> be:
> 1. If the listener.name.for.controller is set, the broker-controller
> communication will go through that listener.
> 2. Otherwise, the controller traffic falls back to use
> inter.broker.listener.name or inter.broker.security.protocol, which is the
> current behavior.
>
> Regarding updating the security protocol with one line change v.s two-lines
> change, I am a little confused, can you elaborate?
>
> Regarding the possibility of hurry and misreading. It is the system admin's
> responsibility to configure the right listener to ensure that different
> kinds of traffic are using the correct endpoints. So I think it is better
> that we always follow the same of convention instead of doing it in
> different ways.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
>
>
> On Fri, Aug 17, 2018 at 4:34 AM, Lucas Wang  wrote:
>
> > Thanks for the review, Becket.
> >
> > (1) After comparing the two approaches, I still feel the current writeup
> is
> > a little better.
> > a. The current writeup asks for an explicit endpoint while reusing the
> > existing "inter.broker.listener.name" with the exactly same semantic,
> > and your proposed change asks for a new listener name for controller
> while
> > reusing the existing "advertised.listeners" config with a slight semantic
> > change since a new controller endpoint needs to be added to it.
> > Hence conceptually the current writeup requires one config change instead
> > of two.
> > Also with one listener name, e.g. INTERNAL, for inter broker traffic,
> > instead of two, e.g. "INTERNAL" and "CONTROLLER",
> > if an operator decides to switch from PLAINTEXT to SSL for internal
> > traffic, chances are that she wants to upgrade
> > both controller connections and data connections, she only needs to
> update
> > one line in
> > the "listener.security.protocol.map" config, and avoids possible
> mistakes.
> >
> >
> > b. When this KIP is picked up by an operator who is in a hurry without
> > reading the docs, if she sees a
> > new listener name for controller is required, and chances are there is
> > already a list of listeners,
> > it's possible for her to simply choose an existing listener name, without
> > explicitly creating
> > the new CONTROLLER listener and endpoints. If this is done, Kafka will be
> > run with the existing
> > behavior, defeating the purpose of this KIP.
> > In comparison, if she sees a separate endpoint is being asked, I feel
> it's
> > unlikely for her to
> > copy and paste an existing endpoint.
> >
> > Please let me know your comments.
> >
> > (2) Good catch, it's a typo, and it's been fixed.
> >
> > Thanks,
> > Lucas
> >
>

[jira] [Reopened] (KAFKA-6753) Speed up event processing on the controller

[Lucas Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lucas Wang reopened KAFKA-6753:
---

> Speed up event processing on the controller 
> 
>
> Key: KAFKA-6753
> URL: https://issues.apache.org/jira/browse/KAFKA-6753
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Lucas Wang
>Assignee: Lucas Wang
>Priority: Minor
> Fix For: 2.1.0
>
> Attachments: Screen Shot 2018-04-04 at 7.08.55 PM.png
>
>
> The existing controller code updates metrics after processing every event. 
> This can slow down event processing on the controller tremendously. In one 
> profiling we see that updating metrics takes nearly 100% of the CPU for the 
> controller event processing thread. Specifically the slowness can be 
> attributed to two factors:
> 1. Each invocation to update the metrics is expensive. Specifically trying to 
> calculate the offline partitions count requires iterating through all the 
> partitions in the cluster to check if the partition is offline; and 
> calculating the preferred replica imbalance count requires iterating through 
> all the partitions in the cluster to check if a partition has a leader other 
> than the preferred leader. In a large cluster, the number of partitions can 
> be quite large, all seen by the controller. Even if the time spent to check a 
> single partition is small, the accumulation effect of so many partitions in 
> the cluster can make the invocation to update metrics quite expensive. One 
> might argue that maybe the logic for processing each single partition is not 
> optimized, we checked the CPU percentage of leaf nodes in the profiling 
> result, and found that inside the loops of collection objects, e.g. the set 
> of all partitions, no single function dominates the processing. Hence the 
> large number of the partitions in a cluster is the main contributor to the 
> slowness of one invocation to update the metrics.
> 2. The invocation to update metrics is called many times when the is a high 
> number of events to be processed by the controller, one invocation after 
> processing any event.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] KIP-361: Add Consumer Configuration to Disable Auto Topic Creation

[Matthias J. Sax]

Thanks for the KIP Dhruvil!

I agree with Jason's comment. An alternative might be to use "suppress"
what would revert the logic of "allow". Not sure which one is more
intuitive and I am fine with both (no personal preference). Just wanted
to mention it as an alternative.

Don't have any further comments/question so far.


-Matthias



On 8/21/18 4:42 PM, Jason Gustafson wrote:
> Hey Dhruvil,
> 
> I would suggest using the verb "allow" rather than "enable. The consumer
> cannot enable auto topic creation because it is configured on the broker.
> All it can do is prevent it from happening if it is enabled.
> 
> -Jason
> 
> On Tue, Aug 21, 2018 at 3:56 PM, Dhruvil Shah  wrote:
> 
>> Hi,
>>
>> I would like to start discussion on KIP-361 that proposes we add a consumer
>> configuration to disable auto topic creation.
>>
>> Link to the KIP:
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-361%3A+Add+Consumer+
>> Configuration+to+Disable+Auto+Topic+Creation
>>
>> Suggestions and feedback are welcome!
>>
>> Thanks,
>> Dhruvil
>>
> 



signature.asc
Description: OpenPGP digital signature

Re: [DISCUSS] KIP-110: Add Codec for ZStandard Compression (Updated)

[Jason Gustafson]

Hi Dongjin,

One of the complications is that old versions of the API will not expect a
new error code. However, since we expect this to be a fatal error anyway
for old clients, it may still be more useful to return the correct error
code. For example, the Kafka clients use the following code to convert the
error code:

public static Errors forCode(short code) {
Errors error = codeToError.get(code);
if (error != null) {
return error;
} else {
log.warn("Unexpected error code: {}.", code);
return UNKNOWN_SERVER_ERROR;
}
}

If we return an unsupported error code, it will be converted to an UNKNOWN
error, but at least we will get the message in the log with the correct
code. That seems preferable to returning a misleading error code. So I
wonder if we can use the new UNSUPPORTED_COMPRESSION_TYPE error even for
older versions.

Also, one question just to check my understanding. I think we would only
use this error code when we /know/ that zstd was in use and the client
doesn't support it? This is true if either 1) the message needs
down-conversion and we encounter a zstd compressed message, or 2) if the
topic is explicitly configured to use zstd. However, if the compression
type is set to "producer," then the fetched data may or may not be
compressed with zstd. In this case, we return the data to the client and
expect it to fail parsing. Is that correct?

Thanks,
Jason



On Tue, Aug 21, 2018 at 9:08 AM, Dongjin Lee  wrote:

> Ismael, Jason and all,
>
> I rewrote the backward compatibility strategy & its alternatives like
> following, based on Ismael & Jason's comments. Since it is not updated to
> the wiki yet, don't hesitate to give me a message if you have any opinion
> on it.
>
> ```
> *Backward Compatibility*
>
> We need to establish some backward-compatibility strategy for the case an
> old client subscribes a topic using ZStandard implicitly (i.e.,
> 'compression.type' configuration of given topic is 'producer' and the
> producer compressed the records with ZStandard). We have the following
> options for this situation:
>
> *A. Support ZStandard to the old clients which can understand v0, v1
> messages only.*
>
> This strategy necessarily requires the down-conversion of v2 message
> compressed with Zstandard into v0 or v1 messages, which means a
> considerable performance degradation. So we rejected this strategy.
>
> *B. Bump the API version and support only v2-available clients*
>
> With this approach, we can message the old clients that they are old and
> should be upgraded. However, there are still several options for the Error
> code.
>
> *B.1. INVALID_REQUEST (42)*
>
> This option gives the client so little information; the user can be
> confused about why the client worked correctly in the past suddenly
> encounters a problem. So we rejected this strategy.
>
> *B.2. CORRUPT_MESSAGE (2)*
>
> This option gives inaccurate information; the user can be surprised and
> misunderstand that the log files are broken in some way. So we rejected
> this strategy.
>
> *B.3 UNSUPPORTED_FOR_MESSAGE_FORMAT (43)*
>
> The advantage of this approach is we don't need to define a new error code;
> we can reuse it and that's all.
>
> The disadvantage of this approach is that it is also a little bit vague;
> This error code is defined as a work for KIP-98[^1] and now returned in the
> transaction error.
>
> *B.4. UNSUPPORTED_COMPRESSION_TYPE (new)*
>
> The advantage of this approach is that it is clear and provides an exact
> description. The disadvantage is we need to add a new error code.
> ```
>
> *It seems like what we need to choose is now so clear:
> UNSUPPORTED_FOR_MESSAGE_FORMAT (B.3) or UNSUPPORTED_COMPRESSION_TYPE
> (B.4).*
> The first one doesn't need a new error message but the latter is more
> explicit. Which one do you prefer? Since all of you have much more
> experience and knowledge than me, I will follow your decision. The wiki
> page will be updated following the decision also.
>
> Best,
> Dongjin
>
> [^1]: https://issues.apache.org/jira/browse/KAFKA-4990
>
> On Sun, Aug 19, 2018 at 4:58 AM Ismael Juma  wrote:
>
> > Sounds reasonable to me.
> >
> > Ismael
> >
> > On Sat, 18 Aug 2018, 12:20 Jason Gustafson,  wrote:
> >
> > > Hey Ismael,
> > >
> > > Your summary looks good to me. I think it might also be a good idea to
> > add
> > > a new UNSUPPORTED_COMPRESSION_TYPE error code to go along with the
> > version
> > > bumps. We won't be able to use it for old api versions since the
> clients
> > > will not understand it, but we can use it going forward so that we're
> not
> > > stuck in a similar situation with a new message format and a new codec
> to
> > > support. Another option is to use UNSUPPORTED_FOR_MESSAGE_FORMAT, but
> it
> > is
> > > not as explicit.
> > >
> > > -Jason
> > >
> > > On Fri, Aug 17, 2018 at 5:19 PM, Ismael Juma 
> wrote:
> > >
> > > > Hi Dongjin and Jason,
> > > >
> > > > I would agree. My summary:
> > > >
>

Re: [DISCUSS] KIP-291: Have separate queues for control requests and data requests

[Lucas Wang]

Hi Eno,

I fully agree with Becket here. If the motivation section makes sense, and
we know we can get burnt by this problem,
then the exact numbers (which vary case by case according to the config
settings and traffic pattern)
are no longer as important.

Thanks,
Lucas


On Tue, Aug 21, 2018 at 9:39 AM Becket Qin  wrote:

> Hi Eno,
>
> Thanks for the comments. This KIP is not really about improving the
> performance in general. It is about ensuring the cluster state can still be
> updated quickly even if the brokers are under heavy load.
>
> We have seen quite often that it took dozens of seconds for a broker to
> process the requests sent by the controller when the cluster is under heavy
> load. This leads to the issues Lucas mentioned in the motivation part.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
> > On Aug 20, 2018, at 11:33 PM, Eno Thereska 
> wrote:
> >
> > Hi folks,
> >
> > I looked at the previous numbers that Lucas provided (thanks!) but it's
> > still not clear to me whether the performance benefits justify the added
> > complexity. I'm looking for some intuition here (a graph would be great
> but
> > not required): for a small/medium/large cluster, what are the expected
> > percentage of control requests today that will benefit from the change?
> > It's a bit hard to go through this level of detail without knowing the
> > expected end-to-end benefit. The best folks to answer this might be ones
> > running such clusters, and ideally should pitch in with some data.
> >
> > Thanks
> > Eno
> >
> > On Mon, Aug 20, 2018 at 7:29 AM, Becket Qin 
> wrote:
> >
> >> Hi Lucas,
> >>
> >> In KIP-103, we introduced a convention to define and look up the
> listeners.
> >> So it would be good if the later KIPs can follow the same convention.
> >>
> >> From what I understand, the advertised.listeners is actually designed
> for
> >> our purpose, i.e. providing a list of listeners that can be used in
> >> different cases. In KIP-103 it was used to separate internal traffic
> from
> >> the external traffic. It is not just for the user traffic or data
> >> only. So adding
> >> a controller listener is not repurposing the config. Also, ZK structure
> is
> >> only visible to brokers, the clients will still only see the listeners
> they
> >> are seeing today.
> >>
> >> For this KIP, we are essentially trying to separate the controller
> traffic
> >> from the inter-broker data traffic. So adding a new
> >> listener.name.for.controller config seems reasonable. The behavior would
> >> be:
> >> 1. If the listener.name.for.controller is set, the broker-controller
> >> communication will go through that listener.
> >> 2. Otherwise, the controller traffic falls back to use
> >> inter.broker.listener.name or inter.broker.security.protocol, which is
> the
> >> current behavior.
> >>
> >> Regarding updating the security protocol with one line change v.s
> two-lines
> >> change, I am a little confused, can you elaborate?
> >>
> >> Regarding the possibility of hurry and misreading. It is the system
> admin's
> >> responsibility to configure the right listener to ensure that different
> >> kinds of traffic are using the correct endpoints. So I think it is
> better
> >> that we always follow the same of convention instead of doing it in
> >> different ways.
> >>
> >> Thanks,
> >>
> >> Jiangjie (Becket) Qin
> >>
> >>
> >>
> >> On Fri, Aug 17, 2018 at 4:34 AM, Lucas Wang 
> wrote:
> >>
> >>> Thanks for the review, Becket.
> >>>
> >>> (1) After comparing the two approaches, I still feel the current
> writeup
> >> is
> >>> a little better.
> >>> a. The current writeup asks for an explicit endpoint while reusing the
> >>> existing "inter.broker.listener.name" with the exactly same semantic,
> >>> and your proposed change asks for a new listener name for controller
> >> while
> >>> reusing the existing "advertised.listeners" config with a slight
> semantic
> >>> change since a new controller endpoint needs to be added to it.
> >>> Hence conceptually the current writeup requires one config change
> instead
> >>> of two.
> >>> Also with one listener name, e.g. INTERNAL, for inter broker traffic,
> >>> instead of two, e.g. "INTERNAL" and "CONTROLLER",
> >>> if an operator decides to switch from PLAINTEXT to SSL for internal
> >>> traffic, chances are that she wants to upgrade
> >>> both controller connections and data connections, she only needs to
> >> update
> >>> one line in
> >>> the "listener.security.protocol.map" config, and avoids possible
> >> mistakes.
> >>>
> >>>
> >>> b. When this KIP is picked up by an operator who is in a hurry without
> >>> reading the docs, if she sees a
> >>> new listener name for controller is required, and chances are there is
> >>> already a list of listeners,
> >>> it's possible for her to simply choose an existing listener name,
> without
> >>> explicitly creating
> >>> the new CONTROLLER listener and endpoints. If this is done, Kafka will
> be
> >>> run with the existing
> >>> behavior, defeating the p

Re: [ANNOUNCE] New Kafka PMC member: Dong Lin

[Dhruvil Shah]

Congratulations, Dong!

On Tue, Aug 21, 2018 at 4:38 PM Jason Gustafson  wrote:

> Congrats!
>
> On Tue, Aug 21, 2018 at 10:03 AM, Ray Chiang  wrote:
>
> > Congrats Dong!
> >
> > -Ray
> >
> >
> > On 8/21/18 9:33 AM, Becket Qin wrote:
> >
> >> Congrats, Dong!
> >>
> >> On Aug 21, 2018, at 11:03 PM, Eno Thereska 
> >>> wrote:
> >>>
> >>> Congrats Dong!
> >>>
> >>> Eno
> >>>
> >>> On Tue, Aug 21, 2018 at 7:05 AM, Ted Yu  wrote:
> >>>
> >>> Congratulation Dong!
> 
>  On Tue, Aug 21, 2018 at 1:59 AM Viktor Somogyi-Vass <
>  viktorsomo...@gmail.com>
>  wrote:
> 
>  Congrats Dong! :)
> >
> > On Tue, Aug 21, 2018 at 10:09 AM James Cheng 
> >
>  wrote:
> 
> > Congrats Dong!
> >>
> >> -James
> >>
> >> On Aug 20, 2018, at 3:54 AM, Ismael Juma  wrote:
> >>>
> >>> Hi everyone,
> >>>
> >>> Dong Lin became a committer in March 2018. Since then, he has
> >>>
> >> remained
> 
> > active in the community and contributed a number of patches, reviewed
> >>> several pull requests and participated in numerous KIP
> discussions. I
> >>>
> >> am
> >
> >> happy to announce that Dong is now a member of the
> >>> Apache Kafka PMC.
> >>>
> >>> Congratulation Dong! Looking forward to your future contributions.
> >>>
> >>> Ismael, on behalf of the Apache Kafka PMC
> >>>
> >>
> >>
> >
>

[jira] [Created] (KAFKA-7323) add replication factor doesn't work

[superheizai (JIRA)]

superheizai created KAFKA-7323:
--

 Summary: add replication factor doesn't work
 Key: KAFKA-7323
 URL: https://issues.apache.org/jira/browse/KAFKA-7323
 Project: Kafka
  Issue Type: Bug
  Components: controller
Affects Versions: 0.11.0.2
Reporter: superheizai


I have topic with 256 parititons.

Firstly, I generate the  topic partitions with their brokerIds with 
kafka-reassign-partitions generate.

Seconld, I add a brokerId for each partition.

Then, I run kafka-reassign-partitions, some partitions increased their 
replication factor, but the others stoped.

When I read log controller.log,  some partitions' replication factors 
increased. Then I remove these paritions which replication factor base been 
increased and run kafka-reassign-partitions again, but no log in 
controller.log, all paritions are "still in progress", no network flow changed 
when watch zabbix network.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-7324) NPE due to lack of SASLExtensions in SASL/OAUTHBEARER

[Ron Dagostino (JIRA)]

Ron Dagostino created KAFKA-7324:


 Summary: NPE due to lack of SASLExtensions in SASL/OAUTHBEARER
 Key: KAFKA-7324
 URL: https://issues.apache.org/jira/browse/KAFKA-7324
 Project: Kafka
  Issue Type: Bug
  Components: clients
Affects Versions: 2.0.1
Reporter: Ron Dagostino
Assignee: Ron Dagostino
 Fix For: 2.0.1


When there are no SASL extensions in an OAUTHBEARER request (or the callback 
handler does not support SaslExtensionsCallback) the 
OAuthBearerSaslClient.retrieveCustomExtensions() method returns null.  This 
null value is then passed to the OAuthBearerClientInitialResponse constructor, 
and that results in an NPE:

java.lang.NullPointerException
at 
org.apache.kafka.common.security.oauthbearer.internals.OAuthBearerClientInitialResponse.validateExtensions(OAuthBearerClientInitialResponse.java:115)
at 
org.apache.kafka.common.security.oauthbearer.internals.OAuthBearerClientInitialResponse.(OAuthBearerClientInitialResponse.java:81)
at 
org.apache.kafka.common.security.oauthbearer.internals.OAuthBearerClientInitialResponse.(OAuthBearerClientInitialResponse.java:75)
at 
org.apache.kafka.common.security.oauthbearer.internals.OAuthBearerSaslClient.evaluateChallenge(OAuthBearerSaslClient.java:99)




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Request for contributor permissions

[楊宗訓]

JIRA ID: jackyoh


Thanks in advance!


-- 
*
研究發展部 - 資深軟體工程師

楊宗訓 jack
e-mail:j...@is-land.com.tw

亦思科技股份有限公司  www.is-land.com.tw
300 新竹科學園區展業二路4號3樓
*

Re: [DISCUSS] KIP-361: Add Consumer Configuration to Disable Auto Topic Creation

[Ismael Juma]

Thanks for the KIP. A few questions/comments:

1. It seems hard to reason about if we just disregard the config for older
brokers. Maybe we should throw an error if the brokers don't support it and
let users explicitly change the config if they want to.

2. We probably want to switch the default and eventually remove this config
in a future version. What's the path to making that happen? One option
would be to warn if people rely on the default as a first step (or warn
every time it's used).

Ismael

On 21 Aug 2018 3:56 pm, "Dhruvil Shah"  wrote:

Hi,

I would like to start discussion on KIP-361 that proposes we add a consumer
configuration to disable auto topic creation.

Link to the KIP:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-361%3A+Add+Consumer+Configuration+to+Disable+Auto+Topic+Creation

Suggestions and feedback are welcome!

Thanks,
Dhruvil

Re: [DISCUSS] KIP-158: Kafka Connect should allow source connectors to set topic-specific settings for new topics

[Randall Hauch]

Okay, after much delay let's try this again for AK 2.1. Has anyone found
any concerns? Stephane suggested that we allow updating topic
configurations (everything but partition count). I'm unconvinced that it's
worth the additional complexity in the implementation and the documentation
to explain the behavior. Changing several of the topic-specific
configurations have significant impact on broker behavior / functionality,
so IMO we need to proceed more cautiously.

Stephane, do you have a particular use case in mind for updating topic
configurations on an existing topic?

Randall


On Fri, Jan 26, 2018 at 4:20 PM Randall Hauch  wrote:

> The KIP deadline for 1.1 has already passed, but I'd like to restart this
> discussion so that we make the next release. I've not yet addressed the
> previous comment about *existing* topics, but I'll try to do that over the
> next few weeks. Any other comments/suggestions/questions?
>
> Best regards,
>
> Randall
>
> On Thu, Oct 5, 2017 at 12:13 AM, Randall Hauch  wrote:
>
>> Oops. Yes, I meant “replication factor”.
>>
>> > On Oct 4, 2017, at 7:18 PM, Ted Yu  wrote:
>> >
>> > Randall:
>> > bq. AdminClient currently allows changing the replication factory.
>> >
>> > By 'replication factory' did you mean 'replication factor' ?
>> >
>> > Cheers
>> >
>> >> On Wed, Oct 4, 2017 at 9:58 AM, Randall Hauch 
>> wrote:
>> >>
>> >> Currently the KIP's scope is only topics that don't yet exist, and we
>> have
>> >> to cognizant of race conditions between tasks with the same connector.
>> I
>> >> think it is worthwhile to consider whether the KIP's scope should
>> expand to
>> >> also address *existing* partitions, though it may not be appropriate to
>> >> have as much control when changing the topic settings for an existing
>> >> topic. For example, changing the number of partitions (which the KIP
>> >> considers a "topic-specific setting" even though technically it is not)
>> >> shouldn't be done blindly due to the partitioning impacts, and IIRC you
>> >> can't reduce them (which we could verify before applying). Also, I
>> don't
>> >> think the AdminClient currently allows changing the replication
>> factory. I
>> >> think changing the topic configs is less problematic both from what
>> makes
>> >> sense for connectors to verify/change and from what the AdminClient
>> >> supports.
>> >>
>> >> Even if we decide that it's not appropriate to change the settings on
>> an
>> >> existing topic, I do think it's advantageous to at least notify the
>> >> connector (or task) prior to the first record sent to a given topic so
>> that
>> >> the connector can fail or issue a warning if it doesn't meet its
>> >> requirements.
>> >>
>> >> Best regards,
>> >>
>> >> Randall
>> >>
>> >> On Wed, Oct 4, 2017 at 12:52 AM, Stephane Maarek <
>> >> steph...@simplemachines.com.au> wrote:
>> >>
>> >>> Hi Randall,
>> >>>
>> >>> Thanks for the KIP. I like it
>> >>> What happens when the target topic is already created but the configs
>> do
>> >>> not match?
>> >>> i.e. wrong RF, num partitions, or missing / additional configs? Will
>> you
>> >>> attempt to apply the necessary changes or throw an error?
>> >>>
>> >>> Thanks!
>> >>> Stephane
>> >>>
>> >>>
>> >>> On 24/5/17, 5:59 am, "Mathieu Fenniak" > >
>> >>> wrote:
>> >>>
>> >>>Ah, yes, I see you a highlighted part that should've made this
>> clear
>> >>>to me the first read. :-)  Much clearer now!
>> >>>
>> >>>By the way, enjoyed your Debezium talk in NYC.
>> >>>
>> >>>Looking forward to this Kafka Connect change; it will allow me to
>> >>>remove a post-deployment tool that I hacked together for the
>> purpose
>> >>>of ensuring auto-created topics have the right config.
>> >>>
>> >>>Mathieu
>> >>>
>> >>>
>> >>>On Tue, May 23, 2017 at 11:38 AM, Randall Hauch 
>> >>> wrote:
>>  Thanks for the quick feedback, Mathieu. Yes, the first
>> >> configuration
>> >>> rule
>>  whose regex matches will be applied, and no other rules will be
>> >>> used. I've
>>  updated the KIP to try to make this more clear, but let me know if
>> >>> it's
>>  still not clear.
>> 
>>  Best regards,
>> 
>>  Randall
>> 
>>  On Tue, May 23, 2017 at 10:07 AM, Mathieu Fenniak <
>>  mathieu.fenn...@replicon.com> wrote:
>> 
>> > Hi Randall,
>> >
>> > Awesome, very much looking forward to this.
>> >
>> > It isn't 100% clear from the KIP how multiple config-based rules
>> >>> would
>> > be applied; it looks like the first configuration rule whose regex
>> > matches the topic name will be used, and no other rules will be
>> > applied.  Is that correct?  (I wasn't sure if it might cascade
>> > together multiple matching rules...)
>> >
>> > Looks great,
>> >
>> > Mathieu
>> >
>> >
>> > On Mon, May 22, 2017 at 1:43 PM, Randall Hauch 
>> >>> wrote:
>> >> Hi, all.
>> >>
>> >> We recently added the ability for Kafka Connect to create
>> >>> *internal*

Re: [DISCUSS] KIP-110: Add Codec for ZStandard Compression (Updated)

[Ismael Juma]

Jason, that's an interesting point regarding the Java client. Do we know
what clients in other languages do in these cases?

Ismael

On Tue, 21 Aug 2018, 17:30 Jason Gustafson,  wrote:

> Hi Dongjin,
>
> One of the complications is that old versions of the API will not expect a
> new error code. However, since we expect this to be a fatal error anyway
> for old clients, it may still be more useful to return the correct error
> code. For example, the Kafka clients use the following code to convert the
> error code:
>
> public static Errors forCode(short code) {
> Errors error = codeToError.get(code);
> if (error != null) {
> return error;
> } else {
> log.warn("Unexpected error code: {}.", code);
> return UNKNOWN_SERVER_ERROR;
> }
> }
>
> If we return an unsupported error code, it will be converted to an UNKNOWN
> error, but at least we will get the message in the log with the correct
> code. That seems preferable to returning a misleading error code. So I
> wonder if we can use the new UNSUPPORTED_COMPRESSION_TYPE error even for
> older versions.
>
> Also, one question just to check my understanding. I think we would only
> use this error code when we /know/ that zstd was in use and the client
> doesn't support it? This is true if either 1) the message needs
> down-conversion and we encounter a zstd compressed message, or 2) if the
> topic is explicitly configured to use zstd. However, if the compression
> type is set to "producer," then the fetched data may or may not be
> compressed with zstd. In this case, we return the data to the client and
> expect it to fail parsing. Is that correct?
>
> Thanks,
> Jason
>
>
>
> On Tue, Aug 21, 2018 at 9:08 AM, Dongjin Lee  wrote:
>
> > Ismael, Jason and all,
> >
> > I rewrote the backward compatibility strategy & its alternatives like
> > following, based on Ismael & Jason's comments. Since it is not updated to
> > the wiki yet, don't hesitate to give me a message if you have any opinion
> > on it.
> >
> > ```
> > *Backward Compatibility*
> >
> > We need to establish some backward-compatibility strategy for the case an
> > old client subscribes a topic using ZStandard implicitly (i.e.,
> > 'compression.type' configuration of given topic is 'producer' and the
> > producer compressed the records with ZStandard). We have the following
> > options for this situation:
> >
> > *A. Support ZStandard to the old clients which can understand v0, v1
> > messages only.*
> >
> > This strategy necessarily requires the down-conversion of v2 message
> > compressed with Zstandard into v0 or v1 messages, which means a
> > considerable performance degradation. So we rejected this strategy.
> >
> > *B. Bump the API version and support only v2-available clients*
> >
> > With this approach, we can message the old clients that they are old and
> > should be upgraded. However, there are still several options for the
> Error
> > code.
> >
> > *B.1. INVALID_REQUEST (42)*
> >
> > This option gives the client so little information; the user can be
> > confused about why the client worked correctly in the past suddenly
> > encounters a problem. So we rejected this strategy.
> >
> > *B.2. CORRUPT_MESSAGE (2)*
> >
> > This option gives inaccurate information; the user can be surprised and
> > misunderstand that the log files are broken in some way. So we rejected
> > this strategy.
> >
> > *B.3 UNSUPPORTED_FOR_MESSAGE_FORMAT (43)*
> >
> > The advantage of this approach is we don't need to define a new error
> code;
> > we can reuse it and that's all.
> >
> > The disadvantage of this approach is that it is also a little bit vague;
> > This error code is defined as a work for KIP-98[^1] and now returned in
> the
> > transaction error.
> >
> > *B.4. UNSUPPORTED_COMPRESSION_TYPE (new)*
> >
> > The advantage of this approach is that it is clear and provides an exact
> > description. The disadvantage is we need to add a new error code.
> > ```
> >
> > *It seems like what we need to choose is now so clear:
> > UNSUPPORTED_FOR_MESSAGE_FORMAT (B.3) or UNSUPPORTED_COMPRESSION_TYPE
> > (B.4).*
> > The first one doesn't need a new error message but the latter is more
> > explicit. Which one do you prefer? Since all of you have much more
> > experience and knowledge than me, I will follow your decision. The wiki
> > page will be updated following the decision also.
> >
> > Best,
> > Dongjin
> >
> > [^1]: https://issues.apache.org/jira/browse/KAFKA-4990
> >
> > On Sun, Aug 19, 2018 at 4:58 AM Ismael Juma  wrote:
> >
> > > Sounds reasonable to me.
> > >
> > > Ismael
> > >
> > > On Sat, 18 Aug 2018, 12:20 Jason Gustafson, 
> wrote:
> > >
> > > > Hey Ismael,
> > > >
> > > > Your summary looks good to me. I think it might also be a good idea
> to
> > > add
> > > > a new UNSUPPORTED_COMPRESSION_TYPE error code to go along with the
> > > version
> > > > bumps. We won't be able to use it for old api versions since the
> > clients
> > > > will

Re: Request for contributor permissions

[Guozhang Wang]

Hello Jack,

I've added you to the contributor list. Cheers.

Guozhang

2018-08-21 19:07 GMT-07:00 楊宗訓 :

> JIRA ID: jackyoh
>
>
> Thanks in advance!
>
>
> --
> *
> 研究發展部 - 資深軟體工程師
>
> 楊宗訓 jack
> e-mail:j...@is-land.com.tw
>
> 亦思科技股份有限公司  www.is-land.com.tw
> 300 新竹科學園區展業二路4號3樓
> *
>



-- 
-- Guozhang

Re: [ANNOUNCE] New Kafka PMC member: Dong Lin

[Abhimanyu Nagrath]

Congratulations, Dong!

On Wed, Aug 22, 2018 at 6:20 AM Dhruvil Shah  wrote:

> Congratulations, Dong!
>
> On Tue, Aug 21, 2018 at 4:38 PM Jason Gustafson 
> wrote:
>
> > Congrats!
> >
> > On Tue, Aug 21, 2018 at 10:03 AM, Ray Chiang  wrote:
> >
> > > Congrats Dong!
> > >
> > > -Ray
> > >
> > >
> > > On 8/21/18 9:33 AM, Becket Qin wrote:
> > >
> > >> Congrats, Dong!
> > >>
> > >> On Aug 21, 2018, at 11:03 PM, Eno Thereska 
> > >>> wrote:
> > >>>
> > >>> Congrats Dong!
> > >>>
> > >>> Eno
> > >>>
> > >>> On Tue, Aug 21, 2018 at 7:05 AM, Ted Yu  wrote:
> > >>>
> > >>> Congratulation Dong!
> > 
> >  On Tue, Aug 21, 2018 at 1:59 AM Viktor Somogyi-Vass <
> >  viktorsomo...@gmail.com>
> >  wrote:
> > 
> >  Congrats Dong! :)
> > >
> > > On Tue, Aug 21, 2018 at 10:09 AM James Cheng  >
> > >
> >  wrote:
> > 
> > > Congrats Dong!
> > >>
> > >> -James
> > >>
> > >> On Aug 20, 2018, at 3:54 AM, Ismael Juma 
> wrote:
> > >>>
> > >>> Hi everyone,
> > >>>
> > >>> Dong Lin became a committer in March 2018. Since then, he has
> > >>>
> > >> remained
> > 
> > > active in the community and contributed a number of patches,
> reviewed
> > >>> several pull requests and participated in numerous KIP
> > discussions. I
> > >>>
> > >> am
> > >
> > >> happy to announce that Dong is now a member of the
> > >>> Apache Kafka PMC.
> > >>>
> > >>> Congratulation Dong! Looking forward to your future
> contributions.
> > >>>
> > >>> Ismael, on behalf of the Apache Kafka PMC
> > >>>
> > >>
> > >>
> > >
> >
>

Re: Request for contributor permissions

[楊宗訓]

Thank you.

2018-08-22 12:00 GMT+08:00 Guozhang Wang :

> Hello Jack,
>
> I've added you to the contributor list. Cheers.
>
> Guozhang
>
> 2018-08-21 19:07 GMT-07:00 楊宗訓 :
>
> > JIRA ID: jackyoh
> >
> >
> > Thanks in advance!
> >
> >
> > --
> > *
> > 研究發展部 - 資深軟體工程師
> >
> > 楊宗訓 jack
> > e-mail:j...@is-land.com.tw
> >
> > 亦思科技股份有限公司  www.is-land.com.tw
> > 300 新竹科學園區展業二路4號3樓
> > *
> >
>
>
>
> --
> -- Guozhang
>



-- 
*
研究發展部 - 資深軟體工程師

楊宗訓 jack
e-mail:j...@is-land.com.tw

亦思科技股份有限公司  www.is-land.com.tw
300 新竹科學園區展業二路4號3樓
*

Build failed in Jenkins: kafka-trunk-jdk8 #2915

[Apache Jenkins Server]

See 


Changes:

[wangguoz] MINOR: Small refactorings on KTable joins (#5540)

[wangguoz] KAFKA-7301: Fix streams Scala join ambiguous overload (#5502)

--
[...truncated 2.48 MB...]
org.apache.kafka.streams.processor.internals.ProcessorStateManagerTest > 
shouldWriteCheckpointForPersistentLogEnabledStore PASSED

org.apache.kafka.streams.processor.internals.ProcessorStateManagerTest > 
testRegisterPersistentStore STARTED

org.apache.kafka.streams.processor.internals.ProcessorStateManagerTest > 
testRegisterPersistentStore PASSED

org.apache.kafka.streams.processor.internals.ProcessorStateManagerTest > 
shouldThrowIllegalArgumentExceptionOnRegisterWhenStoreHasAlreadyBeenRegistered 
STARTED

org.apache.kafka.streams.processor.internals.ProcessorStateManagerTest > 
shouldThrowIllegalArgumentExceptionOnRegisterWhenStoreHasAlreadyBeenRegistered 
PASSED

org.apache.kafka.streams.processor.internals.ProcessorStateManagerTest > 
shouldRestoreStoreWithSinglePutRestoreSpecification STARTED

org.apache.kafka.streams.processor.internals.ProcessorStateManagerTest > 
shouldRestoreStoreWithSinglePutRestoreSpecification PASSED

org.apache.kafka.streams.processor.internals.ProcessorStateManagerTest > 
shouldNotChangeOffsetsIfAckedOffsetsIsNull STARTED

org.apache.kafka.streams.processor.internals.ProcessorStateManagerTest > 
shouldNotChangeOffsetsIfAckedOffsetsIsNull PASSED

org.apache.kafka.streams.processor.internals.ProcessorStateManagerTest > 
shouldThrowIllegalArgumentExceptionIfStoreNameIsSameAsCheckpointFileName STARTED

org.apache.kafka.streams.processor.internals.ProcessorStateManagerTest > 
shouldThrowIllegalArgumentExceptionIfStoreNameIsSameAsCheckpointFileName PASSED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldCloseStateManagerWithOffsets STARTED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldCloseStateManagerWithOffsets PASSED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldProcessRecordsForTopic STARTED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldProcessRecordsForTopic PASSED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldInitializeProcessorTopology STARTED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldInitializeProcessorTopology PASSED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldInitializeContext STARTED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldInitializeContext PASSED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldCheckpointOffsetsWhenStateIsFlushed STARTED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldCheckpointOffsetsWhenStateIsFlushed PASSED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldThrowStreamsExceptionWhenKeyDeserializationFails STARTED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldThrowStreamsExceptionWhenKeyDeserializationFails PASSED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldInitializeStateManager STARTED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldInitializeStateManager PASSED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldProcessRecordsForOtherTopic STARTED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldProcessRecordsForOtherTopic PASSED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldNotThrowStreamsExceptionWhenValueDeserializationFails STARTED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldNotThrowStreamsExceptionWhenValueDeserializationFails PASSED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldThrowStreamsExceptionWhenValueDeserializationFails STARTED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldThrowStreamsExceptionWhenValueDeserializationFails PASSED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldNotThrowStreamsExceptionWhenKeyDeserializationFailsWithSkipHandler STARTED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldNotThrowStreamsExceptionWhenKeyDeserializationFailsWithSkipHandler PASSED

org.apache.kafka.streams.processor.internals.StreamsMetricsImplTest > 
testThroughputMetrics STARTED

org.apache.kafka.streams.processor.internals.StreamsMetricsImplTest > 
testThroughputMetrics PASSED

org.apache.kafka.streams.processor.internals.StreamsMetricsImplTest > 
testLatencyMetrics STARTED

org.apache.kafka.streams.processor.internals.StreamsMetricsImplTest > 
testLatencyMetrics PASSED

org.apache.kafka.streams.processor.internals.StreamsMetricsImplTest > 
testRemoveSensor STARTED

org.apache.kafka.streams.processor.internals.StreamsMetricsIm

Re: [DISCUSS] KIP-358: Migrate Streams API to Duration instead of long ms times

[Nikolay Izhikov]

Dear, commiters.

Please, pay attention to this KIP and share your opinion.

В Вт, 21/08/2018 в 11:14 -0500, John Roesler пишет:
> I'll solicit more reviews. Let's get at least one committer to chime in
> before we start a vote (since we need their approval anyway).
> -John
> 
> On Mon, Aug 20, 2018 at 12:39 PM Nikolay Izhikov 
> wrote:
> 
> > Hello, Ted.
> > 
> > Thanks for the comment.
> > 
> > I've edit KIP and change proposal to `windowSize`.
> > 
> > Guys, any other comments?
> > 
> > 
> > В Вс, 19/08/2018 в 14:57 -0700, Ted Yu пишет:
> > > bq. // or just Duration windowSize();
> > > 
> > > +1 to the above choice.
> > > The duration is obvious from the return type. For getter methods, we
> > 
> > don't
> > > use get as prefix (as least for new code).
> > > 
> > > Cheers
> > > 
> > > On Sun, Aug 19, 2018 at 8:03 AM Nikolay Izhikov 
> > 
> > wrote:
> > > 
> > > > Hello, John.
> > > > 
> > > > Thank you very much for your feedback!
> > > > I've addressed all your comments.
> > > > Please, see my answers and let my know is anything in KIP [1] needs to
> > 
> > be
> > > > improved.
> > > > 
> > > > > The correct choice is actually "Instant", not> "LocalDateTime"
> > > > 
> > > > I've changed the methods proposed in KIP [1] to use Instant.
> > > > 
> > > > > I noticed some recent APIs are> missing (see KIP-328)
> > > > > those APIs were just added and have never been released... you can
> > 
> > just
> > > > 
> > > > replace them.
> > > > 
> > > > I've added new methods to KIP [1].
> > > > Not released methods marked for remove.
> > > > 
> > > > > any existing method that's already deprecated, don't bother
> > > > 
> > > > transitioning it to Duration.
> > > > 
> > > > Fixed.
> > > > 
> > > > > IllegalArgumentException... we should plan to mention this in the
> > > > 
> > > > javadoc for those methods.
> > > > 
> > > > Got it.
> > > > 
> > > > > In Stores, windowSize and segmentInterval should also be durations.
> > > > 
> > > > Fixed.
> > > > 
> > > > > StreamsMetrics, recordLatency ... this one is better left alone.
> > > > 
> > > > OK. I removed this method from KIP [1].
> > > > 
> > > > Two more questions question about implementation:
> > > > 
> > > > 1. We have serveral methods without parameters.
> > > > In java we can't have two methods with parameters with the same name.
> > > > It wouldn't compile.
> > > > So we have to rename new methods. Please, see suggested names and share
> > > > your thoughts:
> > > > 
> > > > Windows {
> > > > long size() -> Duration windowSize();
> > > > }
> > > > 
> > > > Window {
> > > > long start() -> Instant startTime();
> > > > long end() -> Instant endTime();
> > > > }
> > > > 
> > > > SessionWindows {
> > > > long inactivityGap() -> Duration inactivityGapDuration();
> > > > }
> > > > 
> > > > TimeWindowedDeserializer {
> > > > Long getWindowSize() -> Duration getWindowSizeDuration(); // or
> > 
> > just
> > > > Duration windowSize();
> > > > }
> > > > 
> > > > SessionBytesStoreSupplier {
> > > > long retentionPeriod() -> Duration retentionPeriodDuration();
> > > > }
> > > > 
> > > > WindowBytesStoreSupplier {
> > > > long windowSize() -> Duration windowSizeDuration();
> > > > long retentionPeriod() -> Duration retentionPeriodDuration();
> > > > }
> > > > 
> > > > 2. Do we want to use Duration and Instant inside API implementations?
> > > > 
> > > > IGNITE-7277: "Durations potentially worsen memory pressure and gc
> > > > performance, so internally, we will still use longMs as the
> > 
> > representation."
> > > > IGNITE-7222: Duration used to store retention.
> > > > 
> > > > [1]
> > > > 
> > 
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-358%3A+Migrate+Streams+API+to+Duration+instead+of+long+ms+times
> > > > [2]
> > > > 
> > 
> > https://github.com/apache/kafka/commit/b3771ba22acad7870e38ff7f58820c5b50946787#diff-47289575d3e3e2449f27b3a7b6788e1aR64
> > > > 
> > > > В Пт, 17/08/2018 в 14:46 -0500, John Roesler пишет:
> > > > > Hi Nikolay,
> > > > > 
> > > > > Thanks for this very nice KIP!
> > > > > 
> > > > > To answer your questions:
> > > > > 1. Correct, we should not delete existing methods that have been
> > > > 
> > > > released,
> > > > > but ...
> > > > > 
> > > > > 2. Yes, we should deprecate the 'long' variants so that we can drop
> > 
> > them
> > > > > later on. Personally, I like to mention which version deprecated the
> > > > 
> > > > method
> > > > > so everyone can see later on how long it's been deprecated, but this
> > 
> > may
> > > > 
> > > > be
> > > > > controversial, so let's let other weigh in.
> > > > > 
> > > > > 3. I think you're asking whether it's appropriate to drop the "Ms"
> > > > 
> > > > suffix,
> > > > > and I think yes. So "long inactivityGapMs" would become "Duration
> > > > > inactivityGap".
> > > > > In the places where the parameter's name is just "duration", I think
> > 
> > we
> > > > 
> > > > can
> > > > > pick something more descriptive (I realize it was already
> > 
> > "durati