Re: [VOTE] 1.1.0 RC1

[Damian Guy]

Hi Jeff,

Thanks, we will look into this.

Regards,
Damian

On Thu, 8 Mar 2018 at 18:27 Jeff Chao  wrote:

> Hello,
>
> We at Heroku have run 1.1.0 RC1 through our normal performance and
> regression test suite and have found performance to be comparable to 1.0.0.
>
> That said, we're however -1 (non-binding) since this release includes
> Zookeeper 3.4.11  which
> is affected by the critical regression ZOOKEEPER-2960
> . As 3.4.12 isn't
> released yet, it might be better to have 3.4.10 included instead.
>
> Jeff
> Heroku
>
>
> On Tue, Mar 6, 2018 at 1:19 PM, Ted Yu  wrote:
>
> > +1
> >
> > Checked signature
> > Ran test suite - apart from flaky testMetricsLeak, other tests passed.
> >
> > On Tue, Mar 6, 2018 at 2:45 AM, Damian Guy  wrote:
> >
> > > Hello Kafka users, developers and client-developers,
> > >
> > > This is the second candidate for release of Apache Kafka 1.1.0.
> > >
> > > This is minor version release of Apache Kakfa. It Includes 29 new KIPs.
> > > Please see the release plan for more details:
> > >
> > > https://cwiki.apache.org/confluence/pages/viewpage.
> > action?pageId=71764913
> > >
> > > A few highlights:
> > >
> > > * Significant Controller improvements (much faster and session
> expiration
> > > edge cases fixed)
> > > * Data balancing across log directories (JBOD)
> > > * More efficient replication when the number of partitions is large
> > > * Dynamic Broker Configs
> > > * Delegation tokens (KIP-48)
> > > * Kafka Streams API improvements (KIP-205 / 210 / 220 / 224 / 239)
> > >
> > > Release notes for the 1.1.0 release:
> > > http://home.apache.org/~damianguy/kafka-1.1.0-rc1/RELEASE_NOTES.html
> > >
> > > *** Please download, test and vote by Friday, March 9th, 5pm PT
> > >
> > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > http://kafka.apache.org/KEYS
> > >
> > > * Release artifacts to be voted upon (source and binary):
> > > http://home.apache.org/~damianguy/kafka-1.1.0-rc1/
> > >
> > > * Maven artifacts to be voted upon:
> > > https://repository.apache.org/content/groups/staging/
> > >
> > > * Javadoc:
> > > http://home.apache.org/~damianguy/kafka-1.1.0-rc1/javadoc/
> > >
> > > * Tag to be voted upon (off 1.1 branch) is the 1.1.0 tag:
> > > https://github.com/apache/kafka/tree/1.1.0-rc1
> > >
> > >
> > > * Documentation:
> > > http://kafka.apache.org/11/documentation.html
> > >
> > > * Protocol:
> > > http://kafka.apache.org/11/protocol.html
> > >
> > > * Successful Jenkins builds for the 1.1 branch:
> > > Unit/integration tests:
> https://builds.apache.org/job/kafka-1.1-jdk7/68
> > > System tests: https://jenkins.confluent.io/
> > job/system-test-kafka/job/1.1/
> > > 30/
> > >
> > > /**
> > >
> > > Thanks,
> > > Damian Guy
> > >
> >
>

Re: Request to be added to the contributor list so that i can assign JIRAs to myself

[Sanket Band]

Its the same - sband. I did not have an account on apache jira so I created
one with the same ID.

Thanks
Sanket

On Fri, 9 Mar 2018, 12:40 Matthias J. Sax,  wrote:

> I meant your JIRA ID -- sorry for the confusion.
>
>
> -Matthias
>
> On 3/8/18 10:48 PM, Sanket Band wrote:
> > you mean the github id ? it is sband
> >
> > Thanks
> > Sanket Band
> >
> > On Fri, Mar 9, 2018 at 11:22 AM, Matthias J. Sax 
> > wrote:
> >
> >> What is your user ID?
> >>
> >> -Matthias
> >>
> >> On 3/8/18 8:11 PM, Sanket Band wrote:
> >>> Thanks
> >>> Sanket
> >>>
> >>
> >>
> >
>
>

[jira] [Created] (KAFKA-6632) Very slow hashCode methods in Kafka Connect types

[JIRA]

Maciej Bryński created KAFKA-6632:
-

 Summary: Very slow hashCode methods in Kafka Connect types
 Key: KAFKA-6632
 URL: https://issues.apache.org/jira/browse/KAFKA-6632
 Project: Kafka
  Issue Type: Bug
Affects Versions: 1.0.0
Reporter: Maciej Bryński


hashCode method of ConnectSchema (and Field) is used a lot in SMT.

Example:

[https://github.com/apache/kafka/blob/e5d6c9a79a4ca9b82502b8e7f503d86ddaddb7fb/connect/transforms/src/main/java/org/apache/kafka/connect/transforms/InsertField.java#L164]

Unfortunately it's using Objects.hash which is very slow.

I rewrite this to own implementation and gain 6x speedup.

Microbencharks gives:
 * Original ConnectSchema hashCode: 2995ms
 * My implementation: 517ms

(1 iterations of calculating: hashCode for on new 
ConnectSchema(Schema.Type.STRING))
{code:java}
@Override
public int hashCode() {
int result = 5;
result = 31 * result + type.hashCode();
result = 31 * result + (optional ? 1 : 0);
result = 31 * result + (defaultValue == null ? 0 : defaultValue.hashCode());
if (fields != null) {
for (Field f : fields) {
result = 31 * result + f.hashCode();
}
}
result = 31 * result + (keySchema == null ? 0 : keySchema.hashCode());
result = 31 * result + (valueSchema == null ? 0 : valueSchema.hashCode());
result = 31 * result + (name == null ? 0 : name.hashCode());
result = 31 * result + (version == null ? 0 : version);
result = 31 * result + (doc == null ? 0 : doc.hashCode());
if (parameters != null) {
for (String s : parameters.keySet()) {
result = 31 * result + s.hashCode() + parameters.get(s).hashCode();
}
}
return result;
}{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-6633) Is KafkaProducer still thread safe in version 1.0.1

[Herbert Koelman (JIRA)]

Herbert Koelman created KAFKA-6633:
--

 Summary: Is KafkaProducer still thread safe in version 1.0.1
 Key: KAFKA-6633
 URL: https://issues.apache.org/jira/browse/KAFKA-6633
 Project: Kafka
  Issue Type: Bug
  Components: clients
Affects Versions: 1.0.1
Reporter: Herbert Koelman


The javadoc of version 
[0.8|https://kafka.apache.org/082/javadoc/org/apache/kafka/clients/producer/KafkaProducer.html]
 states that producers are thread safe:
{quote}{{The producer is _thread safe_ and should generally be shared among all 
threads for best performance.}}
{quote}

Is it still the case in version 1.0.1 ? I failed to find this information in 
the javadoc of version 1.0.1.

Can I share one producer with many threads ?

(I posted this question as a bug, because I didn't know where else I could post 
questions. Sorry)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-6633) Is KafkaProducer still thread safe in version 1.0.1

[Mickael Maison (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mickael Maison resolved KAFKA-6633.
---
Resolution: Not A Problem

> Is KafkaProducer still thread safe in version 1.0.1
> ---
>
> Key: KAFKA-6633
> URL: https://issues.apache.org/jira/browse/KAFKA-6633
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 1.0.1
>Reporter: Herbert Koelman
>Priority: Minor
>
> The javadoc of version 
> [0.8|https://kafka.apache.org/082/javadoc/org/apache/kafka/clients/producer/KafkaProducer.html]
>  states that producers are thread safe:
> {quote}{{The producer is _thread safe_ and should generally be shared among 
> all threads for best performance.}}
> {quote}
> Is it still the case in version 1.0.1 ? I failed to find this information in 
> the javadoc of version 1.0.1.
> Can I share one producer with many threads ?
> (I posted this question as a bug, because I didn't know where else I could 
> post questions. Sorry)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] KIP-253: Support in-order message delivery with partition expansion

[Clemens Valiente]

I think it's fair to assume that topics will always be increased by an integer 
factor - e.g. from 2 partitions to 4 partitions. Then the mapping is much 
easier.

Why anyone would increase partitions by lass than x2 is a mystery to me. If 
your two partitions cannot handle the load, then with three partitions each one 
will still get 67% of that load which is still way too dangerous.


So in your case we go from

part1: A B C D

part2: E F G H


to


part1: A C

part2: B D

part3: E F

part4: G H



From: Matthias J. Sax 
Sent: 09 March 2018 07:53
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-253: Support in-order message delivery with 
partition expansion

@Jan: You suggest to copy the data from one topic to a new topic, and
provide an "offset mapping" from the old to the new topic for the
consumers. I don't quite understand how this would work.

Let's say there are 2 partitions in the original topic and 3 partitions
in the new topic. If we assume that we don't use linear hashing as you
suggest, there is no guarantee how data will be distributed in the new
topic and also no guarantee about ordering of records in the new topic.

Example (I hope I got it right -- please correct me if it's wrong)

A B C D
E F G H

could be copied to:

A C H
B E F
D G

If the consumer was at offset 1 and 2 in the first topic how would the
mapping be computed? We need to enures that B C D as well as G H are
read after the switch. Thus, offset would need to be 1 0 0. I am not
sure how this would be computed?

Furthermore, I want to point out that the new offsets would imply that E
is consumed a second time by the consumer. E and F were consumed
originally, but E is copied after B that was not yet consumed.

Or is there a way that we can ensure that this "flip" does never happen
while we copy the data?


-Matthias



On 3/8/18 10:32 PM, Matthias J. Sax wrote:
> As I just mentioned joins:
>
> For Kafka Streams it might also be required to change the partition
> count for multiple topics in a coordinated way that allows to maintain
> the co-partitioning property that Kafka Streams uses to computed joins.
>
> Any thoughts how this could be handled?
>
>
> -Matthias
>
> On 3/8/18 10:08 PM, Matthias J. Sax wrote:
>> Jun,
>>
>> There is one more case: non-windowed aggregations. For windowed
>> aggregation, the changelog topic will be compact+delete. However, for
>> non-windowed aggregation the policy is compact only.
>>
>> Even if we assume that windowed aggregations are dominant and
>> non-windowed aggregation are used rarely, it seems to be bad to not
>> support the feature is a non-windowed aggregation is used. Also,
>> non-windowed aggregation volume depends on input-stream volume that
>> might be high.
>>
>> Furthermore, we support stream-table join and this requires that the
>> stream and the table are co-partitioned. Thus, even if the table would
>> have lower volume but the stream must be scaled out, we also need to
>> scale out the table to preserve co-partitioning.
>>
>>
>> -Matthias
>>
>> On 3/8/18 6:44 PM, Jun Rao wrote:
>>> Hi, Matthis,
>>>
>>> My understanding is that in KStream, the only case when a changelog topic
>>> needs to be compacted is when the corresponding input is a KTable. In all
>>> other cases, the changelog topics are of compacted + deletion. So, if most
>>> KTables are not high volume, there may not be a need to expand its
>>> partitions and therefore the partitions of the corresponding changelog
>>> topic.
>>>
>>> Thanks,
>>>
>>> Jun
>>>
>>> On Wed, Mar 7, 2018 at 2:34 PM, Matthias J. Sax 
>>> wrote:
>>>
 Jun,

 thanks for your comment. This should actually work for Streams, because
 we don't rely on producer "hashing" but specify the partition number
 explicitly on send().

 About not allowing to change the number of partition for changelog
 topics: for Streams, this seems to imply that we need to create a second
 changelog topic for each store with the new partition count. However, it
 would be unclear when/if we can delete the old topic. Thus, it moves the
 "problem" into the application layer. It's hard to judge for me atm what
 the impact would be, but it's something we should pay attention to.


 -Matthias

 On 3/6/18 3:45 PM, Jun Rao wrote:
> Hi, Mattias,
>
> Regarding your comment "If it would be time-delay based, it might be
> problematic
> for Kafka Streams: if we get the information that the new input
 partitions
> are available for producing, we need to enable the new changelog
 partitions
> for producing, too. If those would not be available yet, because the
> time-delay did not trigger yet, it would be problematic to avoid
> crashing.", could you just enable the changelog topic to write to its new
> partitions immediately?  The input topic can be configured with a delay
 in
> writing to the new partitions. Initially, there won't b

Want to contribute

[Eduardo Mello]

Hello,
I want to begin to code for open sources projects in my free time and I
want to help the Apache Kafka!
I already have read the How to Contribute and Code Guidelines.
Do u guys have a label for "easy" Jira's issue or something like this?

Att,
Eduardo Mello

Re: Want to contribute

[Damian Guy]

Hi Eduardo,

There are usually JIRAs that have the newbie label. This would be a good
place to start.

Cheers,
Damian

On Fri, 9 Mar 2018 at 13:57 Eduardo Mello  wrote:

> Hello,
> I want to begin to code for open sources projects in my free time and I
> want to help the Apache Kafka!
> I already have read the How to Contribute and Code Guidelines.
> Do u guys have a label for "easy" Jira's issue or something like this?
>
> Att,
> Eduardo Mello
>

Re: Want to contribute

[Sanket Band]

Hi Damian, Matthias

Could you add me to the KAFKA project on jira, my jira id is sband

Thanks
Sanket

On Fri, 9 Mar 2018, 20:23 Damian Guy,  wrote:

> Hi Eduardo,
>
> There are usually JIRAs that have the newbie label. This would be a good
> place to start.
>
> Cheers,
> Damian
>
> On Fri, 9 Mar 2018 at 13:57 Eduardo Mello  wrote:
>
> > Hello,
> > I want to begin to code for open sources projects in my free time and I
> > want to help the Apache Kafka!
> > I already have read the How to Contribute and Code Guidelines.
> > Do u guys have a label for "easy" Jira's issue or something like this?
> >
> > Att,
> > Eduardo Mello
> >
>

Re: Want to contribute

[Damian Guy]

Hi Sanket,

Done.
Thanks,
Damian

On Fri, 9 Mar 2018 at 14:56 Sanket Band  wrote:

> Hi Damian, Matthias
>
> Could you add me to the KAFKA project on jira, my jira id is sband
>
> Thanks
> Sanket
>
> On Fri, 9 Mar 2018, 20:23 Damian Guy,  wrote:
>
> > Hi Eduardo,
> >
> > There are usually JIRAs that have the newbie label. This would be a good
> > place to start.
> >
> > Cheers,
> > Damian
> >
> > On Fri, 9 Mar 2018 at 13:57 Eduardo Mello  wrote:
> >
> > > Hello,
> > > I want to begin to code for open sources projects in my free time and I
> > > want to help the Apache Kafka!
> > > I already have read the How to Contribute and Code Guidelines.
> > > Do u guys have a label for "easy" Jira's issue or something like this?
> > >
> > > Att,
> > > Eduardo Mello
> > >
> >
>

Re: Want to contribute

[Sanket Band]

That was quick ! Thanks

Thanks
Sanket

On Fri, Mar 9, 2018 at 8:29 PM, Damian Guy  wrote:

> Hi Sanket,
>
> Done.
> Thanks,
> Damian
>
> On Fri, 9 Mar 2018 at 14:56 Sanket Band  wrote:
>
> > Hi Damian, Matthias
> >
> > Could you add me to the KAFKA project on jira, my jira id is sband
> >
> > Thanks
> > Sanket
> >
> > On Fri, 9 Mar 2018, 20:23 Damian Guy,  wrote:
> >
> > > Hi Eduardo,
> > >
> > > There are usually JIRAs that have the newbie label. This would be a
> good
> > > place to start.
> > >
> > > Cheers,
> > > Damian
> > >
> > > On Fri, 9 Mar 2018 at 13:57 Eduardo Mello 
> wrote:
> > >
> > > > Hello,
> > > > I want to begin to code for open sources projects in my free time
> and I
> > > > want to help the Apache Kafka!
> > > > I already have read the How to Contribute and Code Guidelines.
> > > > Do u guys have a label for "easy" Jira's issue or something like
> this?
> > > >
> > > > Att,
> > > > Eduardo Mello
> > > >
> > >
> >
>

Re: Want to contribute

[Eduardo Mello]

Thank you!

Ill look for this label!

I created a Jira user too: emello

Thanks,
Eduardo Mello

On Fri, Mar 9, 2018 at 12:00 PM, Sanket Band 
wrote:

> That was quick ! Thanks
>
> Thanks
> Sanket
>
> On Fri, Mar 9, 2018 at 8:29 PM, Damian Guy  wrote:
>
> > Hi Sanket,
> >
> > Done.
> > Thanks,
> > Damian
> >
> > On Fri, 9 Mar 2018 at 14:56 Sanket Band  wrote:
> >
> > > Hi Damian, Matthias
> > >
> > > Could you add me to the KAFKA project on jira, my jira id is sband
> > >
> > > Thanks
> > > Sanket
> > >
> > > On Fri, 9 Mar 2018, 20:23 Damian Guy,  wrote:
> > >
> > > > Hi Eduardo,
> > > >
> > > > There are usually JIRAs that have the newbie label. This would be a
> > good
> > > > place to start.
> > > >
> > > > Cheers,
> > > > Damian
> > > >
> > > > On Fri, 9 Mar 2018 at 13:57 Eduardo Mello 
> > wrote:
> > > >
> > > > > Hello,
> > > > > I want to begin to code for open sources projects in my free time
> > and I
> > > > > want to help the Apache Kafka!
> > > > > I already have read the How to Contribute and Code Guidelines.
> > > > > Do u guys have a label for "easy" Jira's issue or something like
> > this?
> > > > >
> > > > > Att,
> > > > > Eduardo Mello
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-267: Add Processor Unit Test Support to Kafka Streams Test Utils

[Bill Bejeck]

John,

Sorry for the delayed response.  Thanks for the KIP, I'm +1 on it, and I
don't have any further comments on the KIP itself aside from the comments
that others have raised.

Regarding the existing MockProcessorContext and its removal in favor of the
one added from this KIP, I'm actually in favor of keeping both.

IMHO it's reasonable to have both because the testing requirements are
different.  Most users are trying to verify their logic works as expected
within a Kafka Streams application and aren't concerned (or shouldn't be at
least, again IMHO) with testing Kafka Streams itself, that is the
responsibility of the Kafka Streams developers and contributors.

However, for the developers and contributors of Kafka Streams, the need to
test the internals of how Streams works is the primary concern and could at
times require different logic or available methods from a given mock object.

I have a couple of thoughts on mitigation of having two
MockProcessorContext objects

   1. Leave the current MockProcessorContext in the o.a.k.test package but
   rename it to InternalMockProcessorContext and add some documentation as to
   why it's there.
   2. Create a new package under o.a.k.test, called internals and move the
   existing MockProcessorContext there, but that would require a change to the
   visibility of the MockProcessorContext#allStateStores() method to public.

Just wanted to throw in my 2 cents.

Thanks,
Bill

On Thu, Mar 8, 2018 at 11:51 PM, John Roesler  wrote:

> I think what you're suggesting is to:
> 1. compile the main streams code, but not the tests
> 2. compile test-utils (and compile and run the test-utils tests)
> 3. compile and run the streams tests
>
> This works in theory, since the test-utils depends on the main streams
> code, but not the streams tests. and the streams tests depend on test-utils
> while the main streams code does not.
>
> But after poking around a bit and reading up on it, I think this is not
> possible, or at least not mainstream.
>
> The issue is that dependencies are formed between projects, in this case
> streams and streams:test-utils. The upstream project must be built before
> the dependant one, regardless of whether the dependency is for compiling
> the main code or the test code. This means we do have a circular dependency
> on our hands if we want the tests in streams to use the test-utils, since
> they'd both have to be built before the other.
>
> Gradle seems to be quite scriptable, so there may be some way to achieve
> this, but increasing the complexity of the build also introduces a project
> maintenance concern.
>
> The MockProcessorContext itself is pretty simple, so I'm tempted to argue
> that we should just have one for internal unit tests and another for
> test-utils, however this situation also afflicts KAFKA-6474
> , and the
> TopologyTestDriver is not so trivial.
>
> I think the best thing at this point is to go ahead and fold the test-utils
> into the streams project. We can put it into a separate "testutils" package
> to make it easy to identify which code is for test support and which code
> is Kafka Streams. The biggest bummer about this suggestion is that it we
> *just* introduced the test-utils artifact, so folks would to add that
> artifact in 1.1 to write their tests and then have to drop it again in 1.2.
>
> The other major solution is to create a new gradle project for the streams
> unit tests, which depends on streams and test-utils and move all the
> streams unit tests there. I'm pretty sure we can configure gradle just to
> include this project for running tests and not actually package any
> artifacts. This structure basically expresses your observation that the
> test code is essentially a separate module from the main streams code.
>
> Of course, I'm open to alternatives, especially if someone with more
> experience in Gradle is aware of a solution.
>
> Thanks,
> -John
>
>
> On Thu, Mar 8, 2018 at 3:39 PM, Matthias J. Sax 
> wrote:
>
> > Isn't MockProcessorContext in o.a.k.test part of the unit-test package
> > but not the main package?
> >
> > This should resolve the dependency issue.
> >
> > -Matthias
> >
> > On 3/8/18 3:32 PM, John Roesler wrote:
> > > Actually, replacing the MockProcessorContext in o.a.k.test could be a
> bit
> > > tricky, since it would make the "streams" module depend on
> > > "streams:test-utils", but "streams:test-utils" already depends on
> > "streams".
> > >
> > > At first glance, it seems like the options are:
> > > 1. leave the two separate implementations in place. This shouldn't be
> > > underestimated, especially since our internal tests may need different
> > > things from a mocked P.C. than our API users.
> > > 2. move the public testing artifacts into the regular streams module
> > > 3. move the unit tests for Streams into a third module that depends on
> > both
> > > streams and test-utils. Yuck!
> > >
> > > Thanks,
> > > -John
> > >
> > > On

Re: [DISCUSS] KIP-267: Add Processor Unit Test Support to Kafka Streams Test Utils

[Guozhang Wang]

Hmm.. it seems to be a general issue then, since we were planning to also
replace the KStreamTestDriver and ProcessorTopologyTestDriver with the new
TopologyTestDriver soon, so if the argument that testing dependency could
still cause circular dependencies holds it means we cannot do that as well.

My understanding on gradle dependencies has been that test dependencies are
not required to compile when compiling the project, but only required when
testing the project; and the way we script gradle follows the way that for
any test tasks of the project we require compiling it first so this is
fine. John / Bill, could you elaborate a bit more on the maintenance
complexity concerns?


Guozhang

On Fri, Mar 9, 2018 at 7:40 AM, Bill Bejeck  wrote:

> John,
>
> Sorry for the delayed response.  Thanks for the KIP, I'm +1 on it, and I
> don't have any further comments on the KIP itself aside from the comments
> that others have raised.
>
> Regarding the existing MockProcessorContext and its removal in favor of the
> one added from this KIP, I'm actually in favor of keeping both.
>
> IMHO it's reasonable to have both because the testing requirements are
> different.  Most users are trying to verify their logic works as expected
> within a Kafka Streams application and aren't concerned (or shouldn't be at
> least, again IMHO) with testing Kafka Streams itself, that is the
> responsibility of the Kafka Streams developers and contributors.
>
> However, for the developers and contributors of Kafka Streams, the need to
> test the internals of how Streams works is the primary concern and could at
> times require different logic or available methods from a given mock
> object.
>
> I have a couple of thoughts on mitigation of having two
> MockProcessorContext objects
>
>1. Leave the current MockProcessorContext in the o.a.k.test package but
>rename it to InternalMockProcessorContext and add some documentation as
> to
>why it's there.
>2. Create a new package under o.a.k.test, called internals and move the
>existing MockProcessorContext there, but that would require a change to
> the
>visibility of the MockProcessorContext#allStateStores() method to
> public.
>
> Just wanted to throw in my 2 cents.
>
> Thanks,
> Bill
>
> On Thu, Mar 8, 2018 at 11:51 PM, John Roesler  wrote:
>
> > I think what you're suggesting is to:
> > 1. compile the main streams code, but not the tests
> > 2. compile test-utils (and compile and run the test-utils tests)
> > 3. compile and run the streams tests
> >
> > This works in theory, since the test-utils depends on the main streams
> > code, but not the streams tests. and the streams tests depend on
> test-utils
> > while the main streams code does not.
> >
> > But after poking around a bit and reading up on it, I think this is not
> > possible, or at least not mainstream.
> >
> > The issue is that dependencies are formed between projects, in this case
> > streams and streams:test-utils. The upstream project must be built before
> > the dependant one, regardless of whether the dependency is for compiling
> > the main code or the test code. This means we do have a circular
> dependency
> > on our hands if we want the tests in streams to use the test-utils, since
> > they'd both have to be built before the other.
> >
> > Gradle seems to be quite scriptable, so there may be some way to achieve
> > this, but increasing the complexity of the build also introduces a
> project
> > maintenance concern.
> >
> > The MockProcessorContext itself is pretty simple, so I'm tempted to argue
> > that we should just have one for internal unit tests and another for
> > test-utils, however this situation also afflicts KAFKA-6474
> > , and the
> > TopologyTestDriver is not so trivial.
> >
> > I think the best thing at this point is to go ahead and fold the
> test-utils
> > into the streams project. We can put it into a separate "testutils"
> package
> > to make it easy to identify which code is for test support and which code
> > is Kafka Streams. The biggest bummer about this suggestion is that it we
> > *just* introduced the test-utils artifact, so folks would to add that
> > artifact in 1.1 to write their tests and then have to drop it again in
> 1.2.
> >
> > The other major solution is to create a new gradle project for the
> streams
> > unit tests, which depends on streams and test-utils and move all the
> > streams unit tests there. I'm pretty sure we can configure gradle just to
> > include this project for running tests and not actually package any
> > artifacts. This structure basically expresses your observation that the
> > test code is essentially a separate module from the main streams code.
> >
> > Of course, I'm open to alternatives, especially if someone with more
> > experience in Gradle is aware of a solution.
> >
> > Thanks,
> > -John
> >
> >
> > On Thu, Mar 8, 2018 at 3:39 PM, Matthias J. Sax 
> > wrote:
> >
> > > Isn't MockProc

Re: [DISCUSS] KIP-267: Add Processor Unit Test Support to Kafka Streams Test Utils

[John Roesler]

Hi Guozhang and Bill,

I'll summarize what I'm currently thinking in light of all the discussion:

Mock Processor Context:
===

Here's how I see the use cases for the two mocks differing:

1. o.a.k.test.MPC: Crafted for testing Streams use cases. Implements
AbstractProcessorContext, actually forward to child processor nodes, allow
restoring a state store. Most importantly, the freedom to do stuff
convenient for our tests without impacting anyone.

2. (test-utils) MPC: Crafted for testing community Processors (and
friends). Very flat and simple implementation (so people can read it in one
sitting); i.e., doesn't drag in other data models like RecordContext. Test
one processor in isolation, so generally don't bother with complex logic
like scheduling punctuators, forwarding results, or restoring state stores.
Most importantly, an API that can be stable.

So, I really am leaning toward keeping both implementations. I like Bill's
suggestion of renaming the unit testing class to
InternalMockProcessorContext, since having classes with the same name in
different packages is confusing. I look forward to the day when Java 9
takes off and we can actually hide internal classes from the public
interface.

test-utils module:
=

This is actually out of scope for this KIP if we keep both MPC
implementations, but it has been a major feature of this discussion, so we
may as well see it though.

I've waffled a bit on this point, but right now I would propose we
restructure the streams directory thusly:

streams/ (artifact name := "streams", the actual streams code lives here)
- test-utils/ (this is the current test-utils artifact, depends on
"streams")
- tests/ (new module, depends on "streams" and "test-utils", *NO published
artifact*)

This gets us out of the circular dependency without having to engage in any
Gradle shenanigans while preserving "test-utils" as a separate artifact.
This is good because: 1) the test-utils don't need to be in production
code, so it's nice to have a separate artifact, 2) test-utils is already
public in 1.1, and it's a bummer to introduce users' code when we can so
easily avoid it.

Note, though, that if we agree to keep both MPC implementations, then this
really is just important for rewriting our tests to use TopologyTestDriver,
and in fact only the tests that need it should move to "streams/tests/".

What say you?

-John

On Fri, Mar 9, 2018 at 9:01 AM, Guozhang Wang  wrote:

> Hmm.. it seems to be a general issue then, since we were planning to also
> replace the KStreamTestDriver and ProcessorTopologyTestDriver with the new
> TopologyTestDriver soon, so if the argument that testing dependency could
> still cause circular dependencies holds it means we cannot do that as well.
>
> My understanding on gradle dependencies has been that test dependencies are
> not required to compile when compiling the project, but only required when
> testing the project; and the way we script gradle follows the way that for
> any test tasks of the project we require compiling it first so this is
> fine. John / Bill, could you elaborate a bit more on the maintenance
> complexity concerns?
>
>
> Guozhang
>
> On Fri, Mar 9, 2018 at 7:40 AM, Bill Bejeck  wrote:
>
> > John,
> >
> > Sorry for the delayed response.  Thanks for the KIP, I'm +1 on it, and I
> > don't have any further comments on the KIP itself aside from the comments
> > that others have raised.
> >
> > Regarding the existing MockProcessorContext and its removal in favor of
> the
> > one added from this KIP, I'm actually in favor of keeping both.
> >
> > IMHO it's reasonable to have both because the testing requirements are
> > different.  Most users are trying to verify their logic works as expected
> > within a Kafka Streams application and aren't concerned (or shouldn't be
> at
> > least, again IMHO) with testing Kafka Streams itself, that is the
> > responsibility of the Kafka Streams developers and contributors.
> >
> > However, for the developers and contributors of Kafka Streams, the need
> to
> > test the internals of how Streams works is the primary concern and could
> at
> > times require different logic or available methods from a given mock
> > object.
> >
> > I have a couple of thoughts on mitigation of having two
> > MockProcessorContext objects
> >
> >1. Leave the current MockProcessorContext in the o.a.k.test package
> but
> >rename it to InternalMockProcessorContext and add some documentation
> as
> > to
> >why it's there.
> >2. Create a new package under o.a.k.test, called internals and move
> the
> >existing MockProcessorContext there, but that would require a change
> to
> > the
> >visibility of the MockProcessorContext#allStateStores() method to
> > public.
> >
> > Just wanted to throw in my 2 cents.
> >
> > Thanks,
> > Bill
> >
> > On Thu, Mar 8, 2018 at 11:51 PM, John Roesler  wrote:
> >
> > > I think what you're suggesting is to:
> > > 1. compile the main streams code, but n

RE: [DISCUSS]KIP-235 DNS alias and secured connections

[Skrzypek, Jonathan]

Hi,

There has been further discussion on the ticket and it seems having an 
additional option to trigger the DNS lookup behaviour would be the best 
approach.

https://issues.apache.org/jira/browse/KAFKA-6195

Updated the KIP 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-235%3A+Add+DNS+alias+support+for+secured+connection

Would value your opinions.

Jonathan Skrzypek 


-Original Message-
From: Skrzypek, Jonathan [Tech] 
Sent: 22 February 2018 16:21
To: 'dev@kafka.apache.org'
Subject: RE: [DISCUSS]KIP-235 DNS alias and secured connections

Hi,

Could anyone take a look at the pull request, so that if ok I can start a VOTE 
thread ?

Regards,

Jonathan Skrzypek 

-Original Message-
From: Skrzypek, Jonathan [Tech]
Sent: 09 February 2018 13:57
To: 'dev@kafka.apache.org'
Subject: RE: [DISCUSS]KIP-235 DNS alias and secured connections

Hi,

I have raised a PR https://github.com/apache/kafka/pull/4485 with suggested 
code changes.
There are however reported failures, don't understand what's the issue since 
tests are passing.
Any ideas ?


Jonathan Skrzypek 

-Original Message-
From: Skrzypek, Jonathan [Tech]
Sent: 29 January 2018 16:51
To: dev@kafka.apache.org
Subject: RE: [DISCUSS]KIP-235 DNS alias and secured connections

Hi,

Yes I believe this might address what you're seeing as well.

Jonathan Skrzypek
Middleware Engineering
Messaging Engineering
Goldman Sachs International

-Original Message-
From: Stephane Maarek [mailto:steph...@simplemachines.com.au]
Sent: 06 December 2017 10:43
To: dev@kafka.apache.org
Subject: RE: [DISCUSS]KIP-235 DNS alias and secured connections

Hi Jonathan

I think this will be very useful. I reported something similar here :
https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_KAFKA-2D4781&d=DwIFaQ&c=7563p3e2zaQw0AB1wrFVgyagb2IE5rTZOYPxLxfZlX4&r=nNmJlu1rR_QFAPdxGlafmDu9_r6eaCbPOM0NM1EHo-E&m=3R1dVnw5Ttyz1YbVIMSRNMz2gjWsQmbTNXl63kwXvKo&s=MywacMwh18eVH_NvLY6Ffhc3CKMh43Tai3WMUf9PsjM&e=
 

Please confirm your kip will address it ?

Stéphane

On 6 Dec. 2017 8:20 pm, "Skrzypek, Jonathan" 
wrote:

> True, amended the KIP, thanks.
>
> Jonathan Skrzypek
> Middleware Engineering
> Messaging Engineering
> Goldman Sachs International
>
>
> -Original Message-
> From: Tom Bentley [mailto:t.j.bent...@gmail.com]
> Sent: 05 December 2017 18:19
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS]KIP-235 DNS alias and secured connections
>
> Hi Jonathan,
>
> It might be worth mentioning in the KIP that this is necessary only 
> for
> *Kerberos* on SASL, and not other SASL mechanisms. Reading the JIRA it 
> makes sensem, but I was confused up until that point.
>
> Cheers,
>
> Tom
>
> On 5 December 2017 at 17:53, Skrzypek, Jonathan 
> 
> wrote:
>
> > Hi,
> >
> > I would like to discuss a KIP I've submitted :
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.or
> > g_
> > confluence_display_KAFKA_KIP-2D&d=DwIBaQ&c=7563p3e2zaQw0AB1wrFVgyagb
> > 2I
> > E5rTZOYPxLxfZlX4&r=nNmJlu1rR_QFAPdxGlafmDu9_r6eaCbPOM0NM1EHo-E&m=GWK
> > XA
> > ILbqxFU2j7LtoOx9MZ00uy_jJcGWWIG92CyAuc&s=fv5WAkOgLhVOmF4vhEzq_39CWnE
> > o0 q0AJbqhAuDFDT0&e=
> > 235%3A+Add+DNS+alias+support+for+secured+connection
> >
> > Feedback and suggestions welcome !
> >
> > Regards,
> > Jonathan Skrzypek
> > Middleware Engineering
> > Messaging Engineering
> > Goldman Sachs International
> > Christchurch Court - 10-15 Newgate Street London EC1A 7HD
> > Tel: +442070512977
> >
> >
>

Re: [DISCUSS] KIP-258: Allow to Store Record Timestamps in RocksDB

[Bill Bejeck]

Matthias,

Thanks for the KIP, it's a +1 from me.

I do have one question regarding the retrieval methods on the new
interfaces.

Would want to consider adding one method with a Predicate that would allow
for filtering records by the timestamp stored with the record?  Or is this
better left for users to implement themselves once the data has been
retrieved?

Thanks,
Bill

On Thu, Mar 8, 2018 at 7:14 PM, Ted Yu  wrote:

> Matthias:
> For my point #1, I don't have preference as to which separator is chosen.
> Given the background you mentioned, current choice is good.
>
> For #2, I think my proposal is better since it is closer to English
> grammar.
>
> Would be good to listen to what other people think.
>
> On Thu, Mar 8, 2018 at 4:02 PM, Matthias J. Sax 
> wrote:
>
> > Thanks for the comments!
> >
> > @Guozhang:
> >
> > So far, there is one PR for the rebalance metadata upgrade fix
> > (addressing the mentioned
> > https://issues.apache.org/jira/browse/KAFKA-6054) It give a first
> > impression how the metadata upgrade works including a system test:
> > https://github.com/apache/kafka/pull/4636
> >
> > I can share other PRs as soon as they are ready. I agree that the KIP is
> > complex am I ok with putting out more code to give better discussion
> > context.
> >
> > @Ted:
> >
> > I picked `_` instead of `-` to align with the `processing.guarantee`
> > parameter that accepts `at_least_one` and `exactly_once` as values.
> > Personally, I don't care about underscore vs dash but I prefer
> > consistency. If you feel strong about it, we can also change it to `-`.
> >
> > About the interface name: I am fine either way -- I stripped the `With`
> > to keep the name a little shorter. Would be good to get feedback from
> > others and pick the name the majority prefers.
> >
> > @John:
> >
> > We can certainly change it. I agree that it would not make a difference.
> > I'll dig into the code to see if any of the two version might introduce
> > undesired complexity and update the KIP if I don't hit an issue with
> > putting the `-v2` to the store directory instead of `rocksdb-v2`
> >
> >
> > -Matthias
> >
> >
> > On 3/8/18 2:44 PM, John Roesler wrote:
> > > Hey Matthias,
> > >
> > > The KIP looks good to me. I had several questions queued up, but they
> > were
> > > all in the "rejected alternatives" section... oh, well.
> > >
> > > One very minor thought re changing the state directory from
> > "//<
> > > application.id>//rocksdb/storeName/" to "//<
> > > application.id>//rocksdb-v2/storeName/": if you put the "v2"
> > > marker on the storeName part of the path (i.e., "//<
> > > application.id>//rocksdb/storeName-v2/"), then you get the
> same
> > > benefits without altering the high-level directory structure.
> > >
> > > It may not matter, but I could imagine people running scripts to
> monitor
> > > rocksdb disk usage for each task, or other such use cases.
> > >
> > > Thanks,
> > > -John
> > >
> > > On Thu, Mar 8, 2018 at 2:02 PM, Ted Yu  wrote:
> > >
> > >> Matthias:
> > >> Nicely written KIP.
> > >>
> > >> "in_place" : can this be "in-place" ? Underscore may sometimes be miss
> > >> typed (as '-'). I think using '-' is more friendly to user.
> > >>
> > >> public interface ReadOnlyKeyValueTimestampStore {
> > >>
> > >> Is ReadOnlyKeyValueStoreWithTimestamp better name for the class ?
> > >>
> > >> Thanks
> > >>
> > >> On Thu, Mar 8, 2018 at 1:29 PM, Guozhang Wang 
> > wrote:
> > >>
> > >>> Hello Matthias, thanks for the KIP.
> > >>>
> > >>> I've read through the upgrade patch section and it looks good to me,
> if
> > >> you
> > >>> already have a WIP PR for it could you also share it here so that
> > people
> > >>> can take a look?
> > >>>
> > >>> I'm +1 on the KIP itself. But large KIPs like this there are always
> > some
> > >>> devil hidden in the details, so I think it is better to have the
> > >>> implementation in parallel along with the design discussion :)
> > >>>
> > >>>
> > >>> Guozhang
> > >>>
> > >>>
> > >>> On Wed, Mar 7, 2018 at 2:12 PM, Matthias J. Sax <
> matth...@confluent.io
> > >
> > >>> wrote:
> > >>>
> >  Hi,
> > 
> >  I want to propose KIP-258 for the Streams API to allow storing
> >  timestamps in RocksDB. This feature is the basis to resolve multiple
> >  tickets (issues and feature requests).
> > 
> >  Looking forward to your comments about this!
> > 
> >  https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >  258%3A+Allow+to+Store+Record+Timestamps+in+RocksDB
> > 
> > 
> >  -Matthias
> > 
> > 
> > 
> > >>>
> > >>>
> > >>> --
> > >>> -- Guozhang
> > >>>
> > >>
> > >
> >
> >
>

Build failed in Jenkins: kafka-trunk-jdk7 #3237

[Apache Jenkins Server]

See 


Changes:

[github] MINOR: Fix record conversion time in metrics (#4671)

--
[...truncated 414.91 KB...]
kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithConcurrentTransactionsOnAddPartitionsWhenStateIsPrepareCommit 
PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldIncrementEpochAndUpdateMetadataOnHandleInitPidWhenExistingCompleteTransaction
 STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldIncrementEpochAndUpdateMetadataOnHandleInitPidWhenExistingCompleteTransaction
 PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldGenerateNewProducerIdIfEpochsExhausted STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldGenerateNewProducerIdIfEpochsExhausted PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithNotCoordinatorOnInitPidWhenNotCoordinator STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithNotCoordinatorOnInitPidWhenNotCoordinator PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithConcurrentTransactionOnAddPartitionsWhenStateIsPrepareAbort 
STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithConcurrentTransactionOnAddPartitionsWhenStateIsPrepareAbort 
PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldInitPidWithEpochZeroForNewTransactionalId STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldInitPidWithEpochZeroForNewTransactionalId PASSED

kafka.coordinator.transaction.ProducerIdManagerTest > testExceedProducerIdLimit 
STARTED

kafka.coordinator.transaction.ProducerIdManagerTest > testExceedProducerIdLimit 
PASSED

kafka.coordinator.transaction.ProducerIdManagerTest > testGetProducerId STARTED

kafka.coordinator.transaction.ProducerIdManagerTest > testGetProducerId PASSED

kafka.coordinator.transaction.TransactionMarkerChannelManagerTest > 
shouldSaveForLaterWhenLeaderUnknownButNotAvailable STARTED

kafka.coordinator.transaction.TransactionMarkerChannelManagerTest > 
shouldSaveForLaterWhenLeaderUnknownButNotAvailable PASSED

kafka.coordinator.transaction.TransactionMarkerChannelManagerTest > 
shouldGenerateEmptyMapWhenNoRequestsOutstanding STARTED

kafka.coordinator.transaction.TransactionMarkerChannelManagerTest > 
shouldGenerateEmptyMapWhenNoRequestsOutstanding PASSED

kafka.coordinator.transaction.TransactionMarkerChannelManagerTest > 
shouldCreateMetricsOnStarting STARTED

kafka.coordinator.transaction.TransactionMarkerChannelManagerTest > 
shouldCreateMetricsOnStarting PASSED

kafka.coordinator.transaction.TransactionMarkerChannelManagerTest > 
shouldAbortAppendToLogOnEndTxnWhenNotCoordinatorError STARTED

kafka.coordinator.transaction.TransactionMarkerChannelManagerTest > 
shouldAbortAppendToLogOnEndTxnWhenNotCoordinatorError PASSED

kafka.coordinator.transaction.TransactionMarkerChannelManagerTest > 
shouldRetryAppendToLogOnEndTxnWhenCoordinatorNotAvailableError STARTED

kafka.coordinator.transaction.TransactionMarkerChannelManagerTest > 
shouldRetryAppendToLogOnEndTxnWhenCoordinatorNotAvailableError PASSED

kafka.coordinator.transaction.TransactionMarkerChannelManagerTest > 
shouldCompleteAppendToLogOnEndTxnWhenSendMarkersSucceed STARTED

kafka.coordinator.transaction.TransactionMarkerChannelManagerTest > 
shouldCompleteAppendToLogOnEndTxnWhenSendMarkersSucceed PASSED

kafka.coordinator.transaction.TransactionMarkerChannelManagerTest > 
shouldGenerateRequestPerPartitionPerBroker STARTED

kafka.coordinator.transaction.TransactionMarkerChannelManagerTest > 
shouldGenerateRequestPerPartitionPerBroker PASSED

kafka.coordinator.transaction.TransactionMarkerChannelManagerTest > 
shouldRemoveMarkersForTxnPartitionWhenPartitionEmigrated STARTED

kafka.coordinator.transaction.TransactionMarkerChannelManagerTest > 
shouldRemoveMarkersForTxnPartitionWhenPartitionEmigrated PASSED

kafka.coordinator.transaction.TransactionMarkerChannelManagerTest > 
shouldSkipSendMarkersWhenLeaderNotFound STARTED

kafka.coordinator.transaction.TransactionMarkerChannelManagerTest > 
shouldSkipSendMarkersWhenLeaderNotFound PASSED

kafka.coordinator.transaction.TransactionLogTest > shouldReadWriteMessages 
STARTED

kafka.coordinator.transaction.TransactionLogTest > shouldReadWriteMessages 
PASSED

kafka.coordinator.transaction.TransactionLogTest > 
shouldThrowExceptionWriteInvalidTxn STARTED

kafka.coordinator.transaction.TransactionLogTest > 
shouldThrowExceptionWriteInvalidTxn PASSED

kafka.coordinator.transaction.TransactionStateManagerTest > 
testAppendTransactionToLogWhileProducerFenced STARTED

kafka.coordinator.transaction.TransactionStateManagerTest > 
testAppendTransactionToLogWhileProducerFenced PASSED

kafka.coordinator.transaction.TransactionStateManagerTest > 
testCompleteTransitionWhenApp

Re: [DISCUSS] KIP-267: Add Processor Unit Test Support to Kafka Streams Test Utils

[Guozhang Wang]

Hey John,

Re: Mock Processor Context:

That's a good point, I'm convinced that we should keep them as two classes.


Re: test-utils module:

I think I agree with your proposed changes, in fact in order to not scatter
the test classes in two places maybe it's better to move all of them to the
new module. One caveat is that it will make streams' project hierarchy
inconsistent with other projects where the unit test classes are maintained
inside the main artifact package, but I think it is a good cost to pay,
plus once we start publishing test-util artifacts for other projects like
client and connect, we may face the same issue and need to do this
refactoring as well.



Guozhang




On Fri, Mar 9, 2018 at 9:54 AM, John Roesler  wrote:

> Hi Guozhang and Bill,
>
> I'll summarize what I'm currently thinking in light of all the discussion:
>
> Mock Processor Context:
> ===
>
> Here's how I see the use cases for the two mocks differing:
>
> 1. o.a.k.test.MPC: Crafted for testing Streams use cases. Implements
> AbstractProcessorContext, actually forward to child processor nodes, allow
> restoring a state store. Most importantly, the freedom to do stuff
> convenient for our tests without impacting anyone.
>
> 2. (test-utils) MPC: Crafted for testing community Processors (and
> friends). Very flat and simple implementation (so people can read it in one
> sitting); i.e., doesn't drag in other data models like RecordContext. Test
> one processor in isolation, so generally don't bother with complex logic
> like scheduling punctuators, forwarding results, or restoring state stores.
> Most importantly, an API that can be stable.
>
> So, I really am leaning toward keeping both implementations. I like Bill's
> suggestion of renaming the unit testing class to
> InternalMockProcessorContext, since having classes with the same name in
> different packages is confusing. I look forward to the day when Java 9
> takes off and we can actually hide internal classes from the public
> interface.
>
> test-utils module:
> =
>
> This is actually out of scope for this KIP if we keep both MPC
> implementations, but it has been a major feature of this discussion, so we
> may as well see it though.
>
> I've waffled a bit on this point, but right now I would propose we
> restructure the streams directory thusly:
>
> streams/ (artifact name := "streams", the actual streams code lives here)
> - test-utils/ (this is the current test-utils artifact, depends on
> "streams")
> - tests/ (new module, depends on "streams" and "test-utils", *NO published
> artifact*)
>
> This gets us out of the circular dependency without having to engage in any
> Gradle shenanigans while preserving "test-utils" as a separate artifact.
> This is good because: 1) the test-utils don't need to be in production
> code, so it's nice to have a separate artifact, 2) test-utils is already
> public in 1.1, and it's a bummer to introduce users' code when we can so
> easily avoid it.
>
> Note, though, that if we agree to keep both MPC implementations, then this
> really is just important for rewriting our tests to use TopologyTestDriver,
> and in fact only the tests that need it should move to "streams/tests/".
>
> What say you?
>
> -John
>
> On Fri, Mar 9, 2018 at 9:01 AM, Guozhang Wang  wrote:
>
> > Hmm.. it seems to be a general issue then, since we were planning to also
> > replace the KStreamTestDriver and ProcessorTopologyTestDriver with the
> new
> > TopologyTestDriver soon, so if the argument that testing dependency could
> > still cause circular dependencies holds it means we cannot do that as
> well.
> >
> > My understanding on gradle dependencies has been that test dependencies
> are
> > not required to compile when compiling the project, but only required
> when
> > testing the project; and the way we script gradle follows the way that
> for
> > any test tasks of the project we require compiling it first so this is
> > fine. John / Bill, could you elaborate a bit more on the maintenance
> > complexity concerns?
> >
> >
> > Guozhang
> >
> > On Fri, Mar 9, 2018 at 7:40 AM, Bill Bejeck  wrote:
> >
> > > John,
> > >
> > > Sorry for the delayed response.  Thanks for the KIP, I'm +1 on it, and
> I
> > > don't have any further comments on the KIP itself aside from the
> comments
> > > that others have raised.
> > >
> > > Regarding the existing MockProcessorContext and its removal in favor of
> > the
> > > one added from this KIP, I'm actually in favor of keeping both.
> > >
> > > IMHO it's reasonable to have both because the testing requirements are
> > > different.  Most users are trying to verify their logic works as
> expected
> > > within a Kafka Streams application and aren't concerned (or shouldn't
> be
> > at
> > > least, again IMHO) with testing Kafka Streams itself, that is the
> > > responsibility of the Kafka Streams developers and contributors.
> > >
> > > However, for the developers and contributors of Kafka Streams, the need
> > t

Re: [DISCUSS] KIP-267: Add Processor Unit Test Support to Kafka Streams Test Utils

[John Roesler]

Sweet! I think this pretty much wraps up all the discussion points.

I'll update the KIP with all the relevant aspects we discussed and call for
a vote.

I'll also comment on the TopologyTestDriver ticket noting this modular test
strategy.

Thanks, everyone.
-John

On Fri, Mar 9, 2018 at 10:57 AM, Guozhang Wang  wrote:

> Hey John,
>
> Re: Mock Processor Context:
>
> That's a good point, I'm convinced that we should keep them as two classes.
>
>
> Re: test-utils module:
>
> I think I agree with your proposed changes, in fact in order to not scatter
> the test classes in two places maybe it's better to move all of them to the
> new module. One caveat is that it will make streams' project hierarchy
> inconsistent with other projects where the unit test classes are maintained
> inside the main artifact package, but I think it is a good cost to pay,
> plus once we start publishing test-util artifacts for other projects like
> client and connect, we may face the same issue and need to do this
> refactoring as well.
>
>
>
> Guozhang
>
>
>
>
> On Fri, Mar 9, 2018 at 9:54 AM, John Roesler  wrote:
>
> > Hi Guozhang and Bill,
> >
> > I'll summarize what I'm currently thinking in light of all the
> discussion:
> >
> > Mock Processor Context:
> > ===
> >
> > Here's how I see the use cases for the two mocks differing:
> >
> > 1. o.a.k.test.MPC: Crafted for testing Streams use cases. Implements
> > AbstractProcessorContext, actually forward to child processor nodes,
> allow
> > restoring a state store. Most importantly, the freedom to do stuff
> > convenient for our tests without impacting anyone.
> >
> > 2. (test-utils) MPC: Crafted for testing community Processors (and
> > friends). Very flat and simple implementation (so people can read it in
> one
> > sitting); i.e., doesn't drag in other data models like RecordContext.
> Test
> > one processor in isolation, so generally don't bother with complex logic
> > like scheduling punctuators, forwarding results, or restoring state
> stores.
> > Most importantly, an API that can be stable.
> >
> > So, I really am leaning toward keeping both implementations. I like
> Bill's
> > suggestion of renaming the unit testing class to
> > InternalMockProcessorContext, since having classes with the same name in
> > different packages is confusing. I look forward to the day when Java 9
> > takes off and we can actually hide internal classes from the public
> > interface.
> >
> > test-utils module:
> > =
> >
> > This is actually out of scope for this KIP if we keep both MPC
> > implementations, but it has been a major feature of this discussion, so
> we
> > may as well see it though.
> >
> > I've waffled a bit on this point, but right now I would propose we
> > restructure the streams directory thusly:
> >
> > streams/ (artifact name := "streams", the actual streams code lives here)
> > - test-utils/ (this is the current test-utils artifact, depends on
> > "streams")
> > - tests/ (new module, depends on "streams" and "test-utils", *NO
> published
> > artifact*)
> >
> > This gets us out of the circular dependency without having to engage in
> any
> > Gradle shenanigans while preserving "test-utils" as a separate artifact.
> > This is good because: 1) the test-utils don't need to be in production
> > code, so it's nice to have a separate artifact, 2) test-utils is already
> > public in 1.1, and it's a bummer to introduce users' code when we can so
> > easily avoid it.
> >
> > Note, though, that if we agree to keep both MPC implementations, then
> this
> > really is just important for rewriting our tests to use
> TopologyTestDriver,
> > and in fact only the tests that need it should move to "streams/tests/".
> >
> > What say you?
> >
> > -John
> >
> > On Fri, Mar 9, 2018 at 9:01 AM, Guozhang Wang 
> wrote:
> >
> > > Hmm.. it seems to be a general issue then, since we were planning to
> also
> > > replace the KStreamTestDriver and ProcessorTopologyTestDriver with the
> > new
> > > TopologyTestDriver soon, so if the argument that testing dependency
> could
> > > still cause circular dependencies holds it means we cannot do that as
> > well.
> > >
> > > My understanding on gradle dependencies has been that test dependencies
> > are
> > > not required to compile when compiling the project, but only required
> > when
> > > testing the project; and the way we script gradle follows the way that
> > for
> > > any test tasks of the project we require compiling it first so this is
> > > fine. John / Bill, could you elaborate a bit more on the maintenance
> > > complexity concerns?
> > >
> > >
> > > Guozhang
> > >
> > > On Fri, Mar 9, 2018 at 7:40 AM, Bill Bejeck  wrote:
> > >
> > > > John,
> > > >
> > > > Sorry for the delayed response.  Thanks for the KIP, I'm +1 on it,
> and
> > I
> > > > don't have any further comments on the KIP itself aside from the
> > comments
> > > > that others have raised.
> > > >
> > > > Regarding the existing MockProcessorContext and its remova

Build failed in Jenkins: kafka-1.1-jdk7 #71

[Apache Jenkins Server]

See 


Changes:

[rajinisivaram] MINOR: Fix record conversion time in metrics (#4671)

--
[...truncated 416.07 KB...]
kafka.tools.ConsoleConsumerTest > shouldParseValidNewConsumerValidConfig PASSED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewConsumerConfigWithAutoOffsetResetAndMatchingFromBeginning 
STARTED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewConsumerConfigWithAutoOffsetResetAndMatchingFromBeginning 
PASSED

kafka.tools.ConsoleConsumerTest > shouldStopWhenOutputCheckErrorFails STARTED

kafka.tools.ConsoleConsumerTest > shouldStopWhenOutputCheckErrorFails PASSED

kafka.tools.ConsoleConsumerTest > 
shouldExitOnInvalidConfigWithAutoOffsetResetAndConflictingFromBeginningNewConsumer
 STARTED

kafka.tools.ConsoleConsumerTest > 
shouldExitOnInvalidConfigWithAutoOffsetResetAndConflictingFromBeginningNewConsumer
 PASSED

kafka.tools.ConsoleConsumerTest > 
shouldSetAutoResetToSmallestWhenFromBeginningConfigured STARTED

kafka.tools.ConsoleConsumerTest > 
shouldSetAutoResetToSmallestWhenFromBeginningConfigured PASSED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewConsumerConfigWithAutoOffsetResetEarliest STARTED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewConsumerConfigWithAutoOffsetResetEarliest PASSED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewSimpleConsumerValidConfigWithStringOffset STARTED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewSimpleConsumerValidConfigWithStringOffset PASSED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidOldConsumerConfigWithAutoOffsetResetLargest STARTED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidOldConsumerConfigWithAutoOffsetResetLargest PASSED

kafka.tools.ConsoleConsumerTest > shouldParseConfigsFromFile STARTED

kafka.tools.ConsoleConsumerTest > shouldParseConfigsFromFile PASSED

kafka.tools.ConsoleConsumerTest > groupIdsProvidedInDifferentPlacesMustMatch 
STARTED

kafka.tools.ConsoleConsumerTest > groupIdsProvidedInDifferentPlacesMustMatch 
PASSED

kafka.tools.ConsoleConsumerTest > 
shouldExitOnInvalidConfigWithAutoOffsetResetAndConflictingFromBeginningOldConsumer
 STARTED

kafka.tools.ConsoleConsumerTest > 
shouldExitOnInvalidConfigWithAutoOffsetResetAndConflictingFromBeginningOldConsumer
 PASSED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewSimpleConsumerValidConfigWithNumericOffset STARTED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewSimpleConsumerValidConfigWithNumericOffset PASSED

kafka.tools.ConsoleConsumerTest > testDefaultConsumer STARTED

kafka.tools.ConsoleConsumerTest > testDefaultConsumer PASSED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewConsumerConfigWithAutoOffsetResetLatest STARTED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewConsumerConfigWithAutoOffsetResetLatest PASSED

kafka.tools.ConsoleConsumerTest > shouldParseValidOldConsumerValidConfig STARTED

kafka.tools.ConsoleConsumerTest > shouldParseValidOldConsumerValidConfig PASSED

kafka.message.MessageCompressionTest > testCompressSize STARTED

kafka.message.MessageCompressionTest > testCompressSize PASSED

kafka.message.MessageCompressionTest > testSimpleCompressDecompress STARTED

kafka.message.MessageCompressionTest > testSimpleCompressDecompress PASSED

kafka.message.MessageTest > testChecksum STARTED

kafka.message.MessageTest > testChecksum PASSED

kafka.message.MessageTest > testInvalidTimestamp STARTED

kafka.message.MessageTest > testInvalidTimestamp PASSED

kafka.message.MessageTest > testIsHashable STARTED

kafka.message.MessageTest > testIsHashable PASSED

kafka.message.MessageTest > testInvalidTimestampAndMagicValueCombination STARTED

kafka.message.MessageTest > testInvalidTimestampAndMagicValueCombination PASSED

kafka.message.MessageTest > testExceptionMapping STARTED

kafka.message.MessageTest > testExceptionMapping PASSED

kafka.message.MessageTest > testFieldValues STARTED

kafka.message.MessageTest > testFieldValues PASSED

kafka.message.MessageTest > testInvalidMagicByte STARTED

kafka.message.MessageTest > testInvalidMagicByte PASSED

kafka.message.MessageTest > testEquality STARTED

kafka.message.MessageTest > testEquality PASSED

kafka.message.ByteBufferMessageSetTest > testMessageWithProvidedOffsetSeq 
STARTED

kafka.message.ByteBufferMessageSetTest > testMessageWithProvidedOffsetSeq PASSED

kafka.message.ByteBufferMessageSetTest > testValidBytes STARTED

kafka.message.ByteBufferMessageSetTest > testValidBytes PASSED

kafka.message.ByteBufferMessageSetTest > testValidBytesWithCompression STARTED

kafka.message.ByteBufferMessageSetTest > testValidBytesWithCompression PASSED

kafka.message.ByteBufferMessageSetTest > 
testWriteToChannelThatConsumesPartially STARTED

kafka.message.ByteBufferMessageSetTest > 
testWriteToChannelThatConsumesPartially PASSED

kafka.message.ByteBufferMessageSetTest > testIteratorIsConsistent STARTED

kafka.message.ByteBufferM

Re: [DISCUSS] KIP-258: Allow to Store Record Timestamps in RocksDB

[James Cheng]

Matthias,

For all the upgrade paths, is it possible to get rid of the 2nd rolling bounce?

For the in-place upgrade, it seems like primary difference between the 1st 
rolling bounce and the 2nd rolling bounce is to decide whether to send 
Subscription Version 2 or Subscription Version 3.  (Actually, there is another 
difference mentioned in that the KIP says that the 2nd rolling bounce should 
happen after all new state stores are created by the background thread. 
However, within the 2nd rolling bounce, we say that there is still a background 
thread, so it seems like is no actual requirement to wait for the new state 
stores to be created.)

The 2nd rolling bounce already knows how to deal with mixed-mode (having both 
Version 2 and Version 3 in the same consumer group). It seems like we could get 
rid of the 2nd bounce if we added logic (somehow/somewhere) such that:
* Instances send Subscription Version 2 until all instances are running the new 
code.
* Once all the instances are running the new code, then one at a time, the 
instances start sending Subscription V3. Leader still hands out Assignment 
Version 2, until all new state stores are ready.
* Once all instances report that new stores are ready, Leader sends out 
Assignment Version 3.
* Once an instance receives an Assignment Version 3, it can delete the old 
state store.

Doing it that way seems like it would reduce a lot of operator/deployment 
overhead. No need to do 2 rolling restarts. No need to monitor logs for state 
store rebuild. You just deploy it, and the instances update themselves.

What do you think?

The thing that made me think of this is that the "2 rolling bounces" is similar 
to what Kafka brokers have to do changes in inter.broker.protocol.version and 
log.message.format.version. And in the broker case, it seems like it would be 
possible (with some work of course) to modify kafka to allow us to do similar 
auto-detection of broker capabilities and automatically do a switchover from 
old/new versions. 

-James


> On Mar 9, 2018, at 10:38 AM, Bill Bejeck  wrote:
> 
> Matthias,
> 
> Thanks for the KIP, it's a +1 from me.
> 
> I do have one question regarding the retrieval methods on the new
> interfaces.
> 
> Would want to consider adding one method with a Predicate that would allow
> for filtering records by the timestamp stored with the record?  Or is this
> better left for users to implement themselves once the data has been
> retrieved?
> 
> Thanks,
> Bill
> 
> On Thu, Mar 8, 2018 at 7:14 PM, Ted Yu  wrote:
> 
>> Matthias:
>> For my point #1, I don't have preference as to which separator is chosen.
>> Given the background you mentioned, current choice is good.
>> 
>> For #2, I think my proposal is better since it is closer to English
>> grammar.
>> 
>> Would be good to listen to what other people think.
>> 
>> On Thu, Mar 8, 2018 at 4:02 PM, Matthias J. Sax 
>> wrote:
>> 
>>> Thanks for the comments!
>>> 
>>> @Guozhang:
>>> 
>>> So far, there is one PR for the rebalance metadata upgrade fix
>>> (addressing the mentioned
>>> https://issues.apache.org/jira/browse/KAFKA-6054) It give a first
>>> impression how the metadata upgrade works including a system test:
>>> https://github.com/apache/kafka/pull/4636
>>> 
>>> I can share other PRs as soon as they are ready. I agree that the KIP is
>>> complex am I ok with putting out more code to give better discussion
>>> context.
>>> 
>>> @Ted:
>>> 
>>> I picked `_` instead of `-` to align with the `processing.guarantee`
>>> parameter that accepts `at_least_one` and `exactly_once` as values.
>>> Personally, I don't care about underscore vs dash but I prefer
>>> consistency. If you feel strong about it, we can also change it to `-`.
>>> 
>>> About the interface name: I am fine either way -- I stripped the `With`
>>> to keep the name a little shorter. Would be good to get feedback from
>>> others and pick the name the majority prefers.
>>> 
>>> @John:
>>> 
>>> We can certainly change it. I agree that it would not make a difference.
>>> I'll dig into the code to see if any of the two version might introduce
>>> undesired complexity and update the KIP if I don't hit an issue with
>>> putting the `-v2` to the store directory instead of `rocksdb-v2`
>>> 
>>> 
>>> -Matthias
>>> 
>>> 
>>> On 3/8/18 2:44 PM, John Roesler wrote:
 Hey Matthias,
 
 The KIP looks good to me. I had several questions queued up, but they
>>> were
 all in the "rejected alternatives" section... oh, well.
 
 One very minor thought re changing the state directory from
>>> "//<
 application.id>//rocksdb/storeName/" to "//<
 application.id>//rocksdb-v2/storeName/": if you put the "v2"
 marker on the storeName part of the path (i.e., "//<
 application.id>//rocksdb/storeName-v2/"), then you get the
>> same
 benefits without altering the high-level directory structure.
 
 It may not matter, but I could imagine people running scripts to
>> monitor
 rocksdb disk usage for

Build failed in Jenkins: kafka-trunk-jdk8 #2460

[Apache Jenkins Server]

See 


Changes:

[ismael] MINOR: Remove unused local variable in SocketServer (#4669)

--
[...truncated 3.93 MB...]

org.apache.kafka.streams.processor.internals.ProcessorStateManagerTest > 
testRegisterPersistentStore STARTED

org.apache.kafka.streams.processor.internals.ProcessorStateManagerTest > 
testRegisterPersistentStore PASSED

org.apache.kafka.streams.processor.internals.ProcessorStateManagerTest > 
shouldSuccessfullyReInitializeStateStores STARTED

org.apache.kafka.streams.processor.internals.ProcessorStateManagerTest > 
shouldSuccessfullyReInitializeStateStores PASSED

org.apache.kafka.streams.processor.internals.ProcessorStateManagerTest > 
shouldThrowIllegalArgumentExceptionOnRegisterWhenStoreHasAlreadyBeenRegistered 
STARTED

org.apache.kafka.streams.processor.internals.ProcessorStateManagerTest > 
shouldThrowIllegalArgumentExceptionOnRegisterWhenStoreHasAlreadyBeenRegistered 
PASSED

org.apache.kafka.streams.processor.internals.ProcessorStateManagerTest > 
shouldRestoreStoreWithSinglePutRestoreSpecification STARTED

org.apache.kafka.streams.processor.internals.ProcessorStateManagerTest > 
shouldRestoreStoreWithSinglePutRestoreSpecification PASSED

org.apache.kafka.streams.processor.internals.ProcessorStateManagerTest > 
shouldNotChangeOffsetsIfAckedOffsetsIsNull STARTED

org.apache.kafka.streams.processor.internals.ProcessorStateManagerTest > 
shouldNotChangeOffsetsIfAckedOffsetsIsNull PASSED

org.apache.kafka.streams.processor.internals.ProcessorStateManagerTest > 
shouldThrowIllegalArgumentExceptionIfStoreNameIsSameAsCheckpointFileName STARTED

org.apache.kafka.streams.processor.internals.ProcessorStateManagerTest > 
shouldThrowIllegalArgumentExceptionIfStoreNameIsSameAsCheckpointFileName PASSED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldCloseStateManagerWithOffsets STARTED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldCloseStateManagerWithOffsets PASSED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldProcessRecordsForTopic STARTED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldProcessRecordsForTopic PASSED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldInitializeProcessorTopology STARTED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldInitializeProcessorTopology PASSED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldInitializeContext STARTED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldInitializeContext PASSED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldCheckpointOffsetsWhenStateIsFlushed STARTED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldCheckpointOffsetsWhenStateIsFlushed PASSED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldThrowStreamsExceptionWhenKeyDeserializationFails STARTED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldThrowStreamsExceptionWhenKeyDeserializationFails PASSED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldInitializeStateManager STARTED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldInitializeStateManager PASSED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldProcessRecordsForOtherTopic STARTED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldProcessRecordsForOtherTopic PASSED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldNotThrowStreamsExceptionWhenValueDeserializationFails STARTED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldNotThrowStreamsExceptionWhenValueDeserializationFails PASSED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldThrowStreamsExceptionWhenValueDeserializationFails STARTED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldThrowStreamsExceptionWhenValueDeserializationFails PASSED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldNotThrowStreamsExceptionWhenKeyDeserializationFailsWithSkipHandler STARTED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldNotThrowStreamsExceptionWhenKeyDeserializationFailsWithSkipHandler PASSED

org.apache.kafka.streams.processor.internals.StreamsMetricsImplTest > 
testThroughputMetrics STARTED

org.apache.kafka.streams.processor.internals.StreamsMetricsImplTest > 
testThroughputMetrics PASSED

org.apache.kafka.streams.processor.internals.StreamsMetricsImplTest > 
testLatencyMetrics STARTED

org.apache.kafka.streams.processor.internals.StreamsMetricsImplTest > 
testLatencyMetrics PASSED

org.apache.kafka.streams.processor.internals.StreamsMetricsImplTest > 
testRemoveSensor STARTED

org.apach

Re: [VOTE] KIP-186: Increase offsets retention default to 7 days

[Ewen Cheslack-Postava]

Thanks for voting everyone, KIP is accepted with 5 binding +1 and 5
non-binding +1s.

PR is already merged and there are upgrade notes about the potential memory
impact.

-Ewen

On Wed, Mar 7, 2018 at 9:25 PM, Ted Yu  wrote:

> +1
>  Original message From: Guozhang Wang 
> Date: 3/7/18  9:17 PM  (GMT-08:00) To: dev@kafka.apache.org Subject: Re:
> [VOTE] KIP-186: Increase offsets retention default to 7 days
> +1 (binding).
>
> On Wed, Mar 7, 2018 at 5:04 PM, James Cheng  wrote:
>
> > +1 (non-binding)
> >
> > -James
> >
> > > On Mar 7, 2018, at 1:20 PM, Jay Kreps  wrote:
> > >
> > > +1
> > >
> > > I think we can improve this in the future, but this simple change will
> > > avoid a lot of pain. Thanks for reviving it Ewen.
> > >
> > > -Jay
> > >
> > > On Mon, Mar 5, 2018 at 11:35 AM, Ewen Cheslack-Postava <
> > e...@confluent.io>
> > > wrote:
> > >
> > >> I'd like to kick off voting for KIP-186:
> > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > >> 186%3A+Increase+offsets+retention+default+to+7+days
> > >>
> > >> This is the trivial fix that people in the DISCUSS thread were in
> favor
> > of.
> > >> There are some ideas for further refinements, but I think we can
> follow
> > up
> > >> with those in subsequent KIPs, see the discussion thread for details.
> > Also
> > >> note this is related, but complementary, to
> > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > >> 211%3A+Revise+Expiration+Semantics+of+Consumer+Group+Offsets
> > >> .
> > >>
> > >> And of course +1 (binding) from me.
> > >>
> > >> Thanks,
> > >> Ewen
> > >>
> >
> >
>
>
> --
> -- Guozhang
>

[VOTE] KIP-267: Add Processor Unit Test Support to Kafka Streams Test Utils

[John Roesler]

Hello all,

I'd like to start voting on KIP-267, to introduce a MockProcessorContent
enabling Processor, Transformer, and ValueTransformer authors to unit-test
their components.

https://cwiki.apache.org/confluence/display/KAFKA/KIP-267%3A+Add+Processor+Unit+Test+Support+to+Kafka+Streams+Test+Utils?src=jira

Thanks,
-John

Re: [VOTE] KIP-267: Add Processor Unit Test Support to Kafka Streams Test Utils

[Bill Bejeck]

Thanks for the KIP.

+1

-Bill

On Fri, Mar 9, 2018 at 3:16 PM, John Roesler  wrote:

> Hello all,
>
> I'd like to start voting on KIP-267, to introduce a MockProcessorContent
> enabling Processor, Transformer, and ValueTransformer authors to unit-test
> their components.
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 267%3A+Add+Processor+Unit+Test+Support+to+Kafka+Streams+
> Test+Utils?src=jira
>
> Thanks,
> -John
>

Re: [VOTE] KIP-267: Add Processor Unit Test Support to Kafka Streams Test Utils

[Ted Yu]

+1
 Original message From: Bill Bejeck  Date: 
3/9/18  12:29 PM  (GMT-08:00) To: dev@kafka.apache.org Subject: Re: [VOTE] 
KIP-267: Add Processor Unit Test Support to Kafka Streams Test Utils 
Thanks for the KIP.

+1

-Bill

On Fri, Mar 9, 2018 at 3:16 PM, John Roesler  wrote:

> Hello all,
>
> I'd like to start voting on KIP-267, to introduce a MockProcessorContent
> enabling Processor, Transformer, and ValueTransformer authors to unit-test
> their components.
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 267%3A+Add+Processor+Unit+Test+Support+to+Kafka+Streams+
> Test+Utils?src=jira
>
> Thanks,
> -John
>

Re: Want to contribute

[Matthias J. Sax]

Eduardo,

I added you to the contributor list.

-Matthias

On 3/9/18 7:07 AM, Eduardo Mello wrote:
> Thank you!
> 
> Ill look for this label!
> 
> I created a Jira user too: emello
> 
> Thanks,
> Eduardo Mello
> 
> On Fri, Mar 9, 2018 at 12:00 PM, Sanket Band 
> wrote:
> 
>> That was quick ! Thanks
>>
>> Thanks
>> Sanket
>>
>> On Fri, Mar 9, 2018 at 8:29 PM, Damian Guy  wrote:
>>
>>> Hi Sanket,
>>>
>>> Done.
>>> Thanks,
>>> Damian
>>>
>>> On Fri, 9 Mar 2018 at 14:56 Sanket Band  wrote:
>>>
 Hi Damian, Matthias

 Could you add me to the KAFKA project on jira, my jira id is sband

 Thanks
 Sanket

 On Fri, 9 Mar 2018, 20:23 Damian Guy,  wrote:

> Hi Eduardo,
>
> There are usually JIRAs that have the newbie label. This would be a
>>> good
> place to start.
>
> Cheers,
> Damian
>
> On Fri, 9 Mar 2018 at 13:57 Eduardo Mello 
>>> wrote:
>
>> Hello,
>> I want to begin to code for open sources projects in my free time
>>> and I
>> want to help the Apache Kafka!
>> I already have read the How to Contribute and Code Guidelines.
>> Do u guys have a label for "easy" Jira's issue or something like
>>> this?
>>
>> Att,
>> Eduardo Mello
>>
>

>>>
>>
> 



signature.asc
Description: OpenPGP digital signature

Re: [DISCUSS] KIP-253: Support in-order message delivery with partition expansion

[Matthias J. Sax]

Thanks for your comment Clemens. It make sense what you are saying.
However, your described pattern is to split partitions and use linear
hashing to avoid random key distribution. But this is what Jan thinks we
should not do...

Also, I just picked an example with 2 -> 3 partitions, but if you don't
use linear hashing I think the same issue occurs if you double the
number of partitions.

I am in favor of using linear hashing. Still think, it is also useful to
split single partitions, too, in case load is not balanced and some
partitions are hot spots while others are "idle".

-Matthias


On 3/9/18 5:41 AM, Clemens Valiente wrote:
> I think it's fair to assume that topics will always be increased by an 
> integer factor - e.g. from 2 partitions to 4 partitions. Then the mapping is 
> much easier.
> 
> Why anyone would increase partitions by lass than x2 is a mystery to me. If 
> your two partitions cannot handle the load, then with three partitions each 
> one will still get 67% of that load which is still way too dangerous.
> 
> 
> So in your case we go from
> 
> part1: A B C D
> 
> part2: E F G H
> 
> 
> to
> 
> 
> part1: A C
> 
> part2: B D
> 
> part3: E F
> 
> part4: G H
> 
> 
> 
> From: Matthias J. Sax 
> Sent: 09 March 2018 07:53
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-253: Support in-order message delivery with 
> partition expansion
> 
> @Jan: You suggest to copy the data from one topic to a new topic, and
> provide an "offset mapping" from the old to the new topic for the
> consumers. I don't quite understand how this would work.
> 
> Let's say there are 2 partitions in the original topic and 3 partitions
> in the new topic. If we assume that we don't use linear hashing as you
> suggest, there is no guarantee how data will be distributed in the new
> topic and also no guarantee about ordering of records in the new topic.
> 
> Example (I hope I got it right -- please correct me if it's wrong)
> 
> A B C D
> E F G H
> 
> could be copied to:
> 
> A C H
> B E F
> D G
> 
> If the consumer was at offset 1 and 2 in the first topic how would the
> mapping be computed? We need to enures that B C D as well as G H are
> read after the switch. Thus, offset would need to be 1 0 0. I am not
> sure how this would be computed?
> 
> Furthermore, I want to point out that the new offsets would imply that E
> is consumed a second time by the consumer. E and F were consumed
> originally, but E is copied after B that was not yet consumed.
> 
> Or is there a way that we can ensure that this "flip" does never happen
> while we copy the data?
> 
> 
> -Matthias
> 
> 
> 
> On 3/8/18 10:32 PM, Matthias J. Sax wrote:
>> As I just mentioned joins:
>>
>> For Kafka Streams it might also be required to change the partition
>> count for multiple topics in a coordinated way that allows to maintain
>> the co-partitioning property that Kafka Streams uses to computed joins.
>>
>> Any thoughts how this could be handled?
>>
>>
>> -Matthias
>>
>> On 3/8/18 10:08 PM, Matthias J. Sax wrote:
>>> Jun,
>>>
>>> There is one more case: non-windowed aggregations. For windowed
>>> aggregation, the changelog topic will be compact+delete. However, for
>>> non-windowed aggregation the policy is compact only.
>>>
>>> Even if we assume that windowed aggregations are dominant and
>>> non-windowed aggregation are used rarely, it seems to be bad to not
>>> support the feature is a non-windowed aggregation is used. Also,
>>> non-windowed aggregation volume depends on input-stream volume that
>>> might be high.
>>>
>>> Furthermore, we support stream-table join and this requires that the
>>> stream and the table are co-partitioned. Thus, even if the table would
>>> have lower volume but the stream must be scaled out, we also need to
>>> scale out the table to preserve co-partitioning.
>>>
>>>
>>> -Matthias
>>>
>>> On 3/8/18 6:44 PM, Jun Rao wrote:
 Hi, Matthis,

 My understanding is that in KStream, the only case when a changelog topic
 needs to be compacted is when the corresponding input is a KTable. In all
 other cases, the changelog topics are of compacted + deletion. So, if most
 KTables are not high volume, there may not be a need to expand its
 partitions and therefore the partitions of the corresponding changelog
 topic.

 Thanks,

 Jun

 On Wed, Mar 7, 2018 at 2:34 PM, Matthias J. Sax 
 wrote:

> Jun,
>
> thanks for your comment. This should actually work for Streams, because
> we don't rely on producer "hashing" but specify the partition number
> explicitly on send().
>
> About not allowing to change the number of partition for changelog
> topics: for Streams, this seems to imply that we need to create a second
> changelog topic for each store with the new partition count. However, it
> would be unclear when/if we can delete the old topic. Thus, it moves the
> "problem" into the application laye

Re: [VOTE] KIP-267: Add Processor Unit Test Support to Kafka Streams Test Utils

[Guozhang Wang]

Hi John,

You mentioned you are going to do the renaming for the existing class
to InternalMockProcessorContext
but that seems not reflected on the KIP wiki, could you add that in the
proposed changes section?

Other than that, I'm +1 on the KIP.

Thanks!

Guozhang

On Fri, Mar 9, 2018 at 12:39 PM, Ted Yu  wrote:

> +1
>  Original message From: Bill Bejeck 
> Date: 3/9/18  12:29 PM  (GMT-08:00) To: dev@kafka.apache.org Subject: Re:
> [VOTE] KIP-267: Add Processor Unit Test Support to Kafka Streams Test Utils
> Thanks for the KIP.
>
> +1
>
> -Bill
>
> On Fri, Mar 9, 2018 at 3:16 PM, John Roesler  wrote:
>
> > Hello all,
> >
> > I'd like to start voting on KIP-267, to introduce a MockProcessorContent
> > enabling Processor, Transformer, and ValueTransformer authors to
> unit-test
> > their components.
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 267%3A+Add+Processor+Unit+Test+Support+to+Kafka+Streams+
> > Test+Utils?src=jira
> >
> > Thanks,
> > -John
> >
>



-- 
-- Guozhang

Re: [VOTE] KIP-267: Add Processor Unit Test Support to Kafka Streams Test Utils

[Matthias J. Sax]

Guozhang, renaming the class is an internal change and I think it's not
required to cover it in the KIP.


+1 (binding)


-Matthias

On 3/9/18 1:26 PM, Guozhang Wang wrote:
> Hi John,
> 
> You mentioned you are going to do the renaming for the existing class
> to InternalMockProcessorContext
> but that seems not reflected on the KIP wiki, could you add that in the
> proposed changes section?
> 
> Other than that, I'm +1 on the KIP.
> 
> Thanks!
> 
> Guozhang
> 
> On Fri, Mar 9, 2018 at 12:39 PM, Ted Yu  wrote:
> 
>> +1
>>  Original message From: Bill Bejeck 
>> Date: 3/9/18  12:29 PM  (GMT-08:00) To: dev@kafka.apache.org Subject: Re:
>> [VOTE] KIP-267: Add Processor Unit Test Support to Kafka Streams Test Utils
>> Thanks for the KIP.
>>
>> +1
>>
>> -Bill
>>
>> On Fri, Mar 9, 2018 at 3:16 PM, John Roesler  wrote:
>>
>>> Hello all,
>>>
>>> I'd like to start voting on KIP-267, to introduce a MockProcessorContent
>>> enabling Processor, Transformer, and ValueTransformer authors to
>> unit-test
>>> their components.
>>>
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>>> 267%3A+Add+Processor+Unit+Test+Support+to+Kafka+Streams+
>>> Test+Utils?src=jira
>>>
>>> Thanks,
>>> -John
>>>
>>
> 
> 
> 



signature.asc
Description: OpenPGP digital signature

Re: Want to contribute

[Eduardo Mello]

Matthias,

Thank you!

Thanks,
Eduardo Mello



On Fri, Mar 9, 2018 at 6:14 PM, Matthias J. Sax 
wrote:

> Eduardo,
>
> I added you to the contributor list.
>
> -Matthias
>
> On 3/9/18 7:07 AM, Eduardo Mello wrote:
> > Thank you!
> >
> > Ill look for this label!
> >
> > I created a Jira user too: emello
> >
> > Thanks,
> > Eduardo Mello
> >
> > On Fri, Mar 9, 2018 at 12:00 PM, Sanket Band 
> > wrote:
> >
> >> That was quick ! Thanks
> >>
> >> Thanks
> >> Sanket
> >>
> >> On Fri, Mar 9, 2018 at 8:29 PM, Damian Guy 
> wrote:
> >>
> >>> Hi Sanket,
> >>>
> >>> Done.
> >>> Thanks,
> >>> Damian
> >>>
> >>> On Fri, 9 Mar 2018 at 14:56 Sanket Band 
> wrote:
> >>>
>  Hi Damian, Matthias
> 
>  Could you add me to the KAFKA project on jira, my jira id is sband
> 
>  Thanks
>  Sanket
> 
>  On Fri, 9 Mar 2018, 20:23 Damian Guy,  wrote:
> 
> > Hi Eduardo,
> >
> > There are usually JIRAs that have the newbie label. This would be a
> >>> good
> > place to start.
> >
> > Cheers,
> > Damian
> >
> > On Fri, 9 Mar 2018 at 13:57 Eduardo Mello 
> >>> wrote:
> >
> >> Hello,
> >> I want to begin to code for open sources projects in my free time
> >>> and I
> >> want to help the Apache Kafka!
> >> I already have read the How to Contribute and Code Guidelines.
> >> Do u guys have a label for "easy" Jira's issue or something like
> >>> this?
> >>
> >> Att,
> >> Eduardo Mello
> >>
> >
> 
> >>>
> >>
> >
>
>

Build failed in Jenkins: kafka-trunk-jdk9 #459

[Apache Jenkins Server]

See 


Changes:

[wangguoz] KAFKA-4831: Extract WindowedSerde to public APIs (#3307)

--
[...truncated 1.48 MB...]
kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldAbortExpiredTransactionsInOngoingStateAndBumpEpoch STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldAbortExpiredTransactionsInOngoingStateAndBumpEpoch PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldReturnInvalidTxnRequestOnEndTxnRequestWhenStatusIsCompleteCommitAndResultIsNotCommit
 STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldReturnInvalidTxnRequestOnEndTxnRequestWhenStatusIsCompleteCommitAndResultIsNotCommit
 PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldReturnOkOnEndTxnWhenStatusIsCompleteCommitAndResultIsCommit STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldReturnOkOnEndTxnWhenStatusIsCompleteCommitAndResultIsCommit PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithConcurrentTransactionsOnAddPartitionsWhenStateIsPrepareCommit 
STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithConcurrentTransactionsOnAddPartitionsWhenStateIsPrepareCommit 
PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldIncrementEpochAndUpdateMetadataOnHandleInitPidWhenExistingCompleteTransaction
 STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldIncrementEpochAndUpdateMetadataOnHandleInitPidWhenExistingCompleteTransaction
 PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldGenerateNewProducerIdIfEpochsExhausted STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldGenerateNewProducerIdIfEpochsExhausted PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithNotCoordinatorOnInitPidWhenNotCoordinator STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithNotCoordinatorOnInitPidWhenNotCoordinator PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithConcurrentTransactionOnAddPartitionsWhenStateIsPrepareAbort 
STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithConcurrentTransactionOnAddPartitionsWhenStateIsPrepareAbort 
PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldInitPidWithEpochZeroForNewTransactionalId STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldInitPidWithEpochZeroForNewTransactionalId PASSED

kafka.coordinator.transaction.ProducerIdManagerTest > testExceedProducerIdLimit 
STARTED

kafka.coordinator.transaction.ProducerIdManagerTest > testExceedProducerIdLimit 
PASSED

kafka.coordinator.transaction.ProducerIdManagerTest > testGetProducerId STARTED

kafka.coordinator.transaction.ProducerIdManagerTest > testGetProducerId PASSED

kafka.coordinator.transaction.TransactionMarkerChannelManagerTest > 
shouldSaveForLaterWhenLeaderUnknownButNotAvailable STARTED

kafka.coordinator.transaction.TransactionMarkerChannelManagerTest > 
shouldSaveForLaterWhenLeaderUnknownButNotAvailable PASSED

kafka.coordinator.transaction.TransactionMarkerChannelManagerTest > 
shouldGenerateEmptyMapWhenNoRequestsOutstanding STARTED

kafka.coordinator.transaction.TransactionMarkerChannelManagerTest > 
shouldGenerateEmptyMapWhenNoRequestsOutstanding PASSED

kafka.coordinator.transaction.TransactionMarkerChannelManagerTest > 
shouldCreateMetricsOnStarting STARTED

kafka.coordinator.transaction.TransactionMarkerChannelManagerTest > 
shouldCreateMetricsOnStarting PASSED

kafka.coordinator.transaction.TransactionMarkerChannelManagerTest > 
shouldAbortAppendToLogOnEndTxnWhenNotCoordinatorError STARTED

kafka.coordinator.transaction.TransactionMarkerChannelManagerTest > 
shouldAbortAppendToLogOnEndTxnWhenNotCoordinatorError PASSED

kafka.coordinator.transaction.TransactionMarkerChannelManagerTest > 
shouldRetryAppendToLogOnEndTxnWhenCoordinatorNotAvailableError STARTED

kafka.coordinator.transaction.TransactionMarkerChannelManagerTest > 
shouldRetryAppendToLogOnEndTxnWhenCoordinatorNotAvailableError PASSED

kafka.coordinator.transaction.TransactionMarkerChannelManagerTest > 
shouldCompleteAppendToLogOnEndTxnWhenSendMarkersSucceed STARTED

kafka.coordinator.transaction.TransactionMarkerChannelManagerTest > 
shouldCompleteAppendToLogOnEndTxnWhenSendMarkersSucceed PASSED

kafka.coordinator.transaction.TransactionMarkerChannelManagerTest > 
shouldGenerateRequestPerPartitionPerBroker STARTED

kafka.coordinator.transaction.TransactionMarkerChannelManagerTest > 
shouldGenerateRequestPerPartitionPerBroker PASSED

kafka.coordinator.transaction.TransactionMarkerChannelManagerTest > 
shouldRemoveMarkersForTxnPartitionWhenPartitionEmigrated STARTED

kafka.coordinator.transaction.TransactionMarkerChannelManag

Re: [VOTE] KIP-251: Allow timestamp manipulation in Processor API

[Matthias J. Sax]

+1

I am also closing this vote. The KIP is accepted with

3 binding votes (Damian, Guozhang, Matthias) and
2 non-binding votes (Ted, Bill)


Thanks for your comments and votes!


-Matthias


On 3/8/18 10:36 PM, Guozhang Wang wrote:
> Thanks Matthias, that sounds good to me. I'm +1 on the KIP itself.
> 
> 
> Guozhang
> 
> On Thu, Mar 8, 2018 at 5:46 PM, Matthias J. Sax 
> wrote:
> 
>> Guozhang,
>>
>> I updated the code slightly to avoid object creation and I did some perf
>> investigations.
>>
>> 1) JMH Benchmark with the below topology using TopologyTestDriver to
>> pipe data throw the topology:
>>
>>> StreamsBuilder builder = new StreamsBuilder();
>>> KStream stream = builder.stream("topic").transform(new
>> TransformerSupplier>() {
>>> @Override
>>> public Transformer>
>> get() {
>>> return new Transformer> Object>>() {
>>> ProcessorContext context;
>>>
>>> @Override
>>> public void init(ProcessorContext context) {
>>> this.context = context;
>>> }
>>>
>>> @Override
>>> public KeyValue transform(Object
>> key, Object value) {
>>> context.forward(key, value);
>>> return null;
>>> }
>>>
>>> @Override
>>> public KeyValue punctuate(long
>> timestamp) {
>>> return null;
>>> }
>>>
>>> @Override
>>> public void close() {}
>>> };
>>> }
>>> });
>>
>> I run this with zero, one, and five downstream nodes like:
>>
>>> stream.foreach(new ForeachAction() {
>>> @Override
>>> public void apply(Object key, Object value) {}
>>> });
>>
>> On `trunk` I get the following numbers (5 warmup iterations, 15 test
>> iterations)
>>
>> Zero Downstream Nodes:
>>
>>> # Run complete. Total time: 00:00:20
>>>
>>> Benchmark Mode  CntScore   Error  Units
>>> PapiBenchmark.runTestDriver  thrpt   15  2246686.693 ± 56372.920  ops/s
>>
>> One Downstream Node:
>>
>>> # Run complete. Total time: 00:00:20
>>>
>>> Benchmark Mode  CntScore   Error  Units
>>> PapiBenchmark.runTestDriver  thrpt   15  2206277.298 ± 51855.465  ops/s
>>
>> Five Downstream Nodes:
>>
>>> # Run complete. Total time: 00:00:20
>>>
>>> Benchmark Mode  CntScore   Error  Units
>>> PapiBenchmark.runTestDriver  thrpt   15  1855833.516 ± 46901.811  ops/s
>>
>>
>> Repeating the same on my PR branch I get the following numbers:
>>
>> Zero Downstream Nodes:
>>
>>> # Run complete. Total time: 00:00:20
>>>
>>> Benchmark Mode  CntScore   Error  Units
>>> PapiBenchmark.runTestDriver  thrpt   15  2192891.762 ± 77598.908  ops/s
>>
>> One Downstream Node:
>>
>>> # Run complete. Total time: 00:00:20
>>>
>>> Benchmark Mode  CntScore   Error  Units
>>> PapiBenchmark.runTestDriver  thrpt   15  2190676.716 ± 77030.594  ops/s
>>
>> Five Downstream Nodes:
>>
>>> # Run complete. Total time: 00:00:20
>>>
>>> Benchmark Mode  CntScore   Error  Units
>>> PapiBenchmark.runTestDriver  thrpt   15  1921632.144 ± 66276.232  ops/s
>>
>>
>> I also had a look in GC and did not observe an issues. The objects that
>> get created are all in young gen and thus cleaning them up is cheap.
>>
>> Let me know if this addresses your concerns.
>>
>>
>> -Matthias
>>
>>
>>
>>
>> On 2/11/18 9:36 PM, Guozhang Wang wrote:
>>> Hi Matthias,
>>>
>>> Just clarifying a meta question along side with my vote: we still need to
>>> understand the overhead of the `To` objects during run time to determine
>>> whether we would use it in the final proposal or using overloading
>>> functions. Right?
>>>
>>>
>>> Guozhang
>>>
>>> On Sun, Feb 11, 2018 at 9:33 PM, Guozhang Wang 
>> wrote:
>>>
 +1

 On Fri, Feb 9, 2018 at 5:31 AM, Bill Bejeck  wrote:

> Thanks for the KIP, +1 for me.
>
> -Bill
>
> On Fri, Feb 9, 2018 at 6:45 AM, Damian Guy 
>> wrote:
>
>> Thanks Matthias, +1
>>
>> On Fri, 9 Feb 2018 at 02:42 Ted Yu  wrote:
>>
>>> +1
>>>  Original message From: "Matthias J. Sax" <
>>> matth...@confluent.io> Date: 2/8/18  6:05 PM  (GMT-08:00) To:
>>> dev@kafka.apache.org Subject: [VOTE] KIP-251: Allow timestamp
>>> manipulation in Processor API
>>> Hi,
>>>
>>> I want to start the vote for KIP-251:
>>>
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> 251%3A+Allow+timestamp+manipulation+in+Processor+API
>>>
>>>
>>> -Matthias
>>>
>>>
>>
>



 --
 -- Guozhang

>>>
>>>
>>>
>>
>>
> 
> 



signature.asc
Description: OpenPGP digital signature

Build failed in Jenkins: kafka-trunk-jdk8 #2461

[Apache Jenkins Server]

See 


Changes:

[github] MINOR: Fix record conversion time in metrics (#4671)

[wangguoz] KAFKA-4831: Extract WindowedSerde to public APIs (#3307)

--
[...truncated 419.40 KB...]

kafka.controller.ReplicaStateMachineTest > 
testInvalidOfflineReplicaToReplicaDeletionIneligibleTransition PASSED

kafka.controller.ReplicaStateMachineTest > 
testNewReplicaToOfflineReplicaTransition STARTED

kafka.controller.ReplicaStateMachineTest > 
testNewReplicaToOfflineReplicaTransition PASSED

kafka.controller.ReplicaStateMachineTest > 
testInvalidReplicaDeletionSuccessfulToOnlineReplicaTransition STARTED

kafka.controller.ReplicaStateMachineTest > 
testInvalidReplicaDeletionSuccessfulToOnlineReplicaTransition PASSED

kafka.controller.ReplicaStateMachineTest > 
testInvalidNonexistentReplicaToReplicaDeletionStartedTransition STARTED

kafka.controller.ReplicaStateMachineTest > 
testInvalidNonexistentReplicaToReplicaDeletionStartedTransition PASSED

kafka.controller.ReplicaStateMachineTest > 
testInvalidOnlineReplicaToReplicaDeletionSuccessfulTransition STARTED

kafka.controller.ReplicaStateMachineTest > 
testInvalidOnlineReplicaToReplicaDeletionSuccessfulTransition PASSED

kafka.controller.ReplicaStateMachineTest > 
testInvalidOfflineReplicaToNonexistentReplicaTransition STARTED

kafka.controller.ReplicaStateMachineTest > 
testInvalidOfflineReplicaToNonexistentReplicaTransition PASSED

kafka.controller.ReplicaStateMachineTest > 
testInvalidOnlineReplicaToReplicaDeletionIneligibleTransition STARTED

kafka.controller.ReplicaStateMachineTest > 
testInvalidOnlineReplicaToReplicaDeletionIneligibleTransition PASSED

kafka.controller.ReplicaStateMachineTest > 
testInvalidReplicaDeletionSuccessfulToReplicaDeletionStartedTransition STARTED

kafka.controller.ReplicaStateMachineTest > 
testInvalidReplicaDeletionSuccessfulToReplicaDeletionStartedTransition PASSED

kafka.controller.ReplicaStateMachineTest > 
testInvalidNewReplicaToReplicaDeletionSuccessfulTransition STARTED

kafka.controller.ReplicaStateMachineTest > 
testInvalidNewReplicaToReplicaDeletionSuccessfulTransition PASSED

kafka.controller.ReplicaStateMachineTest > 
testInvalidReplicaDeletionIneligibleToReplicaDeletionStartedTransition STARTED

kafka.controller.ReplicaStateMachineTest > 
testInvalidReplicaDeletionIneligibleToReplicaDeletionStartedTransition PASSED

kafka.controller.ReplicaStateMachineTest > 
testInvalidReplicaDeletionStartedToOfflineReplicaTransition STARTED

kafka.controller.ReplicaStateMachineTest > 
testInvalidReplicaDeletionStartedToOfflineReplicaTransition PASSED

kafka.controller.ReplicaStateMachineTest > 
testInvalidNewReplicaToReplicaDeletionStartedTransition STARTED

kafka.controller.ReplicaStateMachineTest > 
testInvalidNewReplicaToReplicaDeletionStartedTransition PASSED

kafka.controller.ReplicaStateMachineTest > 
testOnlineReplicaToOnlineReplicaTransition STARTED

kafka.controller.ReplicaStateMachineTest > 
testOnlineReplicaToOnlineReplicaTransition PASSED

kafka.controller.ReplicaStateMachineTest > 
testInvalidNewReplicaToReplicaDeletionIneligibleTransition STARTED

kafka.controller.ReplicaStateMachineTest > 
testInvalidNewReplicaToReplicaDeletionIneligibleTransition PASSED

kafka.controller.ReplicaStateMachineTest > 
testInvalidNonexistentReplicaToOfflineReplicaTransition STARTED

kafka.controller.ReplicaStateMachineTest > 
testInvalidNonexistentReplicaToOfflineReplicaTransition PASSED

kafka.controller.ReplicaStateMachineTest > 
testReplicaDeletionStartedToReplicaDeletionSuccessfulTransition STARTED

kafka.controller.ReplicaStateMachineTest > 
testReplicaDeletionStartedToReplicaDeletionSuccessfulTransition PASSED

kafka.controller.ReplicaStateMachineTest > 
testInvalidNonexistentReplicaToOnlineReplicaTransition STARTED

kafka.controller.ReplicaStateMachineTest > 
testInvalidNonexistentReplicaToOnlineReplicaTransition PASSED

kafka.controller.ReplicaStateMachineTest > 
testInvalidOnlineReplicaToNewReplicaTransition STARTED

kafka.controller.ReplicaStateMachineTest > 
testInvalidOnlineReplicaToNewReplicaTransition PASSED

kafka.controller.ReplicaStateMachineTest > 
testReplicaDeletionStartedToReplicaDeletionIneligibleTransition STARTED

kafka.controller.ReplicaStateMachineTest > 
testReplicaDeletionStartedToReplicaDeletionIneligibleTransition PASSED

kafka.controller.ReplicaStateMachineTest > 
testInvalidReplicaDeletionSuccessfulToOfflineReplicaTransition STARTED

kafka.controller.ReplicaStateMachineTest > 
testInvalidReplicaDeletionSuccessfulToOfflineReplicaTransition PASSED

kafka.controller.ReplicaStateMachineTest > 
testInvalidReplicaDeletionIneligibleToNewReplicaTransition STARTED

kafka.controller.ReplicaStateMachineTest > 
testInvalidReplicaDeletionIneligibleToNewReplicaTransition PASSED

kafka.controller.ReplicaStateMachineTest > 
testReplicaDeletionIneligibleToOnlineReplicaTransition STARTED

kafka.controller.Replica

Jenkins build is back to normal : kafka-trunk-jdk7 #3238

[Apache Jenkins Server]

See 

[jira] [Resolved] (KAFKA-6622) GroupMetadataManager.loadGroupsAndOffsets decompresses record batch needlessly

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-6622.

   Resolution: Fixed
Fix Version/s: 1.1.0

> GroupMetadataManager.loadGroupsAndOffsets decompresses record batch needlessly
> --
>
> Key: KAFKA-6622
> URL: https://issues.apache.org/jira/browse/KAFKA-6622
> Project: Kafka
>  Issue Type: Bug
>Reporter: radai rosenblatt
>Assignee: radai rosenblatt
>Priority: Major
> Fix For: 1.1.0
>
> Attachments: kafka batch iteration funtime.png
>
>
> when reading records from a consumer offsets batch, the entire batch is 
> decompressed multiple times (per record) as part of calling 
> `batch.baseOffset`. this is a very expensive operation being called in a loop 
> for no reason:
> !kafka batch iteration funtime.png!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] KIP-258: Allow to Store Record Timestamps in RocksDB

[Matthias J. Sax]

@Bill: I think a filter predicate should be part of user code. And even
if we want to add something like this, I would prefer to do it in a
separate KIP.


@James: I would love to avoid a second rolling bounce. But from my
understanding it would not be possible.

The purpose of the second rolling bounce is indeed to switch from
version 2 to 3. It also has a second purpose, to switch from the old
store to the new store (this happens after the last instance bounces a
second time).

The problem with one round of rolling bounces is, that it's unclear when
to which from version 2 to version 3. The StreamsPartitionsAssignor is
stateless by design, and thus, the information which version it should
use must be passed in from externally -- and we want to use the
StreamsConfig to pass in this information.

During upgrade, all new instanced have no information about the progress
of the upgrade (ie, how many other instanced got upgrades already).
Therefore, it's not safe for them to send a version 3 subscription. The
leader also has this limited view on the world and can only send version
2 assignments back.

Thus, for the 1.2 upgrade, I don't think we can simplify the upgrade.

We did consider to change the metadata to make later upgrades (ie, from
1.2 to 1.x) simpler though (for the case we change the metadata or
storage format again -- as long as we don't change it, a single rolling
bounce is sufficient), by encoding "used version" and "supported
version". This would allow the leader to switch to the new version
earlier and without a second rebalance: leader would receive "used
version == old" and "supported version = old/new" -- as long as at least
one instance sends a "supported version = old" leader sends old version
assignment back. However, encoding both version would allow that the
leader can send a new version assignment back, right after the first
round or rebalance finished (all instances send "supported version =
new"). However, there are still two issues with this:

1) if we switch to the new format right after the last instance bounced,
the new stores might not be ready to be used -- this could lead to
"downtime" as store must be restored before processing can resume.

2) Assume an instance fails and is restarted again. At this point, the
instance will still have "upgrade mode" enabled and thus sends the old
protocol data. However, it would be desirable to never fall back to the
old protocol after the switch to the new protocol.

The second issue is minor and I guess if users set-up the instance
properly it could be avoided. However, the first issue would prevent
"zero downtime" upgrades. Having said this, if we consider that we might
change the metadata protocol only if a future release, encoding both
used and supported version might be an advantage in the future and we
could consider to add this information in 1.2 release to prepare for this.

Btw: monitoring the log, is also only required to give the instances
enough time to prepare the stores in new format. If you would do the
second rolling bounce before this, it would still work -- however, you
might see app "downtime" as the new store must be fully restored before
processing can resume.


Does this make sense?


-Matthias



On 3/9/18 11:36 AM, James Cheng wrote:
> Matthias,
> 
> For all the upgrade paths, is it possible to get rid of the 2nd rolling 
> bounce?
> 
> For the in-place upgrade, it seems like primary difference between the 1st 
> rolling bounce and the 2nd rolling bounce is to decide whether to send 
> Subscription Version 2 or Subscription Version 3.  (Actually, there is 
> another difference mentioned in that the KIP says that the 2nd rolling bounce 
> should happen after all new state stores are created by the background 
> thread. However, within the 2nd rolling bounce, we say that there is still a 
> background thread, so it seems like is no actual requirement to wait for the 
> new state stores to be created.)
> 
> The 2nd rolling bounce already knows how to deal with mixed-mode (having both 
> Version 2 and Version 3 in the same consumer group). It seems like we could 
> get rid of the 2nd bounce if we added logic (somehow/somewhere) such that:
> * Instances send Subscription Version 2 until all instances are running the 
> new code.
> * Once all the instances are running the new code, then one at a time, the 
> instances start sending Subscription V3. Leader still hands out Assignment 
> Version 2, until all new state stores are ready.
> * Once all instances report that new stores are ready, Leader sends out 
> Assignment Version 3.
> * Once an instance receives an Assignment Version 3, it can delete the old 
> state store.
> 
> Doing it that way seems like it would reduce a lot of operator/deployment 
> overhead. No need to do 2 rolling restarts. No need to monitor logs for state 
> store rebuild. You just deploy it, and the instances update themselves.
> 
> What do you think?
> 
> The thing that made me think of this is tha

Build failed in Jenkins: kafka-trunk-jdk8 #2462

[Apache Jenkins Server]

See 


Changes:

[ismael] MINOR: Add PayloadGenerator to Trogdor (#4640)

--
[...truncated 414.80 KB...]

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithConcurrentTransactionsOnAddPartitionsWhenStateIsPrepareCommit 
PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldIncrementEpochAndUpdateMetadataOnHandleInitPidWhenExistingCompleteTransaction
 STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldIncrementEpochAndUpdateMetadataOnHandleInitPidWhenExistingCompleteTransaction
 PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldGenerateNewProducerIdIfEpochsExhausted STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldGenerateNewProducerIdIfEpochsExhausted PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithNotCoordinatorOnInitPidWhenNotCoordinator STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithNotCoordinatorOnInitPidWhenNotCoordinator PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithConcurrentTransactionOnAddPartitionsWhenStateIsPrepareAbort 
STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithConcurrentTransactionOnAddPartitionsWhenStateIsPrepareAbort 
PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldInitPidWithEpochZeroForNewTransactionalId STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldInitPidWithEpochZeroForNewTransactionalId PASSED

kafka.coordinator.transaction.ProducerIdManagerTest > testExceedProducerIdLimit 
STARTED

kafka.coordinator.transaction.ProducerIdManagerTest > testExceedProducerIdLimit 
PASSED

kafka.coordinator.transaction.ProducerIdManagerTest > testGetProducerId STARTED

kafka.coordinator.transaction.ProducerIdManagerTest > testGetProducerId PASSED

kafka.coordinator.transaction.TransactionMarkerChannelManagerTest > 
shouldSaveForLaterWhenLeaderUnknownButNotAvailable STARTED

kafka.coordinator.transaction.TransactionMarkerChannelManagerTest > 
shouldSaveForLaterWhenLeaderUnknownButNotAvailable PASSED

kafka.coordinator.transaction.TransactionMarkerChannelManagerTest > 
shouldGenerateEmptyMapWhenNoRequestsOutstanding STARTED

kafka.coordinator.transaction.TransactionMarkerChannelManagerTest > 
shouldGenerateEmptyMapWhenNoRequestsOutstanding PASSED

kafka.coordinator.transaction.TransactionMarkerChannelManagerTest > 
shouldCreateMetricsOnStarting STARTED

kafka.coordinator.transaction.TransactionMarkerChannelManagerTest > 
shouldCreateMetricsOnStarting PASSED

kafka.coordinator.transaction.TransactionMarkerChannelManagerTest > 
shouldAbortAppendToLogOnEndTxnWhenNotCoordinatorError STARTED

kafka.coordinator.transaction.TransactionMarkerChannelManagerTest > 
shouldAbortAppendToLogOnEndTxnWhenNotCoordinatorError PASSED

kafka.coordinator.transaction.TransactionMarkerChannelManagerTest > 
shouldRetryAppendToLogOnEndTxnWhenCoordinatorNotAvailableError STARTED

kafka.coordinator.transaction.TransactionMarkerChannelManagerTest > 
shouldRetryAppendToLogOnEndTxnWhenCoordinatorNotAvailableError PASSED

kafka.coordinator.transaction.TransactionMarkerChannelManagerTest > 
shouldCompleteAppendToLogOnEndTxnWhenSendMarkersSucceed STARTED

kafka.coordinator.transaction.TransactionMarkerChannelManagerTest > 
shouldCompleteAppendToLogOnEndTxnWhenSendMarkersSucceed PASSED

kafka.coordinator.transaction.TransactionMarkerChannelManagerTest > 
shouldGenerateRequestPerPartitionPerBroker STARTED

kafka.coordinator.transaction.TransactionMarkerChannelManagerTest > 
shouldGenerateRequestPerPartitionPerBroker PASSED

kafka.coordinator.transaction.TransactionMarkerChannelManagerTest > 
shouldRemoveMarkersForTxnPartitionWhenPartitionEmigrated STARTED

kafka.coordinator.transaction.TransactionMarkerChannelManagerTest > 
shouldRemoveMarkersForTxnPartitionWhenPartitionEmigrated PASSED

kafka.coordinator.transaction.TransactionMarkerChannelManagerTest > 
shouldSkipSendMarkersWhenLeaderNotFound STARTED

kafka.coordinator.transaction.TransactionMarkerChannelManagerTest > 
shouldSkipSendMarkersWhenLeaderNotFound PASSED

kafka.coordinator.transaction.TransactionLogTest > shouldReadWriteMessages 
STARTED

kafka.coordinator.transaction.TransactionLogTest > shouldReadWriteMessages 
PASSED

kafka.coordinator.transaction.TransactionLogTest > 
shouldThrowExceptionWriteInvalidTxn STARTED

kafka.coordinator.transaction.TransactionLogTest > 
shouldThrowExceptionWriteInvalidTxn PASSED

kafka.coordinator.transaction.TransactionStateManagerTest > 
testAppendTransactionToLogWhileProducerFenced STARTED

kafka.coordinator.transaction.TransactionStateManagerTest > 
testAppendTransactionToLogWhileProducerFenced PASSED

kafka.coordinator.transaction.TransactionStateManagerTest > 
testCompleteTransitionWhenAppendSu

[jira] [Created] (KAFKA-6634) Delay initiating the txn on producers until initializeTopology with EOS turned on

[Guozhang Wang (JIRA)]

Guozhang Wang created KAFKA-6634:


 Summary: Delay initiating the txn on producers until 
initializeTopology with EOS turned on
 Key: KAFKA-6634
 URL: https://issues.apache.org/jira/browse/KAFKA-6634
 Project: Kafka
  Issue Type: Improvement
  Components: streams
Reporter: Guozhang Wang
Assignee: Guozhang Wang


In Streams EOS implementation, the created producers for tasks will initiate a 
txn immediately after being created in the constructor of `StreamTask`. 
However, the task may not process any data and hence producer may not send any 
records for that started txn for a long time because of the restoration 
process. And with default txn.session.timeout valued at 60 seconds, it means 
that if the restoration takes more than that amount of time, upon starting the 
producer will immediately get the error that its producer epoch is already old.

To fix this, we should consider instantiating the txn only after the 
restoration phase is done. Although this may have a caveat that if the producer 
is already fenced, it will not be notified until then, in initializeTopology. 
But I think this should not be a correctness issue since during the restoration 
process we do not make any changes to the processing state.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] KIP-258: Allow to Store Record Timestamps in RocksDB

[Ted Yu]

Thanks for the details, Matthias.

bq. change the metadata protocol only if a future release, encoding both used
and supported version might be an advantage

Looks like encoding both versions wouldn't be implemented in this KIP.

Please consider logging a JIRA with the encoding proposal.

Cheers

On Fri, Mar 9, 2018 at 2:27 PM, Matthias J. Sax 
wrote:

> @Bill: I think a filter predicate should be part of user code. And even
> if we want to add something like this, I would prefer to do it in a
> separate KIP.
>
>
> @James: I would love to avoid a second rolling bounce. But from my
> understanding it would not be possible.
>
> The purpose of the second rolling bounce is indeed to switch from
> version 2 to 3. It also has a second purpose, to switch from the old
> store to the new store (this happens after the last instance bounces a
> second time).
>
> The problem with one round of rolling bounces is, that it's unclear when
> to which from version 2 to version 3. The StreamsPartitionsAssignor is
> stateless by design, and thus, the information which version it should
> use must be passed in from externally -- and we want to use the
> StreamsConfig to pass in this information.
>
> During upgrade, all new instanced have no information about the progress
> of the upgrade (ie, how many other instanced got upgrades already).
> Therefore, it's not safe for them to send a version 3 subscription. The
> leader also has this limited view on the world and can only send version
> 2 assignments back.
>
> Thus, for the 1.2 upgrade, I don't think we can simplify the upgrade.
>
> We did consider to change the metadata to make later upgrades (ie, from
> 1.2 to 1.x) simpler though (for the case we change the metadata or
> storage format again -- as long as we don't change it, a single rolling
> bounce is sufficient), by encoding "used version" and "supported
> version". This would allow the leader to switch to the new version
> earlier and without a second rebalance: leader would receive "used
> version == old" and "supported version = old/new" -- as long as at least
> one instance sends a "supported version = old" leader sends old version
> assignment back. However, encoding both version would allow that the
> leader can send a new version assignment back, right after the first
> round or rebalance finished (all instances send "supported version =
> new"). However, there are still two issues with this:
>
> 1) if we switch to the new format right after the last instance bounced,
> the new stores might not be ready to be used -- this could lead to
> "downtime" as store must be restored before processing can resume.
>
> 2) Assume an instance fails and is restarted again. At this point, the
> instance will still have "upgrade mode" enabled and thus sends the old
> protocol data. However, it would be desirable to never fall back to the
> old protocol after the switch to the new protocol.
>
> The second issue is minor and I guess if users set-up the instance
> properly it could be avoided. However, the first issue would prevent
> "zero downtime" upgrades. Having said this, if we consider that we might
> change the metadata protocol only if a future release, encoding both
> used and supported version might be an advantage in the future and we
> could consider to add this information in 1.2 release to prepare for this.
>
> Btw: monitoring the log, is also only required to give the instances
> enough time to prepare the stores in new format. If you would do the
> second rolling bounce before this, it would still work -- however, you
> might see app "downtime" as the new store must be fully restored before
> processing can resume.
>
>
> Does this make sense?
>
>
> -Matthias
>
>
>
> On 3/9/18 11:36 AM, James Cheng wrote:
> > Matthias,
> >
> > For all the upgrade paths, is it possible to get rid of the 2nd rolling
> bounce?
> >
> > For the in-place upgrade, it seems like primary difference between the
> 1st rolling bounce and the 2nd rolling bounce is to decide whether to send
> Subscription Version 2 or Subscription Version 3.  (Actually, there is
> another difference mentioned in that the KIP says that the 2nd rolling
> bounce should happen after all new state stores are created by the
> background thread. However, within the 2nd rolling bounce, we say that
> there is still a background thread, so it seems like is no actual
> requirement to wait for the new state stores to be created.)
> >
> > The 2nd rolling bounce already knows how to deal with mixed-mode (having
> both Version 2 and Version 3 in the same consumer group). It seems like we
> could get rid of the 2nd bounce if we added logic (somehow/somewhere) such
> that:
> > * Instances send Subscription Version 2 until all instances are running
> the new code.
> > * Once all the instances are running the new code, then one at a time,
> the instances start sending Subscription V3. Leader still hands out
> Assignment Version 2, until all new state stores are ready.
> > * Once all i

Re: [DISCUSS] KIP-258: Allow to Store Record Timestamps in RocksDB

[John Roesler]

Hey James and Matthias,

It seems clear (to me) that there's no way to avoid a double bounce for
this release.

But I do think we should figure out whether there's a feature we can put in
right now to allow future releases to be single-bounce. I'm just thinking
that this double bounce thing is the kind of operational paper cut that
become super annoying the second or third time you have to do it.

Preliminarily, I think we should consider these two problems as orthogonal:
(1) rolling upgrade to protocol versions
(2) upgrading state store (rocks) format

For (1), I think something like this would probably be a change to the
Consumer API, not Streams, which would be a separate KIP. I think it's
worth thinking about doing something along these lines in 1.2 to facilitate
smooth future releases. But all these concepts are extremely muddy for me,
and I'm aware that the Consumer API is extremely detailed and sensitive, so
I'm planning to do a lot more reading before considering whether I want to
file a KIP.

For (2), it's worth noting that (1) already forces us to do a
double-bounce, so refining the state-store upgrade right now doesn't buy us
any operational simplicity. However, let's assume we separately decide to
make some changes for (1) in 1.2 such that future releases could be
single-bounce. Is there some groundwork that we need to lay in 1.2 to
position us for online state store upgrades?

I think actually we do not. I think I can show that there is some, say 1.3
implementation that can upgrade state stores online independent of any code
in 1.2.

For example, a 1.3 instance can start up, detect its 1.2-formatted store,
include this information in its subscribe(). If it gets the subscription,
it can create a 1.3 store and begin restoring it, monitoring the progress
until it's ready to swap, and then swap. For this whole process to work, we
don't depend on any features to already be in place in 1.2.

Considering both of those points, I think Matthias's existing proposal is
the best we can do within the scope of this KIP.

-John

P.S., it did just occur to me that tackling (2) right now buys us one small
win: operators won't have to monitor the logs for store readiness before
kicking off the second bounce. Is that alone enough of a win to justify
tackling this right now? Not sure.

On Fri, Mar 9, 2018 at 2:27 PM, Matthias J. Sax 
wrote:

> @Bill: I think a filter predicate should be part of user code. And even
> if we want to add something like this, I would prefer to do it in a
> separate KIP.
>
>
> @James: I would love to avoid a second rolling bounce. But from my
> understanding it would not be possible.
>
> The purpose of the second rolling bounce is indeed to switch from
> version 2 to 3. It also has a second purpose, to switch from the old
> store to the new store (this happens after the last instance bounces a
> second time).
>
> The problem with one round of rolling bounces is, that it's unclear when
> to which from version 2 to version 3. The StreamsPartitionsAssignor is
> stateless by design, and thus, the information which version it should
> use must be passed in from externally -- and we want to use the
> StreamsConfig to pass in this information.
>
> During upgrade, all new instanced have no information about the progress
> of the upgrade (ie, how many other instanced got upgrades already).
> Therefore, it's not safe for them to send a version 3 subscription. The
> leader also has this limited view on the world and can only send version
> 2 assignments back.
>
> Thus, for the 1.2 upgrade, I don't think we can simplify the upgrade.
>
> We did consider to change the metadata to make later upgrades (ie, from
> 1.2 to 1.x) simpler though (for the case we change the metadata or
> storage format again -- as long as we don't change it, a single rolling
> bounce is sufficient), by encoding "used version" and "supported
> version". This would allow the leader to switch to the new version
> earlier and without a second rebalance: leader would receive "used
> version == old" and "supported version = old/new" -- as long as at least
> one instance sends a "supported version = old" leader sends old version
> assignment back. However, encoding both version would allow that the
> leader can send a new version assignment back, right after the first
> round or rebalance finished (all instances send "supported version =
> new"). However, there are still two issues with this:
>
> 1) if we switch to the new format right after the last instance bounced,
> the new stores might not be ready to be used -- this could lead to
> "downtime" as store must be restored before processing can resume.
>
> 2) Assume an instance fails and is restarted again. At this point, the
> instance will still have "upgrade mode" enabled and thus sends the old
> protocol data. However, it would be desirable to never fall back to the
> old protocol after the switch to the new protocol.
>
> The second issue is minor and I guess if users set-up t

Re: [DISCUSS] KIP-258: Allow to Store Record Timestamps in RocksDB

[Matthias J. Sax]

Ted,

I still consider changing the KIP to include it right away -- if not,
I'll create a JIRA. Need to think it through in more detail first.

(Same for other open questions like interface names -- I collect
feedback and update the KIP after we reach consensus :))

-Matthias

On 3/9/18 3:35 PM, Ted Yu wrote:
> Thanks for the details, Matthias.
> 
> bq. change the metadata protocol only if a future release, encoding both used
> and supported version might be an advantage
> 
> Looks like encoding both versions wouldn't be implemented in this KIP.
> 
> Please consider logging a JIRA with the encoding proposal.
> 
> Cheers
> 
> On Fri, Mar 9, 2018 at 2:27 PM, Matthias J. Sax 
> wrote:
> 
>> @Bill: I think a filter predicate should be part of user code. And even
>> if we want to add something like this, I would prefer to do it in a
>> separate KIP.
>>
>>
>> @James: I would love to avoid a second rolling bounce. But from my
>> understanding it would not be possible.
>>
>> The purpose of the second rolling bounce is indeed to switch from
>> version 2 to 3. It also has a second purpose, to switch from the old
>> store to the new store (this happens after the last instance bounces a
>> second time).
>>
>> The problem with one round of rolling bounces is, that it's unclear when
>> to which from version 2 to version 3. The StreamsPartitionsAssignor is
>> stateless by design, and thus, the information which version it should
>> use must be passed in from externally -- and we want to use the
>> StreamsConfig to pass in this information.
>>
>> During upgrade, all new instanced have no information about the progress
>> of the upgrade (ie, how many other instanced got upgrades already).
>> Therefore, it's not safe for them to send a version 3 subscription. The
>> leader also has this limited view on the world and can only send version
>> 2 assignments back.
>>
>> Thus, for the 1.2 upgrade, I don't think we can simplify the upgrade.
>>
>> We did consider to change the metadata to make later upgrades (ie, from
>> 1.2 to 1.x) simpler though (for the case we change the metadata or
>> storage format again -- as long as we don't change it, a single rolling
>> bounce is sufficient), by encoding "used version" and "supported
>> version". This would allow the leader to switch to the new version
>> earlier and without a second rebalance: leader would receive "used
>> version == old" and "supported version = old/new" -- as long as at least
>> one instance sends a "supported version = old" leader sends old version
>> assignment back. However, encoding both version would allow that the
>> leader can send a new version assignment back, right after the first
>> round or rebalance finished (all instances send "supported version =
>> new"). However, there are still two issues with this:
>>
>> 1) if we switch to the new format right after the last instance bounced,
>> the new stores might not be ready to be used -- this could lead to
>> "downtime" as store must be restored before processing can resume.
>>
>> 2) Assume an instance fails and is restarted again. At this point, the
>> instance will still have "upgrade mode" enabled and thus sends the old
>> protocol data. However, it would be desirable to never fall back to the
>> old protocol after the switch to the new protocol.
>>
>> The second issue is minor and I guess if users set-up the instance
>> properly it could be avoided. However, the first issue would prevent
>> "zero downtime" upgrades. Having said this, if we consider that we might
>> change the metadata protocol only if a future release, encoding both
>> used and supported version might be an advantage in the future and we
>> could consider to add this information in 1.2 release to prepare for this.
>>
>> Btw: monitoring the log, is also only required to give the instances
>> enough time to prepare the stores in new format. If you would do the
>> second rolling bounce before this, it would still work -- however, you
>> might see app "downtime" as the new store must be fully restored before
>> processing can resume.
>>
>>
>> Does this make sense?
>>
>>
>> -Matthias
>>
>>
>>
>> On 3/9/18 11:36 AM, James Cheng wrote:
>>> Matthias,
>>>
>>> For all the upgrade paths, is it possible to get rid of the 2nd rolling
>> bounce?
>>>
>>> For the in-place upgrade, it seems like primary difference between the
>> 1st rolling bounce and the 2nd rolling bounce is to decide whether to send
>> Subscription Version 2 or Subscription Version 3.  (Actually, there is
>> another difference mentioned in that the KIP says that the 2nd rolling
>> bounce should happen after all new state stores are created by the
>> background thread. However, within the 2nd rolling bounce, we say that
>> there is still a background thread, so it seems like is no actual
>> requirement to wait for the new state stores to be created.)
>>>
>>> The 2nd rolling bounce already knows how to deal with mixed-mode (having
>> both Version 2 and Version 3 in the same consumer group)

Build failed in Jenkins: kafka-1.1-jdk7 #72

[Apache Jenkins Server]

See 


Changes:

[jason] KAFKA-6622; Fix performance issue loading consumer offsets (#4661)

[junrao] MINOR: Fix deadlock in ZooKeeperClient.close() on session expiry 
(#4672)

--
[...truncated 416.32 KB...]
kafka.message.ByteBufferMessageSetTest > testWriteTo PASSED

kafka.message.ByteBufferMessageSetTest > testEquals STARTED

kafka.message.ByteBufferMessageSetTest > testEquals PASSED

kafka.message.ByteBufferMessageSetTest > testSizeInBytes STARTED

kafka.message.ByteBufferMessageSetTest > testSizeInBytes PASSED

kafka.message.ByteBufferMessageSetTest > testIterator STARTED

kafka.message.ByteBufferMessageSetTest > testIterator PASSED

kafka.metrics.KafkaTimerTest > testKafkaTimer STARTED

kafka.metrics.KafkaTimerTest > testKafkaTimer PASSED

kafka.metrics.MetricsTest > testMetricsReporterAfterDeletingTopic STARTED

kafka.metrics.MetricsTest > testMetricsReporterAfterDeletingTopic PASSED

kafka.metrics.MetricsTest > testSessionExpireListenerMetrics STARTED

kafka.metrics.MetricsTest > testSessionExpireListenerMetrics PASSED

kafka.metrics.MetricsTest > 
testBrokerTopicMetricsUnregisteredAfterDeletingTopic STARTED

kafka.metrics.MetricsTest > 
testBrokerTopicMetricsUnregisteredAfterDeletingTopic PASSED

kafka.metrics.MetricsTest > testClusterIdMetric STARTED

kafka.metrics.MetricsTest > testClusterIdMetric PASSED

kafka.metrics.MetricsTest > testControllerMetrics STARTED

kafka.metrics.MetricsTest > testControllerMetrics PASSED

kafka.metrics.MetricsTest > testWindowsStyleTagNames STARTED

kafka.metrics.MetricsTest > testWindowsStyleTagNames PASSED

kafka.metrics.MetricsTest > testMetricsLeak STARTED

kafka.metrics.MetricsTest > testMetricsLeak PASSED

kafka.metrics.MetricsTest > testBrokerTopicMetricsBytesInOut STARTED

kafka.metrics.MetricsTest > testBrokerTopicMetricsBytesInOut PASSED

kafka.javaapi.consumer.ZookeeperConsumerConnectorTest > testBasic STARTED

kafka.javaapi.consumer.ZookeeperConsumerConnectorTest > testBasic PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testSizeInBytesWithCompression 
STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > testSizeInBytesWithCompression 
PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > 
testIteratorIsConsistentWithCompression STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > 
testIteratorIsConsistentWithCompression PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testIteratorIsConsistent 
STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > testIteratorIsConsistent PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testEqualsWithCompression 
STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > testEqualsWithCompression 
PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testWrittenEqualsRead STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > testWrittenEqualsRead PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testEquals STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > testEquals PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testSizeInBytes STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > testSizeInBytes PASSED

kafka.security.token.delegation.DelegationTokenManagerTest > 
testPeriodicTokenExpiry STARTED

kafka.security.token.delegation.DelegationTokenManagerTest > 
testPeriodicTokenExpiry PASSED

kafka.security.token.delegation.DelegationTokenManagerTest > 
testTokenRequestsWithDelegationTokenDisabled STARTED

kafka.security.token.delegation.DelegationTokenManagerTest > 
testTokenRequestsWithDelegationTokenDisabled PASSED

kafka.security.token.delegation.DelegationTokenManagerTest > testDescribeToken 
STARTED

kafka.security.token.delegation.DelegationTokenManagerTest > testDescribeToken 
PASSED

kafka.security.token.delegation.DelegationTokenManagerTest > testCreateToken 
STARTED

kafka.security.token.delegation.DelegationTokenManagerTest > testCreateToken 
PASSED

kafka.security.token.delegation.DelegationTokenManagerTest > testExpireToken 
STARTED

kafka.security.token.delegation.DelegationTokenManagerTest > testExpireToken 
PASSED

kafka.security.token.delegation.DelegationTokenManagerTest > testRenewToken 
STARTED

kafka.security.token.delegation.DelegationTokenManagerTest > testRenewToken 
PASSED

kafka.security.auth.ResourceTypeTest > testJavaConversions STARTED

kafka.security.auth.ResourceTypeTest > testJavaConversions PASSED

kafka.security.auth.ResourceTypeTest > testFromString STARTED

kafka.security.auth.ResourceTypeTest > testFromString PASSED

kafka.security.auth.OperationTest > testJavaConversions STARTED

kafka.security.auth.OperationTest > testJavaConversions PASSED

kafka.security.auth.PermissionTypeTest > testJavaConversions STARTED

kafka.security.auth.PermissionTypeTest > testJavaConversions PASSED

kafka.security.auth.PermissionTypeTest > testFromString STARTED

kafka.security.auth.PermissionTypeTest > testFrom

Build failed in Jenkins: kafka-trunk-jdk7 #3239

[Apache Jenkins Server]

See 


Changes:

[ismael] MINOR: Add PayloadGenerator to Trogdor (#4640)

[jason] KAFKA-6622; Fix performance issue loading consumer offsets (#4661)

[ismael] MINOR: Tag AWS instances with Jenkins build url (#4657)

--
[...truncated 420.73 KB...]
kafka.zk.KafkaZkClientTest > testIsrChangeNotificationsDeletion PASSED

kafka.zk.KafkaZkClientTest > testGetDataAndVersion STARTED

kafka.zk.KafkaZkClientTest > testGetDataAndVersion PASSED

kafka.zk.KafkaZkClientTest > testGetChildren STARTED

kafka.zk.KafkaZkClientTest > testGetChildren PASSED

kafka.zk.KafkaZkClientTest > testSetAndGetConsumerOffset STARTED

kafka.zk.KafkaZkClientTest > testSetAndGetConsumerOffset PASSED

kafka.zk.KafkaZkClientTest > testClusterIdMethods STARTED

kafka.zk.KafkaZkClientTest > testClusterIdMethods PASSED

kafka.zk.KafkaZkClientTest > testEntityConfigManagementMethods STARTED

kafka.zk.KafkaZkClientTest > testEntityConfigManagementMethods PASSED

kafka.zk.KafkaZkClientTest > testUpdateLeaderAndIsr STARTED

kafka.zk.KafkaZkClientTest > testUpdateLeaderAndIsr PASSED

kafka.zk.KafkaZkClientTest > testUpdateBrokerInfo STARTED

kafka.zk.KafkaZkClientTest > testUpdateBrokerInfo PASSED

kafka.zk.KafkaZkClientTest > testCreateRecursive STARTED

kafka.zk.KafkaZkClientTest > testCreateRecursive PASSED

kafka.zk.KafkaZkClientTest > testGetConsumerOffsetNoData STARTED

kafka.zk.KafkaZkClientTest > testGetConsumerOffsetNoData PASSED

kafka.zk.KafkaZkClientTest > testDeleteTopicPathMethods STARTED

kafka.zk.KafkaZkClientTest > testDeleteTopicPathMethods PASSED

kafka.zk.KafkaZkClientTest > testSetTopicPartitionStatesRaw STARTED

kafka.zk.KafkaZkClientTest > testSetTopicPartitionStatesRaw PASSED

kafka.zk.KafkaZkClientTest > testAclManagementMethods STARTED

kafka.zk.KafkaZkClientTest > testAclManagementMethods PASSED

kafka.zk.KafkaZkClientTest > testPreferredReplicaElectionMethods STARTED

kafka.zk.KafkaZkClientTest > testPreferredReplicaElectionMethods PASSED

kafka.zk.KafkaZkClientTest > testPropagateLogDir STARTED

kafka.zk.KafkaZkClientTest > testPropagateLogDir PASSED

kafka.zk.KafkaZkClientTest > testGetDataAndStat STARTED

kafka.zk.KafkaZkClientTest > testGetDataAndStat PASSED

kafka.zk.KafkaZkClientTest > testReassignPartitionsInProgress STARTED

kafka.zk.KafkaZkClientTest > testReassignPartitionsInProgress PASSED

kafka.zk.KafkaZkClientTest > testCreateTopLevelPaths STARTED

kafka.zk.KafkaZkClientTest > testCreateTopLevelPaths PASSED

kafka.zk.KafkaZkClientTest > testIsrChangeNotificationGetters STARTED

kafka.zk.KafkaZkClientTest > testIsrChangeNotificationGetters PASSED

kafka.zk.KafkaZkClientTest > testLogDirEventNotificationsDeletion STARTED

kafka.zk.KafkaZkClientTest > testLogDirEventNotificationsDeletion PASSED

kafka.zk.KafkaZkClientTest > testGetLogConfigs STARTED

kafka.zk.KafkaZkClientTest > testGetLogConfigs PASSED

kafka.zk.KafkaZkClientTest > testBrokerSequenceIdMethods STARTED

kafka.zk.KafkaZkClientTest > testBrokerSequenceIdMethods PASSED

kafka.zk.KafkaZkClientTest > testCreateSequentialPersistentPath STARTED

kafka.zk.KafkaZkClientTest > testCreateSequentialPersistentPath PASSED

kafka.zk.KafkaZkClientTest > testConditionalUpdatePath STARTED

kafka.zk.KafkaZkClientTest > testConditionalUpdatePath PASSED

kafka.zk.KafkaZkClientTest > testDeleteTopicZNode STARTED

kafka.zk.KafkaZkClientTest > testDeleteTopicZNode PASSED

kafka.zk.KafkaZkClientTest > testDeletePath STARTED

kafka.zk.KafkaZkClientTest > testDeletePath PASSED

kafka.zk.KafkaZkClientTest > testGetBrokerMethods STARTED

kafka.zk.KafkaZkClientTest > testGetBrokerMethods PASSED

kafka.zk.KafkaZkClientTest > testCreateTokenChangeNotification STARTED

kafka.zk.KafkaZkClientTest > testCreateTokenChangeNotification PASSED

kafka.zk.KafkaZkClientTest > testGetTopicsAndPartitions STARTED

kafka.zk.KafkaZkClientTest > testGetTopicsAndPartitions PASSED

kafka.zk.KafkaZkClientTest > testRegisterBrokerInfo STARTED

kafka.zk.KafkaZkClientTest > testRegisterBrokerInfo PASSED

kafka.zk.KafkaZkClientTest > testControllerManagementMethods STARTED

kafka.zk.KafkaZkClientTest > testControllerManagementMethods PASSED

kafka.zk.KafkaZkClientTest > testTopicAssignmentMethods STARTED

kafka.zk.KafkaZkClientTest > testTopicAssignmentMethods PASSED

kafka.zk.KafkaZkClientTest > testPropagateIsrChanges STARTED

kafka.zk.KafkaZkClientTest > testPropagateIsrChanges PASSED

kafka.zk.KafkaZkClientTest > testControllerEpochMethods STARTED

kafka.zk.KafkaZkClientTest > testControllerEpochMethods PASSED

kafka.zk.KafkaZkClientTest > testDeleteRecursive STARTED

kafka.zk.KafkaZkClientTest > testDeleteRecursive PASSED

kafka.zk.KafkaZkClientTest > testGetTopicPartitionStates STARTED

kafka.zk.KafkaZkClientTest > testGetTopicPartitionStates PASSED

kafka.zk.KafkaZkClientTest > testCreateConfigChangeNotification STARTED

kafka.zk.KafkaZkClientTest > testCreateConfigChange

Build failed in Jenkins: kafka-1.0-jdk7 #165

[Apache Jenkins Server]

See 


Changes:

[jason] KAFKA-6622; Fix performance issue loading consumer offsets (#4661)

--
[...truncated 373.68 KB...]
kafka.controller.ZookeeperClientTest > testMixedPipeline PASSED

kafka.controller.ZookeeperClientTest > testGetDataExistingZNode STARTED

kafka.controller.ZookeeperClientTest > testGetDataExistingZNode PASSED

kafka.controller.ZookeeperClientTest > testGetACLExistingZNode STARTED

kafka.controller.ZookeeperClientTest > testGetACLExistingZNode PASSED

kafka.controller.ZookeeperClientTest > testDeleteExistingZNode STARTED

kafka.controller.ZookeeperClientTest > testDeleteExistingZNode PASSED

kafka.controller.ZookeeperClientTest > testSetDataNonExistentZNode STARTED

kafka.controller.ZookeeperClientTest > testSetDataNonExistentZNode PASSED

kafka.controller.ZookeeperClientTest > testDeleteNonExistentZNode STARTED

kafka.controller.ZookeeperClientTest > testDeleteNonExistentZNode PASSED

kafka.controller.ZookeeperClientTest > testExistsExistingZNode STARTED

kafka.controller.ZookeeperClientTest > testExistsExistingZNode PASSED

kafka.controller.ZookeeperClientTest > testZNodeChangeHandlerForDeletion STARTED

kafka.controller.ZookeeperClientTest > testZNodeChangeHandlerForDeletion PASSED

kafka.controller.ZookeeperClientTest > testStateChangeHandlerForAuthFailure 
STARTED

kafka.controller.ZookeeperClientTest > testStateChangeHandlerForAuthFailure 
PASSED

kafka.controller.ControllerEventManagerTest > testEventThatThrowsException 
STARTED

kafka.controller.ControllerEventManagerTest > testEventThatThrowsException 
PASSED

kafka.controller.ControllerEventManagerTest > testSuccessfulEvent STARTED

kafka.controller.ControllerEventManagerTest > testSuccessfulEvent PASSED

kafka.controller.ControllerFailoverTest > testHandleIllegalStateException 
STARTED

kafka.controller.ControllerFailoverTest > testHandleIllegalStateException PASSED

kafka.network.SocketServerTest > testGracefulClose STARTED

kafka.network.SocketServerTest > testGracefulClose PASSED

kafka.network.SocketServerTest > controlThrowable STARTED

kafka.network.SocketServerTest > controlThrowable PASSED

kafka.network.SocketServerTest > testRequestMetricsAfterStop STARTED

kafka.network.SocketServerTest > testRequestMetricsAfterStop PASSED

kafka.network.SocketServerTest > testConnectionIdReuse STARTED

kafka.network.SocketServerTest > testConnectionIdReuse PASSED

kafka.network.SocketServerTest > testClientDisconnectionUpdatesRequestMetrics 
STARTED

kafka.network.SocketServerTest > testClientDisconnectionUpdatesRequestMetrics 
PASSED

kafka.network.SocketServerTest > testProcessorMetricsTags STARTED

kafka.network.SocketServerTest > testProcessorMetricsTags PASSED

kafka.network.SocketServerTest > testMaxConnectionsPerIp STARTED

kafka.network.SocketServerTest > testMaxConnectionsPerIp PASSED

kafka.network.SocketServerTest > testConnectionId STARTED

kafka.network.SocketServerTest > testConnectionId PASSED

kafka.network.SocketServerTest > 
testBrokerSendAfterChannelClosedUpdatesRequestMetrics STARTED

kafka.network.SocketServerTest > 
testBrokerSendAfterChannelClosedUpdatesRequestMetrics PASSED

kafka.network.SocketServerTest > testNoOpAction STARTED

kafka.network.SocketServerTest > testNoOpAction PASSED

kafka.network.SocketServerTest > simpleRequest STARTED

kafka.network.SocketServerTest > simpleRequest PASSED

kafka.network.SocketServerTest > closingChannelException STARTED

kafka.network.SocketServerTest > closingChannelException PASSED

kafka.network.SocketServerTest > testIdleConnection STARTED

kafka.network.SocketServerTest > testIdleConnection PASSED

kafka.network.SocketServerTest > 
testClientDisconnectionWithStagedReceivesFullyProcessed STARTED

kafka.network.SocketServerTest > 
testClientDisconnectionWithStagedReceivesFullyProcessed PASSED

kafka.network.SocketServerTest > testMetricCollectionAfterShutdown STARTED

kafka.network.SocketServerTest > testMetricCollectionAfterShutdown PASSED

kafka.network.SocketServerTest > testSessionPrincipal STARTED

kafka.network.SocketServerTest > testSessionPrincipal PASSED

kafka.network.SocketServerTest > configureNewConnectionException STARTED

kafka.network.SocketServerTest > configureNewConnectionException PASSED

kafka.network.SocketServerTest > testMaxConnectionsPerIpOverrides STARTED

kafka.network.SocketServerTest > testMaxConnectionsPerIpOverrides PASSED

kafka.network.SocketServerTest > processNewResponseException STARTED

kafka.network.SocketServerTest > processNewResponseException PASSED

kafka.network.SocketServerTest > processCompletedSendException STARTED

kafka.network.SocketServerTest > processCompletedSendException PASSED

kafka.network.SocketServerTest > processDisconnectedException STARTED

kafka.network.SocketServerTest > processDisconnectedException PASSED

kafka.network.SocketServerTest > sendCancelledKeyException STARTED

kafka.netwo

Re: [VOTE] 1.1.0 RC1

[Ismael Juma]

Thanks Jeff:

https://github.com/apache/kafka/pull/4678

Ismael

On Fri, Mar 9, 2018 at 1:56 AM, Damian Guy  wrote:

> Hi Jeff,
>
> Thanks, we will look into this.
>
> Regards,
> Damian
>
> On Thu, 8 Mar 2018 at 18:27 Jeff Chao  wrote:
>
> > Hello,
> >
> > We at Heroku have run 1.1.0 RC1 through our normal performance and
> > regression test suite and have found performance to be comparable to
> 1.0.0.
> >
> > That said, we're however -1 (non-binding) since this release includes
> > Zookeeper 3.4.11 
> which
> > is affected by the critical regression ZOOKEEPER-2960
> > . As 3.4.12 isn't
> > released yet, it might be better to have 3.4.10 included instead.
> >
> > Jeff
> > Heroku
> >
> >
> > On Tue, Mar 6, 2018 at 1:19 PM, Ted Yu  wrote:
> >
> > > +1
> > >
> > > Checked signature
> > > Ran test suite - apart from flaky testMetricsLeak, other tests passed.
> > >
> > > On Tue, Mar 6, 2018 at 2:45 AM, Damian Guy 
> wrote:
> > >
> > > > Hello Kafka users, developers and client-developers,
> > > >
> > > > This is the second candidate for release of Apache Kafka 1.1.0.
> > > >
> > > > This is minor version release of Apache Kakfa. It Includes 29 new
> KIPs.
> > > > Please see the release plan for more details:
> > > >
> > > > https://cwiki.apache.org/confluence/pages/viewpage.
> > > action?pageId=71764913
> > > >
> > > > A few highlights:
> > > >
> > > > * Significant Controller improvements (much faster and session
> > expiration
> > > > edge cases fixed)
> > > > * Data balancing across log directories (JBOD)
> > > > * More efficient replication when the number of partitions is large
> > > > * Dynamic Broker Configs
> > > > * Delegation tokens (KIP-48)
> > > > * Kafka Streams API improvements (KIP-205 / 210 / 220 / 224 / 239)
> > > >
> > > > Release notes for the 1.1.0 release:
> > > > http://home.apache.org/~damianguy/kafka-1.1.0-rc1/RELEASE_NOTES.html
> > > >
> > > > *** Please download, test and vote by Friday, March 9th, 5pm PT
> > > >
> > > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > > http://kafka.apache.org/KEYS
> > > >
> > > > * Release artifacts to be voted upon (source and binary):
> > > > http://home.apache.org/~damianguy/kafka-1.1.0-rc1/
> > > >
> > > > * Maven artifacts to be voted upon:
> > > > https://repository.apache.org/content/groups/staging/
> > > >
> > > > * Javadoc:
> > > > http://home.apache.org/~damianguy/kafka-1.1.0-rc1/javadoc/
> > > >
> > > > * Tag to be voted upon (off 1.1 branch) is the 1.1.0 tag:
> > > > https://github.com/apache/kafka/tree/1.1.0-rc1
> > > >
> > > >
> > > > * Documentation:
> > > > http://kafka.apache.org/11/documentation.html
> > > >
> > > > * Protocol:
> > > > http://kafka.apache.org/11/protocol.html
> > > >
> > > > * Successful Jenkins builds for the 1.1 branch:
> > > > Unit/integration tests:
> > https://builds.apache.org/job/kafka-1.1-jdk7/68
> > > > System tests: https://jenkins.confluent.io/
> > > job/system-test-kafka/job/1.1/
> > > > 30/
> > > >
> > > > /**
> > > >
> > > > Thanks,
> > > > Damian Guy
> > > >
> > >
> >
>

Build failed in Jenkins: kafka-trunk-jdk9 #460

[Apache Jenkins Server]

See 


Changes:

[ismael] MINOR: Add PayloadGenerator to Trogdor (#4640)

[jason] KAFKA-6622; Fix performance issue loading consumer offsets (#4661)

[ismael] MINOR: Tag AWS instances with Jenkins build url (#4657)

--
[...truncated 1.48 MB...]

kafka.zk.KafkaZkClientTest > testCreateAndGetTopicPartitionStatesRaw PASSED

kafka.zk.KafkaZkClientTest > testLogDirGetters STARTED

kafka.zk.KafkaZkClientTest > testLogDirGetters PASSED

kafka.zk.KafkaZkClientTest > testSetGetAndDeletePartitionReassignment STARTED

kafka.zk.KafkaZkClientTest > testSetGetAndDeletePartitionReassignment PASSED

kafka.zk.KafkaZkClientTest > testIsrChangeNotificationsDeletion STARTED

kafka.zk.KafkaZkClientTest > testIsrChangeNotificationsDeletion PASSED

kafka.zk.KafkaZkClientTest > testGetDataAndVersion STARTED

kafka.zk.KafkaZkClientTest > testGetDataAndVersion PASSED

kafka.zk.KafkaZkClientTest > testGetChildren STARTED

kafka.zk.KafkaZkClientTest > testGetChildren PASSED

kafka.zk.KafkaZkClientTest > testSetAndGetConsumerOffset STARTED

kafka.zk.KafkaZkClientTest > testSetAndGetConsumerOffset PASSED

kafka.zk.KafkaZkClientTest > testClusterIdMethods STARTED

kafka.zk.KafkaZkClientTest > testClusterIdMethods PASSED

kafka.zk.KafkaZkClientTest > testEntityConfigManagementMethods STARTED

kafka.zk.KafkaZkClientTest > testEntityConfigManagementMethods PASSED

kafka.zk.KafkaZkClientTest > testUpdateLeaderAndIsr STARTED

kafka.zk.KafkaZkClientTest > testUpdateLeaderAndIsr PASSED

kafka.zk.KafkaZkClientTest > testUpdateBrokerInfo STARTED

kafka.zk.KafkaZkClientTest > testUpdateBrokerInfo PASSED

kafka.zk.KafkaZkClientTest > testCreateRecursive STARTED

kafka.zk.KafkaZkClientTest > testCreateRecursive PASSED

kafka.zk.KafkaZkClientTest > testGetConsumerOffsetNoData STARTED

kafka.zk.KafkaZkClientTest > testGetConsumerOffsetNoData PASSED

kafka.zk.KafkaZkClientTest > testDeleteTopicPathMethods STARTED

kafka.zk.KafkaZkClientTest > testDeleteTopicPathMethods PASSED

kafka.zk.KafkaZkClientTest > testSetTopicPartitionStatesRaw STARTED

kafka.zk.KafkaZkClientTest > testSetTopicPartitionStatesRaw PASSED

kafka.zk.KafkaZkClientTest > testAclManagementMethods STARTED

kafka.zk.KafkaZkClientTest > testAclManagementMethods PASSED

kafka.zk.KafkaZkClientTest > testPreferredReplicaElectionMethods STARTED

kafka.zk.KafkaZkClientTest > testPreferredReplicaElectionMethods PASSED

kafka.zk.KafkaZkClientTest > testPropagateLogDir STARTED

kafka.zk.KafkaZkClientTest > testPropagateLogDir PASSED

kafka.zk.KafkaZkClientTest > testGetDataAndStat STARTED

kafka.zk.KafkaZkClientTest > testGetDataAndStat PASSED

kafka.zk.KafkaZkClientTest > testReassignPartitionsInProgress STARTED

kafka.zk.KafkaZkClientTest > testReassignPartitionsInProgress PASSED

kafka.zk.KafkaZkClientTest > testCreateTopLevelPaths STARTED

kafka.zk.KafkaZkClientTest > testCreateTopLevelPaths PASSED

kafka.zk.KafkaZkClientTest > testIsrChangeNotificationGetters STARTED

kafka.zk.KafkaZkClientTest > testIsrChangeNotificationGetters PASSED

kafka.zk.KafkaZkClientTest > testLogDirEventNotificationsDeletion STARTED

kafka.zk.KafkaZkClientTest > testLogDirEventNotificationsDeletion PASSED

kafka.zk.KafkaZkClientTest > testGetLogConfigs STARTED

kafka.zk.KafkaZkClientTest > testGetLogConfigs PASSED

kafka.zk.KafkaZkClientTest > testBrokerSequenceIdMethods STARTED

kafka.zk.KafkaZkClientTest > testBrokerSequenceIdMethods PASSED

kafka.zk.KafkaZkClientTest > testCreateSequentialPersistentPath STARTED

kafka.zk.KafkaZkClientTest > testCreateSequentialPersistentPath PASSED

kafka.zk.KafkaZkClientTest > testConditionalUpdatePath STARTED

kafka.zk.KafkaZkClientTest > testConditionalUpdatePath PASSED

kafka.zk.KafkaZkClientTest > testDeleteTopicZNode STARTED

kafka.zk.KafkaZkClientTest > testDeleteTopicZNode PASSED

kafka.zk.KafkaZkClientTest > testDeletePath STARTED

kafka.zk.KafkaZkClientTest > testDeletePath PASSED

kafka.zk.KafkaZkClientTest > testGetBrokerMethods STARTED

kafka.zk.KafkaZkClientTest > testGetBrokerMethods PASSED

kafka.zk.KafkaZkClientTest > testCreateTokenChangeNotification STARTED

kafka.zk.KafkaZkClientTest > testCreateTokenChangeNotification PASSED

kafka.zk.KafkaZkClientTest > testGetTopicsAndPartitions STARTED

kafka.zk.KafkaZkClientTest > testGetTopicsAndPartitions PASSED

kafka.zk.KafkaZkClientTest > testRegisterBrokerInfo STARTED

kafka.zk.KafkaZkClientTest > testRegisterBrokerInfo PASSED

kafka.zk.KafkaZkClientTest > testControllerManagementMethods STARTED

kafka.zk.KafkaZkClientTest > testControllerManagementMethods PASSED

kafka.zk.KafkaZkClientTest > testTopicAssignmentMethods STARTED

kafka.zk.KafkaZkClientTest > testTopicAssignmentMethods PASSED

kafka.zk.KafkaZkClientTest > testPropagateIsrChanges STARTED

kafka.zk.KafkaZkClientTest > testPropagateIsrChanges PASSED

kafka.zk.KafkaZkClientTest > testControllerEpochMethods STARTED

kafka.zk.KafkaZkCl

Re: [VOTE] 1.1.0 RC1

[Jeff Chao]

Great! Have a good weekend.

On Fri, Mar 9, 2018 at 4:41 PM, Ismael Juma  wrote:

> Thanks Jeff:
>
> https://github.com/apache/kafka/pull/4678
>
> Ismael
>
> On Fri, Mar 9, 2018 at 1:56 AM, Damian Guy  wrote:
>
> > Hi Jeff,
> >
> > Thanks, we will look into this.
> >
> > Regards,
> > Damian
> >
> > On Thu, 8 Mar 2018 at 18:27 Jeff Chao  wrote:
> >
> > > Hello,
> > >
> > > We at Heroku have run 1.1.0 RC1 through our normal performance and
> > > regression test suite and have found performance to be comparable to
> > 1.0.0.
> > >
> > > That said, we're however -1 (non-binding) since this release includes
> > > Zookeeper 3.4.11 
> > which
> > > is affected by the critical regression ZOOKEEPER-2960
> > > . As 3.4.12
> isn't
> > > released yet, it might be better to have 3.4.10 included instead.
> > >
> > > Jeff
> > > Heroku
> > >
> > >
> > > On Tue, Mar 6, 2018 at 1:19 PM, Ted Yu  wrote:
> > >
> > > > +1
> > > >
> > > > Checked signature
> > > > Ran test suite - apart from flaky testMetricsLeak, other tests
> passed.
> > > >
> > > > On Tue, Mar 6, 2018 at 2:45 AM, Damian Guy 
> > wrote:
> > > >
> > > > > Hello Kafka users, developers and client-developers,
> > > > >
> > > > > This is the second candidate for release of Apache Kafka 1.1.0.
> > > > >
> > > > > This is minor version release of Apache Kakfa. It Includes 29 new
> > KIPs.
> > > > > Please see the release plan for more details:
> > > > >
> > > > > https://cwiki.apache.org/confluence/pages/viewpage.
> > > > action?pageId=71764913
> > > > >
> > > > > A few highlights:
> > > > >
> > > > > * Significant Controller improvements (much faster and session
> > > expiration
> > > > > edge cases fixed)
> > > > > * Data balancing across log directories (JBOD)
> > > > > * More efficient replication when the number of partitions is large
> > > > > * Dynamic Broker Configs
> > > > > * Delegation tokens (KIP-48)
> > > > > * Kafka Streams API improvements (KIP-205 / 210 / 220 / 224 / 239)
> > > > >
> > > > > Release notes for the 1.1.0 release:
> > > > > http://home.apache.org/~damianguy/kafka-1.1.0-rc1/
> RELEASE_NOTES.html
> > > > >
> > > > > *** Please download, test and vote by Friday, March 9th, 5pm PT
> > > > >
> > > > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > > > http://kafka.apache.org/KEYS
> > > > >
> > > > > * Release artifacts to be voted upon (source and binary):
> > > > > http://home.apache.org/~damianguy/kafka-1.1.0-rc1/
> > > > >
> > > > > * Maven artifacts to be voted upon:
> > > > > https://repository.apache.org/content/groups/staging/
> > > > >
> > > > > * Javadoc:
> > > > > http://home.apache.org/~damianguy/kafka-1.1.0-rc1/javadoc/
> > > > >
> > > > > * Tag to be voted upon (off 1.1 branch) is the 1.1.0 tag:
> > > > > https://github.com/apache/kafka/tree/1.1.0-rc1
> > > > >
> > > > >
> > > > > * Documentation:
> > > > > http://kafka.apache.org/11/documentation.html
> > > > >
> > > > > * Protocol:
> > > > > http://kafka.apache.org/11/protocol.html
> > > > >
> > > > > * Successful Jenkins builds for the 1.1 branch:
> > > > > Unit/integration tests:
> > > https://builds.apache.org/job/kafka-1.1-jdk7/68
> > > > > System tests: https://jenkins.confluent.io/
> > > > job/system-test-kafka/job/1.1/
> > > > > 30/
> > > > >
> > > > > /**
> > > > >
> > > > > Thanks,
> > > > > Damian Guy
> > > > >
> > > >
> > >
> >
>

Build failed in Jenkins: kafka-0.10.2-jdk7 #204

[Apache Jenkins Server]

See 


Changes:

[ismael] MINOR: Tag AWS instances with Jenkins build url (#4657)

--
Started by an SCM change
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on H34 (ubuntu xenial) in workspace 

Cloning the remote Git repository
Cloning repository https://github.com/apache/kafka.git
 > git init  # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # 
 > timeout=10
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/0.10.2^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/0.10.2^{commit} # timeout=10
Checking out Revision b8ac5300a4c1367c1c79ed5bce9b66c08bb7eecf 
(refs/remotes/origin/0.10.2)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f b8ac5300a4c1367c1c79ed5bce9b66c08bb7eecf
Commit message: "MINOR: Tag AWS instances with Jenkins build url (#4657)"
 > git rev-list --no-walk d72a00b1d6dcae422e4663b860b23533ce866a5a # timeout=10
ERROR: No tool found matching GRADLE_2_4_RC_2_HOME
[kafka-0.10.2-jdk7] $ /bin/bash -xe /tmp/jenkins7727641182188119346.sh
+ rm -rf 
+ /bin/gradle
/tmp/jenkins7727641182188119346.sh: line 4: /bin/gradle: No such file or 
directory
Build step 'Execute shell' marked build as failure
Recording test results
ERROR: No tool found matching GRADLE_2_4_RC_2_HOME
ERROR: Step ?Publish JUnit test result report? failed: No test report files 
were found. Configuration error?
ERROR: No tool found matching GRADLE_2_4_RC_2_HOME
Not sending mail to unregistered user ism...@juma.me.uk
Not sending mail to unregistered user rajinisiva...@googlemail.com

Re: [DISCUSS] KIP-255: OAuth Authentication via SASL/OAUTHBEARER

[Ron Dagostino]

Hi Rajini.  Thanks for the great feedback.  See below for my
thoughts/conclusions.  I haven't implemented any of it yet or changed the
KIP, but I will start to work on the areas where we are in agreement
immediately, and I will await your feedback on the areas where an
additional iteration is needed to arrive at a conclusion.

Regarding (1), yes, we can and should eliminate some public API.  See below.

Regarding (2), I will change the exception hierarchy so that it is
unchecked.

Regarding (3) and (4), yes, I agree, the expiring/refresh code can and
should be simplified.  The name of the Login class (I called it
ExpiringCredentialRefreshingLogin) must be part of the public API because
it is the class that must be set via the oauthbearer.sasl.login.class
property.  Its underlying implementation doesn't have to be public, but the
fully-qualified name has to be well-known and fixed so that it can be
associated with that configuration property.  As you point out, we are not
unifying the refresh logic for OAUTHBEARER and GSSAPI, though it could be
undertaken at some point in the future; the name "
ExpiringCredentialRefreshingLogin" should probably be used if/when that
unification happens.  In the meantime, the class that we expose should
probably be called "OAuthBearerLogin", and it's fully-qualified name and
the fact that it recognizes several refresh-related property names in the
config, with certain min/max/default values, are the only things that
should be public.  I also agree from (4) that we can stipulate that
SASL/OAUTHBEARER only supports the case where OAUTHBEARER is the only SASL
mechanism communicated to the code, either because there is only one SASL
mechanism defined for the cluster or because the config is done via the new
dynamic functionality from KIP-226 that eliminates the
mechanism-to-login-module ambiguity associated with declaring multiple SASL
mechanisms in a single JAAS config file.  Given all of this, everything I
defined for token refresh could be internal implementation detail except
for ExpiringCredentialLoginModule, which would no longer be needed, and we
only have to expose a single class called OAuthBearerLogin.

Regarding (5), I'm glad you agree the substitutable module options
functionality is generally useful, and I will publish a separate KIP for
it.  I'm thinking the package will be
org.apache.kafka.common.security.optionsubs (I'll gladly accept anything
better if anyone can come up with something -- "optionsubs" is better than
"smo" but it still isn't that great, and unfortunately it is the best
relatively short thing I can think of at the moment).  I'll also see what I
can do to minimize the surface area of the API; that discussion can be done
separately as part of that KIP's discussion thread.

Regarding (6), I agree that exposing the validated token via a publicly
defined SaslServer negotiated property name eliminates the need for the
OAuthBearerSaslServer interface; I will make this change.

Regarding (7), I agree that retrieving the validated token via a callback
would be consistent with other mechanism implementations.  I do have one
concern about adopting this approach, though.  The token validation
mechanism is typically going to require a decent amount of configuration,
and that configuration is going to live in the login module options.  It
feels natural for the declaration of the validation mechanism to live in
the same place, next to the configuration, which is how it currently
happens in the KIP.  If we change it so that the validation mechanism is
declared via the callback handler then we move that declaration somewhere
else, where the callback handler class is defined, which is separate from
where we define the options that configure it.  I think there is some cost
associated with separating two things that should go together.  Also, now
that I think about it, the public API gets bigger with the callback
approach because we trade OAuthBearerTokenValidator for a callback handler
class and a callback class for it to recognize (the cost of 2 vs. 1 is not
much of a difference, but it does exist).  Have I identified these costs
correctly, and if so, do you feel the benefit of consistency outweighs
them, in which case I will make the change, or should we keep it the way it
is?

Regarding (8), I wonder if the same cost/benefit analysis applies to the
token retriever.  Let's decide on (7) first, then we can decide on (8).

Thanks again for the great feedback.

Ron


On Thu, Mar 8, 2018 at 9:31 AM, Rajini Sivaram 
wrote:

> Hi Ron,
>
> Thanks for the KIP. Sorry for the delay in reviewing this. I have a few
> questions/comments.
>
>
>1. Are all of the classes listed in the KIP intended to be public
>classes/interfaces? Since it requires more effort to maintain public
>classes, it will be good if we can make more of the.code internal. For
>example, some of the classes in `oauthbearer.refresh` and `
>oauthbearer.smo` could be made internal?
>2. Ag

Re: [DISCUSS] KIP-258: Allow to Store Record Timestamps in RocksDB

[Guozhang Wang]

@John:

For the protocol version upgrade, it is only for the encoded metadata bytes
protocol, which are just bytes-in bytes-out from Consumer's pov, so I think
this change should be in the Streams layer as well.

@Matthias:

for 2), I agree that adding a "newest supported version" besides the
"currently used version for encoding" is a good idea to allow either case;
the key is that in Streams we would likely end up with a mapping from the
protocol version to the other persistent data format versions such as
rocksDB, changelog. So with such a map we can actually achieve both
scenarios, i.e. 1) one rolling bounce if the upgraded protocol version's
corresponding data format does not change, e.g. 0.10.0 -> 0.10.1 leaders
can choose to use the newer version in the first rolling bounce directly
and we can document to users that they would not need to set
"upgrade.mode", and 2) two rolling bounce if the upgraded protocol version
does indicate the data format changes, e.g. 1.1 -> 1.2, and then we can
document that "upgrade.mode" needs to be set in the first rolling bounce
and reset in the second.


Besides that, some additional comments:

1) I still think "upgrade.from" is less intuitive for users to set than
"internal.protocol.version" where for the latter users only need to set a
single version, while the Streams will map that version to the Streams
assignor's behavior as well as the data format. But maybe I did not get
your idea about how the  "upgrade.from" config will be set, because in
your Compatibility section how the upgrade.from config will be set for
these two rolling bounces are not very clear: for example, should user
reset it to null in the second rolling bounce?

2) In the upgrade path description, rather than talking about specific
version 0.10.0 -> version 0.10.1 etc, can we just categorize all the
possible scenarios, even for future upgrade versions, what should be the
standard operations? The categorized we can summarize to would be (assuming
user upgrade from version X to version Y, where X and Y are Kafka versions,
with the corresponding supported protocol version x and y):


a. x == y, i.e. metadata protocol does not change, and hence no persistent
data formats have changed.

b. x != y, but all persistent data format remains the same.

b. x !=y, AND some persistene data format like RocksDB format, changelog
format, has been changed.

c. special case: we may need some special handling logic when "current
version" or "newest supported version" are not available in the protocol,
i.e. for X as old as 0.10.0 and before 1.2.


under the above scenarios, how many rolling bounces users need to execute?
how they should set the configs in each rolling bounce? and how Streams
library will execute in these cases?



Guozhang





On Fri, Mar 9, 2018 at 4:01 PM, Matthias J. Sax 
wrote:

> Ted,
>
> I still consider changing the KIP to include it right away -- if not,
> I'll create a JIRA. Need to think it through in more detail first.
>
> (Same for other open questions like interface names -- I collect
> feedback and update the KIP after we reach consensus :))
>
> -Matthias
>
> On 3/9/18 3:35 PM, Ted Yu wrote:
> > Thanks for the details, Matthias.
> >
> > bq. change the metadata protocol only if a future release, encoding both
> used
> > and supported version might be an advantage
> >
> > Looks like encoding both versions wouldn't be implemented in this KIP.
> >
> > Please consider logging a JIRA with the encoding proposal.
> >
> > Cheers
> >
> > On Fri, Mar 9, 2018 at 2:27 PM, Matthias J. Sax 
> > wrote:
> >
> >> @Bill: I think a filter predicate should be part of user code. And even
> >> if we want to add something like this, I would prefer to do it in a
> >> separate KIP.
> >>
> >>
> >> @James: I would love to avoid a second rolling bounce. But from my
> >> understanding it would not be possible.
> >>
> >> The purpose of the second rolling bounce is indeed to switch from
> >> version 2 to 3. It also has a second purpose, to switch from the old
> >> store to the new store (this happens after the last instance bounces a
> >> second time).
> >>
> >> The problem with one round of rolling bounces is, that it's unclear when
> >> to which from version 2 to version 3. The StreamsPartitionsAssignor is
> >> stateless by design, and thus, the information which version it should
> >> use must be passed in from externally -- and we want to use the
> >> StreamsConfig to pass in this information.
> >>
> >> During upgrade, all new instanced have no information about the progress
> >> of the upgrade (ie, how many other instanced got upgrades already).
> >> Therefore, it's not safe for them to send a version 3 subscription. The
> >> leader also has this limited view on the world and can only send version
> >> 2 assignments back.
> >>
> >> Thus, for the 1.2 upgrade, I don't think we can simplify the upgrade.
> >>
> >> We did consider to change the metadata to make later upgrades (ie, from
> >> 1.2 to 1.x) simpler though (