Re: [DISCUSS] KIP-115: Enforce offsets.topic.replication.factor

[Stevo Slavić]

I'm fine if it goes into 0.10.2.0

On Tue, Jan 31, 2017 at 8:53 AM, Onur Karaman  wrote:

> I get where Ewen's coming from but personally, I have trouble understanding
> a use case where end users would specifically rely on existing behavior of
> group coordination and offset commits succeeding with nondeterministic RF
> (which is only later to be manually fixed) during the window of time when a
> legitimate non-quickstart cluster is in the process of being setup.
>
> As Ewen mentioned, server.properties is already blatantly not meant for
> production use cases but actually for quickstart. It chooses /tmp for its
> log.dirs, localhost for its zookeeper.connect, and references
> kafka.server.KafkaConfig for additional details and defaults. It seems that
> setting offsets.topic.replication.factor to 1 in server.properties is the
> right way to go.
>
> That being said, if there actually is opposition, then it would be fair to
> bump it to 0.10.3.0.
>
> On Mon, Jan 30, 2017 at 6:12 PM, Ewen Cheslack-Postava 
> wrote:
>
> > So, we have one other blocker bug in system tests that we're trying to
> make
> > sure can safely be removed, so we've had a bit of slack time with this.
> > Obviously having this all happen very last minute isn't really ideal
> since
> > it didn't allow enough time to address the feedback -- Stevo's questions
> > didn't get a response until today.
> >
> > I'd really like this fix in because I think it actually seems pretty low
> > risk, although it would potentially make some consumers on a new cluster
> be
> > delayed indefinitely if the user is relying on having fewer than the
> > default offset.topic.replication.factor. Many users will have there
> > server.properties files stored elsewhere and not rely on the version we
> > ship (and they shouldn't -- it contains other dangerous settings like
> > log.dirs in /tmp). This does mean some users could be negatively
> affected,
> > although I have to admit it seems like it'd be an extremely small
> > population of our users.
> >
> > All that said, given the late voting and discussion continuing while the
> > VOTE thread proceeded, I'd like to suggest that if anyone objects to
> > including in 0.10.2.0, we bump it to 0.10.3.0 and also follow up on
> > Ismael's question (re: does this need to be in a major version, which
> > afaict was only followed up by Jeff's comment that he'd rather not wait).
> > The patch is there, so those that desperately want it can easily
> > cherry-pick it and build a version themselves if it's that big a problem
> > for them, else they would see the fix in the next release.
> >
> > More generally, I think this KIP (and my experience dealing w/ the KIPs +
> > feature freeze for this release) suggest we need a little more clarity on
> > how KIPs fit into the release process. I'm going to follow up in a new
> > release post-mortem thread further but I think we should a) start to
> > distinguish between feature KIPs and bug fix KIPs and possibly apply
> > different rules for them (this one is admittedly ambiguous) and b) push
> the
> > deadline for feature KIPs to a week before feature freeze (giving a more
> > realistic review period for all that code) and make bugfix KIPs
> acceptable
> > up to a week before code freeze (again, providing a realistic review
> > period), and c) move our freeze dates off of Fridays since having them
> > there implicitly gives people 2 extra days to squeeze things in.
> >
> > -Ewen
> >
> > On Mon, Jan 30, 2017 at 1:46 PM, Onur Karaman
> >  > > wrote:
> >
> > > I've updated the KIP title to reflect this distinction:
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 115%3A+Enforce+offsets.topic.replication.factor+upon+__
> > > consumer_offsets+auto+topic+creation
> > >
> > > On Mon, Jan 30, 2017 at 12:52 AM, Onur Karaman <
> > > onurkaraman.apa...@gmail.com
> > > > wrote:
> > >
> > > > Regarding Joel's comment:
> > > > > On Jan 25, 2017, at 9:26 PM, Joel Koshy 
> wrote:
> > > > >
> > > > > already voted, but one thing worth considering (since this KIP
> speaks
> > > of
> > > > > *enforcement*) is desired behavior if the topic already exists and
> > the
> > > > > config != existing RF.
> > > > >
> > > >
> > > > The short answer: The first line of the KIP specifically states that
> it
> > > > attempts enforcement only upon topic creation and I think anything
> else
> > > > should be beyond the scope of this KIP.
> > > >
> > > > The long answer:
> > > > Mismatch between existing RF and the "offsets.topic.replication.
> > factor"
> > > > config happens with:
> > > > a. topic creation paths 3-5 as defined in the KIP if the size of the
> > > > replicas set resulting from AdminUtils != "offsets.topic.replication.
> > > > factor"
> > > > b. topic creation path 6
> > > > c. a config change to the broker's "offsets.topic.replication.
> factor"
> > > > d. partition reassignments that expand the RF
> > > >
> > > > For all of these scenarios, I believe it all boils down to the intent
> 

Re: [DISCUSS] KIP-110: Add Codec for ZStandard Compression

[Dongjin Lee]

Ismael & All,

After Inspecting the related code & commits, I concluded following:

1. We have to update the masking value which is used to retrieve the used
codec id from the messages, to enable the retrieval of the 3rd bit of
compression type field of the message.
2. The above task is already done; so, we need nothing.

Here is why.

Let's start with the first one, with the scenario Juma proposed. In the
case of receiving the message of unsupported compression type, the receiver
(= broker or consumer) raises IllegalArgumentException[^1][^2]. The key
element in this operation is Record#COMPRESSION_CODEC_MASK, which is used
to extract the codec id. We have to update this value from 2-bits extractor
(=0x03) to 3-bits extractor (=0x07).

But in fact, this task is already done, so its current value is 0x07. We
don't have to update it.

The reason why this task is already done has some story; From the first
time Record.java file was added to the project[^3], the
COMPRESSION_CODEC_MASK was already 2-bits extractor, that is, 0x03. At that
time, Kafka supported only two compression types - GZipCompression and
SnappyCompression.[^4] After that, KAFKA-1456 introduced two additional
codecs of LZ4 and LZ4C[^5]. This update modified COMPRESSION_CODEC_MASK
into 3 bits extractor, 0x07, in the aim of supporting four compression
codecs.

Although its following work, KAFKA-1493, removed the support of LZ4C
codec[^6], this mask was not reverted into 2-bits extractor - by this
reason, we don't need to care about the message format.

Attached screenshot is my test on Juma's scenario. I created topic & sent
some messages using the snapshot version with ZStandard compression and
received the message with the latest version. As you can see, it works
perfectly as expected.

If you have more opinion to this issue, don't hesitate to send me as a
message.

Best,
Dongjin

[^1]: see: Record#compressionType.
[^2]: There is similar routine in Message.scala. But after KAFKA-4390, that
routine is not being used anymore - more precisely, Message class is now
used in ConsoleConsumer only. I think this class should be replaced but
since it is a separated topic, I will send another message for this issue.
[^3]: commit 642da2f (2011.8.2).
[^4]: commit c51b940.
[^5]: commit 547cced.
[^6]: commit 37356bf.

On Thu, Jan 26, 2017 at 12:35 AM, Ismael Juma  wrote:

> So far the discussion was around the performance characteristics of the new
> compression algorithm. Another area that is important and is not covered in
> the KIP is the compatibility implications. For example, what happens if a
> consumer that doesn't support zstd tries to consume a topic compressed with
> it? Or if a broker that doesn't support receives data compressed with it?
> If we go through that exercise, then more changes may be required (like
> bumping the version of produce/fetch protocols).
>
> Ismael
>
> On Wed, Jan 25, 2017 at 3:22 PM, Ben Stopford  wrote:
>
> > Is there more discussion to be had on this KIP, or should it be taken to
> a
> > vote?
> >
> > On Mon, Jan 16, 2017 at 6:37 AM Dongjin Lee  wrote:
> >
> > > I updated KIP-110 with JMH-measured benchmark results. Please have a
> > review
> > > when you are free. (The overall result is not different yet.)
> > >
> > > Regards,
> > > Dongjin
> > >
> > > +1. Could anyone assign KAFKA-4514 to me?
> > >
> > > On Thu, Jan 12, 2017 at 11:39 AM, Dongjin Lee 
> > wrote:
> > >
> > > > Okay, I will have a try.
> > > > Thanks Ewen for the guidance!!
> > > >
> > > > Best,
> > > > Dongjin
> > > >
> > > > On Thu, Jan 12, 2017 at 6:44 AM, Ismael Juma 
> > wrote:
> > > >
> > > >> That's a good point Ewen. Dongjin, you could use the branch that
> Ewen
> > > >> linked for the performance testing. It would also help validate the
> > PR.
> > > >>
> > > >> Ismael
> > > >>
> > > >> On Wed, Jan 11, 2017 at 9:38 PM, Ewen Cheslack-Postava <
> > > e...@confluent.io
> > > >> >
> > > >> wrote:
> > > >>
> > > >> > FYI, there's an outstanding patch for getting some JMH
> benchmarking
> > > >> setup:
> > > >> > https://github.com/apache/kafka/pull/1712 I haven't found time to
> > > >> review
> > > >> > it
> > > >> > (and don't really know JMH well anyway) but it might be worth
> > getting
> > > >> that
> > > >> > landed so we can use it for this as well.
> > > >> >
> > > >> > -Ewen
> > > >> >
> > > >> > On Wed, Jan 11, 2017 at 6:35 AM, Dongjin Lee 
> > > >> wrote:
> > > >> >
> > > >> > > Hi Ismael,
> > > >> > >
> > > >> > > 1. In the case of compression output, yes, lz4 is producing the
> > > >> smaller
> > > >> > > output than gzip. In fact, my benchmark was inspired
> > > >> > > by MessageCompressionTest#testCompressSize unit test and the
> > result
> > > >> is
> > > >> > > same - 396 bytes for gzip and 387 bytes for lz4.
> > > >> > > 2. I agree that my (former) approach can result in unreliable
> > > output.
> > > >> > > However, I am experiencing difficulties on how to acquire the
> > > >> benchmark
> > > >> > > metrics from Kafka. For you reco

Re: [DISCUSS] KIP-110: Add Codec for ZStandard Compression

[Magnus Edenhill]

Hi Dongjin and good work on the KIP,

I understand that ZStandard is generally considered an improvement over
LZ4, but the
benchmark you provided on the KIP-110 wiki doesn't really reflect that, and
even
makes a note that they are comparable:
*> As you can see above, ZStandard shows outstanding performance in both of
compression rate and speed, especially working with the speed-first setting
(level 1). To the extent that only LZ4 can be compared to ZStandard.*

My concern is that this minor improvement (based on the benchmark) over LZ4
does not warrant the work
of adding support for a new compression codec to the broker, official
clients and horde of 3rd party clients, including
upgrade paths, transformations, tests, additional dependencies, etc.

Is it possible to produce more convincing comparisons?

Thanks,
Magnus





2017-01-31 10:28 GMT+01:00 Dongjin Lee :

> Ismael & All,
>
> After Inspecting the related code & commits, I concluded following:
>
> 1. We have to update the masking value which is used to retrieve the used
> codec id from the messages, to enable the retrieval of the 3rd bit of
> compression type field of the message.
> 2. The above task is already done; so, we need nothing.
>
> Here is why.
>
> Let's start with the first one, with the scenario Juma proposed. In the
> case of receiving the message of unsupported compression type, the receiver
> (= broker or consumer) raises IllegalArgumentException[^1][^2]. The key
> element in this operation is Record#COMPRESSION_CODEC_MASK, which is used
> to extract the codec id. We have to update this value from 2-bits extractor
> (=0x03) to 3-bits extractor (=0x07).
>
> But in fact, this task is already done, so its current value is 0x07. We
> don't have to update it.
>
> The reason why this task is already done has some story; From the first
> time Record.java file was added to the project[^3], the
> COMPRESSION_CODEC_MASK was already 2-bits extractor, that is, 0x03. At that
> time, Kafka supported only two compression types - GZipCompression and
> SnappyCompression.[^4] After that, KAFKA-1456 introduced two additional
> codecs of LZ4 and LZ4C[^5]. This update modified COMPRESSION_CODEC_MASK
> into 3 bits extractor, 0x07, in the aim of supporting four compression
> codecs.
>
> Although its following work, KAFKA-1493, removed the support of LZ4C
> codec[^6], this mask was not reverted into 2-bits extractor - by this
> reason, we don't need to care about the message format.
>
> Attached screenshot is my test on Juma's scenario. I created topic & sent
> some messages using the snapshot version with ZStandard compression and
> received the message with the latest version. As you can see, it works
> perfectly as expected.
>
> If you have more opinion to this issue, don't hesitate to send me as a
> message.
>
> Best,
> Dongjin
>
> [^1]: see: Record#compressionType.
> [^2]: There is similar routine in Message.scala. But after KAFKA-4390,
> that routine is not being used anymore - more precisely, Message class is
> now used in ConsoleConsumer only. I think this class should be replaced but
> since it is a separated topic, I will send another message for this issue.
> [^3]: commit 642da2f (2011.8.2).
> [^4]: commit c51b940.
> [^5]: commit 547cced.
> [^6]: commit 37356bf.
>
> On Thu, Jan 26, 2017 at 12:35 AM, Ismael Juma  wrote:
>
>> So far the discussion was around the performance characteristics of the
>> new
>> compression algorithm. Another area that is important and is not covered
>> in
>> the KIP is the compatibility implications. For example, what happens if a
>> consumer that doesn't support zstd tries to consume a topic compressed
>> with
>> it? Or if a broker that doesn't support receives data compressed with it?
>> If we go through that exercise, then more changes may be required (like
>> bumping the version of produce/fetch protocols).
>>
>> Ismael
>>
>> On Wed, Jan 25, 2017 at 3:22 PM, Ben Stopford  wrote:
>>
>> > Is there more discussion to be had on this KIP, or should it be taken
>> to a
>> > vote?
>> >
>> > On Mon, Jan 16, 2017 at 6:37 AM Dongjin Lee  wrote:
>> >
>> > > I updated KIP-110 with JMH-measured benchmark results. Please have a
>> > review
>> > > when you are free. (The overall result is not different yet.)
>> > >
>> > > Regards,
>> > > Dongjin
>> > >
>> > > +1. Could anyone assign KAFKA-4514 to me?
>> > >
>> > > On Thu, Jan 12, 2017 at 11:39 AM, Dongjin Lee 
>> > wrote:
>> > >
>> > > > Okay, I will have a try.
>> > > > Thanks Ewen for the guidance!!
>> > > >
>> > > > Best,
>> > > > Dongjin
>> > > >
>> > > > On Thu, Jan 12, 2017 at 6:44 AM, Ismael Juma 
>> > wrote:
>> > > >
>> > > >> That's a good point Ewen. Dongjin, you could use the branch that
>> Ewen
>> > > >> linked for the performance testing. It would also help validate the
>> > PR.
>> > > >>
>> > > >> Ismael
>> > > >>
>> > > >> On Wed, Jan 11, 2017 at 9:38 PM, Ewen Cheslack-Postava <
>> > > e...@confluent.io
>> > > >> >
>> > > >> wrote:
>> > > >>
>> > > >> > FYI, there's a

Re: [DISCUSS] KIP-110: Add Codec for ZStandard Compression

[Ismael Juma]

Thanks Dongjin. That seems to make things simpler indeed. It would be good
to update the KIP with the relevant details. I suggest describing each flow
and the error that the user would see in each case.

Also, you mentioned an attached screenshot, but it seems like there was
nothing attached to the email.

Ismael

On Tue, Jan 31, 2017 at 9:28 AM, Dongjin Lee  wrote:

> Ismael & All,
>
> After Inspecting the related code & commits, I concluded following:
>
> 1. We have to update the masking value which is used to retrieve the used
> codec id from the messages, to enable the retrieval of the 3rd bit of
> compression type field of the message.
> 2. The above task is already done; so, we need nothing.
>
> Here is why.
>
> Let's start with the first one, with the scenario Juma proposed. In the
> case of receiving the message of unsupported compression type, the receiver
> (= broker or consumer) raises IllegalArgumentException[^1][^2]. The key
> element in this operation is Record#COMPRESSION_CODEC_MASK, which is used
> to extract the codec id. We have to update this value from 2-bits extractor
> (=0x03) to 3-bits extractor (=0x07).
>
> But in fact, this task is already done, so its current value is 0x07. We
> don't have to update it.
>
> The reason why this task is already done has some story; From the first
> time Record.java file was added to the project[^3], the
> COMPRESSION_CODEC_MASK was already 2-bits extractor, that is, 0x03. At that
> time, Kafka supported only two compression types - GZipCompression and
> SnappyCompression.[^4] After that, KAFKA-1456 introduced two additional
> codecs of LZ4 and LZ4C[^5]. This update modified COMPRESSION_CODEC_MASK
> into 3 bits extractor, 0x07, in the aim of supporting four compression
> codecs.
>
> Although its following work, KAFKA-1493, removed the support of LZ4C
> codec[^6], this mask was not reverted into 2-bits extractor - by this
> reason, we don't need to care about the message format.
>
> Attached screenshot is my test on Juma's scenario. I created topic & sent
> some messages using the snapshot version with ZStandard compression and
> received the message with the latest version. As you can see, it works
> perfectly as expected.
>
> If you have more opinion to this issue, don't hesitate to send me as a
> message.
>
> Best,
> Dongjin
>
> [^1]: see: Record#compressionType.
> [^2]: There is similar routine in Message.scala. But after KAFKA-4390,
> that routine is not being used anymore - more precisely, Message class is
> now used in ConsoleConsumer only. I think this class should be replaced but
> since it is a separated topic, I will send another message for this issue.
> [^3]: commit 642da2f (2011.8.2).
> [^4]: commit c51b940.
> [^5]: commit 547cced.
> [^6]: commit 37356bf.
>
> On Thu, Jan 26, 2017 at 12:35 AM, Ismael Juma  wrote:
>
>> So far the discussion was around the performance characteristics of the
>> new
>> compression algorithm. Another area that is important and is not covered
>> in
>> the KIP is the compatibility implications. For example, what happens if a
>> consumer that doesn't support zstd tries to consume a topic compressed
>> with
>> it? Or if a broker that doesn't support receives data compressed with it?
>> If we go through that exercise, then more changes may be required (like
>> bumping the version of produce/fetch protocols).
>>
>> Ismael
>>
>> On Wed, Jan 25, 2017 at 3:22 PM, Ben Stopford  wrote:
>>
>> > Is there more discussion to be had on this KIP, or should it be taken
>> to a
>> > vote?
>> >
>> > On Mon, Jan 16, 2017 at 6:37 AM Dongjin Lee  wrote:
>> >
>> > > I updated KIP-110 with JMH-measured benchmark results. Please have a
>> > review
>> > > when you are free. (The overall result is not different yet.)
>> > >
>> > > Regards,
>> > > Dongjin
>> > >
>> > > +1. Could anyone assign KAFKA-4514 to me?
>> > >
>> > > On Thu, Jan 12, 2017 at 11:39 AM, Dongjin Lee 
>> > wrote:
>> > >
>> > > > Okay, I will have a try.
>> > > > Thanks Ewen for the guidance!!
>> > > >
>> > > > Best,
>> > > > Dongjin
>> > > >
>> > > > On Thu, Jan 12, 2017 at 6:44 AM, Ismael Juma 
>> > wrote:
>> > > >
>> > > >> That's a good point Ewen. Dongjin, you could use the branch that
>> Ewen
>> > > >> linked for the performance testing. It would also help validate the
>> > PR.
>> > > >>
>> > > >> Ismael
>> > > >>
>> > > >> On Wed, Jan 11, 2017 at 9:38 PM, Ewen Cheslack-Postava <
>> > > e...@confluent.io
>> > > >> >
>> > > >> wrote:
>> > > >>
>> > > >> > FYI, there's an outstanding patch for getting some JMH
>> benchmarking
>> > > >> setup:
>> > > >> > https://github.com/apache/kafka/pull/1712 I haven't found time
>> to
>> > > >> review
>> > > >> > it
>> > > >> > (and don't really know JMH well anyway) but it might be worth
>> > getting
>> > > >> that
>> > > >> > landed so we can use it for this as well.
>> > > >> >
>> > > >> > -Ewen
>> > > >> >
>> > > >> > On Wed, Jan 11, 2017 at 6:35 AM, Dongjin Lee > >
>> > > >> wrote:
>> > > >> >
>> > > >> > > Hi Ismael,
>> > >

[DISCUS] consuming messages is polling - is there a reason? new KIP for poll?

[Alexander Binzberger]

Hi there,
I realized that consuming messages is implemented as polling on protocol 
level.

Is there a reason why the client has to ask for messages?
Wouldn't it make more sense to push messages to consumers - at least to 
high-level ones?


I would think about it like:
Broker receives a new message from a produce.
It could push the message first to all replicas and then (bulked) to all 
consumers (at least high-level ones with consumer group). [bulk: 
offsetOfFirstMsg, list(msgs)]
Consumers could ack the (last) message (with the new offset) just after 
processing it. [newOffset]
After the last message is acked the broker would push the next (bulked) 
messages to consume. And so on..


Pros:
This way it seams like the protocol and the high-level consumer would be 
simplified.
Clients have a more natural control over the offset and could ack per 
message or per bulk as needed or performance allows.

Additionally the stream processing path over Kafka would be faster.

Best Regards
Alexander Binzberger

--
Alexander Binzberger
System Designer - WINGcon AG
Tel. +49 7543 966-119

Sitz der Gesellschaft: Langenargen
Registergericht: ULM, HRB 734260
USt-Id.: DE232931635, WEEE-Id.: DE74015979
Vorstand: thomasThomas Ehrle (Vorsitz), Fritz R. Paul (Stellvertreter), Tobias 
Treß
Aufsichtsrat: Jürgen Maucher (Vorsitz), Andreas Paul (Stellvertreter), Martin 
Sauter

Re: [DISCUSS] KIP-110: Add Codec for ZStandard Compression

[Ismael Juma]

Yes, I agree that the dataset being used in the comparison doesn't show the
strength/weakness of the various algorithms (I actually thought that there
was a bug initially). Maybe we could use the test suites mentioned in the
following project:

https://github.com/ning/jvm-compressor-benchmark

Ismael

On Tue, Jan 31, 2017 at 9:47 AM, Magnus Edenhill  wrote:

> Hi Dongjin and good work on the KIP,
>
> I understand that ZStandard is generally considered an improvement over
> LZ4, but the
> benchmark you provided on the KIP-110 wiki doesn't really reflect that, and
> even
> makes a note that they are comparable:
> *> As you can see above, ZStandard shows outstanding performance in both of
> compression rate and speed, especially working with the speed-first setting
> (level 1). To the extent that only LZ4 can be compared to ZStandard.*
>
> My concern is that this minor improvement (based on the benchmark) over LZ4
> does not warrant the work
> of adding support for a new compression codec to the broker, official
> clients and horde of 3rd party clients, including
> upgrade paths, transformations, tests, additional dependencies, etc.
>
> Is it possible to produce more convincing comparisons?
>
> Thanks,
> Magnus
>
>
>
>
>
> 2017-01-31 10:28 GMT+01:00 Dongjin Lee :
>
> > Ismael & All,
> >
> > After Inspecting the related code & commits, I concluded following:
> >
> > 1. We have to update the masking value which is used to retrieve the used
> > codec id from the messages, to enable the retrieval of the 3rd bit of
> > compression type field of the message.
> > 2. The above task is already done; so, we need nothing.
> >
> > Here is why.
> >
> > Let's start with the first one, with the scenario Juma proposed. In the
> > case of receiving the message of unsupported compression type, the
> receiver
> > (= broker or consumer) raises IllegalArgumentException[^1][^2]. The key
> > element in this operation is Record#COMPRESSION_CODEC_MASK, which is used
> > to extract the codec id. We have to update this value from 2-bits
> extractor
> > (=0x03) to 3-bits extractor (=0x07).
> >
> > But in fact, this task is already done, so its current value is 0x07. We
> > don't have to update it.
> >
> > The reason why this task is already done has some story; From the first
> > time Record.java file was added to the project[^3], the
> > COMPRESSION_CODEC_MASK was already 2-bits extractor, that is, 0x03. At
> that
> > time, Kafka supported only two compression types - GZipCompression and
> > SnappyCompression.[^4] After that, KAFKA-1456 introduced two additional
> > codecs of LZ4 and LZ4C[^5]. This update modified COMPRESSION_CODEC_MASK
> > into 3 bits extractor, 0x07, in the aim of supporting four compression
> > codecs.
> >
> > Although its following work, KAFKA-1493, removed the support of LZ4C
> > codec[^6], this mask was not reverted into 2-bits extractor - by this
> > reason, we don't need to care about the message format.
> >
> > Attached screenshot is my test on Juma's scenario. I created topic & sent
> > some messages using the snapshot version with ZStandard compression and
> > received the message with the latest version. As you can see, it works
> > perfectly as expected.
> >
> > If you have more opinion to this issue, don't hesitate to send me as a
> > message.
> >
> > Best,
> > Dongjin
> >
> > [^1]: see: Record#compressionType.
> > [^2]: There is similar routine in Message.scala. But after KAFKA-4390,
> > that routine is not being used anymore - more precisely, Message class is
> > now used in ConsoleConsumer only. I think this class should be replaced
> but
> > since it is a separated topic, I will send another message for this
> issue.
> > [^3]: commit 642da2f (2011.8.2).
> > [^4]: commit c51b940.
> > [^5]: commit 547cced.
> > [^6]: commit 37356bf.
> >
> > On Thu, Jan 26, 2017 at 12:35 AM, Ismael Juma  wrote:
> >
> >> So far the discussion was around the performance characteristics of the
> >> new
> >> compression algorithm. Another area that is important and is not covered
> >> in
> >> the KIP is the compatibility implications. For example, what happens if
> a
> >> consumer that doesn't support zstd tries to consume a topic compressed
> >> with
> >> it? Or if a broker that doesn't support receives data compressed with
> it?
> >> If we go through that exercise, then more changes may be required (like
> >> bumping the version of produce/fetch protocols).
> >>
> >> Ismael
> >>
> >> On Wed, Jan 25, 2017 at 3:22 PM, Ben Stopford  wrote:
> >>
> >> > Is there more discussion to be had on this KIP, or should it be taken
> >> to a
> >> > vote?
> >> >
> >> > On Mon, Jan 16, 2017 at 6:37 AM Dongjin Lee 
> wrote:
> >> >
> >> > > I updated KIP-110 with JMH-measured benchmark results. Please have a
> >> > review
> >> > > when you are free. (The overall result is not different yet.)
> >> > >
> >> > > Regards,
> >> > > Dongjin
> >> > >
> >> > > +1. Could anyone assign KAFKA-4514 to me?
> >> > >
> >> > > On Thu, Jan 12, 2017 at 11:39 A

Re: [DISCUSS] KIP-110: Add Codec for ZStandard Compression

[Dongjin Lee]

No problem. I will elaborate my KIP with the updated benchmark & error
cases.

And here, I resend you the screenshot:


​
Best,
Dongjin

On Tue, Jan 31, 2017 at 7:42 PM, Ismael Juma  wrote:

> Thanks Dongjin. That seems to make things simpler indeed. It would be good
> to update the KIP with the relevant details. I suggest describing each flow
> and the error that the user would see in each case.
>
> Also, you mentioned an attached screenshot, but it seems like there was
> nothing attached to the email.
>
> Ismael
>
> On Tue, Jan 31, 2017 at 9:28 AM, Dongjin Lee  wrote:
>
> > Ismael & All,
> >
> > After Inspecting the related code & commits, I concluded following:
> >
> > 1. We have to update the masking value which is used to retrieve the used
> > codec id from the messages, to enable the retrieval of the 3rd bit of
> > compression type field of the message.
> > 2. The above task is already done; so, we need nothing.
> >
> > Here is why.
> >
> > Let's start with the first one, with the scenario Juma proposed. In the
> > case of receiving the message of unsupported compression type, the
> receiver
> > (= broker or consumer) raises IllegalArgumentException[^1][^2]. The key
> > element in this operation is Record#COMPRESSION_CODEC_MASK, which is used
> > to extract the codec id. We have to update this value from 2-bits
> extractor
> > (=0x03) to 3-bits extractor (=0x07).
> >
> > But in fact, this task is already done, so its current value is 0x07. We
> > don't have to update it.
> >
> > The reason why this task is already done has some story; From the first
> > time Record.java file was added to the project[^3], the
> > COMPRESSION_CODEC_MASK was already 2-bits extractor, that is, 0x03. At
> that
> > time, Kafka supported only two compression types - GZipCompression and
> > SnappyCompression.[^4] After that, KAFKA-1456 introduced two additional
> > codecs of LZ4 and LZ4C[^5]. This update modified COMPRESSION_CODEC_MASK
> > into 3 bits extractor, 0x07, in the aim of supporting four compression
> > codecs.
> >
> > Although its following work, KAFKA-1493, removed the support of LZ4C
> > codec[^6], this mask was not reverted into 2-bits extractor - by this
> > reason, we don't need to care about the message format.
> >
> > Attached screenshot is my test on Juma's scenario. I created topic & sent
> > some messages using the snapshot version with ZStandard compression and
> > received the message with the latest version. As you can see, it works
> > perfectly as expected.
> >
> > If you have more opinion to this issue, don't hesitate to send me as a
> > message.
> >
> > Best,
> > Dongjin
> >
> > [^1]: see: Record#compressionType.
> > [^2]: There is similar routine in Message.scala. But after KAFKA-4390,
> > that routine is not being used anymore - more precisely, Message class is
> > now used in ConsoleConsumer only. I think this class should be replaced
> but
> > since it is a separated topic, I will send another message for this
> issue.
> > [^3]: commit 642da2f (2011.8.2).
> > [^4]: commit c51b940.
> > [^5]: commit 547cced.
> > [^6]: commit 37356bf.
> >
> > On Thu, Jan 26, 2017 at 12:35 AM, Ismael Juma  wrote:
> >
> >> So far the discussion was around the performance characteristics of the
> >> new
> >> compression algorithm. Another area that is important and is not covered
> >> in
> >> the KIP is the compatibility implications. For example, what happens if
> a
> >> consumer that doesn't support zstd tries to consume a topic compressed
> >> with
> >> it? Or if a broker that doesn't support receives data compressed with
> it?
> >> If we go through that exercise, then more changes may be required (like
> >> bumping the version of produce/fetch protocols).
> >>
> >> Ismael
> >>
> >> On Wed, Jan 25, 2017 at 3:22 PM, Ben Stopford  wrote:
> >>
> >> > Is there more discussion to be had on this KIP, or should it be taken
> >> to a
> >> > vote?
> >> >
> >> > On Mon, Jan 16, 2017 at 6:37 AM Dongjin Lee 
> wrote:
> >> >
> >> > > I updated KIP-110 with JMH-measured benchmark results. Please have a
> >> > review
> >> > > when you are free. (The overall result is not different yet.)
> >> > >
> >> > > Regards,
> >> > > Dongjin
> >> > >
> >> > > +1. Could anyone assign KAFKA-4514 to me?
> >> > >
> >> > > On Thu, Jan 12, 2017 at 11:39 AM, Dongjin Lee 
> >> > wrote:
> >> > >
> >> > > > Okay, I will have a try.
> >> > > > Thanks Ewen for the guidance!!
> >> > > >
> >> > > > Best,
> >> > > > Dongjin
> >> > > >
> >> > > > On Thu, Jan 12, 2017 at 6:44 AM, Ismael Juma 
> >> > wrote:
> >> > > >
> >> > > >> That's a good point Ewen. Dongjin, you could use the branch that
> >> Ewen
> >> > > >> linked for the performance testing. It would also help validate
> the
> >> > PR.
> >> > > >>
> >> > > >> Ismael
> >> > > >>
> >> > > >> On Wed, Jan 11, 2017 at 9:38 PM, Ewen Cheslack-Postava <
> >> > > e...@confluent.io
> >> > > >> >
> >> > > >> wrote:
> >> > > >>
> >> > > >> > FYI, there's an outstanding patch for getting some JMH
> >> benchmark

Re: [DISCUSS] KIP-115: Enforce offsets.topic.replication.factor

[Ismael Juma]

Hi all,

First of all, thanks Onur for fixing this.

With regards to where we merge the code, my personal preference would be to
merge it to trunk ASAP ((which means that it would be part of 0.10.3.0).
Generally, I think we should be conservative when it comes to code merged
right before the RC gets cut, i.e. we should limit it to documentation,
tests, fixes for regressions and critical fixes that affect new
functionality (to avoid shipping broken new features). As we now have
frequent time-based releases, the next release is never far away and it
reduces/eliminates the need to fast-track KIPs in my opinion. Also we have
to be particularly careful to avoid a situation where we have frequent, but
slightly broken time-based releases making it difficult for users to choose
a good release.

For this particular KIP, I agree with Onur that people wouldn't want to
rely on a non-deterministic RF, so the code change would only affect people
setting up new clusters with 1 or 2 Kafka nodes using an existing config
(i.e. not the updated server.properties) and with no
offsets.topic.replication.factor set in the config. Hopefully people are
not doing this outside of test clusters and the fix is easy.

The second point is the new config in `server.properties`. The fact that
`server.properties` should only be used for quickstarts is obvious to us,
but seemingly not for other people. We have a couple of JIRAs[1][2] and a
number of mailing list threads about people who are surprised by the fact
that `log.dir` is set to `/tmp` in `server.properties`. I actually think
that we should rename `server.properties` to `server-quickstart.properties`
(or something similar) and have a more reasonable `server.properties`
(maybe with a different name). In any case, that's out of scope for this
KIP, but the point seemed relevant.

Having said all that, if I'm the only one who feels this way about
including this KIP in 0.10.2, then I'm OK with it being included.

Ewen, your suggestions for the post-mortem thread sound reasonable to me
(although I'd like to understand better what we would classify as bug fix
KIPs).

Ismael

[1] https://issues.apache.org/jira/browse/KAFKA-1906
[2] https://issues.apache.org/jira/browse/KAFKA-3925

On Tue, Jan 31, 2017 at 7:53 AM, Onur Karaman  wrote:

> I get where Ewen's coming from but personally, I have trouble understanding
> a use case where end users would specifically rely on existing behavior of
> group coordination and offset commits succeeding with nondeterministic RF
> (which is only later to be manually fixed) during the window of time when a
> legitimate non-quickstart cluster is in the process of being setup.
>
> As Ewen mentioned, server.properties is already blatantly not meant for
> production use cases but actually for quickstart. It chooses /tmp for its
> log.dirs, localhost for its zookeeper.connect, and references
> kafka.server.KafkaConfig for additional details and defaults. It seems that
> setting offsets.topic.replication.factor to 1 in server.properties is the
> right way to go.
>
> That being said, if there actually is opposition, then it would be fair to
> bump it to 0.10.3.0.
>
> On Mon, Jan 30, 2017 at 6:12 PM, Ewen Cheslack-Postava 
> wrote:
>
> > So, we have one other blocker bug in system tests that we're trying to
> make
> > sure can safely be removed, so we've had a bit of slack time with this.
> > Obviously having this all happen very last minute isn't really ideal
> since
> > it didn't allow enough time to address the feedback -- Stevo's questions
> > didn't get a response until today.
> >
> > I'd really like this fix in because I think it actually seems pretty low
> > risk, although it would potentially make some consumers on a new cluster
> be
> > delayed indefinitely if the user is relying on having fewer than the
> > default offset.topic.replication.factor. Many users will have there
> > server.properties files stored elsewhere and not rely on the version we
> > ship (and they shouldn't -- it contains other dangerous settings like
> > log.dirs in /tmp). This does mean some users could be negatively
> affected,
> > although I have to admit it seems like it'd be an extremely small
> > population of our users.
> >
> > All that said, given the late voting and discussion continuing while the
> > VOTE thread proceeded, I'd like to suggest that if anyone objects to
> > including in 0.10.2.0, we bump it to 0.10.3.0 and also follow up on
> > Ismael's question (re: does this need to be in a major version, which
> > afaict was only followed up by Jeff's comment that he'd rather not wait).
> > The patch is there, so those that desperately want it can easily
> > cherry-pick it and build a version themselves if it's that big a problem
> > for them, else they would see the fix in the next release.
> >
> > More generally, I think this KIP (and my experience dealing w/ the KIPs +
> > feature freeze for this release) suggest we need a little more clarity on
> > how KIPs fit into the re

Re: Rewind Kafka Stream consumer offset by timestamp

[Jorge Esteban Quilcate Otoya]

Thanks Matthias!

My comments below.

Regards,
Jorge.

El lun., 30 ene. 2017 a las 18:40, Matthias J. Sax ()
escribió:

> It would be enough, IMHO :)
>
> However, we need to discuss some details about this.
>
> 1) we could extend the reset tool with an flag --start-from-offsets and
> the user can specify an offset per partition
>
> This would give the most flexibility, but it is hard to use. Especially
> if you have many partitions, we do not want to hand in this information
> per command line (maybe an "offset file" would work).
>
> Doing this per topic or even global seems to be of little use because it
> lacks proper semantic interpretation.
>

Agree, this option is of little use but could be helpful to give backward
compatibility for clients that don't have timestamp index but nevertheless
they want to rewind to an specific offset.


>
>
> 2) we could extend the reset tool with an flag --start-from-timestamp
> that could be globally applied to all partitions (I guess, that is what
> you have in mind)
>
> Has the advantage that it is easier to use. However, what should the
> parameter format be? Milliseconds since the Epoch (what is the internal

format) seems hard to use either.
>

I think 'dateTime' and 'duration' are valid options here: you could define
to reprocess since 2017-01-01T09:00:00 and also reprocess since P1M - 1
month ago. XML duration (https://www.w3.org/TR/xmlschema-2/#duration) and
dateTime (https://www.w3.org/TR/xmlschema-2/#dateTime) lexical
representations could work here.


>
> There is also the question, how we deal with out-of-order data. Maybe
> it's sufficient though, to just set to the first tuple with equal of
> greater timestamp than the specified time. (and educate user, that they
> might see some older data, ie, with smaller ts, if there is later
> arriving out of order records).
>

Agree.


>
> We might want to exploit broker's timestamp index. But what about older
> brokers that do not have timestamp index, as we do have client backward
> compatibility now? We might just say, "not supported" though.
>
>
item (1) could be valid to give this option to older brokers.


> What about data, that lacks a proper timestamp and users work with
> custom timestamp extractor? Should we support this, too?
>
>
Haven't thought about this this use-case, could be a valid use case.


> Maybe we need a KIP discussion for this. It seems to be a broader feature.
>
>
Yes, I will love to do that. I also believe this could be a valid use-case
to be added to 'kafka-consumer-groups' command-line tool, and have an
external tool to rewind consumer-groups offsets.


> -Matthias
>
>
>
> On 1/30/17 2:07 AM, Jorge Esteban Quilcate Otoya wrote:
> > Thanks Eno and Matthias for your feedback!
> >
> > I've check KIP-95 and Matthias blog post (
> >
> https://www.confluent.io/blog/data-reprocessing-with-kafka-streams-resetting-a-streams-application/
> )
> > and I have a clearer idea on how stream internals work.
> >
> > In a general use-case, following Application Reset Tool's procedure:
> > ---
> >
> >1. for any specified input topic, it resets all offsets to zero
> >2. for any specified intermediate topic, seeks to the end for all
> >partitions
> >3. for all internal topic
> >   1. resets all offsets to zero
> >   2. deletes the topic
> >
> > ---
> > But instead of resetting input topics to zero, resetting input topics to
> > offset by timestamp wouldn't be enough?
> >
> > I will definitely take a look to StreamsResetter and give a try to
> support
> > this feature.
> >
> >
> > El lun., 30 ene. 2017 a las 1:43, Matthias J. Sax (<
> matth...@confluent.io>)
> > escribió:
> >
> >> You can always built you own little tool similar to StreamsResetter.java
> >> to get this done. Ie, you set the committed offset "manually" based on
> >> timestamps before you start your application.
> >>
> >> But as Eno mentioned, you need to think carefully about what a
> >> consistent reset point would be because you cannot reset the
> >> application's state...
> >>
> >> If you start you application with an empty state, this might be less of
> >> an concern though and seems reasonable.
> >>
> >>
> >> -Matthias
> >>
> >> On 1/29/17 12:55 PM, Eno Thereska wrote:
> >>> Hi Jorge,
> >>>
> >>> This is currently not possible, but it is likely to be considered for
> >> discussion. One challenge is that, if you have multiple topics, it is
> >> difficult to rewind them all back to a consistent point in time. KIP-95,
> >> currently under discussion, is handling the slightly different issue, of
> >> stopping the consuming at a point in time:
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-95%3A+Incremental+Batch+Processing+for+Kafka+Streams
> >> <
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-95:+Incremental+Batch+Processing+for+Kafka+Streams
> >>> .
> >>>
> >>> Thanks
> >>> Eno
>  On 29 Jan 2017, at 19:29, Jorge Esteban Quilcate Otoya <
> >> quilcate.jo...@gmail.com> wrote:
> 
> >>>

[GitHub] kafka pull request #2463: KAFKA-4706

[sharad-develop]

Github user sharad-develop closed the pull request at:

https://github.com/apache/kafka/pull/2463


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #2470: Kafka-4711 :Change Default unclean.leader.election...

[sharad-develop]

GitHub user sharad-develop opened a pull request:

https://github.com/apache/kafka/pull/2470

Kafka-4711 :Change Default unclean.leader.election.enabled from True to 
False (KIP-106)

Kafka-4711 :Change Default unclean.leader.election.enabled from True to 
False (KIP-106)
KAFKA-4623 :Change Default unclean.leader.election.enabled from True to 
False

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sharad-develop/kafka KAFKA-4711

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2470.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2470


commit 7946b2f35c0e3811ba7d706fc9c761d73ece47bb
Author: Damian Guy 
Date:   2017-01-16T19:40:47Z

MINOR: Remove unused constructor param from ProcessorStateManager

Remove applicationId parameter as it is no longer used.

Author: Damian Guy 

Reviewers: Guozhang Wang 

Closes #2385 from dguy/minor-remove-unused-param

commit 621dff22e79dc64b9a8748186dd985774044f91a
Author: Rajini Sivaram 
Date:   2017-01-17T11:16:29Z

KAFKA-4363; Documentation for sasl.jaas.config property

Author: Rajini Sivaram 

Reviewers: Ismael Juma 

Closes #2316 from rajinisivaram/KAFKA-4363

commit e3f4cdd0e249f78a7f4e8f064533bcd15eb11cbf
Author: Rajini Sivaram 
Date:   2017-01-17T12:55:07Z

KAFKA-4590; SASL/SCRAM system tests

Runs sanity test and one replication test using SASL/SCRAM.

Author: Rajini Sivaram 

Reviewers: Ewen Cheslack-Postava , Ismael Juma 


Closes #2355 from rajinisivaram/KAFKA-4590

commit 2b19ad9d8c47fb0f78a6e90d2f5711df6110bf1f
Author: Rajini Sivaram 
Date:   2017-01-17T18:42:55Z

KAFKA-4580; Use sasl.jaas.config for some system tests

Switched console_consumer, verifiable_consumer and verifiable_producer to 
use new sasl.jaas_config property instead of static JAAS configuration file 
when used with SASL_PLAINTEXT.

Author: Rajini Sivaram 

Reviewers: Ewen Cheslack-Postava , Ismael Juma 


Closes #2323 from rajinisivaram/KAFKA-4580

(cherry picked from commit 3f6c4f63c9c17424cf717ca76c74554bcf3b2e9a)
Signed-off-by: Ismael Juma 

commit 60d759a227087079e6fd270c68fd9e38441cb34a
Author: Jason Gustafson 
Date:   2017-01-17T18:42:05Z

MINOR: Some cleanups and additional testing for KIP-88

Author: Jason Gustafson 

Reviewers: Vahid Hashemian , Ismael Juma 


Closes #2383 from hachikuji/minor-cleanup-kip-88

commit c9b9acf6a8b542c2d0d825c17a4a20cf3fa5
Author: Damian Guy 
Date:   2017-01-17T20:33:11Z

KAFKA-4588: Wait for topics to be created in 
QueryableStateIntegrationTest.shouldNotMakeStoreAvailableUntilAllStoresAvailable

After debugging this i can see the times that it fails there is a race 
between when the topic is actually created/ready on the broker and when the 
assignment happens. When it fails `StreamPartitionAssignor.assign(..)` gets 
called with a `Cluster` with no topics. Hence the test hangs as no tasks get 
assigned. To fix this I added a `waitForTopics` method to 
`EmbeddedKafkaCluster`. This will wait until the topics have been created.

Author: Damian Guy 

Reviewers: Matthias J. Sax, Guozhang Wang

Closes #2371 from dguy/integration-test-fix

(cherry picked from commit 825f225bc5706b16af8ec44ca47ee1452c11e6f3)
Signed-off-by: Guozhang Wang 

commit 6f72a5a53c444278187fa6be58031168bcaffb26
Author: Damian Guy 
Date:   2017-01-17T22:13:46Z

KAFKA-3452 Follow-up: Refactoring StateStore hierarchies

This is a follow up of https://github.com/apache/kafka/pull/2166 - 
refactoring the store hierarchies as requested

Author: Damian Guy 

Reviewers: Guozhang Wang 

Closes #2360 from dguy/state-store-refactor

(cherry picked from commit 73b7ae0019d387407375f3865e263225c986a6ce)
Signed-off-by: Guozhang Wang 

commit eb62e5695506ae13bd37102c3c08e8a067eca0c8
Author: Ismael Juma 
Date:   2017-01-18T02:43:10Z

KAFKA-4591; Create Topic Policy follow-up

1. Added javadoc to public classes
2. Removed `s` from config name for consistency with interface name
3. The policy interface now implements Configurable and AutoCloseable as 
per the KIP
4. Use `null` instead of `-1` in `RequestMetadata`
5. Perform all broker validation before invoking the policy
6. Add tests

Author: Ismael Juma 

Reviewers: Jason Gustafson 

Closes #2388 from ijuma/create-topic-policy-docs-and-config-name-change

(cherry picked from commit fd6d7bcf335166a524dc9a29a50c96af8f1c1c02)
Signed-off-by: Ismael Juma 

commit e38794e020951adec5a5d0bbfe42c57294bf67bd
Author: Guozhang Wang 
Date:   2017-01-18T04:29:55Z

KAFKA-3502; move RocksDB

[jira] (KAFKA-4623) Change Default unclean.leader.election.enabled from True to False

[Sharad (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847100#comment-15847100
 ] 

Sharad commented on KAFKA-4623:
---

https://github.com/apache/kafka/pull/2470

> Change Default unclean.leader.election.enabled from True to False
> -
>
> Key: KAFKA-4623
> URL: https://issues.apache.org/jira/browse/KAFKA-4623
> Project: Kafka
>  Issue Type: Bug
>Reporter: Ben Stopford
>Assignee: Sharad
>
> See KIP-106
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-106+-+Change+Default+unclean.leader.election.enabled+from+True+to+False



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] (KAFKA-4711) Change Default unclean.leader.election.enabled from True to False (KIP-106)

[Sharad (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847098#comment-15847098
 ] 

Sharad commented on KAFKA-4711:
---

https://github.com/apache/kafka/pull/2470

> Change Default unclean.leader.election.enabled from True to False (KIP-106)
> ---
>
> Key: KAFKA-4711
> URL: https://issues.apache.org/jira/browse/KAFKA-4711
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.11.0.0
>Reporter: Ben Stopford
>Assignee: Sharad
>
> See 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-106+-+Change+Default+unclean.leader.election.enabled+from+True+to+False



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] (KAFKA-4711) Change Default unclean.leader.election.enabled from True to False (KIP-106)

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-4711.

Resolution: Duplicate

KAFKA-4623.

> Change Default unclean.leader.election.enabled from True to False (KIP-106)
> ---
>
> Key: KAFKA-4711
> URL: https://issues.apache.org/jira/browse/KAFKA-4711
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.11.0.0
>Reporter: Ben Stopford
>Assignee: Sharad
>
> See 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-106+-+Change+Default+unclean.leader.election.enabled+from+True+to+False



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] (KAFKA-4623) Change Default unclean.leader.election.enabled from True to False

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-4623:
---
Fix Version/s: 0.11.0.0

> Change Default unclean.leader.election.enabled from True to False
> -
>
> Key: KAFKA-4623
> URL: https://issues.apache.org/jira/browse/KAFKA-4623
> Project: Kafka
>  Issue Type: Bug
>Reporter: Ben Stopford
>Assignee: Sharad
> Fix For: 0.11.0.0
>
>
> See KIP-106
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-106+-+Change+Default+unclean.leader.election.enabled+from+True+to+False



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] (KAFKA-4711) Change Default unclean.leader.election.enabled from True to False (KIP-106)

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847106#comment-15847106
 ] 

Ismael Juma edited comment on KAFKA-4711 at 1/31/17 4:50 PM:
-

Duplicate of KAFKA-4623.


was (Author: ijuma):
KAFKA-4623.

> Change Default unclean.leader.election.enabled from True to False (KIP-106)
> ---
>
> Key: KAFKA-4711
> URL: https://issues.apache.org/jira/browse/KAFKA-4711
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.11.0.0
>Reporter: Ben Stopford
>Assignee: Sharad
>
> See 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-106+-+Change+Default+unclean.leader.election.enabled+from+True+to+False



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] (KAFKA-2955) Add Prompt to kafka-console-producer

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-2955:
-
Resolution: Fixed
Status: Resolved  (was: Patch Available)

Issue resolved by pull request 1233
[https://github.com/apache/kafka/pull/1233]

> Add Prompt to kafka-console-producer
> 
>
> Key: KAFKA-2955
> URL: https://issues.apache.org/jira/browse/KAFKA-2955
> Project: Kafka
>  Issue Type: Improvement
>  Components: producer 
>Affects Versions: 0.9.0.0
>Reporter: Jesse Anderson
>Assignee: Manikumar Reddy
> Fix For: 0.10.3.0
>
>
> A common source of confusion for people using the kafka-console-producer is a 
> lack of prompt. People think that kafka-console-producer is still starting up 
> or connecting. Adding a ">" prompt to show that the kafka-console-producer is 
> ready will fix that.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #1233: KAFKA-2955; Add a simple ">" prompt to console pro...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1233


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] (KAFKA-2955) Add Prompt to kafka-console-producer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847172#comment-15847172
 ] 

ASF GitHub Bot commented on KAFKA-2955:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1233


> Add Prompt to kafka-console-producer
> 
>
> Key: KAFKA-2955
> URL: https://issues.apache.org/jira/browse/KAFKA-2955
> Project: Kafka
>  Issue Type: Improvement
>  Components: producer 
>Affects Versions: 0.9.0.0
>Reporter: Jesse Anderson
>Assignee: Manikumar Reddy
> Fix For: 0.10.3.0
>
>
> A common source of confusion for people using the kafka-console-producer is a 
> lack of prompt. People think that kafka-console-producer is still starting up 
> or connecting. Adding a ">" prompt to show that the kafka-console-producer is 
> ready will fix that.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #2471: KAFKA-4317: Checkpoint State Stores on commit/flus...

[dguy]

GitHub user dguy opened a pull request:

https://github.com/apache/kafka/pull/2471

KAFKA-4317: Checkpoint State Stores on commit/flush

Currently the checkpoint file is deleted at state store initialization and 
it is only ever written again during a clean shutdown. This can result in 
significant delays during restarts as the entire store needs to be loaded from 
the changelog. 
We can mitigate against this by frequently checkpointing the offsets. The 
checkpointing happens only during the commit phase, i.e, after we have manually 
flushed the store and the producer. So we guarantee that the checkpointed 
offsets are never greater than what has been flushed. 
In the event of hard failure we can recover by reading the checkpoints and 
consuming from the stored offsets.
The checkpoint interval can be controlled by the config 
`statestore.checkpoint.interval.ms` - if this is set to a value <= 0 it 
effectively turns checkpoints off. The interval is only i guide in that the 
minimum checkpoint time is always going to be the commit interval (as we need 
to do this to guarantee consistency)

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dguy/kafka kafka-4317

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2471.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2471


commit 6743dc63293e2d0fca57dcb7d1a0ace5237837b0
Author: Damian Guy 
Date:   2017-01-31T13:37:00Z

checkpoint statestores




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] (KAFKA-4317) RocksDB checkpoint files lost on kill -9

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847179#comment-15847179
 ] 

ASF GitHub Bot commented on KAFKA-4317:
---

GitHub user dguy opened a pull request:

https://github.com/apache/kafka/pull/2471

KAFKA-4317: Checkpoint State Stores on commit/flush

Currently the checkpoint file is deleted at state store initialization and 
it is only ever written again during a clean shutdown. This can result in 
significant delays during restarts as the entire store needs to be loaded from 
the changelog. 
We can mitigate against this by frequently checkpointing the offsets. The 
checkpointing happens only during the commit phase, i.e, after we have manually 
flushed the store and the producer. So we guarantee that the checkpointed 
offsets are never greater than what has been flushed. 
In the event of hard failure we can recover by reading the checkpoints and 
consuming from the stored offsets.
The checkpoint interval can be controlled by the config 
`statestore.checkpoint.interval.ms` - if this is set to a value <= 0 it 
effectively turns checkpoints off. The interval is only i guide in that the 
minimum checkpoint time is always going to be the commit interval (as we need 
to do this to guarantee consistency)

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dguy/kafka kafka-4317

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2471.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2471


commit 6743dc63293e2d0fca57dcb7d1a0ace5237837b0
Author: Damian Guy 
Date:   2017-01-31T13:37:00Z

checkpoint statestores




> RocksDB checkpoint files lost on kill -9
> 
>
> Key: KAFKA-4317
> URL: https://issues.apache.org/jira/browse/KAFKA-4317
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Affects Versions: 0.10.0.1
>Reporter: Greg Fodor
>Assignee: Damian Guy
>Priority: Critical
>  Labels: architecture, user-experience
>
> Right now, the checkpoint files for logged RocksDB stores are written during 
> a graceful shutdown, and removed upon restoration. Unfortunately this means 
> that in a scenario where the process is forcibly killed, the checkpoint files 
> are not there, so all RocksDB stores are rematerialized from scratch on the 
> next launch.
> In a way, this is good, because it simulates bootstrapping a new node (for 
> example, its a good way to see how much I/O is used to rematerialize the 
> stores) however it leads to longer recovery times when a non-graceful 
> shutdown occurs and we want to get the job up and running again.
> It seems that two possible things to consider:
> - Simply do not remove checkpoint files on restoring. This way a kill -9 will 
> result in only repeating the restoration of all the data generated in the 
> source topics since the last graceful shutdown.
> - Continually update the checkpoint files (perhaps on commit) -- this would 
> result in the least amount of overhead/latency in restarting, but the 
> additional complexity may not be worth it.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] (KAFKA-1610) Local modifications to collections generated from mapValues will be lost

[Jozef Koval (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847261#comment-15847261
 ] 

Jozef Koval commented on KAFKA-1610:


What is current status of this issue? There has not been any updates for more 
than a year, may I take it and work on it as my first issue with kafka?

> Local modifications to collections generated from mapValues will be lost
> 
>
> Key: KAFKA-1610
> URL: https://issues.apache.org/jira/browse/KAFKA-1610
> Project: Kafka
>  Issue Type: Bug
>Reporter: Guozhang Wang
>Assignee: Mayuresh Gharat
>  Labels: newbie
> Attachments: KAFKA-1610_2014-08-29_09:51:51.patch, 
> KAFKA-1610_2014-08-29_10:03:55.patch, KAFKA-1610_2014-09-03_11:27:50.patch, 
> KAFKA-1610_2014-09-16_13:08:17.patch, KAFKA-1610_2014-09-16_15:23:27.patch, 
> KAFKA-1610_2014-09-30_23:21:46.patch, KAFKA-1610_2014-10-02_12:07:01.patch, 
> KAFKA-1610_2014-10-02_12:09:46.patch, KAFKA-1610.patch
>
>
> In our current Scala code base we have 40+ usages of mapValues, however it 
> has an important semantic difference with map, which is that "map" creates a 
> new map collection instance, while "mapValues" just create a map view of the 
> original map, and hence any further value changes to the view will be 
> effectively lost.
> Example code:
> {code}
> scala> case class Test(i: Int, var j: Int) {}
> defined class Test
> scala> val a = collection.mutable.Map(1 -> 1)
> a: scala.collection.mutable.Map[Int,Int] = Map(1 -> 1)
> scala> val b = a.mapValues(v => Test(v, v))
> b: scala.collection.Map[Int,Test] = Map(1 -> Test(1,1))
> scala> val c = a.map(v => v._1 -> Test(v._2, v._2))
> c: scala.collection.mutable.Map[Int,Test] = Map(1 -> Test(1,1))
> scala> b.foreach(kv => kv._2.j = kv._2.j + 1)
> scala> b
> res1: scala.collection.Map[Int,Test] = Map(1 -> Test(1,1))
> scala> c.foreach(kv => kv._2.j = kv._2.j + 1)
> scala> c
> res3: scala.collection.mutable.Map[Int,Test] = Map(1 -> Test(1,2))
> scala> a.put(1,3)
> res4: Option[Int] = Some(1)
> scala> b
> res5: scala.collection.Map[Int,Test] = Map(1 -> Test(3,3))
> scala> c
> res6: scala.collection.mutable.Map[Int,Test] = Map(1 -> Test(1,2))
> {code}
> We need to go through all these mapValue to see if they should be changed to 
> map



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] (KAFKA-4317) RocksDB checkpoint files lost on kill -9

[Damian Guy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Damian Guy updated KAFKA-4317:
--
Status: Patch Available  (was: Open)

> RocksDB checkpoint files lost on kill -9
> 
>
> Key: KAFKA-4317
> URL: https://issues.apache.org/jira/browse/KAFKA-4317
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Affects Versions: 0.10.0.1
>Reporter: Greg Fodor
>Assignee: Damian Guy
>Priority: Critical
>  Labels: architecture, user-experience
>
> Right now, the checkpoint files for logged RocksDB stores are written during 
> a graceful shutdown, and removed upon restoration. Unfortunately this means 
> that in a scenario where the process is forcibly killed, the checkpoint files 
> are not there, so all RocksDB stores are rematerialized from scratch on the 
> next launch.
> In a way, this is good, because it simulates bootstrapping a new node (for 
> example, its a good way to see how much I/O is used to rematerialize the 
> stores) however it leads to longer recovery times when a non-graceful 
> shutdown occurs and we want to get the job up and running again.
> It seems that two possible things to consider:
> - Simply do not remove checkpoint files on restoring. This way a kill -9 will 
> result in only repeating the restoration of all the data generated in the 
> source topics since the last graceful shutdown.
> - Continually update the checkpoint files (perhaps on commit) -- this would 
> result in the least amount of overhead/latency in restarting, but the 
> additional complexity may not be worth it.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] (KAFKA-4317) RocksDB checkpoint files lost on kill -9

[Matthias J. Sax (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax updated KAFKA-4317:
---
Labels: architecture needs-kip user-experience  (was: architecture 
user-experience)

> RocksDB checkpoint files lost on kill -9
> 
>
> Key: KAFKA-4317
> URL: https://issues.apache.org/jira/browse/KAFKA-4317
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Affects Versions: 0.10.0.1
>Reporter: Greg Fodor
>Assignee: Damian Guy
>Priority: Critical
>  Labels: architecture, needs-kip, user-experience
>
> Right now, the checkpoint files for logged RocksDB stores are written during 
> a graceful shutdown, and removed upon restoration. Unfortunately this means 
> that in a scenario where the process is forcibly killed, the checkpoint files 
> are not there, so all RocksDB stores are rematerialized from scratch on the 
> next launch.
> In a way, this is good, because it simulates bootstrapping a new node (for 
> example, its a good way to see how much I/O is used to rematerialize the 
> stores) however it leads to longer recovery times when a non-graceful 
> shutdown occurs and we want to get the job up and running again.
> It seems that two possible things to consider:
> - Simply do not remove checkpoint files on restoring. This way a kill -9 will 
> result in only repeating the restoration of all the data generated in the 
> source topics since the last graceful shutdown.
> - Continually update the checkpoint files (perhaps on commit) -- this would 
> result in the least amount of overhead/latency in restarting, but the 
> additional complexity may not be worth it.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] (KAFKA-3265) Create Java Admin Client

[Colin P. McCabe (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847277#comment-15847277
 ] 

Colin P. McCabe commented on KAFKA-3265:


Hi [~granthenke],

I wrote a Java AdminClient.  Check it out.

> Create Java Admin Client
> 
>
> Key: KAFKA-3265
> URL: https://issues.apache.org/jira/browse/KAFKA-3265
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Grant Henke
>Assignee: Grant Henke
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #2472: KAFKA-3265: Create Java Admin Client

[cmccabe]

GitHub user cmccabe opened a pull request:

https://github.com/apache/kafka/pull/2472

KAFKA-3265: Create Java Admin Client

Create a Java AdminClient as described in KAFKA-3265 and KIP-4.

Currently it supports getAllGroups, getAllBrokerVersions, deleteTopics, and 
createTopics.  For each API, it uses the associated Kafka requests rather than 
going through ZooKeeper.

It is called AdministrativeClient to avoid confusing with the 
scala.kafka.admin.AdminClient class.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cmccabe/kafka KAFKA-3265

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2472.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2472


commit f8d425863067614de2644b0260ac9c5c51234c00
Author: Colin P. Mccabe 
Date:   2017-01-27T23:25:42Z

Add AdminClient, AdminClientConfig

commit 11c00df0e160493cba318d1ba78698b79ef9f40e
Author: Colin P. Mccabe 
Date:   2017-01-28T00:25:57Z

Add AdminClientsTest.scala

commit 4022cb19a5ac27f5d1140878bad5e58b42301229
Author: Colin P. Mccabe 
Date:   2017-01-28T00:27:18Z

AdminClient: metricGrpPrefix should be "adminclient"

commit ebafe59fcb29d2a9bacc8e8a4cfd8a95d1edd027
Author: Colin P. Mccabe 
Date:   2017-01-30T21:29:08Z

Implement AdminClient#createTopics, AdminClient#deleteTopics

commit 28c2c272eba66e8f95fb8bafd94b83a85cd0e24e
Author: Colin P. Mccabe 
Date:   2017-01-31T01:51:49Z

AdminClient -> AdministrativeClient

commit 50443b6e23790b6747dd2c16fd1d210d66ed9768
Author: Colin P. Mccabe 
Date:   2017-01-31T17:56:10Z

Add Try

commit 146a2e05e762669a0480c9b52e4cd298b48fe42d
Author: Colin P. Mccabe 
Date:   2017-01-31T17:57:29Z

Implement getAllBrokerVersions

commit 090d4b1b62086182caef6404d0852a151982ccba
Author: Colin P. Mccabe 
Date:   2017-01-31T18:29:30Z

fixups

commit 969fdb75dc5aaffb9a2a7547f8f28af9fe9930e9
Author: Colin P. Mccabe 
Date:   2017-01-31T18:38:20Z

style changes




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] (KAFKA-3265) Create Java Admin Client

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847286#comment-15847286
 ] 

ASF GitHub Bot commented on KAFKA-3265:
---

GitHub user cmccabe opened a pull request:

https://github.com/apache/kafka/pull/2472

KAFKA-3265: Create Java Admin Client

Create a Java AdminClient as described in KAFKA-3265 and KIP-4.

Currently it supports getAllGroups, getAllBrokerVersions, deleteTopics, and 
createTopics.  For each API, it uses the associated Kafka requests rather than 
going through ZooKeeper.

It is called AdministrativeClient to avoid confusing with the 
scala.kafka.admin.AdminClient class.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cmccabe/kafka KAFKA-3265

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2472.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2472


commit f8d425863067614de2644b0260ac9c5c51234c00
Author: Colin P. Mccabe 
Date:   2017-01-27T23:25:42Z

Add AdminClient, AdminClientConfig

commit 11c00df0e160493cba318d1ba78698b79ef9f40e
Author: Colin P. Mccabe 
Date:   2017-01-28T00:25:57Z

Add AdminClientsTest.scala

commit 4022cb19a5ac27f5d1140878bad5e58b42301229
Author: Colin P. Mccabe 
Date:   2017-01-28T00:27:18Z

AdminClient: metricGrpPrefix should be "adminclient"

commit ebafe59fcb29d2a9bacc8e8a4cfd8a95d1edd027
Author: Colin P. Mccabe 
Date:   2017-01-30T21:29:08Z

Implement AdminClient#createTopics, AdminClient#deleteTopics

commit 28c2c272eba66e8f95fb8bafd94b83a85cd0e24e
Author: Colin P. Mccabe 
Date:   2017-01-31T01:51:49Z

AdminClient -> AdministrativeClient

commit 50443b6e23790b6747dd2c16fd1d210d66ed9768
Author: Colin P. Mccabe 
Date:   2017-01-31T17:56:10Z

Add Try

commit 146a2e05e762669a0480c9b52e4cd298b48fe42d
Author: Colin P. Mccabe 
Date:   2017-01-31T17:57:29Z

Implement getAllBrokerVersions

commit 090d4b1b62086182caef6404d0852a151982ccba
Author: Colin P. Mccabe 
Date:   2017-01-31T18:29:30Z

fixups

commit 969fdb75dc5aaffb9a2a7547f8f28af9fe9930e9
Author: Colin P. Mccabe 
Date:   2017-01-31T18:38:20Z

style changes




> Create Java Admin Client
> 
>
> Key: KAFKA-3265
> URL: https://issues.apache.org/jira/browse/KAFKA-3265
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Grant Henke
>Assignee: Grant Henke
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] (KAFKA-3265) Create Java Admin Client

[Colin P. McCabe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin P. McCabe reassigned KAFKA-3265:
--

Assignee: Colin P. McCabe  (was: Grant Henke)

> Create Java Admin Client
> 
>
> Key: KAFKA-3265
> URL: https://issues.apache.org/jira/browse/KAFKA-3265
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Grant Henke
>Assignee: Colin P. McCabe
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] (KAFKA-3265) Create Java Admin Client

[Colin P. McCabe (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847277#comment-15847277
 ] 

Colin P. McCabe edited comment on KAFKA-3265 at 1/31/17 6:49 PM:
-

I wrote a Java `AdministrativeClient`, building off the work already done for 
KIP-4.  Currently it supports `getAllGroups`, `getAllBrokerVersions`, 
`deleteTopics`, and `createTopics`. For each API, it uses the associated Kafka 
requests rather than going through ZooKeeper.  It is called 
`AdministrativeClient` to avoid confusing with the 
`scala.kafka.admin.AdminClient` class.  Take a look


was (Author: cmccabe):
Hi [~granthenke],

I wrote a Java AdminClient.  Check it out.

> Create Java Admin Client
> 
>
> Key: KAFKA-3265
> URL: https://issues.apache.org/jira/browse/KAFKA-3265
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Grant Henke
>Assignee: Colin P. McCabe
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] (KAFKA-3265) Create Java Admin Client

[Colin P. McCabe (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847277#comment-15847277
 ] 

Colin P. McCabe edited comment on KAFKA-3265 at 1/31/17 6:50 PM:
-

I wrote a Java {{AdministrativeClient}}, building off the work already done for 
KIP-4.  Currently it supports {{getAllGroups}}, {{getAllBrokerVersions}}, 
{{deleteTopics}}, and {{createTopics}}. For each API, it uses the associated 
Kafka requests rather than going through ZooKeeper.  It is called 
{{AdministrativeClient}} to avoid confusing with the 
{{scala.kafka.admin.AdminClient}} class.  Take a look


was (Author: cmccabe):
I wrote a Java `AdministrativeClient`, building off the work already done for 
KIP-4.  Currently it supports `getAllGroups`, `getAllBrokerVersions`, 
`deleteTopics`, and `createTopics`. For each API, it uses the associated Kafka 
requests rather than going through ZooKeeper.  It is called 
`AdministrativeClient` to avoid confusing with the 
`scala.kafka.admin.AdminClient` class.  Take a look

> Create Java Admin Client
> 
>
> Key: KAFKA-3265
> URL: https://issues.apache.org/jira/browse/KAFKA-3265
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Grant Henke
>Assignee: Colin P. McCabe
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] (KAFKA-1610) Local modifications to collections generated from mapValues will be lost

[Mayuresh Gharat (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847355#comment-15847355
 ] 

Mayuresh Gharat commented on KAFKA-1610:


[~jozi-k] sure. The patch has been available but we somehow missed this.

> Local modifications to collections generated from mapValues will be lost
> 
>
> Key: KAFKA-1610
> URL: https://issues.apache.org/jira/browse/KAFKA-1610
> Project: Kafka
>  Issue Type: Bug
>Reporter: Guozhang Wang
>Assignee: Mayuresh Gharat
>  Labels: newbie
> Attachments: KAFKA-1610_2014-08-29_09:51:51.patch, 
> KAFKA-1610_2014-08-29_10:03:55.patch, KAFKA-1610_2014-09-03_11:27:50.patch, 
> KAFKA-1610_2014-09-16_13:08:17.patch, KAFKA-1610_2014-09-16_15:23:27.patch, 
> KAFKA-1610_2014-09-30_23:21:46.patch, KAFKA-1610_2014-10-02_12:07:01.patch, 
> KAFKA-1610_2014-10-02_12:09:46.patch, KAFKA-1610.patch
>
>
> In our current Scala code base we have 40+ usages of mapValues, however it 
> has an important semantic difference with map, which is that "map" creates a 
> new map collection instance, while "mapValues" just create a map view of the 
> original map, and hence any further value changes to the view will be 
> effectively lost.
> Example code:
> {code}
> scala> case class Test(i: Int, var j: Int) {}
> defined class Test
> scala> val a = collection.mutable.Map(1 -> 1)
> a: scala.collection.mutable.Map[Int,Int] = Map(1 -> 1)
> scala> val b = a.mapValues(v => Test(v, v))
> b: scala.collection.Map[Int,Test] = Map(1 -> Test(1,1))
> scala> val c = a.map(v => v._1 -> Test(v._2, v._2))
> c: scala.collection.mutable.Map[Int,Test] = Map(1 -> Test(1,1))
> scala> b.foreach(kv => kv._2.j = kv._2.j + 1)
> scala> b
> res1: scala.collection.Map[Int,Test] = Map(1 -> Test(1,1))
> scala> c.foreach(kv => kv._2.j = kv._2.j + 1)
> scala> c
> res3: scala.collection.mutable.Map[Int,Test] = Map(1 -> Test(1,2))
> scala> a.put(1,3)
> res4: Option[Int] = Some(1)
> scala> b
> res5: scala.collection.Map[Int,Test] = Map(1 -> Test(3,3))
> scala> c
> res6: scala.collection.mutable.Map[Int,Test] = Map(1 -> Test(1,2))
> {code}
> We need to go through all these mapValue to see if they should be changed to 
> map



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] (KAFKA-4717) connect jars are missing LICENSE/NOTICE files

[dan norwood (JIRA)]

dan norwood created KAFKA-4717:
--

 Summary: connect jars are missing LICENSE/NOTICE files
 Key: KAFKA-4717
 URL: https://issues.apache.org/jira/browse/KAFKA-4717
 Project: Kafka
  Issue Type: Bug
  Components: KafkaConnect
Reporter: dan norwood


other artifacts in kafka contain LICENSE/NOTICE files in the deployed jars, but 
connect seems to be missing them.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] (KAFKA-4717) connect jars are missing LICENSE/NOTICE files

[Ewen Cheslack-Postava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava updated KAFKA-4717:
-
Priority: Blocker  (was: Major)

> connect jars are missing LICENSE/NOTICE files
> -
>
> Key: KAFKA-4717
> URL: https://issues.apache.org/jira/browse/KAFKA-4717
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Reporter: dan norwood
>Priority: Blocker
> Fix For: 0.10.2.0
>
>
> other artifacts in kafka contain LICENSE/NOTICE files in the deployed jars, 
> but connect seems to be missing them.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] (KAFKA-4717) connect jars are missing LICENSE/NOTICE files

[Ewen Cheslack-Postava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava updated KAFKA-4717:
-
Fix Version/s: 0.10.2.0

> connect jars are missing LICENSE/NOTICE files
> -
>
> Key: KAFKA-4717
> URL: https://issues.apache.org/jira/browse/KAFKA-4717
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Reporter: dan norwood
> Fix For: 0.10.2.0
>
>
> other artifacts in kafka contain LICENSE/NOTICE files in the deployed jars, 
> but connect seems to be missing them.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] (KAFKA-4689) OffsetValidationTest fails validation with "Current position greater than the total number of consumed records"

[Ewen Cheslack-Postava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava updated KAFKA-4689:
-
Fix Version/s: (was: 0.10.2.0)

> OffsetValidationTest fails validation with "Current position greater than the 
> total number of consumed records"
> ---
>
> Key: KAFKA-4689
> URL: https://issues.apache.org/jira/browse/KAFKA-4689
> Project: Kafka
>  Issue Type: Bug
>  Components: clients, consumer, system tests
>Reporter: Ewen Cheslack-Postava
>Assignee: Jason Gustafson
>  Labels: system-test-failure
>
> {quote}
> 
> test_id:
> kafkatest.tests.client.consumer_test.OffsetValidationTest.test_consumer_bounce.clean_shutdown=False.bounce_mode=all
> status: FAIL
> run time:   1 minute 49.834 seconds
> Current position greater than the total number of consumed records
> Traceback (most recent call last):
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.6.0-py2.7.egg/ducktape/tests/runner_client.py",
>  line 123, in run
> data = self.run_test()
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.6.0-py2.7.egg/ducktape/tests/runner_client.py",
>  line 176, in run_test
> return self.test_context.function(self.test)
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.6.0-py2.7.egg/ducktape/mark/_mark.py",
>  line 321, in wrapper
> return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/tests/kafkatest/tests/client/consumer_test.py",
>  line 157, in test_consumer_bounce
> "Current position greater than the total number of consumed records"
> AssertionError: Current position greater than the total number of consumed 
> records
> {quote}
> See also 
> https://issues.apache.org/jira/browse/KAFKA-3513?focusedCommentId=15791790&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15791790
>  which is another instance of this bug, which indicates the issue goes back 
> at least as far as 1/17/2017. Note that I don't think we've seen this in 
> 0.10.1 yet.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] (KAFKA-4689) OffsetValidationTest fails validation with "Current position greater than the total number of consumed records"

[Ewen Cheslack-Postava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava updated KAFKA-4689:
-
Priority: Major  (was: Blocker)

> OffsetValidationTest fails validation with "Current position greater than the 
> total number of consumed records"
> ---
>
> Key: KAFKA-4689
> URL: https://issues.apache.org/jira/browse/KAFKA-4689
> Project: Kafka
>  Issue Type: Bug
>  Components: clients, consumer, system tests
>Reporter: Ewen Cheslack-Postava
>Assignee: Jason Gustafson
>  Labels: system-test-failure
>
> {quote}
> 
> test_id:
> kafkatest.tests.client.consumer_test.OffsetValidationTest.test_consumer_bounce.clean_shutdown=False.bounce_mode=all
> status: FAIL
> run time:   1 minute 49.834 seconds
> Current position greater than the total number of consumed records
> Traceback (most recent call last):
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.6.0-py2.7.egg/ducktape/tests/runner_client.py",
>  line 123, in run
> data = self.run_test()
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.6.0-py2.7.egg/ducktape/tests/runner_client.py",
>  line 176, in run_test
> return self.test_context.function(self.test)
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.6.0-py2.7.egg/ducktape/mark/_mark.py",
>  line 321, in wrapper
> return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/tests/kafkatest/tests/client/consumer_test.py",
>  line 157, in test_consumer_bounce
> "Current position greater than the total number of consumed records"
> AssertionError: Current position greater than the total number of consumed 
> records
> {quote}
> See also 
> https://issues.apache.org/jira/browse/KAFKA-3513?focusedCommentId=15791790&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15791790
>  which is another instance of this bug, which indicates the issue goes back 
> at least as far as 1/17/2017. Note that I don't think we've seen this in 
> 0.10.1 yet.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #2469: MINOR: Logging improvements in consumer internals

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2469


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #2473: KAFKA-4717: Use absolute paths to files in root di...

[ewencp]

GitHub user ewencp opened a pull request:

https://github.com/apache/kafka/pull/2473

KAFKA-4717: Use absolute paths to files in root directory so all jars 
include LICENSE and NOTICE files



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ewencp/kafka 
kafka-4717-connect-license-and-notice-files

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2473.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2473


commit 8868c825fa274610358a4fad4004489f59bed610
Author: Ewen Cheslack-Postava 
Date:   2017-01-31T20:53:02Z

KAFKA-4717: Use absolute paths to files in root directory so all jars 
include LICENSE and NOTICE files




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] (KAFKA-4717) connect jars are missing LICENSE/NOTICE files

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847493#comment-15847493
 ] 

ASF GitHub Bot commented on KAFKA-4717:
---

GitHub user ewencp opened a pull request:

https://github.com/apache/kafka/pull/2473

KAFKA-4717: Use absolute paths to files in root directory so all jars 
include LICENSE and NOTICE files



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ewencp/kafka 
kafka-4717-connect-license-and-notice-files

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2473.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2473


commit 8868c825fa274610358a4fad4004489f59bed610
Author: Ewen Cheslack-Postava 
Date:   2017-01-31T20:53:02Z

KAFKA-4717: Use absolute paths to files in root directory so all jars 
include LICENSE and NOTICE files




> connect jars are missing LICENSE/NOTICE files
> -
>
> Key: KAFKA-4717
> URL: https://issues.apache.org/jira/browse/KAFKA-4717
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Reporter: dan norwood
>Priority: Blocker
> Fix For: 0.10.2.0
>
>
> other artifacts in kafka contain LICENSE/NOTICE files in the deployed jars, 
> but connect seems to be missing them.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #2474: KAFKA-4039: Fix deadlock during shutdown due to lo...

[ijuma]

GitHub user ijuma opened a pull request:

https://github.com/apache/kafka/pull/2474

KAFKA-4039: Fix deadlock during shutdown due to log truncation not allowed



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijuma/kafka 
kafka-4039-deadlock-during-shutdown

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2474.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2474


commit d902c2bc81c4799ad1eb7bc7ec50e67d04d1c9c2
Author: Maysam Yabandeh 
Date:   2016-08-17T18:21:07Z

KAFKA-4039: delay invocation of System.exit via FatalExitException

commit 548ddec8c8715125f25f3da3263fe4ead39deca2
Author: Maysam Yabandeh 
Date:   2016-08-17T22:39:12Z

KAFKA-4039 change FatalExitException to Error

commit 08992e10eacb661d24506bff1f617f42cce02ffb
Author: Maysam Yabandeh 
Date:   2016-08-18T00:33:43Z

KAFKA-4039 fix the default value of test mode in FatalExitError

commit bcf9e9d605fef9bc2cf46380767f38816313
Author: Maysam Yabandeh 
Date:   2016-08-18T14:26:20Z

KAFKA-4039 mock System.exit

commit 98acc3674a610ac329935c2d3a1bbdd032685d25
Author: Maysam Yabandeh 
Date:   2016-08-19T14:38:33Z

KAFKA-4039 exit inside an anonymous thread, add test for deadlock

commit 39a667eda0817bc42beaaedb962e5797b6f106e1
Author: Ismael Juma 
Date:   2017-01-30T14:48:24Z

Merge remote-tracking branch 'apache/trunk' into 
kafka-4039-deadlock-during-shutdown

* apache/trunk: (552 commits)
  MINOR: JavaDoc markup cleanup
  KAFKA-4679: Remove unstable markers from Connect APIs
  KAFKA-4450; Add upgrade tests for 0.10.1 and rename TRUNK to DEV_BRANCH 
to reduce confusion
  KAFKA-4635; Client Compatibility follow-ups
  MINOR: update JavaDocs for Kafka Streams DSL helpers
  KAFKA-4557; Handle Producer.send correctly in expiry callbacks
  MINOR: Update copyright year in the NOTICE file.
  KAFKA-4664; Update docs/protocol.html with KIP-97 information
  KAFKA-4704; Coordinator cache loading fails if groupId is reused for 
offset storage after group is removed
  MINOR: Replace for within for each; replace if-elseif with match
  KAFKA-4644: Improve test coverage of StreamsPartitionAssignor
  MINOR: Update KTable JavaDoc
  MINOR: Include more detail in `ConfigDef.parseType` exception message
  KAFKA-4578; Upgrade notes for 0.10.2.0
  MINOR: Streams API JavaDoc improvements
  MINOR: Add Streams system test for broker backwards compatibility
  MINOR: Close create topics policy during shutdown and more tests
  MINOR: Update JavaDoc for DSL PAPI-API
  KAFKA-4636; Per listener security settings overrides (KIP-103)
  MINOR: Change logging level for ignored maybeAddMetric from debug to trace
  ...

commit 4d130ef7e990ca8215684adc36865f08caa79557
Author: Ismael Juma 
Date:   2017-01-31T21:07:37Z

Try a slightly different approach to solving the shutdown deadlock

* Open the latch in `ShutdownableThread` before calling `exit`
* Introduce `Exit` classes that perform exit/halt and can be changed in 
tests
* Invoke `exit` from the calling thread instead of spawning a new thread
* Updated the tests
* A few clean-ups

commit 1c8032b7419874ef2c56e7cf321d39a3f64e6833
Author: Ismael Juma 
Date:   2017-01-31T21:07:51Z

A few style clean-ups

commit 50b6c571ae51d2c9f5b7288faace349acc156bc8
Author: Ismael Juma 
Date:   2017-01-31T21:08:18Z

Merge remote-tracking branch 'apache/trunk' into 
kafka-4039-deadlock-during-shutdown

* apache/trunk:
  MINOR: Logging improvements in consumer internals
  KAFKA-2955; Add a simple ">" prompt to console producer
  KAFKA-4613: Follow-up to fix JavaDocs
  KAFKA-4613: Treat null-key records the same way for joins and aggreations




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] (KAFKA-4039) Exit Strategy: using exceptions instead of inline invocation of exit/halt

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847520#comment-15847520
 ] 

ASF GitHub Bot commented on KAFKA-4039:
---

GitHub user ijuma opened a pull request:

https://github.com/apache/kafka/pull/2474

KAFKA-4039: Fix deadlock during shutdown due to log truncation not allowed



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijuma/kafka 
kafka-4039-deadlock-during-shutdown

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2474.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2474


commit d902c2bc81c4799ad1eb7bc7ec50e67d04d1c9c2
Author: Maysam Yabandeh 
Date:   2016-08-17T18:21:07Z

KAFKA-4039: delay invocation of System.exit via FatalExitException

commit 548ddec8c8715125f25f3da3263fe4ead39deca2
Author: Maysam Yabandeh 
Date:   2016-08-17T22:39:12Z

KAFKA-4039 change FatalExitException to Error

commit 08992e10eacb661d24506bff1f617f42cce02ffb
Author: Maysam Yabandeh 
Date:   2016-08-18T00:33:43Z

KAFKA-4039 fix the default value of test mode in FatalExitError

commit bcf9e9d605fef9bc2cf46380767f38816313
Author: Maysam Yabandeh 
Date:   2016-08-18T14:26:20Z

KAFKA-4039 mock System.exit

commit 98acc3674a610ac329935c2d3a1bbdd032685d25
Author: Maysam Yabandeh 
Date:   2016-08-19T14:38:33Z

KAFKA-4039 exit inside an anonymous thread, add test for deadlock

commit 39a667eda0817bc42beaaedb962e5797b6f106e1
Author: Ismael Juma 
Date:   2017-01-30T14:48:24Z

Merge remote-tracking branch 'apache/trunk' into 
kafka-4039-deadlock-during-shutdown

* apache/trunk: (552 commits)
  MINOR: JavaDoc markup cleanup
  KAFKA-4679: Remove unstable markers from Connect APIs
  KAFKA-4450; Add upgrade tests for 0.10.1 and rename TRUNK to DEV_BRANCH 
to reduce confusion
  KAFKA-4635; Client Compatibility follow-ups
  MINOR: update JavaDocs for Kafka Streams DSL helpers
  KAFKA-4557; Handle Producer.send correctly in expiry callbacks
  MINOR: Update copyright year in the NOTICE file.
  KAFKA-4664; Update docs/protocol.html with KIP-97 information
  KAFKA-4704; Coordinator cache loading fails if groupId is reused for 
offset storage after group is removed
  MINOR: Replace for within for each; replace if-elseif with match
  KAFKA-4644: Improve test coverage of StreamsPartitionAssignor
  MINOR: Update KTable JavaDoc
  MINOR: Include more detail in `ConfigDef.parseType` exception message
  KAFKA-4578; Upgrade notes for 0.10.2.0
  MINOR: Streams API JavaDoc improvements
  MINOR: Add Streams system test for broker backwards compatibility
  MINOR: Close create topics policy during shutdown and more tests
  MINOR: Update JavaDoc for DSL PAPI-API
  KAFKA-4636; Per listener security settings overrides (KIP-103)
  MINOR: Change logging level for ignored maybeAddMetric from debug to trace
  ...

commit 4d130ef7e990ca8215684adc36865f08caa79557
Author: Ismael Juma 
Date:   2017-01-31T21:07:37Z

Try a slightly different approach to solving the shutdown deadlock

* Open the latch in `ShutdownableThread` before calling `exit`
* Introduce `Exit` classes that perform exit/halt and can be changed in 
tests
* Invoke `exit` from the calling thread instead of spawning a new thread
* Updated the tests
* A few clean-ups

commit 1c8032b7419874ef2c56e7cf321d39a3f64e6833
Author: Ismael Juma 
Date:   2017-01-31T21:07:51Z

A few style clean-ups

commit 50b6c571ae51d2c9f5b7288faace349acc156bc8
Author: Ismael Juma 
Date:   2017-01-31T21:08:18Z

Merge remote-tracking branch 'apache/trunk' into 
kafka-4039-deadlock-during-shutdown

* apache/trunk:
  MINOR: Logging improvements in consumer internals
  KAFKA-2955; Add a simple ">" prompt to console producer
  KAFKA-4613: Follow-up to fix JavaDocs
  KAFKA-4613: Treat null-key records the same way for joins and aggreations




> Exit Strategy: using exceptions instead of inline invocation of exit/halt
> -
>
> Key: KAFKA-4039
> URL: https://issues.apache.org/jira/browse/KAFKA-4039
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.10.0.0
>Reporter: Maysam Yabandeh
>Priority: Critical
>  Labels: reliability
> Attachments: deadlock-stack2
>
>
> The current practice is to directly invoke halt/exit right after the line 
> that intends to terminate the execution. In the case of System.exit this 
> could cause deadlocks if the thread invoking System.exit is holding  a lock 
> 

[jira] (KAFKA-4717) connect jars are missing LICENSE/NOTICE files

[Ewen Cheslack-Postava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava updated KAFKA-4717:
-
Status: Patch Available  (was: Open)

> connect jars are missing LICENSE/NOTICE files
> -
>
> Key: KAFKA-4717
> URL: https://issues.apache.org/jira/browse/KAFKA-4717
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Reporter: dan norwood
>Assignee: Ewen Cheslack-Postava
>Priority: Blocker
> Fix For: 0.10.2.0
>
>
> other artifacts in kafka contain LICENSE/NOTICE files in the deployed jars, 
> but connect seems to be missing them.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] (KAFKA-4717) connect jars are missing LICENSE/NOTICE files

[Ewen Cheslack-Postava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava reassigned KAFKA-4717:


Assignee: Ewen Cheslack-Postava

> connect jars are missing LICENSE/NOTICE files
> -
>
> Key: KAFKA-4717
> URL: https://issues.apache.org/jira/browse/KAFKA-4717
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Reporter: dan norwood
>Assignee: Ewen Cheslack-Postava
>Priority: Blocker
> Fix For: 0.10.2.0
>
>
> other artifacts in kafka contain LICENSE/NOTICE files in the deployed jars, 
> but connect seems to be missing them.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] (KAFKA-3959) __consumer_offsets wrong number of replicas at startup

[Ewen Cheslack-Postava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava updated KAFKA-3959:
-
Fix Version/s: (was: 0.10.2.0)
   0.10.3.0

> __consumer_offsets wrong number of replicas at startup
> --
>
> Key: KAFKA-3959
> URL: https://issues.apache.org/jira/browse/KAFKA-3959
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer, offset manager, replication
>Affects Versions: 0.9.0.1, 0.10.0.0, 0.10.0.1, 0.10.0.2, 0.10.1.0, 
> 0.10.1.1, 0.10.1.2
> Environment: Brokers of 3 kafka nodes running Red Hat Enterprise 
> Linux Server release 7.2 (Maipo)
>Reporter: Alban Hurtaud
>Assignee: Onur Karaman
>Priority: Blocker
>  Labels: needs-kip, reliability
> Fix For: 0.10.3.0
>
>
> When creating a stack of 3 kafka brokers, the consumer is starting faster 
> than kafka nodes and when trying to read a topic, only one kafka node is 
> available.
> So the __consumer_offsets is created with a replication factor set to 1 
> (instead of configured 3) :
> offsets.topic.replication.factor=3
> default.replication.factor=3
> min.insync.replicas=2
> Then, other kafka nodes go up and we have exceptions because the replicas # 
> for __consumer_offsets is 1 and min insync is 2. So exceptions are thrown.
> What I missed is : Why the __consumer_offsets is created with replication to 
> 1 (when 1 broker is running) whereas in server.properties it is set to 3 ?
> To reproduce : 
> - Prepare 3 kafka nodes with the 3 lines above added to servers.properties.
> - Run one kafka,
> - Run one consumer (the __consumer_offsets is created with replicas =1)
> - Run 2 more kafka nodes



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: kafka-trunk-jdk8 #1237

[Apache Jenkins Server]

See 

Changes:

[jason] MINOR: Logging improvements in consumer internals

--
[...truncated 5326 lines...]

kafka.api.SslEndToEndAuthorizationTest > testNoProduceWithDescribeAcl PASSED

kafka.api.SslEndToEndAuthorizationTest > testProduceConsumeViaSubscribe STARTED

kafka.api.SslEndToEndAuthorizationTest > testProduceConsumeViaSubscribe PASSED

kafka.api.SslEndToEndAuthorizationTest > testNoProduceWithoutDescribeAcl STARTED

kafka.api.SslEndToEndAuthorizationTest > testNoProduceWithoutDescribeAcl PASSED

kafka.api.RackAwareAutoTopicCreationTest > testAutoCreateTopic STARTED

kafka.api.RackAwareAutoTopicCreationTest > testAutoCreateTopic PASSED

kafka.api.ClientIdQuotaTest > testProducerConsumerOverrideUnthrottled STARTED

kafka.api.ClientIdQuotaTest > testProducerConsumerOverrideUnthrottled PASSED

kafka.api.ClientIdQuotaTest > testThrottledProducerConsumer STARTED

kafka.api.ClientIdQuotaTest > testThrottledProducerConsumer PASSED

kafka.api.ClientIdQuotaTest > testQuotaOverrideDelete STARTED

kafka.api.ClientIdQuotaTest > testQuotaOverrideDelete PASSED

kafka.api.SslProducerSendTest > testSendNonCompressedMessageWithCreateTime 
STARTED

kafka.api.SslProducerSendTest > testSendNonCompressedMessageWithCreateTime 
PASSED

kafka.api.SslProducerSendTest > testClose STARTED

kafka.api.SslProducerSendTest > testClose PASSED

kafka.api.SslProducerSendTest > testFlush STARTED

kafka.api.SslProducerSendTest > testFlush PASSED

kafka.api.SslProducerSendTest > testSendToPartition STARTED

kafka.api.SslProducerSendTest > testSendToPartition PASSED

kafka.api.SslProducerSendTest > testSendOffset STARTED

kafka.api.SslProducerSendTest > testSendOffset PASSED

kafka.api.SslProducerSendTest > testSendCompressedMessageWithCreateTime STARTED

kafka.api.SslProducerSendTest > testSendCompressedMessageWithCreateTime PASSED

kafka.api.SslProducerSendTest > testCloseWithZeroTimeoutFromCallerThread STARTED

kafka.api.SslProducerSendTest > testCloseWithZeroTimeoutFromCallerThread PASSED

kafka.api.SslProducerSendTest > testCloseWithZeroTimeoutFromSenderThread STARTED

kafka.api.SslProducerSendTest > testCloseWithZeroTimeoutFromSenderThread PASSED

kafka.api.SslProducerSendTest > testSendBeforeAndAfterPartitionExpansion STARTED

kafka.api.SslProducerSendTest > testSendBeforeAndAfterPartitionExpansion PASSED

kafka.api.ApiUtilsTest > testShortStringNonASCII STARTED

kafka.api.ApiUtilsTest > testShortStringNonASCII PASSED

kafka.api.ApiUtilsTest > testShortStringASCII STARTED

kafka.api.ApiUtilsTest > testShortStringASCII PASSED

kafka.api.SaslSslConsumerTest > testCoordinatorFailover STARTED

kafka.api.SaslSslConsumerTest > testCoordinatorFailover PASSED

kafka.api.SaslSslConsumerTest > testSimpleConsumption STARTED

kafka.api.SaslSslConsumerTest > testSimpleConsumption PASSED

kafka.api.SaslPlainPlaintextConsumerTest > testCoordinatorFailover STARTED

kafka.api.SaslPlainPlaintextConsumerTest > testCoordinatorFailover PASSED

kafka.api.SaslPlainPlaintextConsumerTest > testSimpleConsumption STARTED

kafka.api.SaslPlainPlaintextConsumerTest > testSimpleConsumption PASSED

kafka.api.FetchRequestTest > testShuffleWithSingleTopic STARTED

kafka.api.FetchRequestTest > testShuffleWithSingleTopic PASSED

kafka.api.FetchRequestTest > testShuffle STARTED

kafka.api.FetchRequestTest > testShuffle PASSED

kafka.api.SaslScramSslEndToEndAuthorizationTest > 
testTwoConsumersWithDifferentSaslCredentials STARTED

kafka.api.SaslScramSslEndToEndAuthorizationTest > 
testTwoConsumersWithDifferentSaslCredentials PASSED

kafka.api.SaslScramSslEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaSubscribe STARTED

kafka.api.SaslScramSslEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaSubscribe PASSED

kafka.api.SaslScramSslEndToEndAuthorizationTest > testProduceConsumeViaAssign 
STARTED

kafka.api.SaslScramSslEndToEndAuthorizationTest > testProduceConsumeViaAssign 
PASSED

kafka.api.SaslScramSslEndToEndAuthorizationTest > 
testNoConsumeWithDescribeAclViaAssign STARTED

kafka.api.SaslScramSslEndToEndAuthorizationTest > 
testNoConsumeWithDescribeAclViaAssign PASSED

kafka.api.SaslScramSslEndToEndAuthorizationTest > 
testNoConsumeWithDescribeAclViaSubscribe STARTED

kafka.api.SaslScramSslEndToEndAuthorizationTest > 
testNoConsumeWithDescribeAclViaSubscribe PASSED

kafka.api.SaslScramSslEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaAssign STARTED

kafka.api.SaslScramSslEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaAssign PASSED

kafka.api.SaslScramSslEndToEndAuthorizationTest > testNoGroupAcl STARTED

kafka.api.SaslScramSslEndToEndAuthorizationTest > testNoGroupAcl PASSED

kafka.api.SaslScramSslEndToEndAuthorizationTest > testNoProduceWithDescribeAcl 
STARTED

kafka.api.SaslScramSslEndToEndAuthorizationTest > testNoProduceWithDescribeAcl 
PASSED

kafka.api.SaslScramSslEndToEndAuthorizationTest > 
testPr

[GitHub] kafka pull request #2475: MINOR: Use an explicit `Errors` object when possib...

[vahidhashemian]

GitHub user vahidhashemian opened a pull request:

https://github.com/apache/kafka/pull/2475

MINOR: Use an explicit `Errors` object when possible instead of a numeric 
error code



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vahidhashemian/kafka 
minor/use_explicit_Errors_type_when_possible

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2475.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2475


commit dc8a1bb070d0ff68e604cc7fee1e21cba705a26f
Author: Vahid Hashemian 
Date:   2017-01-31T20:20:00Z

MINOR: Use an explicit `Errors` object when possible instead of a numeric 
error code




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUS] consuming messages is polling - is there a reason? new KIP for poll?

[radai]

minimizing the cost of clients is part of what makes kafka scale.
a push model would shift a lot of tracking logic onto the broker.

On Tue, Jan 31, 2017 at 2:47 AM, Alexander Binzberger <
alexander.binzber...@wingcon.com> wrote:

> way it seams like the protocol and the high-level consumer would be
> simplified.
> Clients have a more natural control over the offset and could ack per
> message or per bulk as needed or performance allows.
> Additionally the stream processing path over
>

Re: [DISCUS] consuming messages is polling - is there a reason? new KIP for poll?

[Jason Gustafson]

Also, have you looked at the use of the max wait time in fetch requests (
fetch.max.wait.ms for the new consumer)? The broker will hold the fetch in
purgatory until data is available. Sort of lets you fake a push model.

-Jason

On Tue, Jan 31, 2017 at 2:29 PM, radai  wrote:

> minimizing the cost of clients is part of what makes kafka scale.
> a push model would shift a lot of tracking logic onto the broker.
>
> On Tue, Jan 31, 2017 at 2:47 AM, Alexander Binzberger <
> alexander.binzber...@wingcon.com> wrote:
>
> > way it seams like the protocol and the high-level consumer would be
> > simplified.
> > Clients have a more natural control over the offset and could ack per
> > message or per bulk as needed or performance allows.
> > Additionally the stream processing path over
> >
>

[jira] [Commented] (KAFKA-3455) Connect custom processors with the streams DSL

[Matthias J. Sax (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847718#comment-15847718
 ] 

Matthias J. Sax commented on KAFKA-3455:


Just discovering this. I would close as "not a problem". \cc [~jonathan.bender] 
[~guozhang]

> Connect custom processors with the streams DSL
> --
>
> Key: KAFKA-3455
> URL: https://issues.apache.org/jira/browse/KAFKA-3455
> Project: Kafka
>  Issue Type: Sub-task
>  Components: streams
>Affects Versions: 0.10.0.1
>Reporter: Jonathan Bender
>  Labels: user-experience
> Fix For: 0.10.3.0
>
>
> From the kafka users email thread, we discussed the idea of connecting custom 
> processors with topologies defined from the Streams DSL (and being able to 
> sink data from the processor).  Possibly this could involve exposing the 
> underlying processor's name in the streams DSL so it can be connected with 
> the standard processor API.
> {quote}
> Thanks for the feedback. This is definitely something we wanted to support
> in the Streams DSL.
> One tricky thing, though, is that some operations do not translate to a
> single processor, but a sub-graph of processors (think of a stream-stream
> join, which is translated to actually 5 processors for windowing / state
> queries / merging, each with a different internal name). So how to define
> the API to return the processor name needs some more thinking.
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (KAFKA-4612) Simple Kafka Streams app causes "ClassCastException: java.lang.String cannot be cast to [B"

[Matthias J. Sax (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax resolved KAFKA-4612.

Resolution: Not A Problem

> Simple Kafka Streams app causes "ClassCastException: java.lang.String cannot 
> be cast to [B"
> ---
>
> Key: KAFKA-4612
> URL: https://issues.apache.org/jira/browse/KAFKA-4612
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.1.1
> Environment: Virtual Machine using Debian 8 + Confluent Platform 
> 3.1.1.
>Reporter: Kurt Ostfeld
> Attachments: KafkaIsolatedBug.tar.gz
>
>
> I've attached a minimal single source file project that reliably reproduces 
> this issue.
> This project does the following:
> 1) Create test input data. Produces a single random (String,String) record 
> into two diferent topics "topicInput" and "topicTable"
> 2) Creates and runs a Kafka Streams application:
> val kafkaTable: KTable[String, String] = builder.table(Serdes.String, 
> Serdes.String, "topicTable", "topicTable")
> val incomingRecords: KStream[String, String] = 
> builder.stream(Serdes.String, Serdes.String, "topicInput")
> val reKeyedRecords: KStream[String, String] = 
> incomingRecords.selectKey((k, _) => k)
> val joinedRecords: KStream[String, String] = 
> reKeyedRecords.leftJoin(kafkaTable, (s1: String, _: String) => s1)
> joinedRecords.to(Serdes.String, Serdes.String, "topicOutput")
> This reliably generates the following error:
> [error] (StreamThread-1) java.lang.ClassCastException: java.lang.String 
> cannot be cast to [B
> java.lang.ClassCastException: java.lang.String cannot be cast to [B
>   at 
> org.apache.kafka.common.serialization.ByteArraySerializer.serialize(ByteArraySerializer.java:18)
>   at 
> org.apache.kafka.streams.processor.internals.RecordCollectorImpl.send(RecordCollectorImpl.java:63)
>   at 
> org.apache.kafka.streams.processor.internals.SinkNode.process(SinkNode.java:82)
>   at 
> org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:202)
>   at 
> org.apache.kafka.streams.kstream.internals.KStreamFilter$KStreamFilterProcessor.process(KStreamFilter.java:44)
>   at 
> org.apache.kafka.streams.processor.internals.ProcessorNode.process(ProcessorNode.java:82)
>   at 
> org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:202)
>   at 
> org.apache.kafka.streams.kstream.internals.KStreamMap$KStreamMapProcessor.process(KStreamMap.java:43)
>   at 
> org.apache.kafka.streams.processor.internals.ProcessorNode.process(ProcessorNode.java:82)
>   at 
> org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:202)
>   at 
> org.apache.kafka.streams.processor.internals.SourceNode.process(SourceNode.java:66)
>   at 
> org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:180)
>   at 
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:436)
>   at 
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:242)
> One caveat: I'm running this on a Confluent Platform 3.1.1 instance which 
> uses Kafka 0.10.1.0 since there is no newer Confluent Platform available. The 
> Kafka Streams project is built using "kafka-clients" and "kafka-streams" 
> version 0.10.1.1. If I use 0.10.1.0, I reliably hit bug 
> https://issues.apache.org/jira/browse/KAFKA-4355. I am not sure if there is 
> any issue using 0.10.1.1 libraries with a Confluent Platform running Kafka 
> 0.10.1.0. I will obviously try the next Confluent Platform binary when it is 
> available.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-4686) Null Message payload is shutting down broker

[Jason Gustafson (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847751#comment-15847751
 ] 

Jason Gustafson commented on KAFKA-4686:


[~rodrigo.saramago] Thanks for the updates. I spotted this error which actually 
tells us which partition has the corrupt message:
{code}
[2017-01-30 07:03:34,149] ERROR [ReplicaFetcherThread-0-1003], Error due to  
(kafka.server.ReplicaFetcherThread)
kafka.common.KafkaException: error processing data for partition 
[zupme-gateway,12] offset 106
at 
kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2$$anonfun$apply$mcV$sp$1$$anonfun$apply$2.apply(AbstractFetcherThread.scala:170)
at 
kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2$$anonfun$apply$mcV$sp$1$$anonfun$apply$2.apply(AbstractFetcherThread.scala:141)
at scala.Option.foreach(Option.scala:257)
at 
kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2$$anonfun$apply$mcV$sp$1.apply(AbstractFetcherThread.scala:141)
at 
kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2$$anonfun$apply$mcV$sp$1.apply(AbstractFetcherThread.scala:138)
at 
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at 
kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2.apply$mcV$sp(AbstractFetcherThread.scala:138)
at 
kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2.apply(AbstractFetcherThread.scala:138)
at 
kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2.apply(AbstractFetcherThread.scala:138)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:234)
at 
kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:136)
at 
kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:103)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
Caused by: kafka.common.KafkaException: Message payload is null: Message(magic 
= 0, attributes = 1, crc = 3187236729, key = null, payload = null)
at 
kafka.message.ByteBufferMessageSet$$anon$1.(ByteBufferMessageSet.scala:90)
at 
kafka.message.ByteBufferMessageSet$.deepIterator(ByteBufferMessageSet.scala:85)
at 
kafka.message.ByteBufferMessageSet$$anon$2.makeNextOuter(ByteBufferMessageSet.scala:370)
at 
kafka.message.ByteBufferMessageSet$$anon$2.makeNext(ByteBufferMessageSet.scala:383)
at 
kafka.message.ByteBufferMessageSet$$anon$2.makeNext(ByteBufferMessageSet.scala:338)
at 
kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:64)
at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:56)
at kafka.message.MessageSet.toString(MessageSet.scala:105)
at java.lang.String.valueOf(String.java:2994)
at 
scala.collection.mutable.StringBuilder.append(StringBuilder.scala:200)
at kafka.log.Log.append(Log.scala:395)
at 
kafka.server.ReplicaFetcherThread.processPartitionData(ReplicaFetcherThread.scala:130)
at 
kafka.server.ReplicaFetcherThread.processPartitionData(ReplicaFetcherThread.scala:42)
at 
kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2$$anonfun$apply$mcV$sp$1$$anonfun$apply$2.apply(AbstractFetcherThread.scala:153)
... 13 more
{code}
This may not be the only partition which is corrupt, but at least we can 
analyze it a bit more. A couple questions:

1. Do you know if compaction has been enabled on the topic "zupme-gateway"? 
2. What version of Kafka were you on before upgrading to 0.10.1.1? Were you 
ever able to successfully start the brokers running 0.10.1.1?

Also, could use the DumpLogSegments utility (bin/kafka-run-class 
kafka.tools.DumpLogSegments) on all of the segments for partition 12 of 
"zupme-gateway"? Judging by the offset from that message, it should be the 
first segment which is corrupt. If you run with the "--deep-iteration" option, 
you will probably see the same "Message payload is null" error, but we should 
be able to still do the shallow iteration if you leave that option out. Please 
upload the results if possible.

> Null Message payload is shutting down broker
> 
>
> Key: KAFKA-4686
> URL: https://issues.apache.org/jira/browse/KAFKA-4686
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.1.1
> Environment: Amazon Linux AMI release 2016.03 kernel 
> 4.4.19-29.55.amzn1.x86_64
>Reporter: Rodrigo Queiroz Saramago
> Fix For: 0.10.3.0
>
> Attachments: KAFKA-4686-NullMessagePayloadError.tar.xz, 
> kafkaServer.out
>
>
> Hello, I have a test environment with 3 brokers and 1 zookeeper nodes, in 
> which clients connect using 

[GitHub] kafka pull request #2473: KAFKA-4717: Use absolute paths to files in root di...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2473


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-4717) connect jars are missing LICENSE/NOTICE files

[Ewen Cheslack-Postava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava updated KAFKA-4717:
-
   Resolution: Fixed
Fix Version/s: 0.10.1.2
   0.10.3.0
   Status: Resolved  (was: Patch Available)

Issue resolved by pull request 2473
[https://github.com/apache/kafka/pull/2473]

> connect jars are missing LICENSE/NOTICE files
> -
>
> Key: KAFKA-4717
> URL: https://issues.apache.org/jira/browse/KAFKA-4717
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Reporter: dan norwood
>Assignee: Ewen Cheslack-Postava
>Priority: Blocker
> Fix For: 0.10.3.0, 0.10.1.2, 0.10.2.0
>
>
> other artifacts in kafka contain LICENSE/NOTICE files in the deployed jars, 
> but connect seems to be missing them.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-4717) connect jars are missing LICENSE/NOTICE files

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847761#comment-15847761
 ] 

ASF GitHub Bot commented on KAFKA-4717:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2473


> connect jars are missing LICENSE/NOTICE files
> -
>
> Key: KAFKA-4717
> URL: https://issues.apache.org/jira/browse/KAFKA-4717
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Reporter: dan norwood
>Assignee: Ewen Cheslack-Postava
>Priority: Blocker
> Fix For: 0.10.1.2, 0.10.2.0, 0.10.3.0
>
>
> other artifacts in kafka contain LICENSE/NOTICE files in the deployed jars, 
> but connect seems to be missing them.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (KAFKA-4320) Log compaction docs update

[Ishita Mandhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ishita Mandhan reassigned KAFKA-4320:
-

Assignee: (was: Ishita Mandhan)

> Log compaction docs update
> --
>
> Key: KAFKA-4320
> URL: https://issues.apache.org/jira/browse/KAFKA-4320
> Project: Kafka
>  Issue Type: Bug
>  Components: documentation
>Reporter: Dustin Cote
>Priority: Minor
>  Labels: newbie
>
> The log compaction docs are out of date.  At least the default is said to be 
> that log compaction is disabled which is not true as of 0.9.0.1.  Probably 
> the whole section needs a once over to make sure it's in line with what is 
> currently there.  This is the section:
> [http://kafka.apache.org/documentation#design_compactionconfig]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (KAFKA-2544) Replication tools wiki page needs to be updated

[Ishita Mandhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ishita Mandhan reassigned KAFKA-2544:
-

Assignee: (was: Ishita Mandhan)

> Replication tools wiki page needs to be updated
> ---
>
> Key: KAFKA-2544
> URL: https://issues.apache.org/jira/browse/KAFKA-2544
> Project: Kafka
>  Issue Type: Improvement
>  Components: website
>Affects Versions: 0.8.2.1
>Reporter: Stevo Slavic
>Priority: Minor
>  Labels: documentation, newbie
>
> https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools is 
> outdated, mentions tools which have been heavily refactored or replaced by 
> other tools, e.g. add partition tool, list/create topics tools, etc.
> Please have the replication tools wiki page updated.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (KAFKA-3940) Log should check the return value of dir.mkdirs()

[Ishita Mandhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ishita Mandhan reassigned KAFKA-3940:
-

Assignee: (was: Ishita Mandhan)

> Log should check the return value of dir.mkdirs()
> -
>
> Key: KAFKA-3940
> URL: https://issues.apache.org/jira/browse/KAFKA-3940
> Project: Kafka
>  Issue Type: Bug
>  Components: log
>Affects Versions: 0.10.0.0
>Reporter: Jun Rao
>  Labels: newbie, reliability
>
> In Log.loadSegments(), we call dir.mkdirs() w/o checking the return value and 
> just assume the directory will exist after the call. However, if the 
> directory can't be created (e.g. due to no space), we will hit 
> NullPointerException in the next statement, which will be confusing.
>for(file <- dir.listFiles if file.isFile) {



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (KAFKA-3522) Consider adding version information into rocksDB storage format

[Ishita Mandhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ishita Mandhan reassigned KAFKA-3522:
-

Assignee: (was: Ishita Mandhan)

> Consider adding version information into rocksDB storage format
> ---
>
> Key: KAFKA-3522
> URL: https://issues.apache.org/jira/browse/KAFKA-3522
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Guozhang Wang
>  Labels: architecture
>
> Kafka Streams does not introduce any modifications to the data format in the 
> underlying Kafka protocol, but it does use RocksDB for persistent state 
> storage, and currently its data format is fixed and hard-coded. We want to 
> consider the evolution path in the future we we change the data format, and 
> hence having some version info stored along with the storage file / directory 
> would be useful.
> And this information could be even out of the storage file; for example, we 
> can just use a small "version indicator" file in the rocksdb directory for 
> this purposes. Thoughts? [~enothereska] [~jkreps]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-3522) Consider adding version information into rocksDB storage format

[Ishita Mandhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847779#comment-15847779
 ] 

Ishita Mandhan commented on KAFKA-3522:
---

[~ewencp]I don't think I'll be able to work on this anymore, hope it gets 
picked up soon by someone else!

> Consider adding version information into rocksDB storage format
> ---
>
> Key: KAFKA-3522
> URL: https://issues.apache.org/jira/browse/KAFKA-3522
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Guozhang Wang
>Assignee: Ishita Mandhan
>  Labels: architecture
>
> Kafka Streams does not introduce any modifications to the data format in the 
> underlying Kafka protocol, but it does use RocksDB for persistent state 
> storage, and currently its data format is fixed and hard-coded. We want to 
> consider the evolution path in the future we we change the data format, and 
> hence having some version info stored along with the storage file / directory 
> would be useful.
> And this information could be even out of the storage file; for example, we 
> can just use a small "version indicator" file in the rocksdb directory for 
> this purposes. Thoughts? [~enothereska] [~jkreps]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (KAFKA-3543) Allow a variant of transform() which can emit multiple values

[Matthias J. Sax (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax resolved KAFKA-3543.

Resolution: Duplicate

> Allow a variant of transform() which can emit multiple values
> -
>
> Key: KAFKA-3543
> URL: https://issues.apache.org/jira/browse/KAFKA-3543
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Affects Versions: 0.10.0.0
>Reporter: Greg Fodor
>  Labels: api
> Fix For: 0.10.3.0
>
>
> Right now it seems that if you want to apply an arbitrary stateful 
> transformation to a stream, you either have to use a TransformerSupplier or 
> ProcessorSupplier sent to transform() or process(). The custom processor will 
> allow you to emit multiple new values, but the process() method currently 
> terminates that branch of the topology so you can't apply additional data 
> flow. transform() lets you continue the data flow, but forces you to emit a 
> single value for every input value.
> (It actually doesn't quite force you to do this, since you can hold onto the 
> ProcessorContext and emit multiple, but that's probably not the ideal way to 
> do it :))
> It seems desirable to somehow allow a transformation that emits multiple 
> values per input value. I'm not sure of the best way to factor this inside of 
> the current TransformerSupplier/Transformer architecture in a way that is 
> clean and efficient -- currently I'm doing the workaround above of just 
> calling forward() myself on the context and actually emitting dummy values 
> which are filtered out downstream.
> -
> It is worth considering adding a new flatTransofrm function as 
> {code}
>  KStream transform(TransformerSupplier Iterable>> transformerSupplier, String... stateStoreNames)
> {code}
> which is essentially the same as
> {code} transform().flatMap() {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-3543) Allow a variant of transform() which can emit multiple values

[Matthias J. Sax (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847802#comment-15847802
 ] 

Matthias J. Sax commented on KAFKA-3543:


[~gfodor] I close this as duplicate. Nevertheless one question: I don't 
understand your comment in the JIRA description about "just calling forward() 
myself on the context and actually emitting dummy values which are filtered out 
downstream" ? Actually, using {{context.forward()}} is absolutely fine and 
efficient.

> Allow a variant of transform() which can emit multiple values
> -
>
> Key: KAFKA-3543
> URL: https://issues.apache.org/jira/browse/KAFKA-3543
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Affects Versions: 0.10.0.0
>Reporter: Greg Fodor
>  Labels: api
> Fix For: 0.10.3.0
>
>
> Right now it seems that if you want to apply an arbitrary stateful 
> transformation to a stream, you either have to use a TransformerSupplier or 
> ProcessorSupplier sent to transform() or process(). The custom processor will 
> allow you to emit multiple new values, but the process() method currently 
> terminates that branch of the topology so you can't apply additional data 
> flow. transform() lets you continue the data flow, but forces you to emit a 
> single value for every input value.
> (It actually doesn't quite force you to do this, since you can hold onto the 
> ProcessorContext and emit multiple, but that's probably not the ideal way to 
> do it :))
> It seems desirable to somehow allow a transformation that emits multiple 
> values per input value. I'm not sure of the best way to factor this inside of 
> the current TransformerSupplier/Transformer architecture in a way that is 
> clean and efficient -- currently I'm doing the workaround above of just 
> calling forward() myself on the context and actually emitting dummy values 
> which are filtered out downstream.
> -
> It is worth considering adding a new flatTransofrm function as 
> {code}
>  KStream transform(TransformerSupplier Iterable>> transformerSupplier, String... stateStoreNames)
> {code}
> which is essentially the same as
> {code} transform().flatMap() {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: kafka-trunk-jdk8 #1238

[Apache Jenkins Server]

See 

Changes:

[me] KAFKA-4717: Use absolute paths to files in root directory so all jars

--
[...truncated 3567 lines...]
kafka.admin.ConfigCommandTest > shouldParseArgumentsForClientsEntityType STARTED

kafka.admin.ConfigCommandTest > shouldParseArgumentsForClientsEntityType PASSED

kafka.admin.DeleteTopicTest > testDeleteTopicWithCleaner STARTED

kafka.admin.DeleteTopicTest > testDeleteTopicWithCleaner PASSED

kafka.admin.DeleteTopicTest > testResumeDeleteTopicOnControllerFailover STARTED

kafka.admin.DeleteTopicTest > testResumeDeleteTopicOnControllerFailover PASSED

kafka.admin.DeleteTopicTest > testResumeDeleteTopicWithRecoveredFollower STARTED

kafka.admin.DeleteTopicTest > testResumeDeleteTopicWithRecoveredFollower PASSED

kafka.admin.DeleteTopicTest > testDeleteTopicAlreadyMarkedAsDeleted STARTED

kafka.admin.DeleteTopicTest > testDeleteTopicAlreadyMarkedAsDeleted PASSED

kafka.admin.DeleteTopicTest > testPartitionReassignmentDuringDeleteTopic STARTED

kafka.admin.DeleteTopicTest > testPartitionReassignmentDuringDeleteTopic PASSED

kafka.admin.DeleteTopicTest > testDeleteNonExistingTopic STARTED

kafka.admin.DeleteTopicTest > testDeleteNonExistingTopic PASSED

kafka.admin.DeleteTopicTest > testRecreateTopicAfterDeletion STARTED

kafka.admin.DeleteTopicTest > testRecreateTopicAfterDeletion PASSED

kafka.admin.DeleteTopicTest > testDisableDeleteTopic STARTED

kafka.admin.DeleteTopicTest > testDisableDeleteTopic PASSED

kafka.admin.DeleteTopicTest > testAddPartitionDuringDeleteTopic STARTED

kafka.admin.DeleteTopicTest > testAddPartitionDuringDeleteTopic PASSED

kafka.admin.DeleteTopicTest > testDeleteTopicWithAllAliveReplicas STARTED

kafka.admin.DeleteTopicTest > testDeleteTopicWithAllAliveReplicas PASSED

kafka.admin.DeleteTopicTest > testDeleteTopicDuringAddPartition STARTED

kafka.admin.DeleteTopicTest > testDeleteTopicDuringAddPartition PASSED

kafka.admin.DescribeConsumerGroupTest > 
testDescribeConsumersWithNoAssignedPartitionsWithNewConsumer STARTED

kafka.admin.DescribeConsumerGroupTest > 
testDescribeConsumersWithNoAssignedPartitionsWithNewConsumer PASSED

kafka.admin.DescribeConsumerGroupTest > 
testDescribeWithMultiPartitionTopicAndMultipleConsumersWithNewConsumer STARTED

kafka.admin.DescribeConsumerGroupTest > 
testDescribeWithMultiPartitionTopicAndMultipleConsumersWithNewConsumer PASSED

kafka.admin.DescribeConsumerGroupTest > testDescribeExistingGroup STARTED

kafka.admin.DescribeConsumerGroupTest > testDescribeExistingGroup PASSED

kafka.admin.DescribeConsumerGroupTest > 
testDescribeConsumersWithNoAssignedPartitions STARTED

kafka.admin.DescribeConsumerGroupTest > 
testDescribeConsumersWithNoAssignedPartitions PASSED

kafka.admin.DescribeConsumerGroupTest > 
testDescribeExistingGroupWithNewConsumer STARTED

kafka.admin.DescribeConsumerGroupTest > 
testDescribeExistingGroupWithNewConsumer PASSED

kafka.admin.DescribeConsumerGroupTest > 
testDescribeNonExistingGroupWithNewConsumer STARTED

kafka.admin.DescribeConsumerGroupTest > 
testDescribeNonExistingGroupWithNewConsumer PASSED

kafka.admin.DescribeConsumerGroupTest > 
testDescribeExistingGroupWithNoMembersWithNewConsumer STARTED

kafka.admin.DescribeConsumerGroupTest > 
testDescribeExistingGroupWithNoMembersWithNewConsumer PASSED

kafka.admin.DescribeConsumerGroupTest > testDescribeNonExistingGroup STARTED

kafka.admin.DescribeConsumerGroupTest > testDescribeNonExistingGroup PASSED

kafka.admin.DescribeConsumerGroupTest > testDescribeExistingGroupWithNoMembers 
STARTED

kafka.admin.DescribeConsumerGroupTest > testDescribeExistingGroupWithNoMembers 
PASSED

kafka.admin.BrokerApiVersionsCommandTest > checkBrokerApiVersionCommandOutput 
STARTED

kafka.admin.BrokerApiVersionsCommandTest > checkBrokerApiVersionCommandOutput 
PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > shouldFailIfBlankArg STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > shouldFailIfBlankArg PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowVerifyWithoutReassignmentOption STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowVerifyWithoutReassignmentOption PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowGenerateWithoutBrokersOption STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowGenerateWithoutBrokersOption PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowTopicsOptionWithVerify STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowTopicsOptionWithVerify PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowGenerateWithThrottleOption STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowGenerateWithThrottleOption PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > shouldFailIfNoArgs STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > shouldFailIfNoArgs PASSED

kafka.admin.ReassignPart

[jira] [Created] (KAFKA-4718) Revisit DSL partitioning assumption for KStream source topics

[Matthias J. Sax (JIRA)]

Matthias J. Sax created KAFKA-4718:
--

 Summary: Revisit DSL partitioning assumption for KStream source 
topics
 Key: KAFKA-4718
 URL: https://issues.apache.org/jira/browse/KAFKA-4718
 Project: Kafka
  Issue Type: Improvement
  Components: streams
Reporter: Matthias J. Sax
Priority: Minor


Currently, when reading one or multiple topics via a single call to 
{{KStreamBuilder#stream()}}, it is assumed that the data is correctly 
partitions by key.

For "single topic" {{KStream}}, this is a fair assumption, however, for 
multi-topic {{KStream}}, the assumption is most likely not true if input topics 
have a different number of partitions, because producer use hash partitioning 
per default. Thus, to get correct partitions, all producer for those input 
topics need to use (the same or at least a compatible) custom partitioner.

Making this the default assumption seem rather risky, and we might want to 
revisit this. Or at least update some docs with corresponding hints.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [VOTE] KIP-107 Add purgeDataBefore() API in AdminClient

[Apurva Mehta]

Hi Dong,

It looks like this vote passed. Can you close this thread and update the
KIP table?

Thanks,
Apurva

On Tue, Jan 24, 2017 at 1:30 PM, Jun Rao  wrote:

> Hi, Dong,
>
> The changes sound good to me. Also, thanks for the explanation of returning
> a future from purgeDataFrom(). We can keep it that way.
>
> Thanks,
>
> Jun
>
> On Mon, Jan 23, 2017 at 4:24 PM, Dong Lin  wrote:
>
> > Hi all,
> >
> > When I am implementing the patch, I realized that the current usage of
> > "low_watermark" is a bit confusing. So I made the following interface
> > changes in the KIP:
> >
> > - The newly added checkpoint file will be named
> log-begin-offset-checkpoint
> > - Replace low_watermark with log_begin_offset in FetchRequestPartition
> and
> > FetchResponsePartitionHeader
> >
> > The problem with the previous naming conversion is that, low_watermark
> > implies minimum log begin offset of all replicas (similar to high
> > watermark) and we return this value in the PurgeResponse. In other words,
> > low_watermark can not be incremented if a follower is not live. Therefore
> > we can not use low_watermark in the checkpoint file or in the
> FetchResponse
> > from leader to followers if we want to persists the offset-to-purge
> > received from user across broker rebounce.
> >
> > You can find the changes in KIP here
> >  > pageId=67636826&selectedPageVersions=13&selectedPageVersions=14>.
> > Please let me know if you have any concern with this change.
> >
> > Thanks,
> > Dong
> >
> > On Mon, Jan 23, 2017 at 11:20 AM, Dong Lin  wrote:
> >
> > > Thanks for the comment Jun.
> > >
> > > Yeah, I think there is use-case where this can be useful. Allowing for
> > > asynchronous delete will be useful if an application doesn't need
> strong
> > > guarantee of purgeDataFrom(), e.g. if it is done to help reduce disk
> > usage
> > > of kafka. The application may want to purge data for every time it does
> > > auto-commit without wait for future object to complete. On the other
> > hand,
> > > synchronous delete will be useful if an application wants to make sure
> > that
> > > the sensitive or bad data is definitely deleted. I think returning a
> > future
> > > makes both choice available to user and it doesn't complicate
> > > implementation much.
> > >
> > >
> > > On Mon, Jan 23, 2017 at 10:45 AM, Jun Rao  wrote:
> > >
> > >> I feel that it's simpler to just keep the format of the checkpoint
> file
> > as
> > >> it is and just add a separate checkpoint for low watermark. Low
> > watermark
> > >> and high watermark are maintained independently. So, not sure if there
> > is
> > >> significant benefit of storing them together.
> > >>
> > >> Looking at the KIP again. I actually have another question on the api.
> > Is
> > >> there any benefit of returning a Future in the purgeDataBefore() api?
> > >> Since
> > >> admin apis are used infrequently, it seems that it's simpler to just
> > have
> > >> a
> > >> blocking api and returns Map?
> > >>
> > >> Thanks,
> > >>
> > >> Jun
> > >>
> > >> On Sun, Jan 22, 2017 at 3:56 PM, Dong Lin 
> wrote:
> > >>
> > >> > Thanks for the comment Guozhang. Please don't worry about being
> late.
> > I
> > >> > would like to update the KIP if there is clear benefit of the new
> > >> approach.
> > >> > I am wondering if there is any use-case or operation aspects that
> > would
> > >> > benefit from the new approach.
> > >> >
> > >> > I am not saying that these checkpoint files have the same priority.
> I
> > >> > mentioned other checkpoint files to suggest that it is OK to add one
> > >> more
> > >> > checkpoint file. To me three checkpoint files is not much different
> > from
> > >> > four checkpoint files. I am just inclined to not update the KIP if
> the
> > >> only
> > >> > benefit is to avoid addition of a new checkpoint file.
> > >> >
> > >> >
> > >> >
> > >> > On Sun, Jan 22, 2017 at 3:48 PM, Guozhang Wang 
> > >> wrote:
> > >> >
> > >> > > To me the distinction between recovery-checkpoint and
> > >> > > replication-checkpoint are different from the distinction between
> > >> these
> > >> > two
> > >> > > hw checkpoint values: when broker starts up and act as the leader
> > for
> > >> a
> > >> > > partition, it can live without seeing the recovery checkpoint, but
> > >> just
> > >> > > cannot rely on the existing last log segment and need to fetch
> from
> > >> other
> > >> > > replicas; but if the replication-checkpoint file is missing, it
> is a
> > >> > > correctness issue, as it does not know from where to truncate its
> > >> data,
> > >> > and
> > >> > > also how to respond to a fetch request. That is why I think we can
> > >> > separate
> > >> > > these two types of files, since the latter one is more important
> > than
> > >> the
> > >> > > previous one.
> > >> > >
> > >> > > That being said, I do not want to recall another vote on this
> since
> > >> it is
> > >> > > my bad not responding before the vote is called. Jus

Re: [VOTE] KIP-107 Add purgeDataBefore() API in AdminClient

[Dong Lin]

Hey Apurva,

I think the KIP table in
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
has already been updated. Is there anything I missed?

Thanks,
Dong

On Tue, Jan 31, 2017 at 5:45 PM, Apurva Mehta  wrote:

> Hi Dong,
>
> It looks like this vote passed. Can you close this thread and update the
> KIP table?
>
> Thanks,
> Apurva
>
> On Tue, Jan 24, 2017 at 1:30 PM, Jun Rao  wrote:
>
> > Hi, Dong,
> >
> > The changes sound good to me. Also, thanks for the explanation of
> returning
> > a future from purgeDataFrom(). We can keep it that way.
> >
> > Thanks,
> >
> > Jun
> >
> > On Mon, Jan 23, 2017 at 4:24 PM, Dong Lin  wrote:
> >
> > > Hi all,
> > >
> > > When I am implementing the patch, I realized that the current usage of
> > > "low_watermark" is a bit confusing. So I made the following interface
> > > changes in the KIP:
> > >
> > > - The newly added checkpoint file will be named
> > log-begin-offset-checkpoint
> > > - Replace low_watermark with log_begin_offset in FetchRequestPartition
> > and
> > > FetchResponsePartitionHeader
> > >
> > > The problem with the previous naming conversion is that, low_watermark
> > > implies minimum log begin offset of all replicas (similar to high
> > > watermark) and we return this value in the PurgeResponse. In other
> words,
> > > low_watermark can not be incremented if a follower is not live.
> Therefore
> > > we can not use low_watermark in the checkpoint file or in the
> > FetchResponse
> > > from leader to followers if we want to persists the offset-to-purge
> > > received from user across broker rebounce.
> > >
> > > You can find the changes in KIP here
> > >  > > pageId=67636826&selectedPageVersions=13&selectedPageVersions=14>.
> > > Please let me know if you have any concern with this change.
> > >
> > > Thanks,
> > > Dong
> > >
> > > On Mon, Jan 23, 2017 at 11:20 AM, Dong Lin 
> wrote:
> > >
> > > > Thanks for the comment Jun.
> > > >
> > > > Yeah, I think there is use-case where this can be useful. Allowing
> for
> > > > asynchronous delete will be useful if an application doesn't need
> > strong
> > > > guarantee of purgeDataFrom(), e.g. if it is done to help reduce disk
> > > usage
> > > > of kafka. The application may want to purge data for every time it
> does
> > > > auto-commit without wait for future object to complete. On the other
> > > hand,
> > > > synchronous delete will be useful if an application wants to make
> sure
> > > that
> > > > the sensitive or bad data is definitely deleted. I think returning a
> > > future
> > > > makes both choice available to user and it doesn't complicate
> > > > implementation much.
> > > >
> > > >
> > > > On Mon, Jan 23, 2017 at 10:45 AM, Jun Rao  wrote:
> > > >
> > > >> I feel that it's simpler to just keep the format of the checkpoint
> > file
> > > as
> > > >> it is and just add a separate checkpoint for low watermark. Low
> > > watermark
> > > >> and high watermark are maintained independently. So, not sure if
> there
> > > is
> > > >> significant benefit of storing them together.
> > > >>
> > > >> Looking at the KIP again. I actually have another question on the
> api.
> > > Is
> > > >> there any benefit of returning a Future in the purgeDataBefore()
> api?
> > > >> Since
> > > >> admin apis are used infrequently, it seems that it's simpler to just
> > > have
> > > >> a
> > > >> blocking api and returns Map?
> > > >>
> > > >> Thanks,
> > > >>
> > > >> Jun
> > > >>
> > > >> On Sun, Jan 22, 2017 at 3:56 PM, Dong Lin 
> > wrote:
> > > >>
> > > >> > Thanks for the comment Guozhang. Please don't worry about being
> > late.
> > > I
> > > >> > would like to update the KIP if there is clear benefit of the new
> > > >> approach.
> > > >> > I am wondering if there is any use-case or operation aspects that
> > > would
> > > >> > benefit from the new approach.
> > > >> >
> > > >> > I am not saying that these checkpoint files have the same
> priority.
> > I
> > > >> > mentioned other checkpoint files to suggest that it is OK to add
> one
> > > >> more
> > > >> > checkpoint file. To me three checkpoint files is not much
> different
> > > from
> > > >> > four checkpoint files. I am just inclined to not update the KIP if
> > the
> > > >> only
> > > >> > benefit is to avoid addition of a new checkpoint file.
> > > >> >
> > > >> >
> > > >> >
> > > >> > On Sun, Jan 22, 2017 at 3:48 PM, Guozhang Wang <
> wangg...@gmail.com>
> > > >> wrote:
> > > >> >
> > > >> > > To me the distinction between recovery-checkpoint and
> > > >> > > replication-checkpoint are different from the distinction
> between
> > > >> these
> > > >> > two
> > > >> > > hw checkpoint values: when broker starts up and act as the
> leader
> > > for
> > > >> a
> > > >> > > partition, it can live without seeing the recovery checkpoint,
> but
> > > >> just
> > > >> > > cannot rely on the existing last log segment and need to fetch
> > from
> > > >> other
> > > >> > 

Re: [VOTE] KIP-107 Add purgeDataBefore() API in AdminClient

[Dong Lin]

This thread was been closed on Jan 18. We had more discussion after
Guozhang's feedback on Jan 21. But no major change was made to the KIP
after the discussion.


On Tue, Jan 31, 2017 at 5:47 PM, Dong Lin  wrote:

> Hey Apurva,
>
> I think the KIP table in https://cwiki.apache.org/co
> nfluence/display/KAFKA/Kafka+Improvement+Proposals has already been
> updated. Is there anything I missed?
>
> Thanks,
> Dong
>
> On Tue, Jan 31, 2017 at 5:45 PM, Apurva Mehta  wrote:
>
>> Hi Dong,
>>
>> It looks like this vote passed. Can you close this thread and update the
>> KIP table?
>>
>> Thanks,
>> Apurva
>>
>> On Tue, Jan 24, 2017 at 1:30 PM, Jun Rao  wrote:
>>
>> > Hi, Dong,
>> >
>> > The changes sound good to me. Also, thanks for the explanation of
>> returning
>> > a future from purgeDataFrom(). We can keep it that way.
>> >
>> > Thanks,
>> >
>> > Jun
>> >
>> > On Mon, Jan 23, 2017 at 4:24 PM, Dong Lin  wrote:
>> >
>> > > Hi all,
>> > >
>> > > When I am implementing the patch, I realized that the current usage of
>> > > "low_watermark" is a bit confusing. So I made the following interface
>> > > changes in the KIP:
>> > >
>> > > - The newly added checkpoint file will be named
>> > log-begin-offset-checkpoint
>> > > - Replace low_watermark with log_begin_offset in FetchRequestPartition
>> > and
>> > > FetchResponsePartitionHeader
>> > >
>> > > The problem with the previous naming conversion is that, low_watermark
>> > > implies minimum log begin offset of all replicas (similar to high
>> > > watermark) and we return this value in the PurgeResponse. In other
>> words,
>> > > low_watermark can not be incremented if a follower is not live.
>> Therefore
>> > > we can not use low_watermark in the checkpoint file or in the
>> > FetchResponse
>> > > from leader to followers if we want to persists the offset-to-purge
>> > > received from user across broker rebounce.
>> > >
>> > > You can find the changes in KIP here
>> > > > > > pageId=67636826&selectedPageVersions=13&selectedPageVersions=14>.
>> > > Please let me know if you have any concern with this change.
>> > >
>> > > Thanks,
>> > > Dong
>> > >
>> > > On Mon, Jan 23, 2017 at 11:20 AM, Dong Lin 
>> wrote:
>> > >
>> > > > Thanks for the comment Jun.
>> > > >
>> > > > Yeah, I think there is use-case where this can be useful. Allowing
>> for
>> > > > asynchronous delete will be useful if an application doesn't need
>> > strong
>> > > > guarantee of purgeDataFrom(), e.g. if it is done to help reduce disk
>> > > usage
>> > > > of kafka. The application may want to purge data for every time it
>> does
>> > > > auto-commit without wait for future object to complete. On the other
>> > > hand,
>> > > > synchronous delete will be useful if an application wants to make
>> sure
>> > > that
>> > > > the sensitive or bad data is definitely deleted. I think returning a
>> > > future
>> > > > makes both choice available to user and it doesn't complicate
>> > > > implementation much.
>> > > >
>> > > >
>> > > > On Mon, Jan 23, 2017 at 10:45 AM, Jun Rao  wrote:
>> > > >
>> > > >> I feel that it's simpler to just keep the format of the checkpoint
>> > file
>> > > as
>> > > >> it is and just add a separate checkpoint for low watermark. Low
>> > > watermark
>> > > >> and high watermark are maintained independently. So, not sure if
>> there
>> > > is
>> > > >> significant benefit of storing them together.
>> > > >>
>> > > >> Looking at the KIP again. I actually have another question on the
>> api.
>> > > Is
>> > > >> there any benefit of returning a Future in the purgeDataBefore()
>> api?
>> > > >> Since
>> > > >> admin apis are used infrequently, it seems that it's simpler to
>> just
>> > > have
>> > > >> a
>> > > >> blocking api and returns Map?
>> > > >>
>> > > >> Thanks,
>> > > >>
>> > > >> Jun
>> > > >>
>> > > >> On Sun, Jan 22, 2017 at 3:56 PM, Dong Lin 
>> > wrote:
>> > > >>
>> > > >> > Thanks for the comment Guozhang. Please don't worry about being
>> > late.
>> > > I
>> > > >> > would like to update the KIP if there is clear benefit of the new
>> > > >> approach.
>> > > >> > I am wondering if there is any use-case or operation aspects that
>> > > would
>> > > >> > benefit from the new approach.
>> > > >> >
>> > > >> > I am not saying that these checkpoint files have the same
>> priority.
>> > I
>> > > >> > mentioned other checkpoint files to suggest that it is OK to add
>> one
>> > > >> more
>> > > >> > checkpoint file. To me three checkpoint files is not much
>> different
>> > > from
>> > > >> > four checkpoint files. I am just inclined to not update the KIP
>> if
>> > the
>> > > >> only
>> > > >> > benefit is to avoid addition of a new checkpoint file.
>> > > >> >
>> > > >> >
>> > > >> >
>> > > >> > On Sun, Jan 22, 2017 at 3:48 PM, Guozhang Wang <
>> wangg...@gmail.com>
>> > > >> wrote:
>> > > >> >
>> > > >> > > To me the distinction between recovery-checkpoint and
>> > > >> > > replication-checkpoint are dif

Re: [DISCUSS] KIP-112: Handle disk failure for JBOD

[Dong Lin]

Hi all,

I am going to initiate the vote If there is no further concern with the KIP.

Thanks,
Dong


On Fri, Jan 27, 2017 at 8:08 PM, radai  wrote:

> a few extra points:
>
> 1. broker per disk might also incur more client <--> broker sockets:
> suppose every producer / consumer "talks" to >1 partition, there's a very
> good chance that partitions that were co-located on a single 10-disk broker
> would now be split between several single-disk broker processes on the same
> machine. hard to put a multiplier on this, but likely >x1. sockets are a
> limited resource at the OS level and incur some memory cost (kernel
> buffers)
>
> 2. there's a memory overhead to spinning up a JVM (compiled code and byte
> code objects etc). if we assume this overhead is ~300 MB (order of
> magnitude, specifics vary) than spinning up 10 JVMs would lose you 3 GB of
> RAM. not a ton, but non negligible.
>
> 3. there would also be some overhead downstream of kafka in any management
> / monitoring / log aggregation system. likely less than x10 though.
>
> 4. (related to above) - added complexity of administration with more
> running instances.
>
> is anyone running kafka with anywhere near 100GB heaps? i thought the point
> was to rely on kernel page cache to do the disk buffering 
>
> On Thu, Jan 26, 2017 at 11:00 AM, Dong Lin  wrote:
>
> > Hey Colin,
> >
> > Thanks much for the comment. Please see me comment inline.
> >
> > On Thu, Jan 26, 2017 at 10:15 AM, Colin McCabe 
> wrote:
> >
> > > On Wed, Jan 25, 2017, at 13:50, Dong Lin wrote:
> > > > Hey Colin,
> > > >
> > > > Good point! Yeah we have actually considered and tested this
> solution,
> > > > which we call one-broker-per-disk. It would work and should require
> no
> > > > major change in Kafka as compared to this JBOD KIP. So it would be a
> > good
> > > > short term solution.
> > > >
> > > > But it has a few drawbacks which makes it less desirable in the long
> > > > term.
> > > > Assume we have 10 disks on a machine. Here are the problems:
> > >
> > > Hi Dong,
> > >
> > > Thanks for the thoughtful reply.
> > >
> > > >
> > > > 1) Our stress test result shows that one-broker-per-disk has 15%
> lower
> > > > throughput
> > > >
> > > > 2) Controller would need to send 10X as many LeaderAndIsrRequest,
> > > > MetadataUpdateRequest and StopReplicaRequest. This increases the
> burden
> > > > on
> > > > controller which can be the performance bottleneck.
> > >
> > > Maybe I'm misunderstanding something, but there would not be 10x as
> many
> > > StopReplicaRequest RPCs, would there?  The other requests would
> increase
> > > 10x, but from a pretty low base, right?  We are not reassigning
> > > partitions all the time, I hope (or else we have bigger problems...)
> > >
> >
> > I think the controller will group StopReplicaRequest per broker and send
> > only one StopReplicaRequest to a broker during controlled shutdown.
> Anyway,
> > we don't have to worry about this if we agree that other requests will
> > increase by 10X. One MetadataRequest to send to each broker in the
> cluster
> > every time there is leadership change. I am not sure this is a real
> > problem. But in theory this makes the overhead complexity O(number of
> > broker) and may be a concern in the future. Ideally we should avoid it.
> >
> >
> > >
> > > >
> > > > 3) Less efficient use of physical resource on the machine. The number
> > of
> > > > socket on each machine will increase by 10X. The number of connection
> > > > between any two machine will increase by 100X.
> > > >
> > > > 4) Less efficient way to management memory and quota.
> > > >
> > > > 5) Rebalance between disks/brokers on the same machine will less
> > > > efficient
> > > > and less flexible. Broker has to read data from another broker on the
> > > > same
> > > > machine via socket. It is also harder to do automatic load balance
> > > > between
> > > > disks on the same machine in the future.
> > > >
> > > > I will put this and the explanation in the rejected alternative
> > section.
> > > > I
> > > > have a few questions:
> > > >
> > > > - Can you explain why this solution can help avoid scalability
> > > > bottleneck?
> > > > I actually think it will exacerbate the scalability problem due the
> 2)
> > > > above.
> > > > - Why can we push more RPC with this solution?
> > >
> > > To really answer this question we'd have to take a deep dive into the
> > > locking of the broker and figure out how effectively it can parallelize
> > > truly independent requests.  Almost every multithreaded process is
> going
> > > to have shared state, like shared queues or shared sockets, that is
> > > going to make scaling less than linear when you add disks or
> processors.
> > >  (And clearly, another option is to improve that scalability, rather
> > > than going multi-process!)
> > >
> >
> > Yeah I also think it is better to improve scalability inside kafka code
> if
> > possible. I am not sure we currently have any scalability issue inside
> > Kafka that

Re: [VOTE] KIP-107 Add purgeDataBefore() API in AdminClient

[Apurva Mehta]

got it. I missed the message in the middle declaring that the vote passed.

On Tue, Jan 31, 2017 at 5:51 PM, Dong Lin  wrote:

> This thread was been closed on Jan 18. We had more discussion after
> Guozhang's feedback on Jan 21. But no major change was made to the KIP
> after the discussion.
>
>
> On Tue, Jan 31, 2017 at 5:47 PM, Dong Lin  wrote:
>
> > Hey Apurva,
> >
> > I think the KIP table in https://cwiki.apache.org/co
> > nfluence/display/KAFKA/Kafka+Improvement+Proposals has already been
> > updated. Is there anything I missed?
> >
> > Thanks,
> > Dong
> >
> > On Tue, Jan 31, 2017 at 5:45 PM, Apurva Mehta 
> wrote:
> >
> >> Hi Dong,
> >>
> >> It looks like this vote passed. Can you close this thread and update the
> >> KIP table?
> >>
> >> Thanks,
> >> Apurva
> >>
> >> On Tue, Jan 24, 2017 at 1:30 PM, Jun Rao  wrote:
> >>
> >> > Hi, Dong,
> >> >
> >> > The changes sound good to me. Also, thanks for the explanation of
> >> returning
> >> > a future from purgeDataFrom(). We can keep it that way.
> >> >
> >> > Thanks,
> >> >
> >> > Jun
> >> >
> >> > On Mon, Jan 23, 2017 at 4:24 PM, Dong Lin 
> wrote:
> >> >
> >> > > Hi all,
> >> > >
> >> > > When I am implementing the patch, I realized that the current usage
> of
> >> > > "low_watermark" is a bit confusing. So I made the following
> interface
> >> > > changes in the KIP:
> >> > >
> >> > > - The newly added checkpoint file will be named
> >> > log-begin-offset-checkpoint
> >> > > - Replace low_watermark with log_begin_offset in
> FetchRequestPartition
> >> > and
> >> > > FetchResponsePartitionHeader
> >> > >
> >> > > The problem with the previous naming conversion is that,
> low_watermark
> >> > > implies minimum log begin offset of all replicas (similar to high
> >> > > watermark) and we return this value in the PurgeResponse. In other
> >> words,
> >> > > low_watermark can not be incremented if a follower is not live.
> >> Therefore
> >> > > we can not use low_watermark in the checkpoint file or in the
> >> > FetchResponse
> >> > > from leader to followers if we want to persists the offset-to-purge
> >> > > received from user across broker rebounce.
> >> > >
> >> > > You can find the changes in KIP here
> >> > >  diffpagesbyversion.action?
> >> > > pageId=67636826&selectedPageVersions=13&selectedPageVersions=14>.
> >> > > Please let me know if you have any concern with this change.
> >> > >
> >> > > Thanks,
> >> > > Dong
> >> > >
> >> > > On Mon, Jan 23, 2017 at 11:20 AM, Dong Lin 
> >> wrote:
> >> > >
> >> > > > Thanks for the comment Jun.
> >> > > >
> >> > > > Yeah, I think there is use-case where this can be useful. Allowing
> >> for
> >> > > > asynchronous delete will be useful if an application doesn't need
> >> > strong
> >> > > > guarantee of purgeDataFrom(), e.g. if it is done to help reduce
> disk
> >> > > usage
> >> > > > of kafka. The application may want to purge data for every time it
> >> does
> >> > > > auto-commit without wait for future object to complete. On the
> other
> >> > > hand,
> >> > > > synchronous delete will be useful if an application wants to make
> >> sure
> >> > > that
> >> > > > the sensitive or bad data is definitely deleted. I think
> returning a
> >> > > future
> >> > > > makes both choice available to user and it doesn't complicate
> >> > > > implementation much.
> >> > > >
> >> > > >
> >> > > > On Mon, Jan 23, 2017 at 10:45 AM, Jun Rao 
> wrote:
> >> > > >
> >> > > >> I feel that it's simpler to just keep the format of the
> checkpoint
> >> > file
> >> > > as
> >> > > >> it is and just add a separate checkpoint for low watermark. Low
> >> > > watermark
> >> > > >> and high watermark are maintained independently. So, not sure if
> >> there
> >> > > is
> >> > > >> significant benefit of storing them together.
> >> > > >>
> >> > > >> Looking at the KIP again. I actually have another question on the
> >> api.
> >> > > Is
> >> > > >> there any benefit of returning a Future in the purgeDataBefore()
> >> api?
> >> > > >> Since
> >> > > >> admin apis are used infrequently, it seems that it's simpler to
> >> just
> >> > > have
> >> > > >> a
> >> > > >> blocking api and returns Map?
> >> > > >>
> >> > > >> Thanks,
> >> > > >>
> >> > > >> Jun
> >> > > >>
> >> > > >> On Sun, Jan 22, 2017 at 3:56 PM, Dong Lin 
> >> > wrote:
> >> > > >>
> >> > > >> > Thanks for the comment Guozhang. Please don't worry about being
> >> > late.
> >> > > I
> >> > > >> > would like to update the KIP if there is clear benefit of the
> new
> >> > > >> approach.
> >> > > >> > I am wondering if there is any use-case or operation aspects
> that
> >> > > would
> >> > > >> > benefit from the new approach.
> >> > > >> >
> >> > > >> > I am not saying that these checkpoint files have the same
> >> priority.
> >> > I
> >> > > >> > mentioned other checkpoint files to suggest that it is OK to
> add
> >> one
> >> > > >> more
> >> > > >> > checkpoint file. To me three checkpoint files is not much
> >> different
> >> 

[jira] [Updated] (KAFKA-3478) Finer Stream Flow Control

[Ewen Cheslack-Postava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava updated KAFKA-3478:
-
Fix Version/s: (was: 0.10.2.0)

> Finer Stream Flow Control
> -
>
> Key: KAFKA-3478
> URL: https://issues.apache.org/jira/browse/KAFKA-3478
> Project: Kafka
>  Issue Type: Sub-task
>  Components: streams
>Reporter: Guozhang Wang
>  Labels: user-experience
>
> Today we have a event-time based flow control mechanism in order to 
> synchronize multiple input streams in a best effort manner:
> http://docs.confluent.io/3.0.0/streams/architecture.html#flow-control-with-timestamps
> However, there are some use cases where users would like to have finer 
> control of the input streams, for example, with two input streams, one of 
> them always reading from offset 0 upon (re)-starting, and the other reading 
> for log end offset.
> Today we only have one consumer config "offset.auto.reset" to control that 
> behavior, which means all streams are read either from "earliest" or "latest".
> We should consider how to improve this settings to allow users have finer 
> control over these frameworks.
> =
> A finer flow control could also be used to allow for populating a {{KTable}} 
> (with an "initial" state) before starting the actual processing (this feature 
> was ask for in the mailing list multiple times already). Even if it is quite 
> hard to define, *when* the initial populating phase should end, this might 
> still be useful. There would be the following possibilities:
>  1) an initial fixed time period for populating
>(it might be hard for a user to estimate the correct value)
>  2) an "idle" period, ie, if no update to a KTable for a certain time is
> done, we consider it as populated
>  3) a timestamp cut off point, ie, all records with an older timestamp
> belong to the initial populating phase
>  4) a throughput threshold, ie, if the populating frequency falls below
> the threshold, the KTable is considered "finished"
>  5) maybe something else ??
> The API might look something like this
> {noformat}
> KTable table = builder.table("topic", 1000); // populate the table without 
> reading any other topics until see one record with timestamp 1000.
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (KAFKA-1332) Add functionality to the offsetsBeforeTime() API

[Ewen Cheslack-Postava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava updated KAFKA-1332:
-
Fix Version/s: (was: 0.10.2.0)

> Add functionality to the offsetsBeforeTime() API
> 
>
> Key: KAFKA-1332
> URL: https://issues.apache.org/jira/browse/KAFKA-1332
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Affects Versions: 0.9.0.0
>Reporter: Neha Narkhede
>Assignee: Jason Gustafson
>
> Add functionality to the offsetsBeforeTime() API to load offsets 
> corresponding to a particular timestamp, including earliest and latest offsets



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (KAFKA-3262) Make KafkaStreams debugging friendly

[Ewen Cheslack-Postava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava updated KAFKA-3262:
-
Fix Version/s: (was: 0.10.2.0)

> Make KafkaStreams debugging friendly
> 
>
> Key: KAFKA-3262
> URL: https://issues.apache.org/jira/browse/KAFKA-3262
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.0.0
>Reporter: Yasuhiro Matsuda
>Assignee: Eno Thereska
>  Labels: user-experience
>
> Current KafkaStreams polls records in the same thread as the data processing 
> thread. This makes debugging user code, as well as KafkaStreams itself, 
> difficult. When the thread is suspended by the debugger, the next heartbeat 
> of the consumer tie to the thread won't be send until the thread is resumed. 
> This often results in missed heartbeats and causes a group rebalance. So it 
> may will be a completely different context then the thread hits the break 
> point the next time.
> We should consider using separate threads for polling and processing.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (KAFKA-3302) Pass kerberos keytab and principal as part of client config

[Ewen Cheslack-Postava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava updated KAFKA-3302:
-
Fix Version/s: (was: 0.10.2.0)

> Pass kerberos keytab and principal as part of client config 
> 
>
> Key: KAFKA-3302
> URL: https://issues.apache.org/jira/browse/KAFKA-3302
> Project: Kafka
>  Issue Type: Improvement
>  Components: security
>Reporter: Sriharsha Chintalapani
>Assignee: Sriharsha Chintalapani
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (KAFKA-4495) ThreadCache.maybeEvict can become an infinite loop

[Ewen Cheslack-Postava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava updated KAFKA-4495:
-
Fix Version/s: (was: 0.10.2.0)

> ThreadCache.maybeEvict can become an infinite loop
> --
>
> Key: KAFKA-4495
> URL: https://issues.apache.org/jira/browse/KAFKA-4495
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.1.0
>Reporter: Damian Guy
>Assignee: Damian Guy
>
> ThreadCache.maybeEvict can end up in an infinite loop if the cache size is 
> beyond the maxCacheSizeBytes and the cache for the current namespace becomes 
> empty before the cache size drops below maxCacheSizeBytes



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (KAFKA-4611) Support custom authentication mechanism

[Ewen Cheslack-Postava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava updated KAFKA-4611:
-
Fix Version/s: (was: 0.10.2.0)

> Support custom authentication mechanism
> ---
>
> Key: KAFKA-4611
> URL: https://issues.apache.org/jira/browse/KAFKA-4611
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Affects Versions: 0.10.0.0
>Reporter: mahendiran chandrasekar
>
> Currently there are two login mechanisms supported by kafka client.
> 1) Default Login / Abstract Login which uses JAAS authentication
> 2) Kerberos Login
> Supporting user defined login mechanism's would be nice. 
> This could be achieved by removing the limitation from 
> [here](https://github.com/apache/kafka/blob/0.10.0/clients/src/main/java/org/apache/kafka/common/security/authenticator/LoginManager.java#L44)
>  ... Instead get custom login module implemented by user from the configs, 
> gives users the option to implement custom login mechanism. 
> I am running into an issue in setting JAAS authentication system property on 
> all executors of my spark cluster. Having custom mechanism to authorize kafka 
> would be a good improvement for me



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-3944) After the broker restart, fetchers stopped due to a delayed controlled shutdown message

[James Cheng (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847898#comment-15847898
 ] 

James Cheng commented on KAFKA-3944:


Is this related to KAFKA-1120?

> After the broker restart, fetchers stopped due to a delayed controlled 
> shutdown message
> ---
>
> Key: KAFKA-3944
> URL: https://issues.apache.org/jira/browse/KAFKA-3944
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Reporter: Maysam Yabandeh
>Priority: Minor
>  Labels: reliability
>
> The symptom is that cluster reports under-replicated blocks and some replicas 
> do not seem to catch up ever. It turns out that the corresponding fetchers in 
> those brokers were stopped shortly after the broker's restart. The broker had 
> stopped the fetcher upon receiving stop-replica request from the controller. 
> The controller had issued those request upon processing controlled shutdown 
> request form the same broker. However those requests were all sent before the 
> broker restart but the controller processed them after. Here is the timeline:
> # broker sends controlled shutdown message to controller
> # the process fails and the broker proceeds with an unclean shutdown
> # the broker is restated
> # the controller processes the perviously sent controlled shutdown messages
> # the controller sends stop replica messages to the broker
> # the broker shuts down the fetchers, while it has no intent to shut down 
> again
> # this leads to under-replicated blocks
> Example from logs:
> {code}
> broker19.com:/var/log/kafka$ grep "Retrying controlled shutdow\|unclean 
> shutdown" server.log.2016-07-07.2 
> 2016-07-07 15:58:10,818 WARN server.KafkaServer: [Kafka Server 19], Retrying 
> controlled shutdown after the previous attempt failed...
> 2016-07-07 15:58:45,887 WARN server.KafkaServer: [Kafka Server 19], Retrying 
> controlled shutdown after the previous attempt failed...
> 2016-07-07 15:59:20,927 WARN server.KafkaServer: [Kafka Server 19], Retrying 
> controlled shutdown after the previous attempt failed...
> 2016-07-07 15:59:20,929 WARN server.KafkaServer: [Kafka Server 19], 
> Proceeding to do an unclean shutdown as all the controlled shutdown attempts 
> failed
> broker19.com:/var/log/kafka$ head -1 server.log.2016-07-07.3
> 2016-07-07 16:00:23,191 INFO server.KafkaConfig: KafkaConfig values: 
> {code}
> {code}
> broker13.com:/var/log/kafka$ grep "Shutting down broker 19" 
> controller.log.2016-07-07.1 
> 2016-07-07 15:57:35,822 INFO controller.KafkaController: [Controller 13]: 
> Shutting down broker 19
> 2016-07-07 16:02:45,526 INFO controller.KafkaController: [Controller 13]: 
> Shutting down broker 19
> 2016-07-07 16:05:42,432 INFO controller.KafkaController: [Controller 13]: 
> Shutting down broker 19
> {code}
> which resulted into many stop replica request to broker 19:
> {code}
> broker13.com:/var/log/kafka$ grep "The stop replica request (delete = false) 
> sent to broker 19 is" controller.log.2016-07-07.1 | tail -1
> 2016-07-07 16:06:02,374 DEBUG controller.ControllerBrokerRequestBatch: The 
> stop replica request (delete = false) sent to broker 19 is 
> [Topic=topic-xyz,Partition=6,Replica=19]
> {code}
> broker 19 processes them AFTER its restart:
> {code}
> broker19.com:/var/log/kafka$ grep "handling stop replica (delete=false) for 
> partition .topic-xzy,3." state-change.log.2016-07-07.2 
> 2016-07-07 16:06:00,154 TRACE change.logger: Broker 19 handling stop replica 
> (delete=false) for partition [topic-xzy,3]
> 2016-07-07 16:06:00,154 TRACE change.logger: Broker 19 finished handling stop 
> replica (delete=false) for partition [topic-xyz,3]
> 2016-07-07 16:06:00,155 TRACE change.logger: Broker 19 handling stop replica 
> (delete=false) for partition [topic-xyz,3]
> 2016-07-07 16:06:00,155 TRACE change.logger: Broker 19 finished handling stop 
> replica (delete=false) for partition [topic-xyz,3]
> {code}
> and removes the fetchers:
> {code}
> broker19.com:/var/log/kafka$ grep "Removed fetcher.*topic-xyz.3" 
> server.log.2016-07-07.3 | tail -2
> 2016-07-07 16:06:00,154 INFO server.ReplicaFetcherManager: 
> [ReplicaFetcherManager on broker 19] Removed fetcher for partitions 
> [topic-xyz,3]
> 2016-07-07 16:06:00,155 INFO server.ReplicaFetcherManager: 
> [ReplicaFetcherManager on broker 19] Removed fetcher for partitions 
> [topic-xyz,3]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #2429: KAFKA-4677: Avoid unnecessary task movement across...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2429


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-4677) Avoid unnecessary task movement across threads during rebalance

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847997#comment-15847997
 ] 

ASF GitHub Bot commented on KAFKA-4677:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2429


> Avoid unnecessary task movement across threads during rebalance
> ---
>
> Key: KAFKA-4677
> URL: https://issues.apache.org/jira/browse/KAFKA-4677
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.2.0
>Reporter: Damian Guy
>Assignee: Damian Guy
> Fix For: 0.10.3.0
>
>
> StreamPartitionAssigner tries to follow a sticky assignment policy to avoid 
> expensive task migration. Currently, it does this in a best-effort approach.
> We could observe a case, for which tasks did migrate for no good reason, thus 
> we assume that the current implementation could be improved to be more sticky.
> The concrete scenario is as follows:
> assume we have topology with 3 tasks, A, B, C
> assume we have 3 threads, each executing one task: 1-A, 2-B, 3-C
> for some reason, thread 1 goes down and a rebalance gets triggered
> thread 2 and 3 get their partitions revoked
> sometimes (not sure what the exact condition for this is), the new assignment 
> flips the assignment for task B and C (task A is newly assigned to either 
> thread 2 or 3)
> > possible new assignment 2(A,C) and 3-B
> There is no obvious reason (like load-balancing) why the task assignment for 
> B and C does change to the other thread resulting in unnecessary task 
> migration.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (KAFKA-4677) Avoid unnecessary task movement across threads during rebalance

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-4677:
-
Resolution: Fixed
Status: Resolved  (was: Patch Available)

Issue resolved by pull request 2429
[https://github.com/apache/kafka/pull/2429]

> Avoid unnecessary task movement across threads during rebalance
> ---
>
> Key: KAFKA-4677
> URL: https://issues.apache.org/jira/browse/KAFKA-4677
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.2.0
>Reporter: Damian Guy
>Assignee: Damian Guy
> Fix For: 0.10.3.0
>
>
> StreamPartitionAssigner tries to follow a sticky assignment policy to avoid 
> expensive task migration. Currently, it does this in a best-effort approach.
> We could observe a case, for which tasks did migrate for no good reason, thus 
> we assume that the current implementation could be improved to be more sticky.
> The concrete scenario is as follows:
> assume we have topology with 3 tasks, A, B, C
> assume we have 3 threads, each executing one task: 1-A, 2-B, 3-C
> for some reason, thread 1 goes down and a rebalance gets triggered
> thread 2 and 3 get their partitions revoked
> sometimes (not sure what the exact condition for this is), the new assignment 
> flips the assignment for task B and C (task A is newly assigned to either 
> thread 2 or 3)
> > possible new assignment 2(A,C) and 3-B
> There is no obvious reason (like load-balancing) why the task assignment for 
> B and C does change to the other thread resulting in unnecessary task 
> migration.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #2476: KAFKA-4586; Add purgeDataBefore() API (KIP-107)

[lindong28]

GitHub user lindong28 opened a pull request:

https://github.com/apache/kafka/pull/2476

KAFKA-4586; Add purgeDataBefore() API (KIP-107)



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lindong28/kafka KAFKA-4586

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2476.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2476


commit 3546fcc17e9ed816dc50536e23af8d0df6369591
Author: Dong Lin 
Date:   2017-01-27T17:51:06Z

KAFKA-4586; Add purgeDataBefore() API (KIP-107)




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-4586) Add purgeDataBefore() API in AdminClient

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15848020#comment-15848020
 ] 

ASF GitHub Bot commented on KAFKA-4586:
---

GitHub user lindong28 opened a pull request:

https://github.com/apache/kafka/pull/2476

KAFKA-4586; Add purgeDataBefore() API (KIP-107)



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lindong28/kafka KAFKA-4586

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2476.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2476


commit 3546fcc17e9ed816dc50536e23af8d0df6369591
Author: Dong Lin 
Date:   2017-01-27T17:51:06Z

KAFKA-4586; Add purgeDataBefore() API (KIP-107)




> Add purgeDataBefore() API in AdminClient
> 
>
> Key: KAFKA-4586
> URL: https://issues.apache.org/jira/browse/KAFKA-4586
> Project: Kafka
>  Issue Type: New Feature
>Reporter: Dong Lin
>Assignee: Dong Lin
>
> Please visit 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-107%3A+Add+purgeDataBefore%28%29+API+in+AdminClient
>  for motivation etc.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is back to normal : kafka-trunk-jdk8 #1239

[Apache Jenkins Server]

See