[jira] [Updated] (KAFKA-5343) MockProducer should not handle send/ProducerRecord after close

[Martin Grotzke (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Martin Grotzke updated KAFKA-5343:
--
Description: 
Currently {{MockProducer}} implements {{close}} with void, i.e. it just does 
nothing on {{close}}.

Assuming
* production code built for "max.in.flight.requests.per.connection=1" which 
transforms a list of events/records to several invocations of {{send(record, 
callback)}}
* where the {{callback}} invokes {{close(0, MILLISECONDS)}} in case of an 
exception
* the expectation is that after {{send}} failed for record R1, any record that 
was sent after {{close}} was invoked is *not* handled.

Similarly, in case of the {{MockProducer}}, the expectation is that any record 
after R1 is just ignored, and specifically neither autocompleted nor added to 
the list of completions.

This is not the case, because the {{MockProducer}} just ignores an invocation 
of {{close}}.

The fix should be trivial, does it makes sense to submit a PR for this?


  was:
Currently {{MockProducer}} implements {{close}} with void, i.e. it just does 
nothing on {{close}}.

Assuming
* production code built for "max.in.flight.requests.per.connection=1" which 
transforms a list of events/records to several invocations of {{send(record, 
callback)}}
* where the {{callback}} invokes {{close(0, MILLISECONDS)}} in case of an 
exception
* the expectation is that after {{send}} failed for record R1, any record that 
was sent after R1 is *not* handled.

Similarly, in case of the {{MockProducer}}, the expectation is that any record 
after R1 is just ignored, and specifically neither autocompleted nor added to 
the list of completions.

This is not the case, because the {{MockProducer}} just ignores an invocation 
of {{close}}.

The fix should be trivial, does it makes sense to submit a PR for this?



> MockProducer should not handle send/ProducerRecord after close
> --
>
> Key: KAFKA-5343
> URL: https://issues.apache.org/jira/browse/KAFKA-5343
> Project: Kafka
>  Issue Type: Improvement
>  Components: producer 
>Affects Versions: 0.10.2.1
>Reporter: Martin Grotzke
>Priority: Minor
>
> Currently {{MockProducer}} implements {{close}} with void, i.e. it just does 
> nothing on {{close}}.
> Assuming
> * production code built for "max.in.flight.requests.per.connection=1" which 
> transforms a list of events/records to several invocations of {{send(record, 
> callback)}}
> * where the {{callback}} invokes {{close(0, MILLISECONDS)}} in case of an 
> exception
> * the expectation is that after {{send}} failed for record R1, any record 
> that was sent after {{close}} was invoked is *not* handled.
> Similarly, in case of the {{MockProducer}}, the expectation is that any 
> record after R1 is just ignored, and specifically neither autocompleted nor 
> added to the list of completions.
> This is not the case, because the {{MockProducer}} just ignores an invocation 
> of {{close}}.
> The fix should be trivial, does it makes sense to submit a PR for this?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jira-Spam on Dev-Mailinglist

[marc . schlegel]

Hello everyone

I find it hard to follow this mailinglist due to all the mails generated 
by Jira. Just over this weekend there are 240 new mails.
Would it be possible to setup something like j...@kafka.apache.org where 
everyone can subscribe interested in those Jira mails?

Right now I am going to setup a filter which just deletes the jira-tagged 
mails, but I think the current setup also makes it hard to read through 
the archives.

regards
Marc

Re?? Re: Re: [DISCUSS] KIP-148: Add a connect timeout for client

[????????]

Hi Rajini
Thanks for the comment and kindly to add another config for the 
flexibility. The limitation in the consumer with request.timeout.ms  is need to 
fix, but as Guozhang's comment, this is orthogonal to this KIP, we can fix it 
by another KIP or issue.


Thanks,
David





--  --
??: "Rajini Sivaram";;
: 2017??5??23??(??) 11:57
??: ""<254479...@qq.com>; 
: "dev"; 
: Re: Re: Re: [DISCUSS] KIP-148: Add a connect timeout for client



Hi David,

The only issue with yet another config is that nearly everyone will just leave 
it at the default value and it is hard to configure. But that is just my 
personal opinion. If you need the flexibility, then I don't have any objection 
to another config option.


The specific scenario described in the KIP where a connection doesn't get 
closed cleanly and the system waits for TCP timeout - in my opinion, that is 
the scenario where request.timeout.ms is most useful - to detect a broker crash 
where the OS didn't shutdown cleanly. We don't currently handle this scenario 
during connection establishment, as you have described in the KIP.  For this 
scenario, a single config makes sense to me to ensure that crashes are detected 
and handled regardless of the connection state. The limitation in the consumer 
with request.timeout.ms  is something that needs to be fixed anyway since 5 
minutes is too long to detect a broker crash in any connection state.


There could be other scenarios where you want to use different configs. It will 
be useful to include any scenarios you have in mind to the KIP.






On Tue, May 23, 2017 at 11:16 AM,  <254479...@qq.com> wrote:
Hi Rajini

If we only use the request.timeout for the connect and process request,  it is 
not flexible for some cases, for example we need the connecting phase to a 
short time(5s), and the request for a longer time(40s).

Why not add a connect timeout to handler these different cases.

Thanks,
David

--  --
??: "Rajini Sivaram" ;
: 2017??5??23??(??) 19:27
??: "dev" ;
: Re: Re: [DISCUSS] KIP-148: Add a connect timeout for client



Guozhang,

At the moment we don't have a connect timeout. And the behaviour suggested
in the KIP is useful to address this.

We do however have a request.timeout.ms. This is the amount of time it
would take to detect a crashed broker if the broker crashed after a
connection was established. Unfortunately in the consumer, this was
increased to > 5minutes since JoinRequest can take up to
max.poll.interval.ms, which has a default of  5 minutes. Since the
whole point of this timeout is to detect a crashed broker, 5 minutes is too
large.

My suggestion was to use request.timeout.ms to also detect connection
timeouts to a crashed broker - implement the behavior suggested in the KIP
without adding a new config parameter. As Ismael has said, this will need
to fix request.timeout.ms in the consumer.


On Mon, May 22, 2017 at 1:23 PM, Simon Souter 
wrote:

> The following tickets are probably relevant to this KIP:
>
> https://issues.apache.org/jira/browse/KAFKA-3457
> https://issues.apache.org/jira/browse/KAFKA-1894
> https://issues.apache.org/jira/browse/KAFKA-3834
>
> On 22 May 2017 at 16:30, Rajini Sivaram  wrote:
>
> > Ismael,
> >
> > Yes, agree. My concern was that a connection can be shutdown uncleanly at
> > any time. If a client is in the middle of a request, then it times out
> > after min(request.timeout.ms, tcp-timeout). If we add another config
> > option
> > connect.timeout.ms, then we will sometimes wait for min(
> connect.timeout.ms
> > ,
> > tcp-timeout) and sometimes for min(request.timeout.ms, tcp-timeout),
> > depending
> > on connection state. One config option feels neater to me.
> >
> > On Mon, May 22, 2017 at 11:21 AM, Ismael Juma  wrote:
> >
> > > Rajini,
> > >
> > > For this to have the desired effect, we'd probably need to lower the
> > > default request.timeout.ms for the consumer and fix the underlying
> > reason
> > > why it is a little over 5 minutes at the moment.
> > >
> > > Ismael
> > >
> > > On Mon, May 22, 2017 at 4:15 PM, Rajini Sivaram <
> rajinisiva...@gmail.com
> > >
> > > wrote:
> > >
> > > > Hi David,
> > > >
> > > > Sorry, what I meant was: Can you reuse the existing configuration
> > option
> > > > request.timeout,ms , instead of adding a new config and add the
> > behaviour
> > > > that you have proposed in the KIP for the connection phase using this
> > > > timeout? I think the timeout for connection is useful. I am not sure
> we
> > > > need another configuration option to implement it.
> > > >
> > > > Regards,
> > > >
> > > > Rajini
> > > >
> > > >
> > > > On Mon, May 22, 2017 at 11:06 AM,  <254479...@qq.com> wrote:
> > > >
> > > > > Hi Rajini.
> > > > >
> > > > > When kafka node' machine is shutdown or network is closed, the
> > > connecting
> > > > > phase could not use the request

Re: [DISCUSS] KIP-164 Add unavailablePartitionCount and per-partition Unavailable metrics

[Gwen Shapira]

Hi,

Sounds good. I was sure this existed already for some reason :)

On Sun, May 28, 2017 at 11:06 AM Dong Lin  wrote:

> Hi,
>
> We created KIP-164 to propose adding per-partition metric *Unavailable* and
> per-broker metric *UnavailablePartitionCount*
>
> The KIP wik can be found at
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-164-+Add+unavailablePartitionCount+and+per-partition+Unavailable+metrics
> .
>
> Comments are welcome.
>
> Thanks,
> Dong
>

Re: Kafka Loss of messages

[Vinayak Sharma]

Hi Jun,
0.10.0.0
I have a Kafka Cluster running on AWS instances. I am trying to scale
broker up and down while simultaneously using the cluster to produce and
consume messages.




On 29 May 2017 at 13:03, Sanket Dhopeshwarkar  wrote:

> +Vinayak
>
> On 26 May 2017 at 21:44, Jun Rao  wrote:
>
>> Hi, Vinayak,
>>
>> I am not sure that KAFKA-1561 as reported is an issue. Could you describe
>> the scenario in your test in that jira in a bit more detail?
>>
>> Thanks,
>>
>> Jun
>>
>> On Fri, May 26, 2017 at 5:58 AM, Vinayak Sharma 
>> wrote:
>>
>>> Hi,
>>>
>>> I am experiencing loss of messages while scaling brokers up and down in
>>> kafka cluster.
>>>
>>> I came across this jira issue(link
>>> ). Can you confirm if
>>> this is actually an issue in kafka or can the problem be solved by changing
>>> some configuration parameters.
>>>
>>> I am enclosing the server.properties that I am using.
>>>
>>> Please get to back to me.
>>>
>>> Regards,
>>> Vinayak.
>>>
>>> 
>>
>>
>>
>

-- 
 

Re: Kafka Loss of messages

[Vinayak Sharma]

Hi Jun,

Please ignore the earlier mail.

I have a Kafka Cluster running on AWS machines i.e. 3 Zookeeper and 2
Brokers. I am trying to scale the brokers while producing and consuming
messages.

I am sending 1000 messages by batch producer every second for 200 seconds
for the following topics:

 topics : Topic1(20 partitions) Topic2(40 partitions)
Replication Factor.

 Add 1 broker:
 Produced  Consumed

Topic1   20199950
Topic2   20199975


Remove 1 broker:

   Produced  Consumed

Topic1   20199950
Topic2   20199825

I run a script which reassigns the partitions. And as it can be seen from
the above numbers that we see data loss at the consumer end.

Let me know if you need any further details.

Regards,
Vinayak

On 29 May 2017 at 13:03, Sanket Dhopeshwarkar  wrote:

> +Vinayak
>
> On 26 May 2017 at 21:44, Jun Rao  wrote:
>
>> Hi, Vinayak,
>>
>> I am not sure that KAFKA-1561 as reported is an issue. Could you describe
>> the scenario in your test in that jira in a bit more detail?
>>
>> Thanks,
>>
>> Jun
>>
>> On Fri, May 26, 2017 at 5:58 AM, Vinayak Sharma 
>> wrote:
>>
>>> Hi,
>>>
>>> I am experiencing loss of messages while scaling brokers up and down in
>>> kafka cluster.
>>>
>>> I came across this jira issue(link
>>> ). Can you confirm if
>>> this is actually an issue in kafka or can the problem be solved by changing
>>> some configuration parameters.
>>>
>>> I am enclosing the server.properties that I am using.
>>>
>>> Please get to back to me.
>>>
>>> Regards,
>>> Vinayak.
>>>
>>> 
>>
>>
>>
>

-- 
 

[jira] [Updated] (KAFKA-5008) Kafka-Clients not OSGi ready

[Marc (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marc updated KAFKA-5008:

  Reviewer: Neha Narkhede
Issue Type: Improvement  (was: Bug)

> Kafka-Clients not OSGi ready
> 
>
> Key: KAFKA-5008
> URL: https://issues.apache.org/jira/browse/KAFKA-5008
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Affects Versions: 0.10.2.0
>Reporter: Marc
>Priority: Minor
>
> The kafka-clients artifact does not provide OSGi metadata. This adds an 
> additional barrier for OSGi developers to use the artifact since it has to be 
> [wrapped|http://bnd.bndtools.org/chapters/390-wrapping.html].
> The metadata can automatically be created using bnd.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (KAFKA-5345) Some socket connections not closed after restart of Kafka Streams

[Jeroen van Wilgenburg (JIRA)]

Jeroen van Wilgenburg created KAFKA-5345:


 Summary: Some socket connections not closed after restart of Kafka 
Streams
 Key: KAFKA-5345
 URL: https://issues.apache.org/jira/browse/KAFKA-5345
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 0.10.2.1, 0.10.2.0
 Environment: MacOs 10.12.5 and Ubuntu 14.04
Reporter: Jeroen van Wilgenburg


We ran into a problem that resulted in a "Too many open files" exception 
because some sockets are not closed after a restart.
This problem only occurs with version {{0.10.2.1}} and {{0.10.2.0}}. 
{{0.10.1.1}} and {{0.10.1.0}} both work as expected.
I used the same version for the server and client.

I used https://github.com/kohsuke/file-leak-detector to display the open file 
descriptors. The culprit was :

{noformat}
#146 socket channel by thread:pool-2-thread-1 on Mon May 29 11:20:47 CEST 2017
at sun.nio.ch.SocketChannelImpl.(SocketChannelImpl.java:108)
at 
sun.nio.ch.SelectorProviderImpl.openSocketChannel(SelectorProviderImpl.java:60)
at java.nio.channels.SocketChannel.open(SocketChannel.java:145)
at org.apache.kafka.common.network.Selector.connect(Selector.java:168)
at 
org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:629)
at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:186)
at 
org.apache.kafka.streams.processor.internals.StreamsKafkaClient.ensureOneNodeIsReady(StreamsKafkaClient.java:195)
at 
org.apache.kafka.streams.processor.internals.StreamsKafkaClient.getAnyReadyBrokerId(StreamsKafkaClient.java:233)
at 
org.apache.kafka.streams.processor.internals.StreamsKafkaClient.checkBrokerCompatibility(StreamsKafkaClient.java:300)
at 
org.apache.kafka.streams.KafkaStreams.checkBrokerVersionCompatibility(KafkaStreams.java:401)
at org.apache.kafka.streams.KafkaStreams.start(KafkaStreams.java:425)
{noformat}  


I could narrow the problem down to a reproducable example below (the only 
dependency is 
{{org.apache.kafka:kafka-streams:jar:0.10.2.1}}). 
*IMPORTANT*: You have to run this code in the Intellij IDEA debugger with a 
special breakpoint to see it fail. 
See the comments on the socketChannels variable on how to add this breakpoint. 
When you run this code you will see the number of open SocketChannels increase 
(only on version 0.10.2.x).

{code:title=App.java}
import org.apache.kafka.common.serialization.Serdes;
import org.apache.kafka.streams.KafkaStreams;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.kstream.KStreamBuilder;

import java.nio.channels.SocketChannel;
import java.nio.channels.spi.AbstractInterruptibleChannel;
import java.util.ArrayList;
import java.util.List;
import java.util.Properties;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
import java.util.stream.Collectors;

public class App {

private static KafkaStreams streams;
private static String brokerList;

// Fill socketChannels with entries on line 'Socket socket = 
socketChannel.socket();' (line number 170  on 0.10.2.1)
// of org.apache.kafka.common.network.Selector: Add breakpoint, right click 
on breakpoint.
// - Uncheck 'Suspend'
// - Check 'Evaluate and log' and fill text field with (without quotes) 
'App.socketChannels.add(socketChannel)'
private static final List socketChannels = new ArrayList<>();

public static void main(String[] args) {
brokerList = args[0];
init();

ScheduledExecutorService scheduledThreadPool = 
Executors.newScheduledThreadPool(1);

Runnable command = () -> {
streams.close();

System.out.println("Open socketChannels: " + socketChannels.stream()
.filter(AbstractInterruptibleChannel::isOpen)
.collect(Collectors.toList()).size());

init();
};
scheduledThreadPool.scheduleWithFixedDelay(command, 1L, 2000, 
TimeUnit.MILLISECONDS);

}

private static void init() {
Properties streamsConfiguration = new Properties();
streamsConfiguration.put(StreamsConfig.APPLICATION_ID_CONFIG, 
"JeroenApp");
streamsConfiguration.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, 
brokerList);

StreamsConfig config = new StreamsConfig(streamsConfiguration);

KStreamBuilder builder = new KStreamBuilder();
KStream stream = builder.stream(Serdes.String(), 
Serdes.String(), "HarrieTopic");
stream.foreach((key, value) -> System.out.println(value));

streams = new KafkaStreams(builder, config);
streams.start();
}

}
{code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Kafka Loss of messages

[Sanket Dhopeshwarkar]

+Vinayak

On 26 May 2017 at 21:44, Jun Rao  wrote:

> Hi, Vinayak,
>
> I am not sure that KAFKA-1561 as reported is an issue. Could you describe
> the scenario in your test in that jira in a bit more detail?
>
> Thanks,
>
> Jun
>
> On Fri, May 26, 2017 at 5:58 AM, Vinayak Sharma 
> wrote:
>
>> Hi,
>>
>> I am experiencing loss of messages while scaling brokers up and down in
>> kafka cluster.
>>
>> I came across this jira issue(link
>> ). Can you confirm if
>> this is actually an issue in kafka or can the problem be solved by changing
>> some configuration parameters.
>>
>> I am enclosing the server.properties that I am using.
>>
>> Please get to back to me.
>>
>> Regards,
>> Vinayak.
>>
>> 
>
>
>

-- 
 

Re: [DISCUSS]: KIP-161: streams record processing exception handlers

[Jay Kreps]

Hey Eno,

I think this makes sense. I do think people who spend time running
production stream processing systems will, over time, end up strongly
preferring the current behavior of failing and fixing the root problem
rather than skipping, but we don't need to force this on people as long as
the default is to fail.

One thing I'm confused about is the scope of the proposal. I think the plan
is that this would cover all exceptions that occur whether in serializers
or ANY user code? Is that right? So if I do stream.map(x =>
x.header.timestamp) and that throws a NullPointerException, this would be
triggered? If so what I understand is that what is passed in to me is the
original consumer record, not the value x that produced the null pointer
exception? Is that right? If this understanding is correct then the
name RecordExceptionHandler should maybe be something like
ProcessingExceptionHandler since the exception isn't necessarily directly
tied to an input Record, right?

A couple of other comments:

   - It's important we maintain the original stack trace when we rethrow
   the exception (probably obvious, but thought I'd mention it)
   - As a matter of style I'd advocate for making a single
   DefaultExceptionHandler which logs the error and adding configs for this to
   control when (if ever) it fails. This will allow adding additional useful
   options in a way that can be combined (such as the dead letter thing,
   retries, etc). Basically the point is that these facilities aren't
   "either/or". Also you mention adding configs for these in the existing
   proposal, it'd be good to say what the configs are.
   - I think we should hold off on retries unless we have worked out the
   full usage pattern, people can always implement their own. I think the idea
   is that you send the message to some kind of dead letter queue and then
   replay these later. This obviously destroys all semantic guarantees we are
   working hard to provide right now, which may be okay.
   - I agree that the LogAndThresholdExceptionHandler is closest to what
   most people think they want. I think making the exception handler stateful
   is probably fine since this is inherently an approximate threshold. I do
   think this is a bit more complex then it sounds though since you'll
   obviously need to compute some kind of cheap running rate. Obviously the
   two failure modes you'd need to avoid are that 1/1 failures = 100% OR
   conversely that it runs successfully for one year and then fails 100% of
   the time but that isn't caught because of the excess prior history.

-Jay


On Thu, May 25, 2017 at 2:47 AM, Eno Thereska 
wrote:

> Hi there,
>
> I’ve added a KIP on improving exception handling in streams:
> KIP-161: streams record processing exception handlers.
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 161%3A+streams+record+processing+exception+handlers <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-161:+streams+record+
> processing+exception+handlers>
>
> Discussion and feedback is welcome, thank you.
> Eno

Re: Jira-Spam on Dev-Mailinglist

[Guozhang Wang]

I share your pains. Right now I use filters on my email accounts and it has
been down to about 25 per day.

I think setup a separate mailing list for jirabot and jenkins auto
generated emails is a good idea.


Guozhang


On Mon, May 29, 2017 at 12:58 AM,  wrote:

> Hello everyone
>
> I find it hard to follow this mailinglist due to all the mails generated
> by Jira. Just over this weekend there are 240 new mails.
> Would it be possible to setup something like j...@kafka.apache.org where
> everyone can subscribe interested in those Jira mails?
>
> Right now I am going to setup a filter which just deletes the jira-tagged
> mails, but I think the current setup also makes it hard to read through
> the archives.
>
> regards
> Marc




-- 
-- Guozhang

[GitHub] kafka pull request #3164: KAFKA-5150: Reduce lz4 decompression overhead (wit...

[ijuma]

GitHub user ijuma opened a pull request:

https://github.com/apache/kafka/pull/3164

KAFKA-5150: Reduce lz4 decompression overhead (without thread local buffers)

Temporary PR that has additional changes over 
https://github.com/apache/kafka/pull/2967 for comparison.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijuma/kafka 
kafka-5150-reduce-lz4-decompression-overhead

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3164.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3164


commit 950858e7fae838aecbf31c1ea201c3dbcd67a91d
Author: Xavier Léauté 
Date:   2017-05-03T17:01:07Z

small batch decompression benchmark

commit 0177665f3321e101ccbb2e95ec724125bf784e1c
Author: Xavier Léauté 
Date:   2017-05-03T20:40:45Z

KAFKA-5150 reduce lz4 decompression overhead

- reuse decompression buffers, keeping one per thread
- switch lz4 input stream to operate directly on ByteBuffers
- more tests with both compressible / incompressible data, multiple
  blocks, and various other combinations to increase code coverage
- fixes bug that would cause EOFException instead of invalid block size
  for invalid incompressible blocks

commit 7b553afdd7a6a7d39b122c503cd643b915b9f556
Author: Xavier Léauté 
Date:   2017-05-04T17:26:52Z

remove unnecessary synchronized on reset/mark

commit b4c46ac15aa25e2c8ea6bb9392d700b02b61fccd
Author: Xavier Léauté 
Date:   2017-05-05T16:00:49Z

avoid exception when reaching end of batch

commit 77e1a1d47f9060430257821704045353ec77a8d0
Author: Xavier Léauté 
Date:   2017-05-18T16:47:01Z

remove reflection for LZ4 and add comments

commit e3b68668b6b2e0057bcd9a3de24ab8fce774d8d5
Author: Ismael Juma 
Date:   2017-05-26T15:45:13Z

Simplify DataLogInputStream.nextBatch

commit 213bb77b8a3862a325118492d658a4e58ffd3c29
Author: Ismael Juma 
Date:   2017-05-26T15:56:16Z

Minor comment improvement

commit 9bd10361d70de837ccb58a82b356b402d28bb94f
Author: Ismael Juma 
Date:   2017-05-29T14:18:13Z

Minor tweaks in `DefaultRecord.readFrom`

commit 178d4900a6c848a4f1b0aa0ae68aaa24885f36bc
Author: Ismael Juma 
Date:   2017-05-29T15:22:01Z

Cache decompression buffers in Fetcher instead of thread-locals

This means that this only benefits the consumer for now, which
is the most important case. For the server, we should consider
how this fits with KIP-72.

commit c10b310cc13f5ec110cbaed8fb72f24774c2a2cd
Author: Ismael Juma 
Date:   2017-05-29T15:23:19Z

Tweaks to `KafkaLZ4*Stream` classes and `RecordBatchIterationBenchmark

commit d93444c147430a62f5e9d16492ad14d2c6a0dd38
Author: Ismael Juma 
Date:   2017-05-29T18:18:23Z

Trivial style tweaks to KafkaLZ4Test

commit 419500e848b943f20d9bce1790fe40e64080ae29
Author: Ismael Juma 
Date:   2017-05-29T18:38:55Z

Provide a `NO_CACHING` BufferSupplier




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-5150) LZ4 decompression is 4-5x slower than Snappy on small batches / messages

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16028559#comment-16028559
 ] 

ASF GitHub Bot commented on KAFKA-5150:
---

GitHub user ijuma opened a pull request:

https://github.com/apache/kafka/pull/3164

KAFKA-5150: Reduce lz4 decompression overhead (without thread local buffers)

Temporary PR that has additional changes over 
https://github.com/apache/kafka/pull/2967 for comparison.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijuma/kafka 
kafka-5150-reduce-lz4-decompression-overhead

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3164.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3164


commit 950858e7fae838aecbf31c1ea201c3dbcd67a91d
Author: Xavier Léauté 
Date:   2017-05-03T17:01:07Z

small batch decompression benchmark

commit 0177665f3321e101ccbb2e95ec724125bf784e1c
Author: Xavier Léauté 
Date:   2017-05-03T20:40:45Z

KAFKA-5150 reduce lz4 decompression overhead

- reuse decompression buffers, keeping one per thread
- switch lz4 input stream to operate directly on ByteBuffers
- more tests with both compressible / incompressible data, multiple
  blocks, and various other combinations to increase code coverage
- fixes bug that would cause EOFException instead of invalid block size
  for invalid incompressible blocks

commit 7b553afdd7a6a7d39b122c503cd643b915b9f556
Author: Xavier Léauté 
Date:   2017-05-04T17:26:52Z

remove unnecessary synchronized on reset/mark

commit b4c46ac15aa25e2c8ea6bb9392d700b02b61fccd
Author: Xavier Léauté 
Date:   2017-05-05T16:00:49Z

avoid exception when reaching end of batch

commit 77e1a1d47f9060430257821704045353ec77a8d0
Author: Xavier Léauté 
Date:   2017-05-18T16:47:01Z

remove reflection for LZ4 and add comments

commit e3b68668b6b2e0057bcd9a3de24ab8fce774d8d5
Author: Ismael Juma 
Date:   2017-05-26T15:45:13Z

Simplify DataLogInputStream.nextBatch

commit 213bb77b8a3862a325118492d658a4e58ffd3c29
Author: Ismael Juma 
Date:   2017-05-26T15:56:16Z

Minor comment improvement

commit 9bd10361d70de837ccb58a82b356b402d28bb94f
Author: Ismael Juma 
Date:   2017-05-29T14:18:13Z

Minor tweaks in `DefaultRecord.readFrom`

commit 178d4900a6c848a4f1b0aa0ae68aaa24885f36bc
Author: Ismael Juma 
Date:   2017-05-29T15:22:01Z

Cache decompression buffers in Fetcher instead of thread-locals

This means that this only benefits the consumer for now, which
is the most important case. For the server, we should consider
how this fits with KIP-72.

commit c10b310cc13f5ec110cbaed8fb72f24774c2a2cd
Author: Ismael Juma 
Date:   2017-05-29T15:23:19Z

Tweaks to `KafkaLZ4*Stream` classes and `RecordBatchIterationBenchmark

commit d93444c147430a62f5e9d16492ad14d2c6a0dd38
Author: Ismael Juma 
Date:   2017-05-29T18:18:23Z

Trivial style tweaks to KafkaLZ4Test

commit 419500e848b943f20d9bce1790fe40e64080ae29
Author: Ismael Juma 
Date:   2017-05-29T18:38:55Z

Provide a `NO_CACHING` BufferSupplier




> LZ4 decompression is 4-5x slower than Snappy on small batches / messages
> 
>
> Key: KAFKA-5150
> URL: https://issues.apache.org/jira/browse/KAFKA-5150
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Affects Versions: 0.8.2.2, 0.9.0.1, 0.11.0.0, 0.10.2.1
>Reporter: Xavier Léauté
>Assignee: Xavier Léauté
> Fix For: 0.11.0.0
>
>
> I benchmarked RecordsIteratorDeepRecordsIterator instantiation on small batch 
> sizes with small messages after observing some performance bottlenecks in the 
> consumer. 
> For batch sizes of 1 with messages of 100 bytes, LZ4 heavily underperforms 
> compared to Snappy (see benchmark below). Most of our time is currently spent 
> allocating memory blocks in KafkaLZ4BlockInputStream, due to the fact that we 
> default to larger 64kB block sizes. Some quick testing shows we could improve 
> performance by almost an order of magnitude for small batches and messages if 
> we re-used buffers between instantiations of the input stream.
> [Benchmark 
> Code|https://github.com/xvrl/kafka/blob/small-batch-lz4-benchmark/clients/src/test/java/org/apache/kafka/common/record/DeepRecordsIteratorBenchmark.java#L86]
> {code}
> Benchmark  (compressionType)  
> (messageSize)   Mode  Cnt   Score   Error  Units
> DeepRecordsIteratorBenchmark.measureSingleMessageLZ4  
>   100  thrpt   20   84802.279 ±  1983.847  ops/s
> DeepRecordsIteratorBenchmark.measureSingleMessage

Re: Jira-Spam on Dev-Mailinglist

[Gwen Shapira]

I agree.

Guozhang, do you know how to implement the suggestion? JIRA to Apache
Infra? Or is this something we can do ourselves somehow?

On Mon, May 29, 2017 at 9:33 PM Guozhang Wang  wrote:

> I share your pains. Right now I use filters on my email accounts and it has
> been down to about 25 per day.
>
> I think setup a separate mailing list for jirabot and jenkins auto
> generated emails is a good idea.
>
>
> Guozhang
>
>
> On Mon, May 29, 2017 at 12:58 AM,  wrote:
>
> > Hello everyone
> >
> > I find it hard to follow this mailinglist due to all the mails generated
> > by Jira. Just over this weekend there are 240 new mails.
> > Would it be possible to setup something like j...@kafka.apache.org where
> > everyone can subscribe interested in those Jira mails?
> >
> > Right now I am going to setup a filter which just deletes the jira-tagged
> > mails, but I think the current setup also makes it hard to read through
> > the archives.
> >
> > regards
> > Marc
>
>
>
>
> --
> -- Guozhang
>

Re: [DISCUSS] KIP-148: Add a connect timeout for client

[Guozhang Wang]

On Wed, May 24, 2017 at 9:59 AM, Colin McCabe  wrote:

> On Tue, May 23, 2017, at 19:07, Guozhang Wang wrote:
> > I think using a single config to cover end-to-end latency with connecting
> > and request round-trip may not be best appropriate since 1) some request
> > may need much more time than others since they are parked (fetch request
> > with long polling, join group request etc) or throttled,
>
> Hmm.  My proposal was to implement _both_ end-to-end timeouts and
> per-call timeouts.  In that case, some requests needing much more time
> than others should not be a concern, since we can simply set a higher
> per-call timeout on the requests we think will need more time.
>
> > and 2) some
> > requests are prerequisite of others, like group request to discover the
> > coordinator before the fetch offset request, and implementation wise
> > these
> > request send/receive is embedded in latter ones, hence it is not clear if
> > the `request.timeout.ms` should cover just a single RPC or more.
>
> As far as I know, the request timeout has always covered a single RP  If
> we want to implement a higher level timeout that spans multiple RPCs, we
> can set the per-call timeouts appropriately.  For example:
>
> > long deadline = System.currentTimeMillis() + 6;
> > callA(callTimeout = deadline - System.currentTimeMillis())
> > callB(callTimeout = deadline - System.currentTimeMillis())
>
>
I may have misunderstand your previous email. Just clarifying:

1) On the client we already have some configs for controlling end-to-end
timeout, e.g. "max.block.ms" on producer controls how long "send()" and
"partitionsFor()" will block for, and inside such API calls multiple
request round trips may be sent, and for the first request round trip, a
connecting phase may or may not be included. All of these are be covered in
this "max.block.ms" timeout today. However, as we discussed before not all
request round trips have similar latency expectation, so it is better to
make a per-request "request.timeout.ms" and the overall "max.block.ms"
would need to be at least the max of them.

2) Now back to the question whether we should make "request.timeout.ms"
include potential connection phase as well: assume we are going to add the
pre-request "request.timeout.ms" as suggested above, then we may still have
a tight bound on how long connecting should take. For example, let's say we
make "joingroup.request.timeout.ms" (or "fetch.request.timeout.ms" to be
large since we want really long polling behavior) to be a large value, say
200 seconds, then if the client is trying to connect to the broker while
sending the request, and the broker has died, then we may still be blocked
waiting for 30 seconds while I think David's motivation is to fail-fast in
these cases.


> >
> > So no matter whether we add a `connect.timeout.ms` in addition to `
> > request.timeout.ms`, we should consider adding per-request-type timeout
> > value, and make `request.timeout.ms` a global default; if we add the `
> > connect.timeout.ms` the per-request value is only for the round trip,
> > otherwise it is supposed to include the connecting time. Personally I'd
> > prefer the first option to add a universal `connect.timeout.ms`, and in
> > another KIP consider adding per-request-type timeout overrides.
>
> Why have a special case for time spent connecting, though?  Why would
> the user care where the time went, as long as the timeout was met?  It
> feels like this is just a hack because we couldn't raise
> request.timeout.ms to the value that it "should" have been at for the
> shorter requests.  As someone already commented, it's confusing to have
> all these knobs that we don't really need.
>
>
I think that is exactly what David cares (please correct me if I'm wrong):
for some request I would like to wait long enough for it to be completed,
like join-group request; while at the same time if it has encountered some
issues while trying to connect to the broker to send the join group
request, I want to be notified sooner.


> >
> > BTW if the consumer issue is the only cause that we are having a high
> > default value, I'd suggest we separate the consumer rebalance timeout and
> > not piggy-back on the session timeout. Then we can set the default `
> > request.timeout.ms` to a smaller value, like 10 secs. This is orthogonal
> > to
> > this KIP discussion and we can continue this in a separate thread.
>
> +1
>
> cheers,
> Colin
>
> >
> >
> > Guozhang
> >
> > On Tue, May 23, 2017 at 3:31 PM, Colin McCabe 
> wrote:
> >
> > > Another note-- it would be really nice if timeouts were end-to-end,
> > > rather than being set for particular phases of an RP  From a user point
> > > of view, a 30 second timeout should mean that the call either succeeds
> > > or fails after 30 seconds, regardless of how much time is spent looking
> > > for metadata, connecting to brokers, waiting for brokers, etc.  This is
> > > implemented in AdminClient by setting a deadline when the call is fi

[jira] [Assigned] (KAFKA-5338) There is a Misspell in ResetIntegrationTest

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang reassigned KAFKA-5338:


Assignee: hejiefang

> There is a Misspell in ResetIntegrationTest
> ---
>
> Key: KAFKA-5338
> URL: https://issues.apache.org/jira/browse/KAFKA-5338
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.2.1
>Reporter: hejiefang
>Assignee: hejiefang
> Fix For: 0.11.0.0
>
>
> There is a Misspell in Annotations of ResetIntegrationTest.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (KAFKA-5338) There is a Misspell in ResetIntegrationTest

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-5338:
-
Resolution: Fixed
Status: Resolved  (was: Patch Available)

Issue resolved by pull request 3159
[https://github.com/apache/kafka/pull/3159]

> There is a Misspell in ResetIntegrationTest
> ---
>
> Key: KAFKA-5338
> URL: https://issues.apache.org/jira/browse/KAFKA-5338
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.2.1
>Reporter: hejiefang
>Assignee: hejiefang
> Fix For: 0.11.0.0
>
>
> There is a Misspell in Annotations of ResetIntegrationTest.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #3159: [KAFKA-5338]There is a Misspell in ResetIntegratio...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3159


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-5338) There is a Misspell in ResetIntegrationTest

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16028653#comment-16028653
 ] 

ASF GitHub Bot commented on KAFKA-5338:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3159


> There is a Misspell in ResetIntegrationTest
> ---
>
> Key: KAFKA-5338
> URL: https://issues.apache.org/jira/browse/KAFKA-5338
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.2.1
>Reporter: hejiefang
>Assignee: hejiefang
> Fix For: 0.11.0.0
>
>
> There is a Misspell in Annotations of ResetIntegrationTest.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[DISCUSS] KIP-165: Extend Interactive Queries for return latest update timestamp per key

[Jeyhun Karimov]

Dear community,

I want to share KIP-165 [1] based on issue KAFKA-4304 [2].
I would like to get your comments.

[1]
https://cwiki.apache.org/confluence/display/KAFKA/KIP-165%3A+Extend+Interactive+Queries+for+return+latest+update+timestamp+per+key
[2] https://issues.apache.org/jira/browse/KAFKA-4304

Cheers,
Jeyhun

-- 
-Cheers

Jeyhun

RE?? [DISCUSS] KIP-148: Add a connect timeout for client

[????????]

>I think that is exactly what David cares (please correct me if I'm wrong):
>for some request I would like to wait long enough for it to be completed,
>like join-group request; while at the same time if it has encountered some
>issues while trying to connect to the broker to send the join group
>request, I want to be notified sooner.


+1


I will start a vote  in a few days if we all reach the consensus.


Thanks,
David


--  --
??: "Guozhang Wang";;
: 2017??5??30??(??) 6:46
??: "dev@kafka.apache.org"; 

: Re: [DISCUSS] KIP-148: Add a connect timeout for client



On Wed, May 24, 2017 at 9:59 AM, Colin McCabe  wrote:

> On Tue, May 23, 2017, at 19:07, Guozhang Wang wrote:
> > I think using a single config to cover end-to-end latency with connecting
> > and request round-trip may not be best appropriate since 1) some request
> > may need much more time than others since they are parked (fetch request
> > with long polling, join group request etc) or throttled,
>
> Hmm.  My proposal was to implement _both_ end-to-end timeouts and
> per-call timeouts.  In that case, some requests needing much more time
> than others should not be a concern, since we can simply set a higher
> per-call timeout on the requests we think will need more time.
>
> > and 2) some
> > requests are prerequisite of others, like group request to discover the
> > coordinator before the fetch offset request, and implementation wise
> > these
> > request send/receive is embedded in latter ones, hence it is not clear if
> > the `request.timeout.ms` should cover just a single RPC or more.
>
> As far as I know, the request timeout has always covered a single RP  If
> we want to implement a higher level timeout that spans multiple RPCs, we
> can set the per-call timeouts appropriately.  For example:
>
> > long deadline = System.currentTimeMillis() + 6;
> > callA(callTimeout = deadline - System.currentTimeMillis())
> > callB(callTimeout = deadline - System.currentTimeMillis())
>
>
I may have misunderstand your previous email. Just clarifying:

1) On the client we already have some configs for controlling end-to-end
timeout, e.g. "max.block.ms" on producer controls how long "send()" and
"partitionsFor()" will block for, and inside such API calls multiple
request round trips may be sent, and for the first request round trip, a
connecting phase may or may not be included. All of these are be covered in
this "max.block.ms" timeout today. However, as we discussed before not all
request round trips have similar latency expectation, so it is better to
make a per-request "request.timeout.ms" and the overall "max.block.ms"
would need to be at least the max of them.

2) Now back to the question whether we should make "request.timeout.ms"
include potential connection phase as well: assume we are going to add the
pre-request "request.timeout.ms" as suggested above, then we may still have
a tight bound on how long connecting should take. For example, let's say we
make "joingroup.request.timeout.ms" (or "fetch.request.timeout.ms" to be
large since we want really long polling behavior) to be a large value, say
200 seconds, then if the client is trying to connect to the broker while
sending the request, and the broker has died, then we may still be blocked
waiting for 30 seconds while I think David's motivation is to fail-fast in
these cases.


> >
> > So no matter whether we add a `connect.timeout.ms` in addition to `
> > request.timeout.ms`, we should consider adding per-request-type timeout
> > value, and make `request.timeout.ms` a global default; if we add the `
> > connect.timeout.ms` the per-request value is only for the round trip,
> > otherwise it is supposed to include the connecting time. Personally I'd
> > prefer the first option to add a universal `connect.timeout.ms`, and in
> > another KIP consider adding per-request-type timeout overrides.
>
> Why have a special case for time spent connecting, though?  Why would
> the user care where the time went, as long as the timeout was met?  It
> feels like this is just a hack because we couldn't raise
> request.timeout.ms to the value that it "should" have been at for the
> shorter requests.  As someone already commented, it's confusing to have
> all these knobs that we don't really need.
>
>
I think that is exactly what David cares (please correct me if I'm wrong):
for some request I would like to wait long enough for it to be completed,
like join-group request; while at the same time if it has encountered some
issues while trying to connect to the broker to send the join group
request, I want to be notified sooner.


> >
> > BTW if the consumer issue is the only cause that we are having a high
> > default value, I'd suggest we separate the consumer rebalance timeout and
> > not piggy-back on the session timeout. Then we can set the default `
> > request.timeout.ms` to a smaller value, like 10 secs. This is orthogonal
> > to
> > this

[GitHub] kafka pull request #3165: KAFKA-5316: LogCleaner should account for larger r...

[hachikuji]

GitHub user hachikuji opened a pull request:

https://github.com/apache/kafka/pull/3165

KAFKA-5316: LogCleaner should account for larger record sets after cleaning



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hachikuji/kafka KAFKA-5316-0.10.2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3165.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3165


commit 98539df716ca866f06a708d7556ad39ca4e1caae
Author: Jason Gustafson 
Date:   2017-05-30T05:49:33Z

KAFKA-5316: LogCleaner should account for larger record sets after cleaning




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-5316) Log cleaning can increase message size and cause cleaner to crash with buffer overflow

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16028760#comment-16028760
 ] 

ASF GitHub Bot commented on KAFKA-5316:
---

GitHub user hachikuji opened a pull request:

https://github.com/apache/kafka/pull/3165

KAFKA-5316: LogCleaner should account for larger record sets after cleaning



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hachikuji/kafka KAFKA-5316-0.10.2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3165.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3165


commit 98539df716ca866f06a708d7556ad39ca4e1caae
Author: Jason Gustafson 
Date:   2017-05-30T05:49:33Z

KAFKA-5316: LogCleaner should account for larger record sets after cleaning




> Log cleaning can increase message size and cause cleaner to crash with buffer 
> overflow
> --
>
> Key: KAFKA-5316
> URL: https://issues.apache.org/jira/browse/KAFKA-5316
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
> Fix For: 0.11.0.0
>
>
> We have observed in practice that it is possible for a compressed record set 
> to grow after cleaning. Since the size of the cleaner's input and output 
> buffers are identical, this can lead to overflow of the output buffer:
> {code}
> [2017-05-23 15:05:15,480] ERROR [kafka-log-cleaner-thread-0], Error due to  
> (kafka.log.LogCleaner)
> java.nio.BufferOverflowException
> at java.nio.HeapByteBuffer.put(HeapByteBuffer.java:206)
> at org.apache.kafka.common.record.LogEntry.writeTo(LogEntry.java:104)
> at 
> org.apache.kafka.common.record.MemoryRecords.filterTo(MemoryRecords.java:163)
> at 
> org.apache.kafka.common.record.MemoryRecords.filterTo(MemoryRecords.java:114)
> at kafka.log.Cleaner.cleanInto(LogCleaner.scala:468)
> at 
> kafka.log.Cleaner$$anonfun$cleanSegments$1.apply(LogCleaner.scala:405)
> at 
> kafka.log.Cleaner$$anonfun$cleanSegments$1.apply(LogCleaner.scala:401)
> at scala.collection.immutable.List.foreach(List.scala:318)
> at kafka.log.Cleaner.cleanSegments(LogCleaner.scala:401)
> at kafka.log.Cleaner$$anonfun$clean$4.apply(LogCleaner.scala:363)
> at kafka.log.Cleaner$$anonfun$clean$4.apply(LogCleaner.scala:362)
> at scala.collection.immutable.List.foreach(List.scala:318)
> at kafka.log.Cleaner.clean(LogCleaner.scala:362)
> at 
> kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:241)
> at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:220)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
> [2017-05-23 15:05:15,481] INFO [kafka-log-cleaner-thread-0], Stopped  
> (kafka.log.LogCleaner)
> {code}
> It is also then possible for a compressed message set to grow beyond the max 
> message size. Due to the changes in KIP-74 to alter fetch semantics, the 
> suggestion for this case is to allow the recompressed message set to exceed 
> the max message size. This should be rare in practice and won't prevent 
> consumers from making progress.
> To handle the overflow issue, one option is to allocate a temporary buffer 
> when filtering in {{MemoryRecords.filterTo}} and return it in the result. As 
> an optimization, we can resort to this only when there is a single 
> recompressed message set which is larger than the entire write buffer. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #3126: KAFKA-5260: Producer should not send AbortTxn unle...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3126


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-5260) Producer should not send AbortTxn unless transaction has actually begun

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16028764#comment-16028764
 ] 

ASF GitHub Bot commented on KAFKA-5260:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3126


> Producer should not send AbortTxn unless transaction has actually begun
> ---
>
> Key: KAFKA-5260
> URL: https://issues.apache.org/jira/browse/KAFKA-5260
> Project: Kafka
>  Issue Type: Sub-task
>  Components: clients, core, producer 
>Reporter: Jason Gustafson
>Assignee: Damian Guy
>Priority: Blocker
>  Labels: exactly-once
> Fix For: 0.11.0.0
>
>
> When there is an authorization error in AddOffsets or AddPartitions, the 
> producer will raise an authorization exception. When that happens, the user 
> should abort the transaction. The problem is that in an authorization error, 
> the coordinator will not have transitioned to a new state, so if it suddenly 
> receives an AbortTxnRequest, that request will fail with an InvalidTxnState, 
> which will be propagated to the error. The suggested solution is to keep 
> track locally when we are certain that no transaction has been officially 
> begun and to skip sending the AbortTxnRequest in that case.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (KAFKA-5260) Producer should not send AbortTxn unless transaction has actually begun

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson updated KAFKA-5260:
---
Resolution: Fixed
Status: Resolved  (was: Patch Available)

Issue resolved by pull request 3126
[https://github.com/apache/kafka/pull/3126]

> Producer should not send AbortTxn unless transaction has actually begun
> ---
>
> Key: KAFKA-5260
> URL: https://issues.apache.org/jira/browse/KAFKA-5260
> Project: Kafka
>  Issue Type: Sub-task
>  Components: clients, core, producer 
>Reporter: Jason Gustafson
>Assignee: Damian Guy
>Priority: Blocker
>  Labels: exactly-once
> Fix For: 0.11.0.0
>
>
> When there is an authorization error in AddOffsets or AddPartitions, the 
> producer will raise an authorization exception. When that happens, the user 
> should abort the transaction. The problem is that in an authorization error, 
> the coordinator will not have transitioned to a new state, so if it suddenly 
> receives an AbortTxnRequest, that request will fail with an InvalidTxnState, 
> which will be propagated to the error. The suggested solution is to keep 
> track locally when we are certain that no transaction has been officially 
> begun and to skip sending the AbortTxnRequest in that case.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: kafka-trunk-jdk7 #2282

[Apache Jenkins Server]

See 


Changes:

[jason] KAFKA-5260; Producer should not send AbortTxn unless transaction has

--
[...truncated 904.45 KB...]

kafka.server.DelayedOperationTest > testRequestPurge PASSED

kafka.server.DelayedOperationTest > testRequestExpiry STARTED

kafka.server.DelayedOperationTest > testRequestExpiry PASSED

kafka.server.DelayedOperationTest > 
shouldReturnNilOperationsOnCancelForKeyWhenKeyDoesntExist STARTED

kafka.server.DelayedOperationTest > 
shouldReturnNilOperationsOnCancelForKeyWhenKeyDoesntExist PASSED

kafka.server.DelayedOperationTest > 
shouldCancelForKeyReturningCancelledOperations STARTED

kafka.server.DelayedOperationTest > 
shouldCancelForKeyReturningCancelledOperations PASSED

kafka.server.DelayedOperationTest > testRequestSatisfaction STARTED

kafka.server.DelayedOperationTest > testRequestSatisfaction PASSED

kafka.server.MultipleListenersWithDefaultJaasContextTest > testProduceConsume 
STARTED

kafka.server.MultipleListenersWithDefaultJaasContextTest > testProduceConsume 
PASSED

kafka.server.ThrottledResponseExpirationTest > testThrottledRequest STARTED

kafka.server.ThrottledResponseExpirationTest > testThrottledRequest PASSED

kafka.server.ThrottledResponseExpirationTest > testExpire STARTED

kafka.server.ThrottledResponseExpirationTest > testExpire PASSED

kafka.server.ReplicaManagerQuotasTest > shouldGetBothMessagesIfQuotasAllow 
STARTED

kafka.server.ReplicaManagerQuotasTest > shouldGetBothMessagesIfQuotasAllow 
PASSED

kafka.server.ReplicaManagerQuotasTest > 
shouldExcludeSubsequentThrottledPartitions STARTED

kafka.server.ReplicaManagerQuotasTest > 
shouldExcludeSubsequentThrottledPartitions PASSED

kafka.server.ReplicaManagerQuotasTest > 
shouldGetNoMessagesIfQuotasExceededOnSubsequentPartitions STARTED

kafka.server.ReplicaManagerQuotasTest > 
shouldGetNoMessagesIfQuotasExceededOnSubsequentPartitions PASSED

kafka.server.ReplicaManagerQuotasTest > shouldIncludeInSyncThrottledReplicas 
STARTED

kafka.server.ReplicaManagerQuotasTest > shouldIncludeInSyncThrottledReplicas 
PASSED

kafka.server.SaslApiVersionsRequestTest > 
testApiVersionsRequestWithUnsupportedVersion STARTED

kafka.server.SaslApiVersionsRequestTest > 
testApiVersionsRequestWithUnsupportedVersion PASSED

kafka.server.SaslApiVersionsRequestTest > 
testApiVersionsRequestBeforeSaslHandshakeRequest STARTED

kafka.server.SaslApiVersionsRequestTest > 
testApiVersionsRequestBeforeSaslHandshakeRequest PASSED

kafka.server.SaslApiVersionsRequestTest > 
testApiVersionsRequestAfterSaslHandshakeRequest STARTED

kafka.server.SaslApiVersionsRequestTest > 
testApiVersionsRequestAfterSaslHandshakeRequest PASSED

kafka.server.ControlledShutdownLeaderSelectorTest > testSelectLeader STARTED

kafka.server.ControlledShutdownLeaderSelectorTest > testSelectLeader PASSED

kafka.server.MetadataCacheTest > 
getTopicMetadataWithNonSupportedSecurityProtocol STARTED

kafka.server.MetadataCacheTest > 
getTopicMetadataWithNonSupportedSecurityProtocol PASSED

kafka.server.MetadataCacheTest > getTopicMetadataIsrNotAvailable STARTED

kafka.server.MetadataCacheTest > getTopicMetadataIsrNotAvailable PASSED

kafka.server.MetadataCacheTest > getTopicMetadata STARTED

kafka.server.MetadataCacheTest > getTopicMetadata PASSED

kafka.server.MetadataCacheTest > getTopicMetadataReplicaNotAvailable STARTED

kafka.server.MetadataCacheTest > getTopicMetadataReplicaNotAvailable PASSED

kafka.server.MetadataCacheTest > getTopicMetadataPartitionLeaderNotAvailable 
STARTED

kafka.server.MetadataCacheTest > getTopicMetadataPartitionLeaderNotAvailable 
PASSED

kafka.server.MetadataCacheTest > getAliveBrokersShouldNotBeMutatedByUpdateCache 
STARTED

kafka.server.MetadataCacheTest > getAliveBrokersShouldNotBeMutatedByUpdateCache 
PASSED

kafka.server.MetadataCacheTest > getTopicMetadataNonExistingTopics STARTED

kafka.server.MetadataCacheTest > getTopicMetadataNonExistingTopics PASSED

kafka.server.FetchRequestTest > testBrokerRespectsPartitionsOrderAndSizeLimits 
STARTED

kafka.server.FetchRequestTest > testBrokerRespectsPartitionsOrderAndSizeLimits 
PASSED

kafka.server.FetchRequestTest > testFetchRequestV2WithOversizedMessage STARTED

kafka.server.FetchRequestTest > testFetchRequestV2WithOversizedMessage PASSED

kafka.server.checkpoints.OffsetCheckpointFileTest > 
shouldReturnEmptyMapForEmptyFile STARTED

kafka.server.checkpoints.OffsetCheckpointFileTest > 
shouldReturnEmptyMapForEmptyFile PASSED

kafka.server.checkpoints.OffsetCheckpointFileTest > 
shouldThrowIfVersionIsNotRecognised STARTED

kafka.server.checkpoints.OffsetCheckpointFileTest > 
shouldThrowIfVersionIsNotRecognised PASSED

kafka.server.checkpoints.OffsetCheckpointFileTest > 
shouldPersistAndOverwriteAndReloadFile STARTED

kafka.server.checkpoints.OffsetCheckpointFileTest > 
shouldPersistAndOverwriteAndReloadFile PASSED

kafka.server.checkpoints.OffsetCheckpointFileTest > shoul