Build failed in Jenkins: kafka-trunk-jdk8 #3356

2019-01-31 Thread Apache Jenkins Server 
See 


Changes:

[github] KAFKA-7652: Part II; Add single-point query for SessionStore and use 
for

--
[...truncated 4.56 MB...]

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfTimestampIsDifferentForCompareValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfTimestampIsDifferentForCompareValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualForCompareKeyValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualForCompareKeyValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullExpectedRecordForCompareKeyValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullExpectedRecordForCompareKeyValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareValueTimestampWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareValueTimestampWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullExpectedRecordForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullExpectedRecordForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareValueWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareValueWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueIsEqualForCompareValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueIsEqualForCompareValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordWithExpectedRecordForCompareKeyValueTimestamp 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordWithExpectedRecordForCompareKeyValueTimestamp 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentForCompareKeyValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentForCompareKeyValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordWithExpectedRecordForCompareKeyValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordWithExpectedRecordForCompareKeyValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueIsEqualWithNullForCompareValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueIsEqualWithNullForCompareValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualWithNullForCompareValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualWithNullForCompareValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.Ou

Re: [DISCUSS] KIP-416: Notify SourceTask of ACK'd offsets, metadata

2019-01-31 Thread Andrew Schofield 
As you might expect, I like the overloaded commitRecord() but I think the 
overloaded method should be called in exactly the same situations as the 
previous method. When it does not reflect an ACK, the second parameter could be 
null. The text of the KIP says that the overloaded method is only called when a 
record is ACKed and I would have thought that the connector implementor would 
want to provide only a single variant of commitRecord().

Andrew Schofield
IBM Event Streams

On 31/01/2019, 03:00, "Ryanne Dolan"  wrote:

I've updated the KIP and PR to overload commitRecord instead of adding a
new method. Here's the PR:


https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fkafka%2Fpull%2F6171&data=02%7C01%7C%7Cc627d954fa6f44574f7908d6872838c5%7C84df9e7fe9f640afb435%7C1%7C0%7C636845004151935856&sdata=hxBWSTt5gF7AAVxw2P8%2BZ8duBB0T97gHOOYG6GCkdd8%3D&reserved=0

Ryanne

On Mon, Jan 21, 2019 at 6:29 PM Ryanne Dolan  wrote:

> Andrew Schofield suggested we overload the commitRecord method instead of
> adding a new one. Thoughts?
>
> Ryanne
>
> On Thu, Jan 17, 2019, 5:34 PM Ryanne Dolan 
>> I had to change the KIP number (concurrency is hard!) so the link is now:
>>
>>
>> 
https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-416%253A%2BNotify%2BSourceTask%2Bof%2BACK%2527d%2Boffsets%252C%2Bmetadata&data=02%7C01%7C%7Cc627d954fa6f44574f7908d6872838c5%7C84df9e7fe9f640afb435%7C1%7C0%7C636845004151935856&sdata=VkAFrM8B2ozCRJosPQjgM3aDD1cS%2Bob8KWVuNuuOJ9s%3D&reserved=0
>>
>> Ryanne
>>
>> On Fri, Jan 11, 2019 at 2:43 PM Ryanne Dolan 
>> wrote:
>>
>>> Hey y'all,
>>>
>>> Please review the following small KIP:
>>>
>>>
>>> 
https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-414%253A%2BNotify%2BSourceTask%2Bof%2BACK%2527d%2Boffsets%252C%2Bmetadata&data=02%7C01%7C%7Cc627d954fa6f44574f7908d6872838c5%7C84df9e7fe9f640afb435%7C1%7C0%7C636845004151945855&sdata=2mhXA4hEV3ZvrFaOcTqagO1rYNj1JsYAEDHQsFqkzG8%3D&reserved=0
>>>
>>> Thanks!
>>> Ryanne
>>>
>>

Re: [DISCUSS] KIP-390: Add producer option to adjust compression level

2019-01-31 Thread Dongjin Lee 
Mickael,

It seems like the majority of the community agrees that the new config
scheme proposed by Becket is much better. However, we still need another
KIP to support this kind of config; it is different from the case of
`listener.security.protocol.map,` which is just a concatenation of
available settings (i.e., string type.) For Becket's idea, we need to add
support for the key-value style.

It seems like we need advice from a committer, whether to add a new
prerequisite KIP for the map type configuration to implement Becket's idea.

Committers:

Could you give us some advice on this problem?

Best,
Dongjin

On Wed, Jan 30, 2019 at 11:30 PM Mickael Maison 
wrote:

> Thanks, that's a very interesting KIP!
>
> I agree with Becket that a clearer config format is likely to help.
> Have you considered using a config format similar to listeners (as
> described in the "Updating SSL Keystore of an Existing Listener" in
> the Kafka docs)?
>
> Also worth noting that we already have configs that effectively take a
> map, for example: listener.security.protocol.map so not sure if we
> need an additional KIP
>
> On Wed, Jan 30, 2019 at 5:22 AM Dongjin Lee  wrote:
> >
> > Hello.
> >
> > Do you have any idea on Becket's Idea of new config format (example
> below)?
> >
> > ```
> > compression.config="gzip.compression.level=5, lz4.compression.level=17,
> > zstd.compression.level=22"
> > ```
> >
> > It requires some additional KIP for supporting new config format (map),
> but
> > it can significantly simplify the configuration with flexibility and
> > extensibility. If you prefer this way, I hope to carry the ball.
> >
> > If not, please give me an opinion here or the voting thread.
> >
> > Thanks,
> > Dongjin
> >
> >
> > On Fri, Jan 25, 2019 at 1:25 AM Dongjin Lee  wrote:
> >
> > > Hi Becket,
> > >
> > > Thank you for your opinion. Frankly, I have no strong opinion on
> > > configuration name. In this problem, I will follow the community's
> choice.
> > > (I like your idea in that it has a notion of 'scope' per compression
> codec.
> > > However, it should be implemented on top of new config type like Map;
> It
> > > will require another KIP as a prerequisite, but if the community prefer
> > > this way, I will take the task.)
> > >
> > > (One minor correction: the one who thought 'producer' compression
> config
> > > would cause a problem at broker was me, not Ismael - and Ismael
> reassured
> > > me there will be no problem with it.)
> > >
> > > To All,
> > >
> > > How about Becket's idea of 'compression.config' option?
> > >
> > > Best,
> > > Dongjin
> > >
> > > On Wed, Jan 23, 2019 at 1:16 PM Becket Qin 
> wrote:
> > >
> > >> Hi Dongjin,
> > >>
> > >> Thanks for the KIP and sorry for being a bit late on the discussion.
> > >>
> > >> It makes sense to expose the configuration for compression types. But
> I am
> > >> wondering if there is a better way to do that than what proposed in
> the
> > >> KIP. What I feel confusing is that we are effectively sharing the
> > >> configuration across different compression types, the meaning of the
> > >> configuration are actually kind of different depending on the
> compression
> > >> type. This will end up with issues like what Ismael has brought up
> > >> earlier.
> > >> Say if the broker has compression type of producer (this may result in
> > >> mixed compression type in the same topic), and for some reason the
> broker
> > >> needs to re-compress the topic (e.g. due to log compaction), a single
> > >> topic
> > >> level compression config may not work, because a valid compression
> level
> > >> for lz4 maybe invalid for gzip.
> > >>
> > >> One alternative I am thinking is to provide a "compression.config"
> > >> configuration, inside which it specifies configuration used by each
> > >> specific compression type as k-v pairs. The format could use some name
> > >> space as well. For example,
> > >>
> > >>
> > >>
> compression.config="gzip.compression.level=5,lz4.compression.level=17,zstd.compression.level=22".
> > >>
> > >> Each compression type will just pick whatever configuration they need
> from
> > >> the k-v pairs defined in this config.
> > >>
> > >> Besides clarity, some other benefits are:
> > >> 1. Extensibility, we don't have to add more configuration when we add
> new
> > >> compression types, or expose new config for a particular compression
> type.
> > >> 2. Even for the same config, different compression type may have
> different
> > >> terminologies. With the approach we can honor those terminologies
> instead
> > >> of shoehorning them into the same configuration name.
> > >>
> > >> What do you think?
> > >>
> > >> Thanks,
> > >>
> > >> Jiangjie (Becket) Qin
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> On Wed, Jan 23, 2019 at 12:07 AM Dongjin Lee 
> wrote:
> > >>
> > >> > Hello. I just fixed the draft implementation, with rebasing onto the
> > >> latest
> > >> > trunk. The KIP was also restored.
> > >> >
> > >> > Please have a look, and if there is no major probl

Quotas in Kafka

2019-01-31 Thread Angelo Xavier Rici 
Hello, I'm testing the Quotas in Kafka, but the results are not coherent
I'm configuring the parameters "producer_byte_rate and consumer_byte_rate"

For example:

I configured 10 messages with 100 kb
"./bin/kafka-producer-perf-test.sh --topic first_topic --num-records 10
--record-size 9 --throughput 50 --producer-props client.id =
client1 bootstrap.servers = localhost:9092"

Limitation: and the 50 kb limitation
"./bin/kafka-configs.sh --zookeeper localhost: 2181 --alter --add-config
'producer_byte_rate = 4, consumer_byte_rate = 4' --entity-type
clients --entity-name client1"

The result:
10 records sent, 1.0 records / sec (0.09 MB / sec), 1705.3 ms avg latency,
10074.0 max latency.
10 records sent, 0.959693 records / sec (0.09 MB / sec), 1705.30 ms avg
latency, 10074 ms max latency, 391 ms 50th, 10074 ms 95th, 10074 ms 99th,
10074 ms 99.9th.

How to send a message per second if the limit is half of each shipment?


Thanks


Angelo Xavier Rici

[jira] [Created] (KAFKA-7886) Some partitions are fully truncated during recovery when log.message.format = 0.10.2 & inter.broker.protocol >= 0.11

2019-01-31 Thread JIRA 
Hervé RIVIERE created KAFKA-7886:


 Summary: Some partitions are fully truncated during recovery when 
log.message.format = 0.10.2 & inter.broker.protocol >= 0.11
 Key: KAFKA-7886
 URL: https://issues.apache.org/jira/browse/KAFKA-7886
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 2.1.0, 2.0.1, 0.11.0.0
 Environment: centos 7 
Reporter: Hervé RIVIERE
 Attachments: broker.log

On a cluster of Kafka 2.0.1, and brokers configured with 
* inter.broker.protocol.format = 2.0
* log.message.format.version = 0.10.2

 

In such configuration, when a broker is restarted (clean shutdown), the 
recovery process, for some partitions, is not taking in account the high 
watermark and is truncating and re-downloading the full partition.


Typically for brokers with 500 partitions each / 5 TB of disk usage the 
recovery process with this configuration is during up to 1 hour whereas it 
usually takes less than 10 min in the same broker when 
(inter.broker.protocol.format = log.message.format.version)
Partitions redownloaded seems not predictable : after several restart of the 
same broker, partitions redownloaded are now always the same.


Broker log filter for one specific partition that was redownloaded ( the 
truncate offset : 12878451349 is corresponding to the log-start-offset) :

 
{code:java}
2019-01-31 09:23:34,703 INFO [ProducerStateManager partition=my_topic-11] 
Writing producer snapshot at offset 13132373966 (kafka.log.ProducerStateManager)
2019-01-31 09:25:15,245 INFO [Log partition=my_topic-11, dir=/var/lib/kafka] 
Loading producer state till offset 13132373966 with message format version 1 
(kafka.log.Log)
2019-01-31 09:25:15,245 INFO [ProducerStateManager partition=my_topic-11] 
Writing producer snapshot at offset 13130789408 (kafka.log.ProducerStateManager)
2019-01-31 09:25:15,249 INFO [ProducerStateManager partition=my_topic-11] 
Writing producer snapshot at offset 13131829288 (kafka.log.ProducerStateManager)
2019-01-31 09:25:15,388 INFO [ProducerStateManager partition=my_topic-11] 
Writing producer snapshot at offset 13132373966 (kafka.log.ProducerStateManager)

2019-01-31 09:25:15,388 INFO [Log partition=my_topic-11, dir=/var/lib/kafka] 
Completed load of log with 243 segments, log start offset 12878451349 and log 
end offset 13132373966 in 46273 ms (kafka.log.Log)

2019-01-31 09:28:38,226 INFO Replica loaded for partition my_topic-11 with 
initial high watermark 13132373966 (kafka.cluster.Replica)
2019-01-31 09:28:38,226 INFO Replica loaded for partition my_topic-11 with 
initial high watermark 0 (kafka.cluster.Replica)
2019-01-31 09:28:38,226 INFO Replica loaded for partition my_topic-11 with 
initial high watermark 0 (kafka.cluster.Replica)
2019-01-31 09:28:42,132 INFO The cleaning for partition my_topic-11 is aborted 
and paused (kafka.log.LogCleaner)

2019-01-31 09:28:42,133 INFO [Log partition=my_topic-11, dir=/var/lib/kafka] 
Truncating to offset 12878451349 (kafka.log.Log)

2019-01-31 09:28:42,135 INFO [Log partition=my_topic-11, dir=/var/lib/kafka] 
Scheduling log segment [baseOffset 12879521312, size 536869342] for deletion. 
(kafka.log.Log)
(...)
2019-01-31 09:28:42,521 INFO [Log partition=my_topic-11, dir=/var/lib/kafka] 
Scheduling log segment [baseOffset 13131829288, size 280543535] for deletion. 
(kafka.log.Log)
2019-01-31 09:28:43,870 WARN [ReplicaFetcher replicaId=11, leaderId=13, 
fetcherId=1] Truncating my_topic-11 to offset 12878451349 below high watermark 
13132373966 (kafka.server.ReplicaFetcherThread)
2019-01-31 09:29:03,703 INFO [Log partition=my_topic-11, dir=/var/lib/kafka] 
Found deletable segments with base offsets [12878451349] due to retention time 
25920ms breach (kafka.log.Log)
2019-01-31 09:28:42,550 INFO Compaction for partition my_topic-11 is resumed 
(kafka.log.LogManager)
{code}
 

We sucessfull tried to reproduce the same bug with kafka 0.11, 2.0.1 & 2.1.0

 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] KIP-416: Notify SourceTask of ACK'd offsets, metadata

2019-01-31 Thread Ryanne Dolan 
Andrew, I have considered this, but I think passing null for RecordMetadata
would be surprising and error prone for anyone implementing SourceTask. I
figure the only use-case for overriding this variant (and not the existing
one) is to capture the RecordMetadata. If that's the case, every
implementation would need to check for null. What worries me is that an
implementation that does not check for null will seem to work until an SMT
is configured to filter records, which I believe would be exceedingly rare.
Moreover, the presence of the RecordMetadata parameter strongly implies
that the record has been sent and ACK'd, and it would be surprising to
discover otherwise.

On the other hand, the current PR makes it difficult to distinguish between
records that are filtered vs ACK'd. The implementing class would need to
correlate across poll() and the two commitRecord() invocations in order to
find records that were poll()'d but not ACK'd. In contrast, if we passed
null to commitRecord, the method would trivially know that the record was
filtered. I think this is probably not a common use-case, so I don't think
we should worry about it. In fact, the existing commitRecord callback seems
to purposefully hide this detail from the implementing class, and I don't
know why we'd try to expose it in the new method.

This sort of confusion is why I originally proposed a new method name for
this callback, as does the similar KIP-381. I agree that overloading the
existing method is all-around easier, and I think a casual reader would
make the correct assumption that RecordMetadata in the parameter list
implies that the record was sent and ACK'd.

> the connector implementor would want to provide only a single variant of
commitRecord()

I think this would be true either way. The only reason you'd implement both
variants is to detect that a record has _not_ been ACK'd, which again I
believe is a non-requirement.

Would love to hear if you disagree.

Thanks!
Ryanne


On Thu, Jan 31, 2019 at 3:47 AM Andrew Schofield 
wrote:

> As you might expect, I like the overloaded commitRecord() but I think the
> overloaded method should be called in exactly the same situations as the
> previous method. When it does not reflect an ACK, the second parameter
> could be null. The text of the KIP says that the overloaded method is only
> called when a record is ACKed and I would have thought that the connector
> implementor would want to provide only a single variant of commitRecord().
>
> Andrew Schofield
> IBM Event Streams
>
> On 31/01/2019, 03:00, "Ryanne Dolan"  wrote:
>
> I've updated the KIP and PR to overload commitRecord instead of adding
> a
> new method. Here's the PR:
>
>
> https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fkafka%2Fpull%2F6171&data=02%7C01%7C%7Cc627d954fa6f44574f7908d6872838c5%7C84df9e7fe9f640afb435%7C1%7C0%7C636845004151935856&sdata=hxBWSTt5gF7AAVxw2P8%2BZ8duBB0T97gHOOYG6GCkdd8%3D&reserved=0
>
> Ryanne
>
> On Mon, Jan 21, 2019 at 6:29 PM Ryanne Dolan 
> wrote:
>
> > Andrew Schofield suggested we overload the commitRecord method
> instead of
> > adding a new one. Thoughts?
> >
> > Ryanne
> >
> > On Thu, Jan 17, 2019, 5:34 PM Ryanne Dolan  wrote:
> >
> >> I had to change the KIP number (concurrency is hard!) so the link
> is now:
> >>
> >>
> >>
> https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-416%253A%2BNotify%2BSourceTask%2Bof%2BACK%2527d%2Boffsets%252C%2Bmetadata&data=02%7C01%7C%7Cc627d954fa6f44574f7908d6872838c5%7C84df9e7fe9f640afb435%7C1%7C0%7C636845004151935856&sdata=VkAFrM8B2ozCRJosPQjgM3aDD1cS%2Bob8KWVuNuuOJ9s%3D&reserved=0
> >>
> >> Ryanne
> >>
> >> On Fri, Jan 11, 2019 at 2:43 PM Ryanne Dolan  >
> >> wrote:
> >>
> >>> Hey y'all,
> >>>
> >>> Please review the following small KIP:
> >>>
> >>>
> >>>
> https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-414%253A%2BNotify%2BSourceTask%2Bof%2BACK%2527d%2Boffsets%252C%2Bmetadata&data=02%7C01%7C%7Cc627d954fa6f44574f7908d6872838c5%7C84df9e7fe9f640afb435%7C1%7C0%7C636845004151945855&sdata=2mhXA4hEV3ZvrFaOcTqagO1rYNj1JsYAEDHQsFqkzG8%3D&reserved=0
> >>>
> >>> Thanks!
> >>> Ryanne
> >>>
> >>
>
>
>

[jira] [Created] (KAFKA-7887) Transaction Producer hanging when commiting/aborting transaction after a broker failure

2019-01-31 Thread Cameron (JIRA) 
Cameron created KAFKA-7887:
--

 Summary: Transaction Producer hanging when commiting/aborting 
transaction after a broker failure
 Key: KAFKA-7887
 URL: https://issues.apache.org/jira/browse/KAFKA-7887
 Project: Kafka
  Issue Type: Bug
  Components: clients
Affects Versions: 2.0.1
Reporter: Cameron


I have observed that when a broker fails, the issue with transactional producer 
hanging while trying to commit and/or abort transaction still persists

Within TransactionalRequestResult.class lines 36-42

{code:java}
while(!completed) {
  try {
this.latch.await();
completed = true;
  } catch (InterruptedException var3) {
  }
}
{code}

this.latch.await() never returns

Reproducible by bringing down kafka broker while transactional producing is 
sending records

Suggest (1) calling latch.await() with timeout parameter or (2) allow for 
easier overriding of method by developers to allow them to call await() with a 
timeout 








--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-7887) Transaction Producer hanging when commiting/aborting transaction after a broker failure

2019-01-31 Thread Cameron (JIRA) 


 [ 
https://issues.apache.org/jira/browse/KAFKA-7887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Cameron resolved KAFKA-7887.

Resolution: Duplicate

> Transaction Producer hanging when commiting/aborting transaction after a 
> broker failure
> ---
>
> Key: KAFKA-7887
> URL: https://issues.apache.org/jira/browse/KAFKA-7887
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 2.0.1
>Reporter: Cameron
>Priority: Major
>
> I have observed that when a broker fails, the issue with transactional 
> producer hanging while trying to commit and/or abort transaction still 
> persists
> Within TransactionalRequestResult.class lines 36-42
> {code:java}
> while(!completed) {
>   try {
> this.latch.await();
> completed = true;
>   } catch (InterruptedException var3) {
>   }
> }
> {code}
> this.latch.await() never returns
> Reproducible by bringing down kafka broker while transactional producing is 
> sending records
> Suggest (1) calling latch.await() with timeout parameter or (2) allow for 
> easier overriding of method by developers to allow them to call await() with 
> a timeout 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-7790) Fix Bugs in Trogdor Task Expiration

2019-01-31 Thread Colin P. McCabe (JIRA) 


 [ 
https://issues.apache.org/jira/browse/KAFKA-7790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin P. McCabe resolved KAFKA-7790.

Resolution: Fixed

> Fix Bugs in Trogdor Task Expiration
> ---
>
> Key: KAFKA-7790
> URL: https://issues.apache.org/jira/browse/KAFKA-7790
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Stanislav Kozlovski
>Assignee: Stanislav Kozlovski
>Priority: Major
>
> If an Agent process is restarted, it will be re-sent the worker 
> specifications for any tasks that are not DONE.  The agent will run these 
> tasks for the original time period.  It should be fixed to run them only for 
> the remaining task time.  There is also a bug where the coordinator can 
> sometimes re-create a worker even when the task is DONE.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-7792) Trogdor should have an uptime function

2019-01-31 Thread Colin P. McCabe (JIRA) 


 [ 
https://issues.apache.org/jira/browse/KAFKA-7792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin P. McCabe resolved KAFKA-7792.

Resolution: Fixed

> Trogdor should have an uptime function
> --
>
> Key: KAFKA-7792
> URL: https://issues.apache.org/jira/browse/KAFKA-7792
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Colin P. McCabe
>Assignee: Stanislav Kozlovski
>Priority: Minor
>
> Trogdor should have an uptime function which returns how long the coordinator 
> or agent has been up.  This will also be a good way to test that the daemon 
> is running without fetching a full status.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-7793) Improve the Trogdor command-line

2019-01-31 Thread Colin P. McCabe (JIRA) 


 [ 
https://issues.apache.org/jira/browse/KAFKA-7793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin P. McCabe resolved KAFKA-7793.

Resolution: Fixed

> Improve the Trogdor command-line
> 
>
> Key: KAFKA-7793
> URL: https://issues.apache.org/jira/browse/KAFKA-7793
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Colin P. McCabe
>Assignee: Colin P. McCabe
>Priority: Major
>
> Improve the Trogdor command-line.  It should be easier to launch tasks from a 
> task spec in a file.  It should be easier to list the currently-running tasks 
> in a readable way.  We should be able to filter the currently-running tasks.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: broken build on windows

2019-01-31 Thread Colin McCabe 
Hi Chia-Ping Tsai,

Thanks for the patch.  :)

cheers,
Colin


On Wed, Jan 30, 2019, at 07:31, Chia-Ping Tsai wrote:
> hi folks,
> 
> I love GNU/Linux but it pains me to see broken build on windows. Could 
> someone take a look at PR: https://github.com/apache/kafka/pull/6208 ? 
> It enables kafka to embrace windows again.
> 
> Cheers,
> Chia-Ping
>

[jira] [Created] (KAFKA-7888) kafka cluster not recovering - Shrinking ISR from 14,13 to 13 (kafka.cluster.Partition) continously

2019-01-31 Thread Kemal ERDEN (JIRA) 
Kemal ERDEN created KAFKA-7888:
--

 Summary: kafka cluster not recovering - Shrinking ISR from 14,13 
to 13 (kafka.cluster.Partition) continously
 Key: KAFKA-7888
 URL: https://issues.apache.org/jira/browse/KAFKA-7888
 Project: Kafka
  Issue Type: Bug
  Components: controller, replication, zkclient
Affects Versions: 2.1.0
 Environment: using kafka_2.12-2.1.0

3 ZKs 3 Broker cluster, using 3 boxes (1 ZK and 1 broker on each box), 
default.replication factor: 2, 
offset replication factor was 1 when the error happened, increased to 2 after 
seeing this error by reassigning-partitions.
compression: default (producer) on broker but sending gzip from producers.

linux (redhat) etx4 kafka logs on single local disk
Reporter: Kemal ERDEN
 Attachments: combined.log, producer.log

we're seeing the following repeating logs on our kafka cluster from time to 
time which seems to cause messages expiring on Producers and the cluster going 
into a non-recoverable state. The only fix seems to be to restart brokers.


 {{Shrinking ISR from 14,13 to 13 (kafka.cluster.Partition)}}
 {{Cached zkVersion [21] not equal to that in zookeeper, skip updating ISR 
(kafka.cluster.Partition)}}

 and later on the following log is repeated:

{{Got user-level KeeperException when processing sessionid:0xe046aa4f8e6 
type:setData cxid:0x2df zxid:0xa01fd txntype:-1 reqpath:n/a Error 
Path:/brokers/topics/ucTrade/partitions/6/state Error:KeeperErrorCode = 
BadVersion for /brokers/topics/ucTrade/partitions/6/state}}

We haven't interfered with any of the brokers/zookeepers whilst this happened.

I've attached a combined log which represents a combination of controller, 
server and state change logs from each broker (ids 13,14 and 15, log files have 
the suffix b13, b14, b15 respectively)

We have increased the heaps from 1g to 6g for the brokers and from 512m to 4g 
for the zookeepers since this happened but not sure if it is relevant. the ZK 
logs are unfortunately overwritten so can't provide those.

We produce varying message sizes but some messages are relatively large (6mb) 
but we use compression on the producers (set to gzip).

I've attached some logs from one of our producers as well.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-7889) Kafka broker crashes on windows when log compaction is used

2019-01-31 Thread huzaifa kagazwala (JIRA) 
huzaifa kagazwala created KAFKA-7889:


 Summary: Kafka broker crashes on windows when log compaction is 
used
 Key: KAFKA-7889
 URL: https://issues.apache.org/jira/browse/KAFKA-7889
 Project: Kafka
  Issue Type: Bug
  Components: log
Affects Versions: 2.1.0, 1.1.1, 1.1.0
Reporter: huzaifa kagazwala


I am running kafka on a Windows 10 machine tried the latest "kafka_2.12-2.1.0" 
build. I created a topic with log compaction enabled for testing. After 
publishing some data as soon as the first log rollover happens the server 
crashes with the below error.

 

command used to create the topic.

kafka-topics.bat -create -zookeeper localhost:2181 -topic compact_log__test_2  
-replication-factor 1 -partitions 2 -config segment.bytes=10 -config 
segment.ms=60 -config min.cleanable.dirty.ratio=0.01 -config 
compression.type=snappy -config cleanup.policy=compact -config 
min.compaction.lag.ms=3

 

Error Details:

 

[2019-01-31 11:12:09,763] INFO [ProducerStateManager 
partition=compact_log__test_2-0] Writing producer snapshot at offset 998 
(kafka.log.ProducerStateManager)
[2019-01-31 11:12:09,770] INFO [Log partition=compact_log__test_2-0, 
dir=C:\kafka_logs] Rolled new log segment at offset 998 in 35 ms. 
(kafka.log.Log)
[2019-01-31 11:12:09,791] INFO [ProducerStateManager 
partition=compact_log__test_2-1] Writing producer snapshot at offset 1002 
(kafka.log.ProducerStateManager)
[2019-01-31 11:12:09,796] INFO [Log partition=compact_log__test_2-1, 
dir=C:\kafka_logs] Rolled new log segment at offset 1002 in 17 ms. 
(kafka.log.Log)
[2019-01-31 11:12:34,711] ERROR Failed to clean up log for 
compact_log__test_2-0 in dir C:\kafka_logs due to IOException 
(kafka.server.LogDirFailureChannel)
java.nio.file.FileSystemException: 
C:\kafka_logs\compact_log__test_2-0\.timeindex.cleaned -> 
C:\kafka_logs\compact_log__test_2-0\.timeindex.swap: The 
process cannot access the file because it is being used by another process.

at sun.nio.fs.WindowsException.translateToIOException(Unknown Source)
 at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
 at sun.nio.fs.WindowsFileCopy.move(Unknown Source)
 at sun.nio.fs.WindowsFileSystemProvider.move(Unknown Source)
 at java.nio.file.Files.move(Unknown Source)
 at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:809)
 at kafka.log.AbstractIndex.renameTo(AbstractIndex.scala:205)
 at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:490)
 at kafka.log.Log.$anonfun$replaceSegments$4(Log.scala:1892)
 at kafka.log.Log.$anonfun$replaceSegments$4$adapted(Log.scala:1892)
 at scala.collection.immutable.List.foreach(List.scala:388)
 at kafka.log.Log.replaceSegments(Log.scala:1892)
 at kafka.log.Cleaner.cleanSegments(LogCleaner.scala:583)
 at kafka.log.Cleaner.$anonfun$doClean$6(LogCleaner.scala:515)
 at kafka.log.Cleaner.$anonfun$doClean$6$adapted(LogCleaner.scala:514)
 at scala.collection.immutable.List.foreach(List.scala:388)
 at kafka.log.Cleaner.doClean(LogCleaner.scala:514)
 at kafka.log.Cleaner.clean(LogCleaner.scala:492)
 at kafka.log.LogCleaner$CleanerThread.cleanLog(LogCleaner.scala:353)
 at kafka.log.LogCleaner$CleanerThread.cleanFilthiestLog(LogCleaner.scala:319)
 at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:300)
 at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:82)
 Suppressed: java.nio.file.FileSystemException: 
C:\kafka_logs\compact_log__test_2-0\.timeindex.cleaned -> 
C:\kafka_logs\compact_log__test_2-0\.timeindex.swap: The 
process cannot access the file because it is being used by another process.

at sun.nio.fs.WindowsException.translateToIOException(Unknown Source)
 at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
 at sun.nio.fs.WindowsFileCopy.move(Unknown Source)
 at sun.nio.fs.WindowsFileSystemProvider.move(Unknown Source)
 at java.nio.file.Files.move(Unknown Source)
 at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:806)
 ... 16 more
[2019-01-31 11:12:34,716] INFO [ReplicaManager broker=0] Stopping serving 
replicas in dir C:\kafka_logs (kafka.server.ReplicaManager)
[2019-01-31 11:12:34,723] INFO [ReplicaFetcherManager on broker 0] Removed 
fetcher for partitions Set(__consumer_offsets-22, __consumer_offsets-30, 
__consumer_offsets-8, __consumer_offsets-21, __consumer_offsets-4, 
__consumer_offsets-27, __consumer_offsets-7, __consumer_offsets-9, 
__consumer_offsets-46, __consumer_offsets-25, compact_log__test_2-0, 
__consumer_offsets-35, __consumer_offsets-41, __consumer_offsets-33, 
__consumer_offsets-23, __consumer_offsets-49, __consumer_offsets-47, 
__consumer_offsets-16, __consumer_offsets-28, __consumer_offsets-31, 
__consumer_offsets-36, __consumer_offsets-42, __consumer_offsets-3, 
__consumer_offsets-18, __consumer_offsets-37, __co

[jira] [Resolved] (KAFKA-7859) Replace LeaveGroup request/response with automated protocol

2019-01-31 Thread Boyang Chen (JIRA) 


 [ 
https://issues.apache.org/jira/browse/KAFKA-7859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Boyang Chen resolved KAFKA-7859.

Resolution: Fixed

> Replace LeaveGroup request/response with automated protocol
> ---
>
> Key: KAFKA-7859
> URL: https://issues.apache.org/jira/browse/KAFKA-7859
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Boyang Chen
>Assignee: Boyang Chen
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-7816) Windowed topic should have window size as part of the metadata

2019-01-31 Thread Boyang Chen (JIRA) 


 [ 
https://issues.apache.org/jira/browse/KAFKA-7816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Boyang Chen resolved KAFKA-7816.

Resolution: Won't Fix

Not a real use case

> Windowed topic should have window size as part of the metadata
> --
>
> Key: KAFKA-7816
> URL: https://issues.apache.org/jira/browse/KAFKA-7816
> Project: Kafka
>  Issue Type: Improvement
>  Components: consumer, streams
>Reporter: Boyang Chen
>Assignee: Boyang Chen
>Priority: Major
>
> Currently the Kafka window store topics require a windowed serde to properly 
> deserialize the records. One of the required config is `window.size.ms`, 
> which indicates the diff between (window.end - window.start). For space 
> efficiency, KStream only stores the windowed record with window start time, 
> because as long as the restore consumer knows size of the window, it would 
> properly derive the window end time by adding window.size.ms to window start 
> time.
> However, this makes the reuse of window topic very hard because another user 
> has to config the correct window size in order to deserialize the data. When 
> we extract the customized consumer as a template, every time new user has to 
> define their own window size. If we do wild-card matching consumer, things 
> could be even worse to work because different topics may have different 
> window size and user has to read through the application code to find that 
> info.
> To make the decoding of window topic easier, we are proposing to add a new 
> config to TopicMetadata called `windowSize` which could be used for 
> applications to properly deserialize the data without requirement to config a 
> window size. This could also make client side serde API easier. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: kafka-trunk-jdk11 #253

2019-01-31 Thread Apache Jenkins Server 
See 


Changes:

[colin] KAFKA-7859: Use automatic RPC generation in LeaveGroups (#6188)

--
[...truncated 2.28 MB...]

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testWithSchemaIdentity PASSED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testSchemalessTimestampToDate STARTED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testSchemalessTimestampToDate PASSED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testSchemalessTimestampToTime STARTED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testSchemalessTimestampToTime PASSED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testSchemalessTimestampToUnix STARTED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testSchemalessTimestampToUnix PASSED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testConfigMissingFormat STARTED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testConfigMissingFormat PASSED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testConfigNoTargetType STARTED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testConfigNoTargetType PASSED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testWithSchemaStringToTimestamp STARTED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testWithSchemaStringToTimestamp PASSED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testWithSchemaTimeToTimestamp STARTED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testWithSchemaTimeToTimestamp PASSED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testSchemalessUnixToTimestamp STARTED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testSchemalessUnixToTimestamp PASSED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testWithSchemaTimestampToString STARTED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testWithSchemaTimestampToString PASSED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testConfigInvalidFormat STARTED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testConfigInvalidFormat PASSED

org.apache.kafka.connect.transforms.HoistFieldTest > withSchema STARTED

org.apache.kafka.connect.transforms.HoistFieldTest > withSchema PASSED

org.apache.kafka.connect.transforms.HoistFieldTest > schemaless STARTED

org.apache.kafka.connect.transforms.HoistFieldTest > schemaless PASSED

org.apache.kafka.connect.transforms.InsertFieldTest > 
schemalessInsertConfiguredFields STARTED

org.apache.kafka.connect.transforms.InsertFieldTest > 
schemalessInsertConfiguredFields PASSED

org.apache.kafka.connect.transforms.InsertFieldTest > topLevelStructRequired 
STARTED

org.apache.kafka.connect.transforms.InsertFieldTest > topLevelStructRequired 
PASSED

org.apache.kafka.connect.transforms.InsertFieldTest > 
copySchemaAndInsertConfiguredFields STARTED

org.apache.kafka.connect.transforms.InsertFieldTest > 
copySchemaAndInsertConfiguredFields PASSED

org.apache.kafka.connect.transforms.MaskFieldTest > withSchema STARTED

org.apache.kafka.connect.transforms.MaskFieldTest > withSchema PASSED

org.apache.kafka.connect.transforms.MaskFieldTest > schemaless STARTED

org.apache.kafka.connect.transforms.MaskFieldTest > schemaless PASSED

org.apache.kafka.connect.transforms.ExtractFieldTest > withSchema STARTED

org.apache.kafka.connect.transforms.ExtractFieldTest > withSchema PASSED

org.apache.kafka.connect.transforms.ExtractFieldTest > testNullWithSchema 
STARTED

org.apache.kafka.connect.transforms.ExtractFieldTest > testNullWithSchema PASSED

org.apache.kafka.connect.transforms.ExtractFieldTest > schemaless STARTED

org.apache.kafka.connect.transforms.ExtractFieldTest > schemaless PASSED

org.apache.kafka.connect.transforms.ExtractFieldTest > testNullSchemaless 
STARTED

org.apache.kafka.connect.transforms.ExtractFieldTest > testNullSchemaless PASSED

org.apache.kafka.connect.transforms.TimestampRouterTest > defaultConfiguration 
STARTED

org.apache.kafka.connect.transforms.TimestampRouterTest > defaultConfiguration 
PASSED

org.apache.kafka.connect.transforms.CastTest > 
castWholeDateRecordValueWithSchemaString STARTED

org.apache.kafka.connect.transforms.CastTest > 
castWholeDateRecordValueWithSchemaString PASSED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueSchemalessBooleanTrue STARTED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueSchemalessBooleanTrue PASSED

org.apache.kafka.connect.transforms.CastTest > testConfigInvalidTargetType 
STARTED

org.apache.kafka.connect.transforms.CastTest > testConfigInvalidTargetType 
PASSED

org.apache.kafka.connect.transforms.CastTest > 
testConfigMixWholeAndFieldTransformation STARTED

org.apache.kafka.connect.transforms.CastTest > 
testConfigMixWholeAndFieldTr

Build failed in Jenkins: kafka-trunk-jdk8 #3357

2019-01-31 Thread Apache Jenkins Server 
See 


Changes:

[colin] KAFKA-7859: Use automatic RPC generation in LeaveGroups (#6188)

[colin] MINOR: fix checkstyle suppressions for generated RPC code to work on

--
[...truncated 2.28 MB...]

org.apache.kafka.streams.TopologyTest > 
timeWindowAnonymousMaterializedCountShouldPreserveTopologyStructure PASSED

org.apache.kafka.streams.TopologyTest > shouldFailIfSinkIsParent STARTED

org.apache.kafka.streams.TopologyTest > shouldFailIfSinkIsParent PASSED

org.apache.kafka.streams.TopologyTest > shouldNotAllowToAddSourcesWithSameName 
STARTED

org.apache.kafka.streams.TopologyTest > shouldNotAllowToAddSourcesWithSameName 
PASSED

org.apache.kafka.streams.TopologyTest > 
sourceAndProcessorWithMultipleStatesShouldHaveSingleSubtopology STARTED

org.apache.kafka.streams.TopologyTest > 
sourceAndProcessorWithMultipleStatesShouldHaveSingleSubtopology PASSED

org.apache.kafka.streams.TopologyTest > shouldDescribeEmptyTopology STARTED

org.apache.kafka.streams.TopologyTest > shouldDescribeEmptyTopology PASSED

org.apache.kafka.streams.TopologyTest > shouldFailIfSinkIsItsOwnParent STARTED

org.apache.kafka.streams.TopologyTest > shouldFailIfSinkIsItsOwnParent PASSED

org.apache.kafka.streams.TopologyTest > 
testPatternMatchesAlreadyProvidedTopicSource STARTED

org.apache.kafka.streams.TopologyTest > 
testPatternMatchesAlreadyProvidedTopicSource PASSED

org.apache.kafka.streams.TopologyTest > 
singleSourcePatternShouldHaveSingleSubtopology STARTED

org.apache.kafka.streams.TopologyTest > 
singleSourcePatternShouldHaveSingleSubtopology PASSED

org.apache.kafka.streams.TopologyTest > 
kTableNonMaterializedFilterShouldPreserveTopologyStructure STARTED

org.apache.kafka.streams.TopologyTest > 
kTableNonMaterializedFilterShouldPreserveTopologyStructure PASSED

org.apache.kafka.streams.TopologyTest > 
topologyWithDynamicRoutingShouldDescribeExtractorClass STARTED

org.apache.kafka.streams.TopologyTest > 
topologyWithDynamicRoutingShouldDescribeExtractorClass PASSED

org.apache.kafka.streams.TopologyTest > singleSourceShouldHaveSingleSubtopology 
STARTED

org.apache.kafka.streams.TopologyTest > singleSourceShouldHaveSingleSubtopology 
PASSED

org.apache.kafka.streams.TopologyTest > 
shouldNotAllowNullTopicChooserWhenAddingSink STARTED

org.apache.kafka.streams.TopologyTest > 
shouldNotAllowNullTopicChooserWhenAddingSink PASSED

org.apache.kafka.streams.TopologyTest > 
processorsWithSameSinkShouldHaveSameSubtopology STARTED

org.apache.kafka.streams.TopologyTest > 
processorsWithSameSinkShouldHaveSameSubtopology PASSED

org.apache.kafka.streams.TopologyTest > 
shouldNotAllowNullTopicsWhenAddingSoureWithTopic STARTED

org.apache.kafka.streams.TopologyTest > 
shouldNotAllowNullTopicsWhenAddingSoureWithTopic PASSED

org.apache.kafka.streams.TopologyTest > shouldFailWithUnknownParent STARTED

org.apache.kafka.streams.TopologyTest > shouldFailWithUnknownParent PASSED

org.apache.kafka.streams.TopologyTest > 
kTableAnonymousMaterializedFilterShouldPreserveTopologyStructure STARTED

org.apache.kafka.streams.TopologyTest > 
kTableAnonymousMaterializedFilterShouldPreserveTopologyStructure PASSED

org.apache.kafka.streams.TopologyTest > 
sinkShouldReturnTopicNameExtractorWithDynamicRouting STARTED

org.apache.kafka.streams.TopologyTest > 
sinkShouldReturnTopicNameExtractorWithDynamicRouting PASSED

org.apache.kafka.streams.TopologyTest > 
shouldNotAllowToAddProcessorWithEmptyParents STARTED

org.apache.kafka.streams.TopologyTest > 
shouldNotAllowToAddProcessorWithEmptyParents PASSED

org.apache.kafka.streams.TopologyTest > 
sessionWindowAnonymousMaterializedCountShouldPreserveTopologyStructure STARTED

org.apache.kafka.streams.TopologyTest > 
sessionWindowAnonymousMaterializedCountShouldPreserveTopologyStructure PASSED

org.apache.kafka.streams.TopologyTest > shouldFailIfNodeIsItsOwnParent STARTED

org.apache.kafka.streams.TopologyTest > shouldFailIfNodeIsItsOwnParent PASSED

org.apache.kafka.streams.TopologyTest > shouldNotAllowToAddStateStoreToSource 
STARTED

org.apache.kafka.streams.TopologyTest > shouldNotAllowToAddStateStoreToSource 
PASSED

org.apache.kafka.streams.TopologyTest > 
shouldNotAllowNullNameWhenAddingProcessor STARTED

org.apache.kafka.streams.TopologyTest > 
shouldNotAllowNullNameWhenAddingProcessor PASSED

org.apache.kafka.streams.TopologyTest > shouldNotAllowToAddStoreWithSameName 
STARTED

org.apache.kafka.streams.TopologyTest > shouldNotAllowToAddStoreWithSameName 
PASSED

org.apache.kafka.streams.TopologyTest > 
shouldNotAllowToAddProcessorWithNullParents STARTED

org.apache.kafka.streams.TopologyTest > 
shouldNotAllowToAddProcessorWithNullParents PASSED

org.apache.kafka.streams.TopologyTest > 
shouldNotAllowNullNameWhenAddingSourceWithPattern STARTED

org.apache.kafka.streams.TopologyTest > 
shouldNotAllowNullNameWhenAddingSourceWithPattern PASSED

org.apache.kafka.streams.TopologyTest

[jira] [Created] (KAFKA-7890) Invalidate ClusterConnectionState cache for a broker if the hostname of the broker changes.

2019-01-31 Thread Mark Cho (JIRA) 
Mark Cho created KAFKA-7890:
---

 Summary: Invalidate ClusterConnectionState cache for a broker if 
the hostname of the broker changes.
 Key: KAFKA-7890
 URL: https://issues.apache.org/jira/browse/KAFKA-7890
 Project: Kafka
  Issue Type: Bug
  Components: network
Affects Versions: 2.1.0
Reporter: Mark Cho


We've ran into a similar issue as this ticket: 
[https://issues.apache.org/jira/projects/KAFKA/issues/KAFKA-7755]

The fix for KAFKA-7755 doesn't work for this case as the hostname is not 
updated when resolving the addresses.

`ClusterConnectionStates::connecting` method makes an assumption that broker ID 
will always map to same hostname. In our case, when a broker is terminated in 
AWS, it is replaced by a different instance under the same broker ID. 

In this case, the consumer fails to connect to the right host when the broker 
ID returns to the cluster. For example, we see the following line in DEBUG logs:
{code:java}
Initiating connection to node 100.66.7.94:7101 (id: 1 rack: us-east-1c) using 
address /100.66.14.165
{code}
It tries to connect to the new broker instance using the wrong (old) IP address.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: kafka-trunk-jdk11 #254

2019-01-31 Thread Apache Jenkins Server 
See 


Changes:

[colin] MINOR: fix checkstyle suppressions for generated RPC code to work on

--
[...truncated 2.28 MB...]

org.apache.kafka.connect.transforms.HoistFieldTest > schemaless PASSED

org.apache.kafka.connect.transforms.InsertFieldTest > 
schemalessInsertConfiguredFields STARTED

org.apache.kafka.connect.transforms.InsertFieldTest > 
schemalessInsertConfiguredFields PASSED

org.apache.kafka.connect.transforms.InsertFieldTest > topLevelStructRequired 
STARTED

org.apache.kafka.connect.transforms.InsertFieldTest > topLevelStructRequired 
PASSED

org.apache.kafka.connect.transforms.InsertFieldTest > 
copySchemaAndInsertConfiguredFields STARTED

org.apache.kafka.connect.transforms.InsertFieldTest > 
copySchemaAndInsertConfiguredFields PASSED

org.apache.kafka.connect.transforms.MaskFieldTest > withSchema STARTED

org.apache.kafka.connect.transforms.MaskFieldTest > withSchema PASSED

org.apache.kafka.connect.transforms.MaskFieldTest > schemaless STARTED

org.apache.kafka.connect.transforms.MaskFieldTest > schemaless PASSED

org.apache.kafka.connect.transforms.ExtractFieldTest > withSchema STARTED

org.apache.kafka.connect.transforms.ExtractFieldTest > withSchema PASSED

org.apache.kafka.connect.transforms.ExtractFieldTest > testNullWithSchema 
STARTED

org.apache.kafka.connect.transforms.ExtractFieldTest > testNullWithSchema PASSED

org.apache.kafka.connect.transforms.ExtractFieldTest > schemaless STARTED

org.apache.kafka.connect.transforms.ExtractFieldTest > schemaless PASSED

org.apache.kafka.connect.transforms.ExtractFieldTest > testNullSchemaless 
STARTED

org.apache.kafka.connect.transforms.ExtractFieldTest > testNullSchemaless PASSED

org.apache.kafka.connect.transforms.TimestampRouterTest > defaultConfiguration 
STARTED

org.apache.kafka.connect.transforms.TimestampRouterTest > defaultConfiguration 
PASSED

org.apache.kafka.connect.transforms.CastTest > 
castWholeDateRecordValueWithSchemaString STARTED

org.apache.kafka.connect.transforms.CastTest > 
castWholeDateRecordValueWithSchemaString PASSED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueSchemalessBooleanTrue STARTED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueSchemalessBooleanTrue PASSED

org.apache.kafka.connect.transforms.CastTest > testConfigInvalidTargetType 
STARTED

org.apache.kafka.connect.transforms.CastTest > testConfigInvalidTargetType 
PASSED

org.apache.kafka.connect.transforms.CastTest > 
testConfigMixWholeAndFieldTransformation STARTED

org.apache.kafka.connect.transforms.CastTest > 
testConfigMixWholeAndFieldTransformation PASSED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueSchemalessFloat32 STARTED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueSchemalessFloat32 PASSED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueSchemalessFloat64 STARTED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueSchemalessFloat64 PASSED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueSchemalessUnsupportedType STARTED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueSchemalessUnsupportedType PASSED

org.apache.kafka.connect.transforms.CastTest > castWholeRecordKeySchemaless 
STARTED

org.apache.kafka.connect.transforms.CastTest > castWholeRecordKeySchemaless 
PASSED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueWithSchemaFloat32 STARTED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueWithSchemaFloat32 PASSED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueWithSchemaFloat64 STARTED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueWithSchemaFloat64 PASSED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueWithSchemaString STARTED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueWithSchemaString PASSED

org.apache.kafka.connect.transforms.CastTest > testConfigEmpty STARTED

org.apache.kafka.connect.transforms.CastTest > testConfigEmpty PASSED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueSchemalessInt16 STARTED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueSchemalessInt16 PASSED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueSchemalessInt32 STARTED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueSchemalessInt32 PASSED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueSchemalessInt64 STARTED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueSchemalessInt64 PASSED

org.apache.kafka.connect.transforms.CastTest > castFieldsSchemaless STARTED

org.apache.kafka.connect.transforms.CastTest > castFieldsSchemaless PASSED

org.apache.kafka.connect.transforms.CastTest > testUnsupportedTargetType STARTED

org.apache.ka

Re: [DISCUSS] KIP-390: Add producer option to adjust compression level

2019-01-31 Thread Becket Qin 
Hi Dongjin,

Thanks for the reply. I agree that implementation wise, it is clearer to
add a new "Map" config type. However, practically speaking I don't think
this KIP strongly depends on that. As long as we follow the same string
format in the configuration for all those map-like configurations, whether
to have a Map config type or not seems not that important.

Given that the existing configuration uses the format of
"KEY:VALUE[,KEY:VALUE]", we could just use this string format.

Thanks,

Jiangjie (Becket) Qin

On Thu, Jan 31, 2019 at 7:47 PM Dongjin Lee  wrote:

> Mickael,
>
> It seems like the majority of the community agrees that the new config
> scheme proposed by Becket is much better. However, we still need another
> KIP to support this kind of config; it is different from the case of
> `listener.security.protocol.map,` which is just a concatenation of
> available settings (i.e., string type.) For Becket's idea, we need to add
> support for the key-value style.
>
> It seems like we need advice from a committer, whether to add a new
> prerequisite KIP for the map type configuration to implement Becket's idea.
>
> Committers:
>
> Could you give us some advice on this problem?
>
> Best,
> Dongjin
>
> On Wed, Jan 30, 2019 at 11:30 PM Mickael Maison 
> wrote:
>
> > Thanks, that's a very interesting KIP!
> >
> > I agree with Becket that a clearer config format is likely to help.
> > Have you considered using a config format similar to listeners (as
> > described in the "Updating SSL Keystore of an Existing Listener" in
> > the Kafka docs)?
> >
> > Also worth noting that we already have configs that effectively take a
> > map, for example: listener.security.protocol.map so not sure if we
> > need an additional KIP
> >
> > On Wed, Jan 30, 2019 at 5:22 AM Dongjin Lee  wrote:
> > >
> > > Hello.
> > >
> > > Do you have any idea on Becket's Idea of new config format (example
> > below)?
> > >
> > > ```
> > > compression.config="gzip.compression.level=5, lz4.compression.level=17,
> > > zstd.compression.level=22"
> > > ```
> > >
> > > It requires some additional KIP for supporting new config format (map),
> > but
> > > it can significantly simplify the configuration with flexibility and
> > > extensibility. If you prefer this way, I hope to carry the ball.
> > >
> > > If not, please give me an opinion here or the voting thread.
> > >
> > > Thanks,
> > > Dongjin
> > >
> > >
> > > On Fri, Jan 25, 2019 at 1:25 AM Dongjin Lee 
> wrote:
> > >
> > > > Hi Becket,
> > > >
> > > > Thank you for your opinion. Frankly, I have no strong opinion on
> > > > configuration name. In this problem, I will follow the community's
> > choice.
> > > > (I like your idea in that it has a notion of 'scope' per compression
> > codec.
> > > > However, it should be implemented on top of new config type like Map;
> > It
> > > > will require another KIP as a prerequisite, but if the community
> prefer
> > > > this way, I will take the task.)
> > > >
> > > > (One minor correction: the one who thought 'producer' compression
> > config
> > > > would cause a problem at broker was me, not Ismael - and Ismael
> > reassured
> > > > me there will be no problem with it.)
> > > >
> > > > To All,
> > > >
> > > > How about Becket's idea of 'compression.config' option?
> > > >
> > > > Best,
> > > > Dongjin
> > > >
> > > > On Wed, Jan 23, 2019 at 1:16 PM Becket Qin 
> > wrote:
> > > >
> > > >> Hi Dongjin,
> > > >>
> > > >> Thanks for the KIP and sorry for being a bit late on the discussion.
> > > >>
> > > >> It makes sense to expose the configuration for compression types.
> But
> > I am
> > > >> wondering if there is a better way to do that than what proposed in
> > the
> > > >> KIP. What I feel confusing is that we are effectively sharing the
> > > >> configuration across different compression types, the meaning of the
> > > >> configuration are actually kind of different depending on the
> > compression
> > > >> type. This will end up with issues like what Ismael has brought up
> > > >> earlier.
> > > >> Say if the broker has compression type of producer (this may result
> in
> > > >> mixed compression type in the same topic), and for some reason the
> > broker
> > > >> needs to re-compress the topic (e.g. due to log compaction), a
> single
> > > >> topic
> > > >> level compression config may not work, because a valid compression
> > level
> > > >> for lz4 maybe invalid for gzip.
> > > >>
> > > >> One alternative I am thinking is to provide a "compression.config"
> > > >> configuration, inside which it specifies configuration used by each
> > > >> specific compression type as k-v pairs. The format could use some
> name
> > > >> space as well. For example,
> > > >>
> > > >>
> > > >>
> >
> compression.config="gzip.compression.level=5,lz4.compression.level=17,zstd.compression.level=22".
> > > >>
> > > >> Each compression type will just pick whatever configuration they
> need
> > from
> > > >> the k-v pairs defined in this config.
> > > >>
> > > >> Besides

[jira] [Reopened] (KAFKA-7799) Fix flaky test RestServerTest.testCORSEnabled

2019-01-31 Thread Manikumar (JIRA) 


 [ 
https://issues.apache.org/jira/browse/KAFKA-7799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar reopened KAFKA-7799:
--

This test is failing frequently on Jenkins builds. Reopening.

> Fix flaky test RestServerTest.testCORSEnabled
> -
>
> Key: KAFKA-7799
> URL: https://issues.apache.org/jira/browse/KAFKA-7799
> Project: Kafka
>  Issue Type: Test
>  Components: KafkaConnect
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
>Priority: Major
>
> Starting to see this failure quite a lot, locally and on jenkins:
> {code}
> org.apache.kafka.connect.runtime.rest.RestServerTest.testCORSEnabled
> Failing for the past 7 builds (Since Failed#18600 )
> Took 0.7 sec.
> Error Message
> java.lang.AssertionError: expected: but was:
> Stacktrace
> java.lang.AssertionError: expected: but was:
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:834)
>   at org.junit.Assert.assertEquals(Assert.java:118)
>   at org.junit.Assert.assertEquals(Assert.java:144)
>   at 
> org.apache.kafka.connect.runtime.rest.RestServerTest.checkCORSRequest(RestServerTest.java:221)
>   at 
> org.apache.kafka.connect.runtime.rest.RestServerTest.testCORSEnabled(RestServerTest.java:84)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at org.junit.internal.runners.TestMethod.invoke(TestMethod.java:68)
>   at 
> org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl$PowerMockJUnit44MethodRunner.runTestMethod(PowerMockJUnit44RunnerDelegateImpl.java:326)
>   at org.junit.internal.runners.MethodRoadie$2.run(MethodRoadie.java:89)
>   at 
> org.junit.internal.runners.MethodRoadie.runBeforesThenTestThenAfters(MethodRoadie.java:97)
> {code}
> If it helps, I see an uncaught exception in the stdout:
> {code}
> [2019-01-08 19:35:23,664] ERROR Uncaught exception in REST call to 
> /connector-plugins/FileStreamSource/validate 
> (org.apache.kafka.connect.runtime.rest.errors.ConnectExceptionMapper:61)
> javax.ws.rs.NotFoundException: HTTP 404 Not Found
>   at 
> org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:274)
>   at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272)
>   at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268)
>   at org.glassfish.jersey.internal.Errors.process(Errors.java:316)
>   at org.glassfish.jersey.internal.Errors.process(Errors.java:298)
>   at org.glassfish.jersey.internal.Errors.process(Errors.java:268)
>   at 
> org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289)
>   at 
> org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256)
>   at 
> org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] KIP-417: Allow JmxTool to connect to a secured RMI port

2019-01-31 Thread Manikumar 
Hi,

+1 (binding). Thanks for the KIP.

On Mon, Jan 28, 2019 at 5:37 PM Fangbin Sun  wrote:

> Hi, All:
> I would like to start a vote on KIP-417 which aims at supporting JmxTool
> to connect to a secured RMI port.
>
>
> The KIP:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-417%3A+Allow+JmxTool+to+connect+to+a+secured+RMI+port
>
>
> Thanks!
> Fangbin

[jira] [Created] (KAFKA-7891) Reduce time to start kafka server with clean state

2019-01-31 Thread Pradeep Bansal (JIRA) 
Pradeep Bansal created KAFKA-7891:
-

 Summary: Reduce time to start kafka server  with clean state
 Key: KAFKA-7891
 URL: https://issues.apache.org/jira/browse/KAFKA-7891
 Project: Kafka
  Issue Type: Improvement
Affects Versions: 2.1.0
Reporter: Pradeep Bansal


I am using kafka 2.1.0 and has 6 broker cluster. In this I had a scenario where 
a topic (with replication factor 3 and min insync replica as 2) leader went 
down and we lost its data. When starting this broker with fresh state, it took 
aroun 45 minutes to catch up with replicas (data size of 190 G).

 

Is it expected to take such a large period to recover. are there configuration 
using which this can be optimized.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-7892) kafka broker jvm old increase continually

2019-01-31 Thread wangxingmin (JIRA) 
wangxingmin created KAFKA-7892:
--

 Summary: kafka broker jvm old increase continually
 Key: KAFKA-7892
 URL: https://issues.apache.org/jira/browse/KAFKA-7892
 Project: Kafka
  Issue Type: Bug
  Components: admin
Affects Versions: 0.10.2.1
 Environment: ubuntu 16.04
Reporter: wangxingmin


my kafka cluster jvm old increase continually and trigger fullgc, because on 
toipc in this broker increase messages number produces. I tried the jvm -xmx 
-xms different memory setting，but，the jvm old zone still not release. how can I 
optimize this 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)