Fwd: [DISCUSS] Make org.apache.kafka.clients.Metadata#TOPIC_EXPIRY_MS configurable

[Pavel Moukhataev]

Hello

I'd like to introduce new feature for kafka client:
Making org.apache.kafka.clients.Metadata#TOPIC_EXPIRY_MS configurable
Here is KPI
https://cwiki.apache.org/confluence/display/KAFKA/KIP-375%3A+Make+org.apache.kafka.clients.Metadata%23TOPIC_EXPIRY_MS+configurable

The problem is: if application sends records to some topic rarely then
topic metadata gets expired and sending thread is blocked to wait topic
metadata.

Easy fix is to make TOPIC_EXPIRY_MS configurable.

-- 
Pavel
+7-903-258-5544
skype://pavel.moukhataev

[jira] [Created] (KAFKA-7549) Old ProduceRequest with zstd compression does not return error to client

[Magnus Edenhill (JIRA)]

Magnus Edenhill created KAFKA-7549:
--

 Summary: Old ProduceRequest with zstd compression does not return 
error to client
 Key: KAFKA-7549
 URL: https://issues.apache.org/jira/browse/KAFKA-7549
 Project: Kafka
  Issue Type: Bug
  Components: compression
Reporter: Magnus Edenhill


Kafka broker v2.1.0rc0.

 

KIP-110 states that:

"Zstd will only be allowed for the bumped produce API. That is, for older 
version clients(=below KAFKA_2_1_IV0), we return UNSUPPORTED_COMPRESSION_TYPE 
regardless of the message format."

 

However, sending a ProduceRequest V3 with zstd compression (which is a client 
side bug) closes the connection with the following exception rather than 
returning UNSUPPORTED_COMPRESSION_TYPE in the ProduceResponse:

 
{noformat}
[2018-10-25 11:40:31,813] ERROR Exception while processing request from 
127.0.0.1:60723-127.0.0.1:60656-94 (kafka.network.Processor)
org.apache.kafka.common.errors.InvalidRequestException: Error getting request 
for apiKey: PRODUCE, apiVersion: 3, connectionId: 
127.0.0.1:60723-127.0.0.1:60656-94, listenerName: ListenerName(PLAINTEXT), 
principal: User:ANONYMOUS
Caused by: org.apache.kafka.common.record.InvalidRecordException: Produce 
requests with version 3 are note allowed to use ZStandard compression
{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-7550) 0-byte log segments on topics with log compaction

[Jakub Korab (JIRA)]

Jakub Korab created KAFKA-7550:
--

 Summary: 0-byte log segments on topics with log compaction
 Key: KAFKA-7550
 URL: https://issues.apache.org/jira/browse/KAFKA-7550
 Project: Kafka
  Issue Type: Bug
  Components: log
Affects Versions: 1.0.0
Reporter: Jakub Korab


In production, I am seeing some old log segments that are 0-bytes in length. 
There are no associated .index or .timeindex files.  These topics have log 
compaction turned on.

The segments are not creating any issues.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-5462) Add a configuration for users to specify a template for building a custom principal name

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-5462.
--
   Resolution: Fixed
Fix Version/s: 2.2.0

> Add a configuration for users to specify a template for building a custom 
> principal name
> 
>
> Key: KAFKA-5462
> URL: https://issues.apache.org/jira/browse/KAFKA-5462
> Project: Kafka
>  Issue Type: Bug
>  Components: security
>Affects Versions: 0.10.2.1
>Reporter: Koelli Mungee
>Assignee: Manikumar
>Priority: Major
> Fix For: 2.2.0
>
>
> Add a configuration for users to specify a template for building a custom 
> principal name.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-395) kafka.tools.MirrorMaker black/white list improvement

[JIRA]

 [ 
https://issues.apache.org/jira/browse/KAFKA-395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sönke Liebau resolved KAFKA-395.

Resolution: Workaround

As this issue has not been touched in more than 6 years I think it is fairly 
safe to assume that we can close this.

Any discussions around mirroring functionality are better addressed in the 
MirrorMaker 2.0 KIP discussion.

Regarding the specifics of this issue, that can be worked around by placing 
whitelists topics into a file and the paste-ing that file into the command as 
shown below. I believe that this should be sufficient as a workaround.
{code:java}
// ./kafka-mirror-maker.sh --consumer.config ../config/consumer.properties 
--producer.config ../config/producer.properties --whitelist "$(paste 
whitelist.topics -d'|' -s)" --blacklist "$(paste blacklist.topics -d'|' -s)"
{code}

> kafka.tools.MirrorMaker black/white list improvement
> 
>
> Key: KAFKA-395
> URL: https://issues.apache.org/jira/browse/KAFKA-395
> Project: Kafka
>  Issue Type: Improvement
>  Components: config
>Reporter: Dave DeMaagd
>Priority: Minor
>
> Current black/white list topics are specified directly on the command line, 
> while functional, this has two drawbacks:
> 1) Changes become unwieldy if there are a large number of running instances - 
> potentially many instances to restart, which can have implications for data 
> stream lag
> 2) Maintaining the list itself can become increasingly complex if there are a 
> large number of elements in the list (particularly if they are complex 
> expressions)
> Suggest extending the way that black/white lists can be fed to the mirror 
> maker application, in particular, being able to specify the black/white list 
> as a file (or possibly a generic URI).  Thinking that this could be 
> accomplished either by adding '--whitelistfile' and '--blacklistfile' command 
> line parameters, or modifying the existing '--blacklist' and '--whitelist' 
> parameters to include a 'is this a valid file?' test and decide how to handle 
> it based on that (if it is a file, read it, if not, use current behavior). 
> Follow up suggestion would be to have the mirror maker process check for 
> updates to the list file, and on change, validate and reload it, and run from 
> that point with the new list information. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Frequent under-replicated partitions

[Suman B N]

Still looking for some response here. Pls assist.

On Sat, Oct 20, 2018 at 12:43 AM Suman B N  wrote:

> Rate of ingestion is not 150-200rps. Its 150k-200k rps.
>
> On Fri, Oct 19, 2018 at 11:12 PM Suman B N  wrote:
>
>> Team,
>> We have been observing some partitions being under-replicated. Broker
>> version 0.10.2.1. Below actions were carried out but in vain:
>>
>>- Tried restarting nodes.
>>- Tried increasing replica fetcher threads. Recommend ideal replica
>>fetcher threads for a 20 node cluster with 150-200rps spread across 1000
>>topics and 3000 partitions.
>>- Tried increasing network threads. (I think this doesn't have any
>>effect but still wanted to try). Recommend ideal network threads for a 20
>>node cluster with 150-200rps spread across 1000 topics and 3000 
>> partitions.
>>
>> Logs look very clean. No exceptions. I don't have much idea on how
>> replica fetcher threads and logs can be debugged. So asking for help here.
>> Any help or leads would be appreciated.
>>
>> --
>> *Suman*
>> *OlaCabs*
>>
>
>
> --
> *Suman*
> *OlaCabs*
>


-- 
*Suman*
*OlaCabs*

Request for contributor permission

[Andrew Schofield]

Please could I have contributor permission to Apache Kafka. Here are my details:

JIRA ID: schofielaj
Wiki ID: schofielaj

Thanks,
Andrew Schofield
IBM Event Streams

Re: [VOTE] KIP-377: TopicCommand to use AdminClient

[Viktor Somogyi-Vass]

Thanks for the votes so far.

@Colin: yup, I've added this to the KIP. Thanks.
On Thu, Oct 25, 2018 at 2:30 AM Colin McCabe  wrote:
>
> Thanks, Viktor.  +1 (binding).
>
> One note: can we add a deprecation warning when --zookeeper is used, to 
> indicate that this option will be phased out in the future?
>
> best,
> Colin
>
> On Wed, Oct 24, 2018, at 05:47, Mickael Maison wrote:
> > +1 (non-binding)
> > Thanks for the KIP!
> > On Wed, Oct 24, 2018 at 1:28 PM Viktor Somogyi-Vass
> >  wrote:
> > >
> > > Hi All,
> > >
> > > I'd like to start a vote on KIP-377:
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-377%3A+TopicCommand+to+use+AdminClient.
> > >
> > > Summary:
> > > The KIP basically proposes to add --bootstrap-server and
> > > --command-config option to TopicsCommand and implement topic
> > > administration with AdminClient in a backwards compatible way (so
> > > wouldn't drop or change the --zookeeper option usage).
> > >
> > > I'd appreciate any votes or feedback.
> > >
> > > Viktor

Build failed in Jenkins: kafka-trunk-jdk11 #54

[Apache Jenkins Server]

See 


Changes:

[manikumar.reddy] KAFKA-5462: Add configuration to build custom SSL principal 
name

--
[...truncated 2.34 MB...]

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairs STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKeyAndDefaultTimestamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKeyAndDefaultTimestamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithDefaultTimestamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithDefaultTimestamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKey STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKey PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithOtherTopicNameAndTimestampWithTimetamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithOtherTopicNameAndTimestampWithTimetamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullHeaders STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullHeaders PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithDefaultTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithDefaultTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecord PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 STARTED

org.apache.kafka.stream

Re: [VOTE] KIP-377: TopicCommand to use AdminClient

[Harsha Chintalapani]

Thanks for the KIP. +1 (binding).
-Harsha
On Oct 25, 2018, 10:40 AM -0400, Viktor Somogyi-Vass , 
wrote:
> Thanks for the votes so far.
>
> @Colin: yup, I've added this to the KIP. Thanks.
> On Thu, Oct 25, 2018 at 2:30 AM Colin McCabe  wrote:
> >
> > Thanks, Viktor. +1 (binding).
> >
> > One note: can we add a deprecation warning when --zookeeper is used, to 
> > indicate that this option will be phased out in the future?
> >
> > best,
> > Colin
> >
> > On Wed, Oct 24, 2018, at 05:47, Mickael Maison wrote:
> > > +1 (non-binding)
> > > Thanks for the KIP!
> > > On Wed, Oct 24, 2018 at 1:28 PM Viktor Somogyi-Vass
> > >  wrote:
> > > >
> > > > Hi All,
> > > >
> > > > I'd like to start a vote on KIP-377:
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-377%3A+TopicCommand+to+use+AdminClient.
> > > >
> > > > Summary:
> > > > The KIP basically proposes to add --bootstrap-server and
> > > > --command-config option to TopicsCommand and implement topic
> > > > administration with AdminClient in a backwards compatible way (so
> > > > wouldn't drop or change the --zookeeper option usage).
> > > >
> > > > I'd appreciate any votes or feedback.
> > > >
> > > > Viktor

Re: [DISCUSS] KIP-385: Provide configuration allowing consumer to no throw away prefetched data

[Colin McCabe]

Hi Zahari,

One question we didn't figure out earlier was who would actually want this 
cached data to be thrown away.  If there's nobody who actually wants this, then 
perhaps we can simplify the proposal by just unconditionally retaining the 
cache until the partition is resumed, or we unsubscribe from the partition.  
This would avoid adding a new configuration.

best,
Colin


On Sun, Oct 21, 2018, at 11:54, Zahari Dichev wrote:
> Hi there, although it has been discussed briefly already in this thread
> ,
> I decided to follow the process and initiate a DISCUSS thread. Comments 
> and
> suggestions are more than welcome.
> 
> 
> Zahari Dichev

Re: [VOTE] KIP-377: TopicCommand to use AdminClient

[Kevin Lu]

+1 (non-binding)

Regards,
Kevin

On Thu, Oct 25, 2018 at 8:19 AM Harsha Chintalapani  wrote:

> Thanks for the KIP. +1 (binding).
> -Harsha
> On Oct 25, 2018, 10:40 AM -0400, Viktor Somogyi-Vass <
> viktorsomo...@gmail.com>, wrote:
> > Thanks for the votes so far.
> >
> > @Colin: yup, I've added this to the KIP. Thanks.
> > On Thu, Oct 25, 2018 at 2:30 AM Colin McCabe  wrote:
> > >
> > > Thanks, Viktor. +1 (binding).
> > >
> > > One note: can we add a deprecation warning when --zookeeper is used,
> to indicate that this option will be phased out in the future?
> > >
> > > best,
> > > Colin
> > >
> > > On Wed, Oct 24, 2018, at 05:47, Mickael Maison wrote:
> > > > +1 (non-binding)
> > > > Thanks for the KIP!
> > > > On Wed, Oct 24, 2018 at 1:28 PM Viktor Somogyi-Vass
> > > >  wrote:
> > > > >
> > > > > Hi All,
> > > > >
> > > > > I'd like to start a vote on KIP-377:
> > > > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-377%3A+TopicCommand+to+use+AdminClient
> .
> > > > >
> > > > > Summary:
> > > > > The KIP basically proposes to add --bootstrap-server and
> > > > > --command-config option to TopicsCommand and implement topic
> > > > > administration with AdminClient in a backwards compatible way (so
> > > > > wouldn't drop or change the --zookeeper option usage).
> > > > >
> > > > > I'd appreciate any votes or feedback.
> > > > >
> > > > > Viktor
>

[jira] [Resolved] (KAFKA-7535) KafkaConsumer doesn't report records-lag if isolation.level is read_committed

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-7535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-7535.
--
Resolution: Fixed

> KafkaConsumer doesn't report records-lag if isolation.level is read_committed
> -
>
> Key: KAFKA-7535
> URL: https://issues.apache.org/jira/browse/KAFKA-7535
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 2.0.0
>Reporter: Alexey Vakhrenev
>Assignee: lambdaliu
>Priority: Major
>  Labels: regression
> Fix For: 2.0.1, 2.1.0
>
>
> Starting from 2.0.0, {{KafkaConsumer}} doesn't report {{records-lag}} if 
> {{isolation.level}} is {{read_committed}}. The last version, where it works 
> is {{1.1.1}}.
> The issue can be easily reproduced in {{kafka.api.PlaintextConsumerTest}} by 
> adding {{consumerConfig.setProperty("isolation.level", "read_committed")}} 
> witin related tests:
>  - {{testPerPartitionLagMetricsCleanUpWithAssign}}
>  - {{testPerPartitionLagMetricsCleanUpWithSubscribe}}
>  - {{testPerPartitionLagWithMaxPollRecords}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] KIP-385: Provide configuration allowing consumer to no throw away prefetched data

[Mayuresh Gharat]

Hi Colin/Zahari,

I have created a ticket for the similar/same feature :
https://issues.apache.org/jira/browse/KAFKA-7548
We (Linkedin) had a use case in Samza at Linkedin when they moved from the
SimpleConsumer to KafkaConsumer and they wanted to do this pause and resume
pattern.
They realized there was performance degradation when they started using
KafkaConsumer.assign() and pausing and unPausing partitions. We realized
that not throwing away the prefetched data for paused partitions might
improve the performance. We wrote a benchmark (I can share it if needed) to
prove this. I have attached the findings in the ticket.
We have been running the hotfix internally for quite a while now. When
samza ran this fix in production, they realized 30% improvement in there
app performance.
I have the patch ready on our internal branch and would like to submit a PR
for this on the above ticket asap.
I am not sure, if we need a separate config for this as we haven't seen a
lot of memory overhead due to this in our systems. We have had this running
in production for a considerable amount of time without any issues.
It would be great if you guys can review the PR once its up and see if that
satisfies your requirement. If it doesn't then we can think more on the
config driven approach.
Thoughts??

Thanks,

Mayuresh


On Thu, Oct 25, 2018 at 8:21 AM Colin McCabe  wrote:

> Hi Zahari,
>
> One question we didn't figure out earlier was who would actually want this
> cached data to be thrown away.  If there's nobody who actually wants this,
> then perhaps we can simplify the proposal by just unconditionally retaining
> the cache until the partition is resumed, or we unsubscribe from the
> partition.  This would avoid adding a new configuration.
>
> best,
> Colin
>
>
> On Sun, Oct 21, 2018, at 11:54, Zahari Dichev wrote:
> > Hi there, although it has been discussed briefly already in this thread
> > <
> https://lists.apache.org/thread.html/fbb7e9ccc41084fc2ff8612e6edf307fb400f806126b644d383b4a64@%3Cdev.kafka.apache.org%3E
> >,
> > I decided to follow the process and initiate a DISCUSS thread. Comments
> > and
> > suggestions are more than welcome.
> >
> >
> > Zahari Dichev
>


-- 
-Regards,
Mayuresh R. Gharat
(862) 250-7125

[jira] [Created] (KAFKA-7551) Refactor to create both producer & consumer in Worker

[Magesh kumar Nandakumar (JIRA)]

Magesh kumar Nandakumar created KAFKA-7551:
--

 Summary: Refactor to create both producer & consumer in Worker
 Key: KAFKA-7551
 URL: https://issues.apache.org/jira/browse/KAFKA-7551
 Project: Kafka
  Issue Type: Task
  Components: KafkaConnect
Reporter: Magesh kumar Nandakumar
Assignee: Magesh kumar Nandakumar
 Fix For: 2.2.0


In distributed mode,  the producer is created in the Worker and the consumer is 
created in the WorkerSinkTask. The proposal is to refactor it so that both of 
them are created in Worker. This will not affect any functionality and is just 
a refactoring to make the code consistent.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : kafka-trunk-jdk11 #55

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-7552) StatefulProcessorNode tries to connect state store to processor before it is added

[Adam Bellemare (JIRA)]

Adam Bellemare created KAFKA-7552:
-

 Summary: StatefulProcessorNode tries to connect state store to 
processor before it is added
 Key: KAFKA-7552
 URL: https://issues.apache.org/jira/browse/KAFKA-7552
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 2.0.0, 2.1.0
Reporter: Adam Bellemare


StatefulProcessorNode tries to "connectProcessorAndStateStores" before 
"addStateStore" is called on the state store. This throws an exception. Current 
implementations of Kafka Streams do not appear to test for this, nor do any of 
the kafka streams applications use it. Discovered while looking to use the node 
for another ticket.

[https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/kstream/internals/graph/StatefulProcessorNode.java#L86]

 

Results in "org.apache.kafka.streams.errors.TopologyException: Invalid 
topology: StateStore  is not added yet."



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-7553) Jenkins PR tests hung

[John Roesler (JIRA)]

John Roesler created KAFKA-7553:
---

 Summary: Jenkins PR tests hung
 Key: KAFKA-7553
 URL: https://issues.apache.org/jira/browse/KAFKA-7553
 Project: Kafka
  Issue Type: Bug
Reporter: John Roesler
 Attachments: consoleText.txt

I wouldn't worry about this unless it continues to happen, but I wanted to 
document it.

This was a Java 11 build: 
[https://builds.apache.org/job/kafka-pr-jdk11-scala2.12/266/]

It was for this PR: [https://github.com/apache/kafka/pull/5795]

And this commit: 
[https://github.com/apache/kafka/pull/5795/commits/5bdcd0e023c6f406d585155399f6541bb6a9f9c2]

 

It looks like the tests just hung after 46 minutes, until the build timed out 
at 180 minutes.

End of the output:
{noformat}
...
00:46:27.275 kafka.server.ServerGenerateBrokerIdTest > 
testConsistentBrokerIdFromUserConfigAndMetaProps STARTED
00:46:29.775 
00:46:29.775 kafka.server.ServerGenerateBrokerIdTest > 
testConsistentBrokerIdFromUserConfigAndMetaProps PASSED
03:00:51.124 Build timed out (after 180 minutes). Marking the build as aborted.
03:00:51.440 Build was aborted
03:00:51.492 [FINDBUGS] Skipping publisher since build result is ABORTED
03:00:51.492 Recording test results
03:00:51.495 Setting GRADLE_4_8_1_HOME=/home/jenkins/tools/gradle/4.8.1
03:00:58.017 Setting GRADLE_4_8_1_HOME=/home/jenkins/tools/gradle/4.8.1
03:00:59.330 Setting GRADLE_4_8_1_HOME=/home/jenkins/tools/gradle/4.8.1
03:00:59.331 Adding one-line test results to commit status...
03:00:59.332 Setting GRADLE_4_8_1_HOME=/home/jenkins/tools/gradle/4.8.1
03:00:59.334 Setting GRADLE_4_8_1_HOME=/home/jenkins/tools/gradle/4.8.1
03:00:59.335 Setting status of 5bdcd0e023c6f406d585155399f6541bb6a9f9c2 to 
FAILURE with url https://builds.apache.org/job/kafka-pr-jdk11-scala2.12/266/ 
and message: 'FAILURE
03:00:59.335  9053 tests run, 1 skipped, 0 failed.'
03:00:59.335 Using context: JDK 11 and Scala 2.12
03:00:59.541 Setting GRADLE_4_8_1_HOME=/home/jenkins/tools/gradle/4.8.1
03:00:59.542 Finished: ABORTED{noformat}
 

I did find one test that started but did not finish:
{noformat}
00:23:29.576 kafka.api.PlaintextConsumerTest > 
testLowMaxFetchSizeForRequestAndPartition STARTED
{noformat}
But note that the tests continued to run for another 23 minutes after this one 
started.

 

Just for completeness, there were 4 failures:
{noformat}
00:22:06.875 kafka.admin.ResetConsumerGroupOffsetTest > 
testResetOffsetsNotExistingGroup FAILED
00:22:06.875 java.util.concurrent.ExecutionException: 
org.apache.kafka.common.errors.CoordinatorNotAvailableException: The 
coordinator is not available.
00:22:06.875 at 
org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
00:22:06.875 at 
org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
00:22:06.875 at 
org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
00:22:06.876 at 
org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:262)
00:22:06.876 at 
kafka.admin.ConsumerGroupCommand$ConsumerGroupService.resetOffsets(ConsumerGroupCommand.scala:307)
00:22:06.876 at 
kafka.admin.ResetConsumerGroupOffsetTest.testResetOffsetsNotExistingGroup(ResetConsumerGroupOffsetTest.scala:89)
00:22:06.876 
00:22:06.876 Caused by:
00:22:06.876 
org.apache.kafka.common.errors.CoordinatorNotAvailableException: The 
coordinator is not available.{noformat}
 
{noformat}
00:25:22.175 kafka.api.CustomQuotaCallbackTest > testCustomQuotaCallback FAILED
00:25:22.175 java.lang.AssertionError: Partition [group1_largeTopic,69] 
metadata not propagated after 15000 ms
00:25:22.176 at kafka.utils.TestUtils$.fail(TestUtils.scala:351)
00:25:22.176 at 
kafka.utils.TestUtils$.waitUntilTrue(TestUtils.scala:741)
00:25:22.176 at 
kafka.utils.TestUtils$.waitUntilMetadataIsPropagated(TestUtils.scala:831)
00:25:22.176 at 
kafka.utils.TestUtils$$anonfun$createTopic$2.apply(TestUtils.scala:330)
00:25:22.176 at 
kafka.utils.TestUtils$$anonfun$createTopic$2.apply(TestUtils.scala:329)
00:25:22.176 at 
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
00:25:22.176 at 
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
00:25:22.176 at 
scala.collection.Iterator$class.foreach(Iterator.scala:891)
00:25:22.176 at 
scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
00:25:22.176 at 
scala.collection.MapLike$DefaultKeySet.foreach(MapLike.scala:174)
00:25:22.176 at 
scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
00:25:22.176 at 
scala.collection.AbstractSet.scala$collection$SetLike$$super$map(Set.scala:47)
00:25:22.176 at scala.collection.SetLike$class.map(SetLike.scala:92)
00:25:22.17

[jira] [Created] (KAFKA-7554) zookeeper.session.timeout.ms Value

[BELUGA BEHR (JIRA)]

BELUGA BEHR created KAFKA-7554:
--

 Summary: zookeeper.session.timeout.ms Value
 Key: KAFKA-7554
 URL: https://issues.apache.org/jira/browse/KAFKA-7554
 Project: Kafka
  Issue Type: Improvement
  Components: zkclient
Reporter: BELUGA BEHR


{quote}
zookeeper.session.timeout.ms = 6000 (6s)
zookeeper.connection.timeout.ms = 6000 (6s)
{quote}
- https://kafka.apache.org/documentation/#configuration

Kind of an odd value?  Was it supposed to be 6 (60s) ?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] KIP-385: Provide configuration allowing consumer to no throw away prefetched data

[Zahari Dichev]

Hi there Mayuresh,

Great to heat that this is actually working well in production for some
time now. I have changed the details of the KIP to reflect the fact that as
already discussed - we do not really need any kind of configuration as this
data should not be thrown away at all.  Submitting a PR sounds great,
although I feel a bit jealous you (LinkedIn) beat me to my first kafka
commit  ;)  Not sure how things stand with the voting process ?

Zahari



On Thu, Oct 25, 2018 at 7:39 PM Mayuresh Gharat 
wrote:

> Hi Colin/Zahari,
>
> I have created a ticket for the similar/same feature :
> https://issues.apache.org/jira/browse/KAFKA-7548
> We (Linkedin) had a use case in Samza at Linkedin when they moved from the
> SimpleConsumer to KafkaConsumer and they wanted to do this pause and resume
> pattern.
> They realized there was performance degradation when they started using
> KafkaConsumer.assign() and pausing and unPausing partitions. We realized
> that not throwing away the prefetched data for paused partitions might
> improve the performance. We wrote a benchmark (I can share it if needed) to
> prove this. I have attached the findings in the ticket.
> We have been running the hotfix internally for quite a while now. When
> samza ran this fix in production, they realized 30% improvement in there
> app performance.
> I have the patch ready on our internal branch and would like to submit a PR
> for this on the above ticket asap.
> I am not sure, if we need a separate config for this as we haven't seen a
> lot of memory overhead due to this in our systems. We have had this running
> in production for a considerable amount of time without any issues.
> It would be great if you guys can review the PR once its up and see if that
> satisfies your requirement. If it doesn't then we can think more on the
> config driven approach.
> Thoughts??
>
> Thanks,
>
> Mayuresh
>
>
> On Thu, Oct 25, 2018 at 8:21 AM Colin McCabe  wrote:
>
> > Hi Zahari,
> >
> > One question we didn't figure out earlier was who would actually want
> this
> > cached data to be thrown away.  If there's nobody who actually wants
> this,
> > then perhaps we can simplify the proposal by just unconditionally
> retaining
> > the cache until the partition is resumed, or we unsubscribe from the
> > partition.  This would avoid adding a new configuration.
> >
> > best,
> > Colin
> >
> >
> > On Sun, Oct 21, 2018, at 11:54, Zahari Dichev wrote:
> > > Hi there, although it has been discussed briefly already in this thread
> > > <
> >
> https://lists.apache.org/thread.html/fbb7e9ccc41084fc2ff8612e6edf307fb400f806126b644d383b4a64@%3Cdev.kafka.apache.org%3E
> > >,
> > > I decided to follow the process and initiate a DISCUSS thread. Comments
> > > and
> > > suggestions are more than welcome.
> > >
> > >
> > > Zahari Dichev
> >
>
>
> --
> -Regards,
> Mayuresh R. Gharat
> (862) 250-7125
>

Build failed in Jenkins: kafka-2.0-jdk8 #175

[Apache Jenkins Server]

See 


Changes:

[lindong28] KAFKA-7535; KafkaConsumer doesn't report records-lag if 
isolation.level

--
[...truncated 928.21 KB...]

kafka.security.auth.ZkAuthorizationTest > testConsumerOffsetPathAcls PASSED

kafka.security.auth.ZkAuthorizationTest > testZkMigration STARTED

kafka.security.auth.ZkAuthorizationTest > testZkMigration PASSED

kafka.security.auth.ZkAuthorizationTest > testChroot STARTED

kafka.security.auth.ZkAuthorizationTest > testChroot PASSED

kafka.security.auth.ZkAuthorizationTest > testDelete STARTED

kafka.security.auth.ZkAuthorizationTest > testDelete PASSED

kafka.security.auth.ZkAuthorizationTest > testDeleteRecursive STARTED

kafka.security.auth.ZkAuthorizationTest > testDeleteRecursive PASSED

kafka.security.auth.AclTest > testAclJsonConversion STARTED

kafka.security.auth.AclTest > testAclJsonConversion PASSED

kafka.security.auth.OperationTest > testJavaConversions STARTED

kafka.security.auth.OperationTest > testJavaConversions PASSED

kafka.security.auth.ResourceTest > shouldParseOldTwoPartString STARTED

kafka.security.auth.ResourceTest > shouldParseOldTwoPartString PASSED

kafka.security.auth.ResourceTest > shouldParseOldTwoPartWithEmbeddedSeparators 
STARTED

kafka.security.auth.ResourceTest > shouldParseOldTwoPartWithEmbeddedSeparators 
PASSED

kafka.security.auth.ResourceTest > 
shouldThrowOnTwoPartStringWithUnknownResourceType STARTED

kafka.security.auth.ResourceTest > 
shouldThrowOnTwoPartStringWithUnknownResourceType PASSED

kafka.security.auth.ResourceTest > shouldThrowOnBadResourceTypeSeparator STARTED

kafka.security.auth.ResourceTest > shouldThrowOnBadResourceTypeSeparator PASSED

kafka.security.auth.ResourceTest > shouldParseThreePartString STARTED

kafka.security.auth.ResourceTest > shouldParseThreePartString PASSED

kafka.security.auth.ResourceTest > shouldRoundTripViaString STARTED

kafka.security.auth.ResourceTest > shouldRoundTripViaString PASSED

kafka.security.auth.ResourceTest > shouldParseThreePartWithEmbeddedSeparators 
STARTED

kafka.security.auth.ResourceTest > shouldParseThreePartWithEmbeddedSeparators 
PASSED

kafka.security.auth.ResourceTypeTest > testJavaConversions STARTED

kafka.security.auth.ResourceTypeTest > testJavaConversions PASSED

kafka.security.auth.ResourceTypeTest > testFromString STARTED

kafka.security.auth.ResourceTypeTest > testFromString PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testAuthorizeWithPrefixedResource 
STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testAuthorizeWithPrefixedResource 
PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testAllowAllAccess STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testAllowAllAccess PASSED

kafka.security.auth.SimpleAclAuthorizerTest > 
testLocalConcurrentModificationOfResourceAcls STARTED

kafka.security.auth.SimpleAclAuthorizerTest > 
testLocalConcurrentModificationOfResourceAcls PASSED

kafka.security.auth.SimpleAclAuthorizerTest > 
testDeleteAllAclOnWildcardResource STARTED

kafka.security.auth.SimpleAclAuthorizerTest > 
testDeleteAllAclOnWildcardResource PASSED

kafka.security.auth.SimpleAclAuthorizerTest > 
testHighConcurrencyDeletionOfResourceAcls STARTED

kafka.security.auth.SimpleAclAuthorizerTest > 
testHighConcurrencyDeletionOfResourceAcls PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testNoAclFound STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testNoAclFound PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testAclInheritance STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testAclInheritance PASSED

kafka.security.auth.SimpleAclAuthorizerTest > 
testDistributedConcurrentModificationOfResourceAcls STARTED

kafka.security.auth.SimpleAclAuthorizerTest > 
testDistributedConcurrentModificationOfResourceAcls PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testAddAclsOnWildcardResource 
STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testAddAclsOnWildcardResource 
PASSED

kafka.security.auth.SimpleAclAuthorizerTest > 
testWritesExtendedAclChangeEventWhenInterBrokerProtocolAtLeastKafkaV2 STARTED

kafka.security.auth.SimpleAclAuthorizerTest > 
testWritesExtendedAclChangeEventWhenInterBrokerProtocolAtLeastKafkaV2 PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testAclManagementAPIs STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testAclManagementAPIs PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testWildCardAcls STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testWildCardAcls PASSED

kafka.security.auth.SimpleAclAuthorizerTest > 
testWritesLiteralAclChangeEventWhenInterBrokerProtocolIsKafkaV2 STARTED

kafka.security.auth.SimpleAclAuthorizerTest > 
testWritesLiteralAclChangeEventWhenInterBrokerProtocolIsKafkaV2 PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testTopicAcl STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testTopicAcl PASSED

kafka.security

Build failed in Jenkins: kafka-trunk-jdk11 #56

[Apache Jenkins Server]

See 


Changes:

[lindong28] MINOR: add upgrade note for KIP-336

--
[...truncated 2.35 MB...]

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairs STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKeyAndDefaultTimestamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKeyAndDefaultTimestamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithDefaultTimestamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithDefaultTimestamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKey STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKey PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithOtherTopicNameAndTimestampWithTimetamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithOtherTopicNameAndTimestampWithTimetamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullHeaders STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullHeaders PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithDefaultTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithDefaultTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecord STARTED

org.apache.kafka.streams.te

Re: [VOTE] KIP-377: TopicCommand to use AdminClient

[Gwen Shapira]

+1 (binding)
Thanks for working on this.

On Wed, Oct 24, 2018, 7:28 AM Viktor Somogyi-Vass 
wrote:

> Hi All,
>
> I'd like to start a vote on KIP-377:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-377%3A+TopicCommand+to+use+AdminClient
> .
>
> Summary:
> The KIP basically proposes to add --bootstrap-server and
> --command-config option to TopicsCommand and implement topic
> administration with AdminClient in a backwards compatible way (so
> wouldn't drop or change the --zookeeper option usage).
>
> I'd appreciate any votes or feedback.
>
> Viktor
>

Build failed in Jenkins: kafka-2.0-jdk8 #176

[Apache Jenkins Server]

See 

--
[...truncated 434.91 KB...]
kafka.security.auth.ZkAuthorizationTest > testConsumerOffsetPathAcls STARTED

kafka.security.auth.ZkAuthorizationTest > testConsumerOffsetPathAcls PASSED

kafka.security.auth.ZkAuthorizationTest > testZkMigration STARTED

kafka.security.auth.ZkAuthorizationTest > testZkMigration PASSED

kafka.security.auth.ZkAuthorizationTest > testChroot STARTED

kafka.security.auth.ZkAuthorizationTest > testChroot PASSED

kafka.security.auth.ZkAuthorizationTest > testDelete STARTED

kafka.security.auth.ZkAuthorizationTest > testDelete PASSED

kafka.security.auth.ZkAuthorizationTest > testDeleteRecursive STARTED

kafka.security.auth.ZkAuthorizationTest > testDeleteRecursive PASSED

kafka.security.auth.AclTest > testAclJsonConversion STARTED

kafka.security.auth.AclTest > testAclJsonConversion PASSED

kafka.security.auth.OperationTest > testJavaConversions STARTED

kafka.security.auth.OperationTest > testJavaConversions PASSED

kafka.security.auth.ResourceTest > shouldParseOldTwoPartString STARTED

kafka.security.auth.ResourceTest > shouldParseOldTwoPartString PASSED

kafka.security.auth.ResourceTest > shouldParseOldTwoPartWithEmbeddedSeparators 
STARTED

kafka.security.auth.ResourceTest > shouldParseOldTwoPartWithEmbeddedSeparators 
PASSED

kafka.security.auth.ResourceTest > 
shouldThrowOnTwoPartStringWithUnknownResourceType STARTED

kafka.security.auth.ResourceTest > 
shouldThrowOnTwoPartStringWithUnknownResourceType PASSED

kafka.security.auth.ResourceTest > shouldThrowOnBadResourceTypeSeparator STARTED

kafka.security.auth.ResourceTest > shouldThrowOnBadResourceTypeSeparator PASSED

kafka.security.auth.ResourceTest > shouldParseThreePartString STARTED

kafka.security.auth.ResourceTest > shouldParseThreePartString PASSED

kafka.security.auth.ResourceTest > shouldRoundTripViaString STARTED

kafka.security.auth.ResourceTest > shouldRoundTripViaString PASSED

kafka.security.auth.ResourceTest > shouldParseThreePartWithEmbeddedSeparators 
STARTED

kafka.security.auth.ResourceTest > shouldParseThreePartWithEmbeddedSeparators 
PASSED

kafka.security.auth.ResourceTypeTest > testJavaConversions STARTED

kafka.security.auth.ResourceTypeTest > testJavaConversions PASSED

kafka.security.auth.ResourceTypeTest > testFromString STARTED

kafka.security.auth.ResourceTypeTest > testFromString PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testAuthorizeWithPrefixedResource 
STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testAuthorizeWithPrefixedResource 
PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testAllowAllAccess STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testAllowAllAccess PASSED

kafka.security.auth.SimpleAclAuthorizerTest > 
testLocalConcurrentModificationOfResourceAcls STARTED

kafka.security.auth.SimpleAclAuthorizerTest > 
testLocalConcurrentModificationOfResourceAcls PASSED

kafka.security.auth.SimpleAclAuthorizerTest > 
testDeleteAllAclOnWildcardResource STARTED

kafka.security.auth.SimpleAclAuthorizerTest > 
testDeleteAllAclOnWildcardResource PASSED

kafka.security.auth.SimpleAclAuthorizerTest > 
testHighConcurrencyDeletionOfResourceAcls STARTED

kafka.security.auth.SimpleAclAuthorizerTest > 
testHighConcurrencyDeletionOfResourceAcls PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testNoAclFound STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testNoAclFound PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testAclInheritance STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testAclInheritance PASSED

kafka.security.auth.SimpleAclAuthorizerTest > 
testDistributedConcurrentModificationOfResourceAcls STARTED

kafka.security.auth.SimpleAclAuthorizerTest > 
testDistributedConcurrentModificationOfResourceAcls PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testAddAclsOnWildcardResource 
STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testAddAclsOnWildcardResource 
PASSED

kafka.security.auth.SimpleAclAuthorizerTest > 
testWritesExtendedAclChangeEventWhenInterBrokerProtocolAtLeastKafkaV2 STARTED

kafka.security.auth.SimpleAclAuthorizerTest > 
testWritesExtendedAclChangeEventWhenInterBrokerProtocolAtLeastKafkaV2 PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testAclManagementAPIs STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testAclManagementAPIs PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testWildCardAcls STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testWildCardAcls PASSED

kafka.security.auth.SimpleAclAuthorizerTest > 
testWritesLiteralAclChangeEventWhenInterBrokerProtocolIsKafkaV2 STARTED

kafka.security.auth.SimpleAclAuthorizerTest > 
testWritesLiteralAclChangeEventWhenInterBrokerProtocolIsKafkaV2 PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testTopicAcl STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testTopicAcl PASSED

kafka.security.auth.SimpleAclAuthorizerTest > t

Re: Request for contributor permission

[Matthias J. Sax]

Done.

On 10/25/18 7:27 AM, Andrew Schofield wrote:
> Please could I have contributor permission to Apache Kafka. Here are my 
> details:
> 
> JIRA ID: schofielaj
> Wiki ID: schofielaj
> 
> Thanks,
> Andrew Schofield
> IBM Event Streams
> 



signature.asc
Description: OpenPGP digital signature

Re: [DISCUSS] KIP-385: Provide configuration allowing consumer to no throw away prefetched data

[Mayuresh Gharat]

Hi Zahari,

Oops. We had planned to put this patch upstream but somehow slipped my
mind. We were recently going over hotfixes that we have and this seemed
something that had been due for sometime now. Glad to know that someone
else apart from us might also benefit from this :)

Thanks,

Mayuresh

On Thu, Oct 25, 2018 at 12:25 PM Zahari Dichev 
wrote:

> Hi there Mayuresh,
>
> Great to heat that this is actually working well in production for some
> time now. I have changed the details of the KIP to reflect the fact that as
> already discussed - we do not really need any kind of configuration as this
> data should not be thrown away at all.  Submitting a PR sounds great,
> although I feel a bit jealous you (LinkedIn) beat me to my first kafka
> commit  ;)  Not sure how things stand with the voting process ?
>
> Zahari
>
>
>
> On Thu, Oct 25, 2018 at 7:39 PM Mayuresh Gharat <
> gharatmayures...@gmail.com>
> wrote:
>
> > Hi Colin/Zahari,
> >
> > I have created a ticket for the similar/same feature :
> > https://issues.apache.org/jira/browse/KAFKA-7548
> > We (Linkedin) had a use case in Samza at Linkedin when they moved from
> the
> > SimpleConsumer to KafkaConsumer and they wanted to do this pause and
> resume
> > pattern.
> > They realized there was performance degradation when they started using
> > KafkaConsumer.assign() and pausing and unPausing partitions. We realized
> > that not throwing away the prefetched data for paused partitions might
> > improve the performance. We wrote a benchmark (I can share it if needed)
> to
> > prove this. I have attached the findings in the ticket.
> > We have been running the hotfix internally for quite a while now. When
> > samza ran this fix in production, they realized 30% improvement in there
> > app performance.
> > I have the patch ready on our internal branch and would like to submit a
> PR
> > for this on the above ticket asap.
> > I am not sure, if we need a separate config for this as we haven't seen a
> > lot of memory overhead due to this in our systems. We have had this
> running
> > in production for a considerable amount of time without any issues.
> > It would be great if you guys can review the PR once its up and see if
> that
> > satisfies your requirement. If it doesn't then we can think more on the
> > config driven approach.
> > Thoughts??
> >
> > Thanks,
> >
> > Mayuresh
> >
> >
> > On Thu, Oct 25, 2018 at 8:21 AM Colin McCabe  wrote:
> >
> > > Hi Zahari,
> > >
> > > One question we didn't figure out earlier was who would actually want
> > this
> > > cached data to be thrown away.  If there's nobody who actually wants
> > this,
> > > then perhaps we can simplify the proposal by just unconditionally
> > retaining
> > > the cache until the partition is resumed, or we unsubscribe from the
> > > partition.  This would avoid adding a new configuration.
> > >
> > > best,
> > > Colin
> > >
> > >
> > > On Sun, Oct 21, 2018, at 11:54, Zahari Dichev wrote:
> > > > Hi there, although it has been discussed briefly already in this
> thread
> > > > <
> > >
> >
> https://lists.apache.org/thread.html/fbb7e9ccc41084fc2ff8612e6edf307fb400f806126b644d383b4a64@%3Cdev.kafka.apache.org%3E
> > > >,
> > > > I decided to follow the process and initiate a DISCUSS thread.
> Comments
> > > > and
> > > > suggestions are more than welcome.
> > > >
> > > >
> > > > Zahari Dichev
> > >
> >
> >
> > --
> > -Regards,
> > Mayuresh R. Gharat
> > (862) 250-7125
> >
>


-- 
-Regards,
Mayuresh R. Gharat
(862) 250-7125

Re: [DISCUSS] KIP-310: Add a Kafka Source Connector to Kafka Connect

[McCaig, Rhys]

Hi All,

Based on the feedback in this thread, and in light of Ryanne’s excellent 
proposal (KIP-382: MirrorMaker 2.0) which incorporates and extends the goals of 
KIP-310, I have updated the status of KIP-310 to “Discarded" and added a 
comment that KIP-382 supersedes it.

Thank you all for the discussion and feedback - this is my first KIP and I 
appreciate the community providing feedback on my contributions!

Rhys

> On Sep 26, 2018, at 10:42 AM, Konstantine Karantasis 
>  wrote:
> 
> Hi Rhys,
> 
> thanks for the proposal and apologies for the late feedback. Utilizing
> Connect to mirror Kafka topics is definitely a plausible proposal for a
> very useful use case.
> 
> However, I don't think the apache/kafka repository is the right place to
> host such a Connector. Currently, no full-featured, production-ready
> connectors are hosted in AK. The only two connectors shipped with AK
> (FileStreamSourceConnector and FileStreamSinkConnector) are there to
> demonstrate implementations only as examples.
> 
> I find this approach very appealing. AK focuses on providing the core
> infrastructure for Connect, that is required in every Kafka Connect
> deployment, as well as offering the means to generically install, deploy
> and operate connectors. But all the connectors reside outside AK and
> comprise a vibrant ecosystem of open source and proprietary components
> that, essentially - even for the most useful and ubiquitous of the
> connectors - are optional for users to install and use. This seems simple
> and flexible, both in terms of releasing and using/deploying software
> related to Kafka Connect. I might even say that I'd be in favor of
> extending this approach to all the Connect components, including
> Transformations and Converters.
> 
> I'm aware that MirrorMaker is part of AK, but to me this refers to the
> early days of Apache Kafka, when the size of the project and the ecosystem
> was smaller, Connect and Streams had not been implemented yet, and
> mirroring topics between Kafka clusters was already a basic need. With a
> much more rich ecosystem now and more sizable and well defined packages in
> AK, I think the approach that decouples connectors from the Connect
> framework itself is a good one.
> 
> In my opinion, the fact that this connector targets Kafka itself as a
> source is not an adequate reason to include it in apache/kafka within the
> Connect framework. It seems it can evolve naturally, as every other
> connector, in its own repository.
> 
> Regards,
> Konstantine
> 
> 
> On Sat, Aug 4, 2018 at 7:20 PM McCaig, Rhys  wrote:
> 
>> Hi All,
>> 
>> If there are no further comments on this KIP I’ll start a vote early this
>> week.
>> 
>> Rhys
>> 
>> On Aug 1, 2018, at 12:32 AM, McCaig, Rhys > > wrote:
>> 
>> Hi All,
>> 
>> I’ve updated the proposal to include the improvements suggested by
>> Stephane.
>> 
>> I have also submitted a PR to implement this functionality into Kafka.
>> https://github.com/apache/kafka/pull/5438
>> 
>> I don’t have a benchmark against MirrorMaker yet, as I only currently have
>> a local docker stack available to me, though I have seen very good
>> performance in that test stack (200k messages/sec@100bytes on limited
>> compute resource containers). Further benchmarking might take a few days.
>> 
>> Review and comments would be appreciated.
>> 
>> Cheers,
>> Rhys
>> 
>> 
>> On Jun 18, 2018, at 9:00 AM, McCaig, Rhys > > wrote:
>> 
>> Hi Stephane,
>> 
>> Thanks for your feedback and apologies for the delay in my response.
>> 
>> Are there any performance benchmarks against Mirror Maker available? I'm
>> interested to know if this is more performant / scalable.
>> Regarding the implementation, here's some feedback:
>> 
>> 
>> Currently I don’t have any performance benchmarks, but I think this is a
>> great idea, ill see if I can set up something one the next week or so.
>> 
>> - I think it's worth mentioning that this solution does not rely on
>> consumer groups, and therefore tracking progress may be tricky. Can you
>> think of a way to expose that?
>> 
>> This is a reasonable concern. I’m not sure how to track this other than
>> looking at the Kafka connect offsets. Once a messages is passed to the
>> framework, I'm unaware of a way to get at the commit offsets on the
>> producer side. Any thoughts?
>> 
>> - Some code can be in config Validator I believe:
>> 
>> https://github.com/Comcast/MirrorTool-for-Kafka-Connect/blob/master/src/main/java/com/comcast/kafka/connect/kafka/KafkaSourceConnector.java#L47
>> 
>> - I think your kip mentions `source.admin.` and `source.consumer.` but I
>> don't see it reflected yet in the code
>> 
>> - Is there a way to be flexible and merge list and regex, or offer the two
>> simultaneously ? source_topics=my_static_topic,prefix.* ?
>> 
>> Agree on all of the above - I will incorporate into the code later this
>> week as ill get some time back to work on this.
>

Re: [VOTE] KIP-349 Priorities for Source Topics

[nick]

The reporter of KAFKA-6690 (Bala) replied in the JIra ticket to my question to 
elaborate about his use-case.  I don’t think he’s on the dev list.  Here’s his 
response:  

  Bala:  Sorry about the delay in reply. We use Kafka to process the 
asynchronous events of our Document Management System such as preview 
generation, indexing for search etc. The traffic gets generated via Web and 
Desktop Sync application. In such cases, we had to prioritize the traffic from 
web and consume them first. But this might lead to the starvation of events 
from sync if the consumer speed is slow and the event rate is high from web. A 
solution to handle the starvation with a timeout after which the events are 
consumed normally for a specified period of time would be great and help us use 
our resources effectively.

--
  Nick




> On Oct 18, 2018, at 12:23 PM, n...@afshartous.com wrote:
> 
>> On Oct 12, 2018, at 5:06 PM, Colin McCabe  wrote:
>> 
>> Maybe there's some really cool use-case that I haven't thought of.  But so 
>> far I can't really think of any time I would need topic priorities if I was 
>> muting topics and offloading blocking operations in a reasonable way.  It 
>> would be good to identify use-cases 
> 
> 
> Hi Colin,
> 
> How about the use-case where there are multiple streams/topics, and the 
> intent is to have a single consumer interleave the messages so that higher 
> priority messages are processed first ?
> That seems to be what the reporter of the associated Jira ticket
> 
>   https://issues.apache.org/jira/browse/KAFKA-6690 
> 
> 
> has identified as a use-case he frequently encounters.  I’ve asked him to 
> elaborate on the dev list though he has not responded yet.
> 
> Best,
> --
>  Nick
> 
> 
> 

Re: [DISCUSS] KIP-385: Provide configuration allowing consumer to no throw away prefetched data

[Mayuresh Gharat]

Hi Zahari,

Created the patch here : https://github.com/apache/kafka/pull/5844

Thanks,

Mayuresh

On Thu, Oct 25, 2018 at 4:42 PM Mayuresh Gharat 
wrote:

> Hi Zahari,
>
> Oops. We had planned to put this patch upstream but somehow slipped my
> mind. We were recently going over hotfixes that we have and this seemed
> something that had been due for sometime now. Glad to know that someone
> else apart from us might also benefit from this :)
>
> Thanks,
>
> Mayuresh
>
> On Thu, Oct 25, 2018 at 12:25 PM Zahari Dichev 
> wrote:
>
>> Hi there Mayuresh,
>>
>> Great to heat that this is actually working well in production for some
>> time now. I have changed the details of the KIP to reflect the fact that
>> as
>> already discussed - we do not really need any kind of configuration as
>> this
>> data should not be thrown away at all.  Submitting a PR sounds great,
>> although I feel a bit jealous you (LinkedIn) beat me to my first kafka
>> commit  ;)  Not sure how things stand with the voting process ?
>>
>> Zahari
>>
>>
>>
>> On Thu, Oct 25, 2018 at 7:39 PM Mayuresh Gharat <
>> gharatmayures...@gmail.com>
>> wrote:
>>
>> > Hi Colin/Zahari,
>> >
>> > I have created a ticket for the similar/same feature :
>> > https://issues.apache.org/jira/browse/KAFKA-7548
>> > We (Linkedin) had a use case in Samza at Linkedin when they moved from
>> the
>> > SimpleConsumer to KafkaConsumer and they wanted to do this pause and
>> resume
>> > pattern.
>> > They realized there was performance degradation when they started using
>> > KafkaConsumer.assign() and pausing and unPausing partitions. We realized
>> > that not throwing away the prefetched data for paused partitions might
>> > improve the performance. We wrote a benchmark (I can share it if
>> needed) to
>> > prove this. I have attached the findings in the ticket.
>> > We have been running the hotfix internally for quite a while now. When
>> > samza ran this fix in production, they realized 30% improvement in there
>> > app performance.
>> > I have the patch ready on our internal branch and would like to submit
>> a PR
>> > for this on the above ticket asap.
>> > I am not sure, if we need a separate config for this as we haven't seen
>> a
>> > lot of memory overhead due to this in our systems. We have had this
>> running
>> > in production for a considerable amount of time without any issues.
>> > It would be great if you guys can review the PR once its up and see if
>> that
>> > satisfies your requirement. If it doesn't then we can think more on the
>> > config driven approach.
>> > Thoughts??
>> >
>> > Thanks,
>> >
>> > Mayuresh
>> >
>> >
>> > On Thu, Oct 25, 2018 at 8:21 AM Colin McCabe 
>> wrote:
>> >
>> > > Hi Zahari,
>> > >
>> > > One question we didn't figure out earlier was who would actually want
>> > this
>> > > cached data to be thrown away.  If there's nobody who actually wants
>> > this,
>> > > then perhaps we can simplify the proposal by just unconditionally
>> > retaining
>> > > the cache until the partition is resumed, or we unsubscribe from the
>> > > partition.  This would avoid adding a new configuration.
>> > >
>> > > best,
>> > > Colin
>> > >
>> > >
>> > > On Sun, Oct 21, 2018, at 11:54, Zahari Dichev wrote:
>> > > > Hi there, although it has been discussed briefly already in this
>> thread
>> > > > <
>> > >
>> >
>> https://lists.apache.org/thread.html/fbb7e9ccc41084fc2ff8612e6edf307fb400f806126b644d383b4a64@%3Cdev.kafka.apache.org%3E
>> > > >,
>> > > > I decided to follow the process and initiate a DISCUSS thread.
>> Comments
>> > > > and
>> > > > suggestions are more than welcome.
>> > > >
>> > > >
>> > > > Zahari Dichev
>> > >
>> >
>> >
>> > --
>> > -Regards,
>> > Mayuresh R. Gharat
>> > (862) 250-7125
>> >
>>
>
>
> --
> -Regards,
> Mayuresh R. Gharat
> (862) 250-7125
>


-- 
-Regards,
Mayuresh R. Gharat
(862) 250-7125

Jenkins build is back to normal : kafka-2.0-jdk8 #177

[Apache Jenkins Server]

See 

[VOTE] 2.0.1 RC0

[Manikumar]

Hello Kafka users, developers and client-developers,

This is the first candidate for release of Apache Kafka 2.0.1.

This is a bug fix release closing 49 tickets:
https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+2.0.1

Release notes for the 2.0.1 release:
http://home.apache.org/~manikumar/kafka-2.0.1-rc0/RELEASE_NOTES.html

*** Please download, test and vote by  Tuesday, October 30, end of day

Kafka's KEYS file containing PGP keys we use to sign the release:
http://kafka.apache.org/KEYS

* Release artifacts to be voted upon (source and binary):
http://home.apache.org/~manikumar/kafka-2.0.1-rc0/

* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/

* Javadoc:
http://home.apache.org/~manikumar/kafka-2.0.1-rc0/javadoc/

* Tag to be voted upon (off 2.0 branch) is the 2.0.1 tag:
https://github.com/apache/kafka/releases/tag/2.0.1-rc0

* Documentation:
http://kafka.apache.org/20/documentation.html

* Protocol:
http://kafka.apache.org/20/protocol.html

* Successful Jenkins builds for the 2.0 branch:
Unit/integration tests: https://builds.apache.org/job/kafka-2.0-jdk8/177/

/**

Thanks,
Manikumar

Jenkins build is back to normal : kafka-trunk-jdk11 #57

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-7555) If the disk is full, Kafka will suddenly fail.

[hajin kim (JIRA)]

hajin kim created KAFKA-7555:


 Summary: If the disk is full, Kafka will suddenly fail. 
 Key: KAFKA-7555
 URL: https://issues.apache.org/jira/browse/KAFKA-7555
 Project: Kafka
  Issue Type: Bug
Reporter: hajin kim


If any of the log repositories are full, Kafka will turn off and will not 
restart. After that, I can restart by manually removing the saved logs, but 
this seems obviously a Kafka problem. Is there a way to prevent or avoid this 
problem?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)