Build failed in Jenkins: kafka-trunk-jdk11 #720

[Apache Jenkins Server]

See 


Changes:

[cmccabe] KAFKA-8345: KIP-455 Protocol changes (part 1) (#7114)

--
[...truncated 6.29 MB...]
kafka.api.SslProducerSendTest > testCloseWithZeroTimeoutFromCallerThread PASSED

kafka.api.SslProducerSendTest > testCloseWithZeroTimeoutFromSenderThread STARTED

kafka.api.SslProducerSendTest > testCloseWithZeroTimeoutFromSenderThread PASSED

kafka.api.SslProducerSendTest > testSendBeforeAndAfterPartitionExpansion STARTED

kafka.api.SslProducerSendTest > testSendBeforeAndAfterPartitionExpansion PASSED

kafka.api.ConsumerBounceTest > testCloseDuringRebalance STARTED

kafka.api.ConsumerBounceTest > testCloseDuringRebalance PASSED

kafka.api.ConsumerBounceTest > testClose STARTED

kafka.api.ConsumerBounceTest > testClose PASSED

kafka.api.ConsumerBounceTest > testSeekAndCommitWithBrokerFailures STARTED

kafka.api.ConsumerBounceTest > testSeekAndCommitWithBrokerFailures PASSED

kafka.api.ConsumerBounceTest > 
testConsumerReceivesFatalExceptionWhenGroupPassesMaxSize STARTED

kafka.api.ConsumerBounceTest > 
testConsumerReceivesFatalExceptionWhenGroupPassesMaxSize PASSED

kafka.api.ConsumerBounceTest > testSubscribeWhenTopicUnavailable STARTED

kafka.api.ConsumerBounceTest > testSubscribeWhenTopicUnavailable PASSED

kafka.api.ConsumerBounceTest > 
testRollingBrokerRestartsWithSmallerMaxGroupSizeConfigDisruptsBigGroup STARTED

kafka.api.ConsumerBounceTest > 
testRollingBrokerRestartsWithSmallerMaxGroupSizeConfigDisruptsBigGroup PASSED

kafka.api.ConsumerBounceTest > testConsumptionWithBrokerFailures STARTED

kafka.api.ConsumerBounceTest > testConsumptionWithBrokerFailures SKIPPED

kafka.api.PlaintextProducerSendTest > 
testSendCompressedMessageWithLogAppendTime STARTED

kafka.api.PlaintextProducerSendTest > 
testSendCompressedMessageWithLogAppendTime PASSED

kafka.api.PlaintextProducerSendTest > testAutoCreateTopic STARTED

kafka.api.PlaintextProducerSendTest > testAutoCreateTopic PASSED

kafka.api.PlaintextProducerSendTest > testSendWithInvalidCreateTime STARTED

kafka.api.PlaintextProducerSendTest > testSendWithInvalidCreateTime PASSED

kafka.api.PlaintextProducerSendTest > testBatchSizeZero STARTED

kafka.api.PlaintextProducerSendTest > testBatchSizeZero PASSED

kafka.api.PlaintextProducerSendTest > testWrongSerializer STARTED

kafka.api.PlaintextProducerSendTest > testWrongSerializer PASSED

kafka.api.PlaintextProducerSendTest > 
testSendNonCompressedMessageWithLogAppendTime STARTED

kafka.api.PlaintextProducerSendTest > 
testSendNonCompressedMessageWithLogAppendTime PASSED

kafka.api.PlaintextProducerSendTest > 
testSendNonCompressedMessageWithCreateTime STARTED

kafka.api.PlaintextProducerSendTest > 
testSendNonCompressedMessageWithCreateTime PASSED

kafka.api.PlaintextProducerSendTest > testClose STARTED

kafka.api.PlaintextProducerSendTest > testClose PASSED

kafka.api.PlaintextProducerSendTest > testFlush STARTED

kafka.api.PlaintextProducerSendTest > testFlush PASSED

kafka.api.PlaintextProducerSendTest > testSendToPartition STARTED

kafka.api.PlaintextProducerSendTest > testSendToPartition PASSED

kafka.api.PlaintextProducerSendTest > testSendOffset STARTED

kafka.api.PlaintextProducerSendTest > testSendOffset PASSED

kafka.api.PlaintextProducerSendTest > testSendCompressedMessageWithCreateTime 
STARTED

kafka.api.PlaintextProducerSendTest > testSendCompressedMessageWithCreateTime 
PASSED

kafka.api.PlaintextProducerSendTest > testCloseWithZeroTimeoutFromCallerThread 
STARTED

kafka.api.PlaintextProducerSendTest > testCloseWithZeroTimeoutFromCallerThread 
PASSED

kafka.api.PlaintextProducerSendTest > testCloseWithZeroTimeoutFromSenderThread 
STARTED

kafka.api.PlaintextProducerSendTest > testCloseWithZeroTimeoutFromSenderThread 
PASSED

kafka.api.PlaintextProducerSendTest > testSendBeforeAndAfterPartitionExpansion 
STARTED

kafka.api.PlaintextProducerSendTest > testSendBeforeAndAfterPartitionExpansion 
PASSED

kafka.api.SaslSslAdminClientIntegrationTest > testAclDescribe STARTED

kafka.api.SaslSslAdminClientIntegrationTest > testAclDescribe PASSED

kafka.api.SaslSslAdminClientIntegrationTest > 
testLegacyAclOpsNeverAffectOrReturnPrefixed STARTED

kafka.api.SaslSslAdminClientIntegrationTest > 
testLegacyAclOpsNeverAffectOrReturnPrefixed PASSED

kafka.api.SaslSslAdminClientIntegrationTest > testAttemptToCreateInvalidAcls 
STARTED

kafka.api.SaslSslAdminClientIntegrationTest > testAttemptToCreateInvalidAcls 
PASSED

kafka.api.SaslSslAdminClientIntegrationTest > testAclAuthorizationDenied STARTED

kafka.api.SaslSslAdminClientIntegrationTest > testAclAuthorizationDenied PASSED

kafka.api.SaslSslAdminClientIntegrationTest > testAclOperations STARTED

kafka.api.SaslSslAdminClientIntegrationTest > testAclOperations PASSED

kafka.api.SaslSslAdminClientIntegrationTest > testAclOperations2 STARTED

kafka.api.SaslSslAdminClientIntegrationTest > testAclOperations2 PASSED

kafka.api.Sas

[jira] [Resolved] (KAFKA-822) Reassignment of partitions needs a cleanup

[JIRA]

 [ 
https://issues.apache.org/jira/browse/KAFKA-822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sönke Liebau resolved KAFKA-822.

Resolution: Abandoned

Closing this as abandoned after asking for feedback on the dev list. Its 
probably also fixed, but not absolutely sure of that.

> Reassignment of partitions needs a cleanup
> --
>
> Key: KAFKA-822
> URL: https://issues.apache.org/jira/browse/KAFKA-822
> Project: Kafka
>  Issue Type: Bug
>  Components: controller, tools
>Affects Versions: 0.8.0
>Reporter: Swapnil Ghike
>Assignee: Swapnil Ghike
>Priority: Major
>  Labels: bugs
>
> 1. This is probably a left-over from when the ReassignPartitionsCommand used 
> to be blocking: 
> Currently, for each partition that is reassigned, controller deletes the 
> /admin/reassign_partitions zk path, and populates it with a new list with the 
> reassigned partition removed from the original list. This is probably an 
> overkill, and we can delete the zk path completely once the reassignment of 
> all partitions has completed successfully or in error. 
> 2. It will help to clarify that there could be no replicas that have started 
> and are not in the ISR when KafkaController.onPartitionReassignment() is 
> called.
> 3. We should batch the requests in 
> KafkaController.StopOldReplicasOfReassignedPartition()
> 4. Update controllerContext.partitionReplicaAssignment only once in 
> KafkaController.updateAssignedReplicasForPartition().
> 5. Need to thoroughly test.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Resolved] (KAFKA-1016) Broker should limit purgatory size

[JIRA]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sönke Liebau resolved KAFKA-1016.
-
Resolution: Not A Problem

Closing this as "not a problem", I believe the Purgatory redesign should help 
with the issue described here to a large extent.


> Broker should limit purgatory size
> --
>
> Key: KAFKA-1016
> URL: https://issues.apache.org/jira/browse/KAFKA-1016
> Project: Kafka
>  Issue Type: Bug
>  Components: purgatory
>Affects Versions: 0.8.0
>Reporter: Chris Riccomini
>Assignee: Joel Koshy
>Priority: Major
>
> I recently ran into a case where a poorly configured Kafka consumer was able 
> to trigger out of memory exceptions in multiple Kafka brokers. The consumer 
> was configured to have a fetcher.max.wait of Int.MaxInt.
> For low volume topics, this configuration causes the consumer to block for 
> frequently, and for long periods of time. [~junrao] informs me that the fetch 
> request will time out after the socket timeout is reached. In our case, this 
> was set to 30s.
> With several thousand consumer threads, the fetch request purgatory got into 
> the 100,000-400,000 range, which we believe triggered the out of memory 
> exception. [~nehanarkhede] claims to have seem similar behavior in other high 
> volume clusters.
> It kind of seems like a bad thing that a poorly configured consumer can 
> trigger out of memory exceptions in the broker. I was thinking maybe it makes 
> sense to have the broker try and protect itself from this situation. Here are 
> some potential solutions:
> 1. Have a broker-side max wait config for fetch requests.
> 2. Threshold the purgatory size, and either drop the oldest connections in 
> purgatory, or reject the newest fetch requests when purgatory is full.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Resolved] (KAFKA-1099) StopReplicaRequest and StopReplicaResponse should also carry the replica ids

[JIRA]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sönke Liebau resolved KAFKA-1099.
-
Resolution: Abandoned

Closing this as abandoned after asking for feedback on the dev list and 
receiving no objections.

> StopReplicaRequest and StopReplicaResponse should also carry the replica ids
> 
>
> Key: KAFKA-1099
> URL: https://issues.apache.org/jira/browse/KAFKA-1099
> Project: Kafka
>  Issue Type: Bug
>  Components: controller
>Affects Versions: 0.8.1
>Reporter: Neha Narkhede
>Assignee: Neha Narkhede
>Priority: Major
>
> The stop replica request and response only contain a list of partitions for 
> which a replica should be moved to offline/nonexistent state. But the replica 
> id information is implicit in the network layer as the receiving broker. This 
> complicates stop replica response handling on the controller. This blocks the 
> right fix for KAFKA-1097 since it requires invoking callback for processing a 
> StopReplicaResponse and it requires to know the replica id from the 
> StopReplicaResponse.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Resolved] (KAFKA-1111) Broker prematurely accepts TopicMetadataRequests on startup

[JIRA]

 [ 
https://issues.apache.org/jira/browse/KAFKA-?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sönke Liebau resolved KAFKA-.
-
Resolution: Abandoned

Closing as abandoned after no objections on dev list. If this is indeed still 
an issue we can always reopen this.

> Broker prematurely accepts TopicMetadataRequests on startup
> ---
>
> Key: KAFKA-
> URL: https://issues.apache.org/jira/browse/KAFKA-
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jason Rosenberg
>Assignee: Neha Narkhede
>Priority: Major
>
> I have an issue where on startup, the broker starts accepting 
> TopicMetadataRequests before it has had metadata sync'd from the controller.  
> This results in a bunch of log entries that look like this:
> 013-11-01 03:26:01,577  INFO [kafka-request-handler-0] admin.AdminUtils$ - 
> Topic creation { "partitions":{ "0":[ 9, 10 ] }, "version":1 }
> 2013-11-01 03:26:07,767  INFO [kafka-request-handler-1] admin.AdminUtils$ - 
> Topic creation { "partitions":{ "0":[ 9, 11 ] }, "version":1 }
> 2013-11-01 03:26:07,823  INFO [kafka-request-handler-1] admin.AdminUtils$ - 
> Topic creation { "partitions":{ "0":[ 10, 11 ] }, "version":1 }
> 2013-11-01 03:26:11,183  INFO [kafka-request-handler-2] admin.AdminUtils$ - 
> Topic creation { "partitions":{ "0":[ 10, 11 ] }, "version":1 }
> From an email thread, Neha remarks:
> Before a broker receives the first
> LeaderAndIsrRequest/UpdateMetadataRequest, it is technically not ready to
> start serving any request. But it still ends up serving
> TopicMetadataRequest which can re-create topics accidentally. It shouldn't
> succeed, but this is still a problem.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Re: [VOTE] KIP-492 Add java security providers in Kafka Security config

[Rajini Sivaram]

+1 (binding)

Thanks for the KIP, Sandeep!

Regards,

Rajini

On Tue, Jul 30, 2019 at 4:40 AM Satish Duggana 
wrote:

> +1 (non-binding)
>
> Thanks,
> Satish.
>
> On Tue, Jul 30, 2019 at 5:18 AM Harsha Chintalapani 
> wrote:
> >
> > Thanks for the KIP Sandeep.
> >
> > +1 (binding)
> >
> > Thanks,
> > Harsha
> > On Jul 29, 2019, 12:22 PM -0700, Sandeep Mopuri ,
> wrote:
> > > Hi all, after some good discussion
> > > 
> about the
> > > KIP
> > > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
> >,
> > > I'm starting the voting.
> > >
> > > This KIP proposes adding new security configuration to accept custom
> > > security providers that can provide algorithms for SSL or SASL.
> > >
> > > --
> > > Thanks,
> > > M.Sai Sandeep
>

[jira] [Created] (KAFKA-8732) specifying a non-existent broker to ./bin/kafka-reassign-partitions.sh leads to reassignment never getting completed.

[Raunak (JIRA)]

Raunak created KAFKA-8732:
-

 Summary: specifying a non-existent broker to 
./bin/kafka-reassign-partitions.sh leads to reassignment never getting 
completed.
 Key: KAFKA-8732
 URL: https://issues.apache.org/jira/browse/KAFKA-8732
 Project: Kafka
  Issue Type: Bug
  Components: controller, tools
Affects Versions: 0.10.1.1
 Environment: Ubuntu-VERSION="14.04.5 LTS"
Reporter: Raunak


Specifying a non-existent broker to ./bin/kafka-reassign-partitions.sh leads to 
reassignment never getting completed. 

 My reassignment is getting struck if I provide non-existing broker ID. My 
kafka version is 0.10.1.1.

 

 
{code:java}
./kafka-reassign-partitions.sh --zookeeper zk:2181 --reassignment-json-file 
le.json --execute
Current partition replica assignment

{"version":1,"partitions":[{"topic":"cv-topic","partition":0,"replicas":[1011131,101067,98,101240]}]}
Save this to use as the --reassignment-json-file option during rollback
Successfully started reassignment of partitions.
{code}
In this 98 is the non-existing broker. Deleting reassign_partitions znode is of 
no use as well. As when I describe the same topic the 98 broker is out of sync.

 

 
{code:java}
Topic:cv-topic PartitionCount:1 ReplicationFactor:4 Configs:
Topic: cv-topic Partition: 0 Leader: 1011131 Replicas: 1011131,101067,98,101240 
Isr: 1011131,101067,101240

{code}
Now 98 will always be out of sync.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Re: [DISCUSS] KIP-470: TopologyTestDriver test input and output usability improvements

[John Roesler]

Hey Jukka,

Sorry for the delay.

For what it's worth, I think 3, 4, and 5 are all good options. I guess my
own preference is 5.

It seems like the migration pain is a one-time concern vs. having more
maintainable code for years thereafter.

Thanks,
-John



On Tue, Jul 2, 2019 at 4:03 AM Jukka Karvanen 
wrote:

> Hi Matthias,
>
> Generally I think using Instant and Duration make the test more readable
> and that's why I modified KIP based on your suggestion.
> Now a lot of code use time with long or Long and that make the change more
> complicated.
>
> What I tried to say about the migration is the lines without timestamp or
> if long timestamp is supported can be migrated mainly with search &
> recplace:
>
>
> testDriver.pipeInput(recordFactory.create(WordCountLambdaExample.inputTopic,
> nullKey, "Hello", 1L));
>
> ->
>
> *inputTopic*.pipeInput(nullKey, "Hello", 1L);
>
> If long is not supported as timestamp, the same is not so easy:
>
>
> testDriver.pipeInput(recordFactory.create(WordCountLambdaExample.inputTopic,
> nullKey, "Hello", 1L));
>
> ->
>
> *inputTopic1*.pipeInput(nullKey, "Hello", Instant.ofEpochMilli(1L));
>
> Also if you need to convert arbitrary long timestamps to proper time
> Instants, it require you need to understand the logic of the test. So
> mechanical search & replace is not possible.
>
>
> I see there are several alternatives for long vs Instant / Duration:
>
> 1. All times as long/Long like in this version:
>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=119550011
>
> (startTimestampMs, autoAdvanceMs as parameter of  createInputTopic
> instead of configureTiming)
>
> 2. Auto timestamping configured with Instant and Duration, pipeInput
> and TestRecord with long:
>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=120722523
>
>
> 3. (CURRENT) Auto timestamping configured with Instant and Duration,
> pipeInput and TestRecord with Instant, version with long deprecated:
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-470%3A+TopologyTestDriver+test+input+and+output+usability+improvements
>
>
> 4. Auto timestamping configured with Instant and Duration, pipeInput
> and TestRecord with Instant and long parallel (with long not
> deprecated):
>
> 5. Auto timestamping configured with Instant and Duration, pipeInput
> and TestRecord with Instant only
>
> I hope to get feedback.
>
> My own preference currently is alternative 3. or 4.
>
>
> If somebody want to test, there is a implementation of this version
> available in Github:
>
> https://github.com/jukkakarvanen/kafka-streams-test-topics
>
> which can be used directly from public Maven repository:
>
> 
> com.github.jukkakarvanen
> kafka-streams-test-topics
> 0.0.1-beta3
> test
> 
>
> Also is this approach in KIP-470 preferred over KIP-456, so can we close
> it:
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-456%3A+Helper+classes+to+make+it+simpler+to+write+test+logic+with+TopologyTestDriver
>
> Jukka
>
> .
>
>
> pe 28. kesäk. 2019 klo 1.10 Matthias J. Sax (matth...@confluent.io)
> kirjoitti:
>
> > Thanks Jukka!
> >
> > The idea to use `Instant/Duration` was a proposal. If we think it's not
> > a good one, we could still stay with `long`. Because `ProducerRecord`
> > and `ConsumerRecord` are both based on `long,` it might make sense to
> > keep `long`?
> >
> > > The result of converting millis to Instant directly generates
> > >> rather non readable test code and changing from long to Instant
> > correctly
> > >> require understand what is the case it is testing.
> >
> > This might be a good indicator the using `Instant/Duration` might not be
> > a good idea.
> >
> > Would be nice to get feedback from others.
> >
> > About adding new methods that we deprecate immediately: I don't think we
> > should do this. IMHO, there are two kind of users, one that immediately
> > rewrite their code to move off deprecated methods. Those won't use the
> > new+deprecated ones anyway. Other uses migrate their code slowly and
> > would just not rewrite their code at all, and thus also not use the
> > new+deprecated methods.
> >
> > > Checking my own tests I was able to migrate the most of my code with
> > > search&replace without further thinking about the logic to this new
> > > approach. The result of converting millis to Instant directly generates
> > > rather non readable test code and changing from long to Instant
> correctly
> > > require understand what is the case it is testing.
> >
> > Not sure if I can follow here. You first say, you could easily migrate
> > your code, but than you say it was not easily possible? Can you clarify
> > your experience upgrading your test code?
> >
> >
> > -Matthias
> >
> >
> > On 6/27/19 12:21 AM, Jukka Karvanen wrote:
> > > Hi,
> > >
> > >>> (4) Should we switch from `long` for timestamps to `Instant` and
> > > `Duration` ?
> > >> This version startTimestamp is Instant and autoAdvance Duration in
> > > Initializatio

JIRA and KIP contributor permissions

[Gokul Ramanan Subramanian]

Hello Community,

In order to start contributing to Apache Kafka project, could I please request 
contributor access to JIRA and be granted write permissions to the Kafka wiki?

My name: Gokul Ramanan Subramanian
JIRA username: gokul2411s
Confluence username: gokul2411s 
(https://cwiki.apache.org/confluence/display/~gokul2411s)
Committer email: gokus...@amazon.com

Thank you in advance.
Gokul Ramanan Subramanian

Build failed in Jenkins: kafka-trunk-jdk8 #3818

[Apache Jenkins Server]

See 

--
[...truncated 6.46 MB...]

org.apache.kafka.connect.json.JsonConverterTest > structToConnect PASSED

org.apache.kafka.connect.json.JsonConverterTest > stringToJson STARTED

org.apache.kafka.connect.json.JsonConverterTest > stringToJson PASSED

org.apache.kafka.connect.json.JsonConverterTest > nullSchemaAndArrayToJson 
STARTED

org.apache.kafka.connect.json.JsonConverterTest > nullSchemaAndArrayToJson 
PASSED

org.apache.kafka.connect.json.JsonConverterTest > byteToConnect STARTED

org.apache.kafka.connect.json.JsonConverterTest > byteToConnect PASSED

org.apache.kafka.connect.json.JsonConverterTest > nullSchemaPrimitiveToConnect 
STARTED

org.apache.kafka.connect.json.JsonConverterTest > nullSchemaPrimitiveToConnect 
PASSED

org.apache.kafka.connect.json.JsonConverterTest > byteToJson STARTED

org.apache.kafka.connect.json.JsonConverterTest > byteToJson PASSED

org.apache.kafka.connect.json.JsonConverterTest > intToConnect STARTED

org.apache.kafka.connect.json.JsonConverterTest > intToConnect PASSED

org.apache.kafka.connect.json.JsonConverterTest > dateToJson STARTED

org.apache.kafka.connect.json.JsonConverterTest > dateToJson PASSED

org.apache.kafka.connect.json.JsonConverterTest > noSchemaToConnect STARTED

org.apache.kafka.connect.json.JsonConverterTest > noSchemaToConnect PASSED

org.apache.kafka.connect.json.JsonConverterTest > noSchemaToJson STARTED

org.apache.kafka.connect.json.JsonConverterTest > noSchemaToJson PASSED

org.apache.kafka.connect.json.JsonConverterTest > nullSchemaAndPrimitiveToJson 
STARTED

org.apache.kafka.connect.json.JsonConverterTest > nullSchemaAndPrimitiveToJson 
PASSED

org.apache.kafka.connect.json.JsonConverterTest > 
decimalToConnectOptionalWithDefaultValue STARTED

org.apache.kafka.connect.json.JsonConverterTest > 
decimalToConnectOptionalWithDefaultValue PASSED

org.apache.kafka.connect.json.JsonConverterTest > mapToJsonStringKeys STARTED

org.apache.kafka.connect.json.JsonConverterTest > mapToJsonStringKeys PASSED

org.apache.kafka.connect.json.JsonConverterTest > arrayToJson STARTED

org.apache.kafka.connect.json.JsonConverterTest > arrayToJson PASSED

org.apache.kafka.connect.json.JsonConverterTest > nullToConnect STARTED

org.apache.kafka.connect.json.JsonConverterTest > nullToConnect PASSED

org.apache.kafka.connect.json.JsonConverterTest > timeToJson STARTED

org.apache.kafka.connect.json.JsonConverterTest > timeToJson PASSED

org.apache.kafka.connect.json.JsonConverterTest > structToJson STARTED

org.apache.kafka.connect.json.JsonConverterTest > structToJson PASSED

org.apache.kafka.connect.json.JsonConverterTest > 
testConnectSchemaMetadataTranslation STARTED

org.apache.kafka.connect.json.JsonConverterTest > 
testConnectSchemaMetadataTranslation PASSED

org.apache.kafka.connect.json.JsonConverterTest > shortToJson STARTED

org.apache.kafka.connect.json.JsonConverterTest > shortToJson PASSED

org.apache.kafka.connect.json.JsonConverterTest > dateToConnect STARTED

org.apache.kafka.connect.json.JsonConverterTest > dateToConnect PASSED

org.apache.kafka.connect.json.JsonConverterTest > doubleToJson STARTED

org.apache.kafka.connect.json.JsonConverterTest > doubleToJson PASSED

org.apache.kafka.connect.json.JsonConverterTest > testStringHeaderToJson STARTED

org.apache.kafka.connect.json.JsonConverterTest > testStringHeaderToJson PASSED

org.apache.kafka.connect.json.JsonConverterTest > decimalToConnectOptional 
STARTED

org.apache.kafka.connect.json.JsonConverterTest > decimalToConnectOptional 
PASSED

org.apache.kafka.connect.json.JsonConverterTest > structSchemaIdentical STARTED

org.apache.kafka.connect.json.JsonConverterTest > structSchemaIdentical PASSED

org.apache.kafka.connect.json.JsonConverterTest > timeToConnectWithDefaultValue 
STARTED

org.apache.kafka.connect.json.JsonConverterTest > timeToConnectWithDefaultValue 
PASSED

org.apache.kafka.connect.json.JsonConverterTest > timeToConnect STARTED

org.apache.kafka.connect.json.JsonConverterTest > timeToConnect PASSED

org.apache.kafka.connect.json.JsonConverterTest > 
dateToConnectOptionalWithDefaultValue STARTED

org.apache.kafka.connect.json.JsonConverterTest > 
dateToConnectOptionalWithDefaultValue PASSED

org.apache.kafka.connect.json.JsonConverterTest > mapToConnectStringKeys STARTED

org.apache.kafka.connect.json.JsonConverterTest > mapToConnectStringKeys PASSED

org.apache.kafka.connect.json.JsonConverterTest > dateToConnectOptional STARTED

org.apache.kafka.connect.json.JsonConverterTest > dateToConnectOptional PASSED

org.apache.kafka.connect.json.JsonConverterTest > 
timeToConnectOptionalWithDefaultValue STARTED

org.apache.kafka.connect.json.JsonConverterTest > 
timeToConnectOptionalWithDefaultValue PASSED

org.apache.kafka.connect.json.JsonConverterTest > floatToConnect STARTED

org.apache.kafka.connect.json.JsonConverterTest > floatToConnect PASSED

org.apache.kafka.conn

[jira] [Resolved] (KAFKA-8640) Replace OffsetFetch request/response with automated protocol

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-8640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-8640.

Resolution: Fixed

> Replace OffsetFetch request/response with automated protocol
> 
>
> Key: KAFKA-8640
> URL: https://issues.apache.org/jira/browse/KAFKA-8640
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Boyang Chen
>Assignee: Boyang Chen
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Resolved] (KAFKA-8442) Inconsistent ISR output in topic command when using --bootstrap-server

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-8442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-8442.

   Resolution: Fixed
Fix Version/s: 2.4.0

> Inconsistent ISR output in topic command when using --bootstrap-server
> --
>
> Key: KAFKA-8442
> URL: https://issues.apache.org/jira/browse/KAFKA-8442
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jason Gustafson
>Assignee: huxihx
>Priority: Major
> Fix For: 2.4.0
>
>
> If there is no leader for a partition, the Metadata API returns an empty ISR. 
> When using the `--bootstrap-server` option with `kafka-topics.sh`, this leads 
> to the following output:
> {code}
> Topic:foo   PartitionCount:1ReplicationFactor:2 
> Configs:segment.bytes=1073741824
> Topic: foo  Partition: 0Leader: noneReplicas: 1,3   Isr: 
> {code}
> When using `--zookeeper`, we display the current ISR correctly:
> {code}
> Topic:foo   PartitionCount:1ReplicationFactor:2 Configs:
> Topic: foo  Partition: 0Leader: -1  Replicas: 1,3   Isr: 1
> {code}
> To avoid confusion, we should make this output consistent or at least not 
> misleading. We should either change the Metadata API to print the ISR when we 
> have it or we can change the output of the topic command to `N/A` or 
> something like that.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Resolved] (KAFKA-8717) Use cached hw/lso offset metadata when reading from log

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-8717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-8717.

   Resolution: Fixed
Fix Version/s: 2.4.0

> Use cached hw/lso offset metadata when reading from log
> ---
>
> Key: KAFKA-8717
> URL: https://issues.apache.org/jira/browse/KAFKA-8717
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
>Priority: Major
> Fix For: 2.4.0
>
>
> The broker caches log offset metadata (e.g. segment position) for the high 
> watermark and last stable offset in order to avoid additional index lookups 
> when handling fetches. Currently this metadata is only used when determining 
> delayed fetch satisfaction. We can also use it when reading from the log in 
> order to avoid additional redundant index lookups.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Configuration for custom authorizer

[Jeff Zemerick]

Hi all,

I'm implementing a custom authorizer using the Authorizer interface. In it
there is a configure function that takes a map of properties. When using
kafka-acl.sh those properties are passed via --authorizer-properties. How
do I pass those properties to Kafka when the server starts in order to
parameterize some functions of the authorizer?

Thanks,
Jeff

[jira] [Created] (KAFKA-8733) Offline partitions occur when leader's disk is slow in reads while responding to follower fetch requests.

[Satish Duggana (JIRA)]

Satish Duggana created KAFKA-8733:
-

 Summary: Offline partitions occur when leader's disk is slow in 
reads while responding to follower fetch requests.
 Key: KAFKA-8733
 URL: https://issues.apache.org/jira/browse/KAFKA-8733
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 1.1.2, 2.4.0
Reporter: Satish Duggana
Assignee: Satish Duggana


We found offline partitions issue multiple times on some of the hosts in our 
clusters. After going through the broker logs and hosts’s disk stats, it looks 
like this issue occurs whenever the read/write operations take more time on 
that disk. In a particular case where read time is more than the 
replica.lag.time.max.ms, follower replicas will be out of sync as their earlier 
fetch requests are stuck while reading the local log and their fetch status is 
not yet updated as mentioned in the below code of `ReplicaManager`. If there is 
an issue in reading the data from the log for a duration more than 
replica.lag.time.max.ms then all the replicas will be out of sync and partition 
becomes offline if min.isr.replicas > 1 and unclean.leader.election is disabled.

 
{code:java}
def readFromLog(): Seq[(TopicPartition, LogReadResult)] = {
  val result = readFromLocalLog( // this call took more than 
`replica.lag.time.max.ms`
  replicaId = replicaId,
  fetchOnlyFromLeader = fetchOnlyFromLeader,
  readOnlyCommitted = fetchOnlyCommitted,
  fetchMaxBytes = fetchMaxBytes,
  hardMaxBytesLimit = hardMaxBytesLimit,
  readPartitionInfo = fetchInfos,
  quota = quota,
  isolationLevel = isolationLevel)
  if (isFromFollower) updateFollowerLogReadResults(replicaId, result). // fetch 
time gets updated here, but mayBeShrinkIsr should have been already called and 
the replica is removed from sir
 else result
 }

val logReadResults = readFromLog()
{code}

I will raise a KIP describing options on how to handle this scenario.

 



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Re: [VOTE] KIP-492 Add java security providers in Kafka Security config

[Manikumar]

+1 (binding)

Thanks for the KIP. LGTM.

Thanks.
Manikumar

On Tue, Jul 30, 2019 at 1:42 PM Rajini Sivaram 
wrote:

> +1 (binding)
>
> Thanks for the KIP, Sandeep!
>
> Regards,
>
> Rajini
>
> On Tue, Jul 30, 2019 at 4:40 AM Satish Duggana 
> wrote:
>
> > +1 (non-binding)
> >
> > Thanks,
> > Satish.
> >
> > On Tue, Jul 30, 2019 at 5:18 AM Harsha Chintalapani 
> > wrote:
> > >
> > > Thanks for the KIP Sandeep.
> > >
> > > +1 (binding)
> > >
> > > Thanks,
> > > Harsha
> > > On Jul 29, 2019, 12:22 PM -0700, Sandeep Mopuri ,
> > wrote:
> > > > Hi all, after some good discussion
> > > > 
> > about the
> > > > KIP
> > > > <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
> > >,
> > > > I'm starting the voting.
> > > >
> > > > This KIP proposes adding new security configuration to accept custom
> > > > security providers that can provide algorithms for SSL or SASL.
> > > >
> > > > --
> > > > Thanks,
> > > > M.Sai Sandeep
> >
>

Re: JIRA and KIP contributor permissions

[Bill Bejeck]

Thanks for your interest in Apache Kafka.

You should be set now.

Thanks,
Bill

On Tue, Jul 30, 2019 at 9:40 AM Gokul Ramanan Subramanian <
gokul24...@gmail.com> wrote:

> Hello Community,
>
> In order to start contributing to Apache Kafka project, could I please
> request contributor access to JIRA and be granted write permissions to the
> Kafka wiki?
>
> My name: Gokul Ramanan Subramanian
> JIRA username: gokul2411s
> Confluence username: gokul2411s (
> https://cwiki.apache.org/confluence/display/~gokul2411s)
> Committer email: gokus...@amazon.com
>
> Thank you in advance.
> Gokul Ramanan Subramanian
>
>

Re: [VOTE] KIP-221: Enhance KStream with Connecting Topic Creation and Repartition Hint

[Levani Kokhreidze]

Hello,

Still waiting for feedback on this KIP.
Please let me know if you have any concerns and/or questions.

Regards,
Levani


> On Jul 24, 2019, at 8:20 PM, Sophie Blee-Goldman  wrote:
> 
> Looks good! Thanks Levani,
> 
> +1 (non-binding)
> 
> Sophie
> 
> On Tue, Jul 23, 2019 at 2:16 PM Levani Kokhreidze 
> wrote:
> 
>> Hello,
>> 
>> I’d like to initialize voting on KIP-221:
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-221%3A+Enhance+KStream+with+Connecting+Topic+Creation+and+Repartition+Hint
>> <
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-221:+Enhance+KStream+with+Connecting+Topic+Creation+and+Repartition+Hint
>>> 
>> If there’re any more concerns about the KIP, happy to discuss further.
>> 
>> Regards,
>> Levani

Re: [DISCUSS] KIP-478 Strongly Typed Processor API

[John Roesler]

Thanks, Matthias, and thanks again for raising the concern.
-John

On Mon, Jul 29, 2019 at 4:58 PM Matthias J. Sax 
wrote:

> Thanks for the details!
>
> Also talked to Guozhang about a potential upgrade path. This KIP seems
> not to put us into an bad position to provide a clean upgrade path if we
> change the `ProcessorContext` in the future.
>
> Thus, I think we can move forward.
>
>
> -Matthias
>
> On 7/24/19 3:32 PM, John Roesler wrote:
> > Hey again Matthias,
> >
> > I think it might help to frame the evaluation of the Context question if
> we
> > have a "spitball" proposal for what change we might want to make to the
> > context.
> >
> > Currently, the ProcessorContext is referenced in the following public
> > interfaces:
> >
> > org.apache.kafka.streams.errors.DeserializationExceptionHandler#handle
> > org.apache.kafka.streams.kstream.Transformer#init
> > org.apache.kafka.streams.kstream.ValueTransformer#init
> > org.apache.kafka.streams.kstream.ValueTransformerWithKey#init
> > org.apache.kafka.streams.processor.Processor#init
> > org.apache.kafka.streams.processor.StateStore#init
> >
> > We can sub-divide the ProcessorContext into broad categories:
> > General Information:
> > * a handle on the config
> > * information about the execution context (what is the task id, the
> > application id, etc)
> > Record Information:
> > * extra information about the current record
> > Store Support:
> > * the ability to register state restore callbacks
> > Processor Support:
> > * the ability to schedule punctuations
> > * the ability to get registered state stores
> > * the ability to schedule punctuations
> > * the ability to forward records
> > * the ability to request commits
> >
> > We could imagine slicing the Processor Context into four new component
> > interfaces, and making ProcessorContext just implement them. Then, we
> could
> > mix-and-match those new component interfaces for use elsewhere.
> >
> > E.g.,:
> > org.apache.kafka.streams.errors.DeserializationExceptionHandler#handle
> > * only gets the informational context
> >
> > org.apache.kafka.streams.kstream.Transformer#init
> > org.apache.kafka.streams.kstream.ValueTransformer#init
> > org.apache.kafka.streams.kstream.ValueTransformerWithKey#init
> > * information context
> > * the ability to get registered state stores
> > Also
> > * the ability to schedule punctuations
> > * restricted ability to forward (only obeying the rules of the particular
> > interface, for example)
> > Or maybe just:
> > * no ability to foraward
> > * the ability to schedule special punctuators that can return one or more
> > keys or values when fired
> >
> > org.apache.kafka.streams.processor.Processor#init
> > * all the contexts, except the ability to register state restore
> callbacks
> >
> > org.apache.kafka.streams.processor.StateStore#init
> > * information contexts
> > * the ability to register state restore callbacks
> > * maybe punctuations and forwards, could be discussed further
> >
> >
> > The operative question for us right now is whether there is a clean path
> to
> > something like this from the current KIP, or whether we'd be forced to
> > deprecate an interface we are only just now adding. Note that the only
> > interfaces we're adding right now are :
> > * org.apache.kafka.streams.processor.api.Processor
> > * org.apache.kafka.streams.processor.api.ProcessorSupplier
> > And the only thing we need to make the above spitball proposal compatible
> > with these proposed interfaces is to deprecate the ability to register
> > state restore callbacks from the ProcessorContext.
> >
> > Otherwise, we would at that time be able to propose new Transformer
> > interfaces that take (e.g.) TransformerContexts, likewise with
> > DeserializationExceptionHandler and StateStore.
> >
> > In other words, I _think_ that we have a clean migration path to address
> > the Context problem in follow-on work. But clearly my motivation may be
> > corrupt. What do you think?
> >
> > Thanks,
> > -John
> >
> >
> > On Wed, Jul 24, 2019 at 5:06 PM John Roesler  wrote:
> >
> >> Hey Matthias,
> >>
> >> I agree, it's worth double-checking to make sure that the upgrade path
> >> would be smooth. There's no point in putting ourselves in an awkward
> jam.
> >> I'll look into it and report back.
> >>
> >> Regarding the global store logic, I confirmed that the "state update
> >> processor" shouldn't be forwarding anything, so we can safely bound its
> >> output type to `Void`. I've updated the KIP.
> >>
> >> Thanks,
> >> -John
> >>
> >> On Wed, Jul 24, 2019 at 3:08 PM Matthias J. Sax 
> >> wrote:
> >>
> >>> If we don't fix the `ProcessorContext` now, how would an upgrade path
> >>> look like?
> >>>
> >>> We woudl deprecate existing `init()` and add a new `init()`, and during
> >>> runtime need to call both? This sound rather error prone to me and
> might
> >>> be confusing to users? Hence, it might be beneficial to fix it right
> now.
> >>>
> >>> If my concerns are not valid, and we think 

Re: [VOTE] KIP-478 Strongly Typed Streams Processor API

[John Roesler]

Thanks, everyone, for the really good discussion.

The vote has been open for 6 days, and has three binding votes (Guozhang,
Bill, Matthias), in addition to my own non-binding +1, so the KIP vote
passes!

Next, I'll close my POC PR and put together an actual change set for review.

Thanks again, all,
-John

On Mon, Jul 29, 2019 at 4:58 PM Matthias J. Sax 
wrote:

> +1 (binding)
>
> On 7/29/19 11:59 AM, Bill Bejeck wrote:
> > Thanks for the KIP.
> >
> > +1 (binding)
> >
> > -Bill
> >
> > On Wed, Jul 24, 2019 at 12:12 PM Guozhang Wang 
> wrote:
> >
> >> Yeah I think I agree with you.
> >>
> >> +1 (binding) from me.
> >>
> >>
> >> Guozhang
> >>
> >>
> >> On Wed, Jul 24, 2019 at 7:43 AM John Roesler  wrote:
> >>
> >>> Hi Guozhang,
> >>>
> >>> Thanks! I just replied in the discuss thread. I agree with what you're
> >>> proposing, but would like to consider it outside the scope of this KIP,
> >> if
> >>> that's ok with you.
> >>>
> >>> -John
> >>>
> >>> On Tue, Jul 23, 2019 at 5:38 PM Guozhang Wang 
> >> wrote:
> >>>
>  Hi John,
> 
>  I left another question regarding Transformer in the DISCUSS thread.
> >>> Other
>  than that I think this KIP is ready. Thanks!
> 
> 
>  Guozhang
> 
> 
>  On Tue, Jul 16, 2019 at 9:01 AM John Roesler 
> >> wrote:
> 
> > Hi Dev,
> >
> > After a good discussion, I'd like to start the vote for KIP-478
> > (https://cwiki.apache.org/confluence/x/2SkLBw).
> >
> > The proposal is to deprecate the existing interface
> > org.apache.kafka.streams.processor.Processor in favor of a
> > new one, org.apache.kafka.streams.processor.api.Processor > KOut, VOut> that parameterizes both the input and output types.
> >
> > This change enables both the Streams DSL internal code and external
> > Processor API code to improve their type safety and protect
> >> themselves
> > from type-level bugs.
> >
> > Thanks,
> > -John
> >
> 
> 
>  --
>  -- Guozhang
> 
> >>>
> >>
> >>
> >> --
> >> -- Guozhang
> >>
> >
>
>

[jira] [Created] (KAFKA-8734) Remove PartitionAssignorAdapter and deprecated PartitionAssignor interface

[Sophie Blee-Goldman (JIRA)]

Sophie Blee-Goldman created KAFKA-8734:
--

 Summary: Remove PartitionAssignorAdapter and deprecated 
PartitionAssignor interface
 Key: KAFKA-8734
 URL: https://issues.apache.org/jira/browse/KAFKA-8734
 Project: Kafka
  Issue Type: Task
  Components: clients
Affects Versions: 3.0.0
Reporter: Sophie Blee-Goldman


In 2.4 we deprecated the consumer.internal.PartitionAssignor interface and 
migrated all assignors to the [new public consumer.ConsumerPartitionAssignor 
interface|[https://github.com/apache/kafka/pull/7108|https://github.com/apache/kafka/pull/7108/files]].
 Although internal, we provided an [adapter 
|[https://github.com/apache/kafka/pull/7110]]for those who may have implemented 
a custom PartitionAssignor to avoid breaking changes. These should be removed 
in the next major release.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Re: [VOTE] KIP-221: Enhance KStream with Connecting Topic Creation and Repartition Hint

[Bill Bejeck]

Thanks for the KIP Levani.

+1 (binding)

-Bill

On Tue, Jul 30, 2019 at 3:37 PM Levani Kokhreidze 
wrote:

> Hello,
>
> Still waiting for feedback on this KIP.
> Please let me know if you have any concerns and/or questions.
>
> Regards,
> Levani
>
>
> > On Jul 24, 2019, at 8:20 PM, Sophie Blee-Goldman 
> wrote:
> >
> > Looks good! Thanks Levani,
> >
> > +1 (non-binding)
> >
> > Sophie
> >
> > On Tue, Jul 23, 2019 at 2:16 PM Levani Kokhreidze <
> levani.co...@gmail.com>
> > wrote:
> >
> >> Hello,
> >>
> >> I’d like to initialize voting on KIP-221:
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-221%3A+Enhance+KStream+with+Connecting+Topic+Creation+and+Repartition+Hint
> >> <
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-221:+Enhance+KStream+with+Connecting+Topic+Creation+and+Repartition+Hint
> >>>
> >> If there’re any more concerns about the KIP, happy to discuss further.
> >>
> >> Regards,
> >> Levani
>
>

[jira] [Created] (KAFKA-8735) BrokerMetadataCheckPoint should check metadata.properties existence itself

[Qinghui Xu (JIRA)]

Qinghui Xu created KAFKA-8735:
-

 Summary: BrokerMetadataCheckPoint should check metadata.properties 
existence itself 
 Key: KAFKA-8735
 URL: https://issues.apache.org/jira/browse/KAFKA-8735
 Project: Kafka
  Issue Type: Improvement
Reporter: Qinghui Xu


BrokerMetadataCheckPoint tries to read metadata.properties from log directory 
during server start up. And it relies on org.apache.kafka.common.util.Utils 
(from org.apache.kafka:kafka-clients) to load the properties file in a given 
directory.

During the process, we need to handle the case in which the properties file 
does not exist (not as an error). Currently, BrokerMetadataCheckPoint relies on 
the behavior of  `org.apache.kafka.common.util.Utils#loadProps` to find out if 
the file exists or not: if the properties file is absent, it is expecting 
NoSuchFileException (for branch 2.1 and above), and it was expecting 
FileNotFoundException (for branch 2.0 and before). Knowing that 
`org.apache.kafka.common.util.Utils#loadProps` signature throws only 
IOException, this exception pattern matching is thus sort of leak of 
abstraction making BrokerMetadataCheckPoint relies on the implementation 
details of `org.apache.kafka.common.util.Utils#loadProps`. 

This makes BrokerMetadataCheckPoint very fragile, especially when 
`org.apache.kafka.common.util.Utils` and 
`kafka.server.BrokerMetadataCheckPoint` are from different artifacts, an 
example that I just ran into:
 * We have a project A that depends on project B, and project B has a compile 
time dependency on `org.apache.kafka:kafka-clients`. A is relying 
`org.apach.kafka:kafka_2.11` in its tests: it will spawn some kafka brokers in 
the tests.
 * At first A and B are both using kafka libraries 2.0.1, and everything is 
working fine
 * At some point a newer version of B upgrades `org.apache.kafka:kafka-clients` 
to 2.3.0
 * When A wants to use the newer version of B, its tests are broken because 
kafka brokers fail to start: now `org.apache.kafka.common.util.Utils` (2.3.0) 
throws NoSucheFileException while BrokerMetadataCheckPoint (2.0.1) expects to 
catch FileNotFoundException

It would be much more reliable for BrokerMetadataCheckPoint to check the file 
existence before trying to load the properties from the file.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Re: [DISCUSS] KIP-447: Producer scalability for exactly once semantics

[Boyang Chen]

Thank you Guozhang for the reply. We will consider the interface change
from 429 as a backup plan for 447.

And bumping this thread for more discussion.

On Mon, Jul 22, 2019 at 6:28 PM Guozhang Wang  wrote:

> On Sat, Jul 20, 2019 at 9:50 AM Boyang Chen 
> wrote:
>
> > Thank you Guozhang for the suggestion! I would normally prefer naming a
> > flag corresponding to its functionality. Seems to me `isolation_level`
> > makes us another hop on information track.
> >
> > Fair enough, let's use a separate flag name then :)
>
>
> > As for the generation.id exposure, I'm fine leveraging the new API from
> > 429, but however is that design finalized yet, and whether the API will
> be
> > added on the generic Consumer interface?
> >
> > The current PartitionAssignor is inside `internals` package and in
> KIP-429
> we are going to create a new interface out of `internals` to really make it
> public APIs, and as part of that we are refactoring some of its method
> signatures. I just feel some of the newly introduced classes can be reused
> in your KIP as well, i.e. just for code succinctness, but no semantical
> indications.
>
>
> > Boyang
> >
> > On Fri, Jul 19, 2019 at 3:57 PM Guozhang Wang 
> wrote:
> >
> > > Boyang, thanks for the updated proposal!
> > >
> > > 3.a. As Jason mentioned, with EOS enabled we still need to augment the
> > > offset fetch request with a boolean to indicate "give me an retriable
> > error
> > > code if there's pending offset, rather than sending me the committed
> > offset
> > > immediately". Personally I still feel it is okay to piggy-back on the
> > > ISOLATION_LEVEL boolean, but I'm also fine with another
> > `await_transaction`
> > > boolean if you feel strongly about it.
> > >
> > > 10. About the exposure of generation id, there may be some refactoring
> > work
> > > coming from KIP-429 that can benefit KIP-447 as well since we are
> > wrapping
> > > the consumer subscription / assignment data in new classes. Note that
> > > current proposal does not `generationId` since with the cooperative
> > sticky
> > > assignor we think it is not necessary for correctness, but also if we
> > agree
> > > it is okay to expose it we can potentially include it in
> > > `ConsumerAssignmentData` as well.
> > >
> > >
> > > Guozhang
> > >
> > >
> > > On Thu, Jul 18, 2019 at 3:55 PM Boyang Chen <
> reluctanthero...@gmail.com>
> > > wrote:
> > >
> > > > Thank you Jason for the ideas.
> > > >
> > > > On Mon, Jul 15, 2019 at 5:28 PM Jason Gustafson 
> > > > wrote:
> > > >
> > > > > Hi Boyang,
> > > > >
> > > > > Thanks for the updates. A few comments below:
> > > > >
> > > > > 1. The KIP mentions that `transaction.timeout.ms` should be
> reduced
> > to
> > > > > 10s.
> > > > > I think this makes sense for Kafka Streams which is tied to the
> > > consumer
> > > > > group semantics and uses a default 10s session timeout. However, it
> > > > seems a
> > > > > bit dangerous to make this change for the producer generally. Could
> > we
> > > > just
> > > > > change it for streams?
> > > > >
> > > > > That sounds good to me.
> > > >
> > > > > 2. The new `initTransactions` API takes a `Consumer` instance. I
> > think
> > > > the
> > > > > idea is to basically put in a backdoor to give the producer access
> to
> > > the
> > > > > group generationId. It's not clear to me how this would work given
> > > > package
> > > > > restrictions. I wonder if it would be better to just expose the
> state
> > > we
> > > > > need from the consumer. I know we have been reluctant to do this so
> > far
> > > > > because we treat the generationId as an implementation detail.
> > > However, I
> > > > > think we might just bite the bullet and expose it rather than
> coming
> > up
> > > > > with a messy hack. Concepts such as memberIds have already been
> > exposed
> > > > in
> > > > > the AdminClient, so maybe it is not too bad. Alternatively, we
> could
> > > use
> > > > an
> > > > > opaque type. For example:
> > > > >
> > > > > // public
> > > > > interface GroupMetadata {}
> > > > >
> > > > > // private
> > > > > interface ConsumerGroupMetadata {
> > > > >   final int generationId;
> > > > >   final String memberId;
> > > > > }
> > > > >
> > > > > // Consumer API
> > > > > public GroupMetadata groupMetadata();
> > > > >
> > > > > I am probably leaning toward just exposing the state we need.
> > > > >
> > > > > Yes, also to mention that Kafka Streams use generic Cosnumer API
> > which
> > > > doesn't have rich
> > > > states like a full `KafkaConsumer`. The hack will not work as
> expected.
> > > >
> > > > Instead, just exposing the consumer generation.id seems a way easier
> > > work.
> > > > We could consolidate
> > > > the API and make it
> > > >
> > > > 3. Given that we are already providing a way to propagate group state
> > > from
> > > > > the consumer to the producer, I wonder if we may as well include
> the
> > > > > memberId and groupInstanceId. This would make the validation we do
> > for
> > > > > TxnOffsetCommit consis

Re: [VOTE] KIP-455: Create an Administrative API for Replica Reassignment

[Jun Rao]

Hi, Colin,

Thanks for the KIP. Sorry for the late reply. LGTM overall. A few detailed
comments below.

10. The KIP adds two new fields (AddingReplicas and RemovingReplicas) to
LeaderAndIsr request. Could you explain how these 2 fields will be used?
Should we include those two fields in UpdateMetadata and potentially
Metadata requests too?

11. "If a new reassignment is issued during an on-going one, we cancel the
current one by emptying out both AR and RR, constructing them from (the
updated from the last-reassignment) R and TR, and starting anew." In this
case, it seems that the controller needs to issue a StopReplica request to
remove those unneeded replicas.

12. "Essentially, once a cancellation is called we subtract AR from R,
empty out both AR and RR, and send LeaderAndIsr requests to cancel the
replica movements that have not yet completed." Similar to the above, it
seems the controller needs to issue a StopReplica request to remove those
unneeded replicas.

13. Since we changed the format of the topics/[topic] zNode, should we bump
up the version number in the json value?

Jun

On Mon, Jul 22, 2019 at 8:38 AM Colin McCabe  wrote:

> Hi all,
>
> With three non-binding +1 votes from Viktor Somogyi-Vass, Robert Barrett,
> and George Li, and 3 binding +1 votes from Gwen Shapira, Jason Gustafson,
> and myself, the vote passes.  Thanks, everyone!
>
> best,
> Colin
>
> On Fri, Jul 19, 2019, at 08:55, Robert Barrett wrote:
> > +1 (non-binding). Thanks for the KIP!
> >
> > On Thu, Jul 18, 2019 at 5:59 PM George Li  .invalid>
> > wrote:
> >
> > >  +1 (non-binding)
> > >
> > >
> > >
> > > Thanks for addressing the comments.
> > > George
> > >
> > > On Thursday, July 18, 2019, 05:03:58 PM PDT, Gwen Shapira <
> > > g...@confluent.io> wrote:
> > >
> > >  Renewing my +1, thank you Colin and Stan for working through all the
> > > questions, edge cases, requests and alternatives. We ended up with a
> > > great protocol.
> > >
> > > On Thu, Jul 18, 2019 at 4:54 PM Jason Gustafson 
> > > wrote:
> > > >
> > > > +1 Thanks for the KIP. Really looking forward to this!
> > > >
> > > > -Jason
> > > >
> > > > On Wed, Jul 17, 2019 at 1:41 PM Colin McCabe 
> wrote:
> > > >
> > > > > Thanks, Stanislav.  Let's restart the vote to reflect the fact that
> > > we've
> > > > > made significant changes.  The new vote will go for 3 days as
> usual.
> > > > >
> > > > > I'll start with my +1 (binding).
> > > > >
> > > > > best,
> > > > > Colin
> > > > >
> > > > >
> > > > > On Wed, Jul 17, 2019, at 08:56, Stanislav Kozlovski wrote:
> > > > > > Hey everybody,
> > > > > >
> > > > > > We have further iterated on the KIP in the accompanying
> discussion
> > > thread
> > > > > > and I'd like to propose we resume the vote.
> > > > > >
> > > > > > Some notable changes:
> > > > > > - we will store reassignment information in the
> > > `/brokers/topics/[topic]`
> > > > > > - we will internally use two collections to represent a
> reassignment
> > > -
> > > > > > "addingReplicas" and "removingReplicas". LeaderAndIsr has been
> > > updated
> > > > > > accordingly
> > > > > > - the Alter API will still use the "targetReplicas" collection,
> but
> > > the
> > > > > > List API will now return three separate collections - the full
> > > replica
> > > > > set,
> > > > > > the replicas we are adding as part of this reassignment
> > > > > ("addingReplicas")
> > > > > > and the replicas we are removing ("removingReplicas")
> > > > > > - cancellation of a reassignment now means a proper rollback of
> the
> > > > > > assignment to its original state prior to the API call
> > > > > >
> > > > > > As always, you can re-read the KIP here
> > > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-455%3A+Create+an+Administrative+API+for+Replica+Reassignment
> > > > > >
> > > > > > Best,
> > > > > > Stanislav
> > > > > >
> > > > > > On Wed, May 22, 2019 at 6:12 PM Colin McCabe  >
> > > wrote:
> > > > > >
> > > > > > > Hi George,
> > > > > > >
> > > > > > > Thanks for taking a look.  I am working on getting a PR done
> as a
> > > > > > > proof-of-concept.  I'll post it soon.  Then we'll finish up the
> > > vote.
> > > > > > >
> > > > > > > best,
> > > > > > > Colin
> > > > > > >
> > > > > > > On Tue, May 21, 2019, at 17:33, George Li wrote:
> > > > > > > >  Hi Colin,
> > > > > > > >
> > > > > > > >  Great! Looking forward to these features.+1
> (non-binding)
> > > > > > > >
> > > > > > > > What is the estimated timeline to have this implemented?  If
> any
> > > help
> > > > > > > > is needed in the implementation of cancelling
> reassignments,  I
> > > can
> > > > > > > > help if there is spare cycle.
> > > > > > > >
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > George
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >On Thursday, May 16, 2019, 9:48:56 AM PDT, Colin McCabe
> > > > > > > >  wrote:
> > > > > > > >
> > > > > > > >  Hi George,
> > > > > > > >
> > > > > > > > Yes, KIP-455 allows the reassignme

Re: [DISCUSS] KIP-490: log when consumer groups lose a message because offset has been deleted

[Jose M]

Hello Stanislav,

Thanks again for your comments.

I understand, and Im happy to hear that my usecase is rare. The reason
is that before going to real production, we are forced to build a
prototype, with limited resources, but still resilient enough to pass
acceptance tests.

I agree the whitelist on configuration does not look great, and that
the cardinality exposed on the broker would be high and it is better
to avoid that. Currently many of my consumer groups only have 1
consumer, but it is a trade off that I can accept to expose the metric
on consumer side, even if it means I will not have it in case of crash
loop.

I understand the metric could be something like this on consumer side:
```
 metricMessagesLost +=  ( firstAvailableOffset >  currentOffset ?
firstAvailableOffset - currentOffset : 0 )
```
I will update the KIP with your inputs.

Thanks a lot for you help and your time,


Jose M

On Mon, Jul 29, 2019 at 3:55 PM Stanislav Kozlovski
 wrote:
>
> Hey Jose,
>
> Thanks for sharing your use cases.
> From my experience, it is uncommon to run with a retention.ms setting small
> enough that it can make you lose messages when your consumers can't catch
> up. If you are concerned with data loss, I think the cost investment into
> hardware is generally worth it.
> I think your use case might benefit from setting `retention.bytes` to
> ensure you don't go over a specific size and a higher retention.ms. I
> assume that might be more deterministic as it is likely you have a better
> idea of how much data these files will be (and can get) rather than how
> long they'd take to process.
>
> In any case, I think it's an exception to have to manually configure and
> modify retention.ms in real time according to consumer lag. This metric (if
> enabled) would be the highest cardinality metric in the server, as it is
> per consumer group *and* partition. I know the current proposal suggests we
> enable it through a whitelist config, but I think that would be intuitive
> to users and I'm not sure if it's a good idea to guard metrics according to
> configurations.
> In general, I believe we should aim to limit the raw number of metrics
> exposed from the broker when there is another way to solve the problem.
>
> I think the metric should go on the broker side, in case the consumer
> > is not even be instantiated, or it is crashing in a loop.
>
> We would need *all* consumers in the consumer group to not be available in
> order to not have the information exposed. Also, it is generally expected
> to have your consumers run all the time (Kafka is a real-time streaming
> platform) and batch use cases are the exception.
> If they are all crashing in a loop, there is an outage to be solved and you
> should increase your retention if there is a chance it deleted unconsumed
> data.
> Because of the rareness of needing the information in real-time, I still
> think having it in the consumer is a good approach.
>
> Let me know if that makes sense.
>
> Thanks,
> Stanislav
>
> On Sun, Jul 28, 2019 at 5:00 PM Jose M  wrote:
>
> > Hello,
> >
> > Thanks for taking the time to review my KIP!
> >
> > I will describe some production scenarios I faced to better explain
> > the reasons for this KIP.
> >
> > * Usecase 1: batch processing of files.
> > A batch is producing huge files that must be processed. Each line of
> > the file will be a message produced to a topic. It means the topic
> > storing this messages will go from 0 lag to lets say 5 million lag, in
> > a few seconds. I will adjust the retention time on the topic based on
> > the processing rate on the consumer of this topic. Ex: 5 million
> > messages at 100 TPS needs ~14 hours retention time. In practice we set
> > up bigger retention time, just in case. If a second file arrives
> > before the first one has been processed and the processing ratio is
> > slower than I thought, I will lose the end of the first file, without
> > notice.
> >
> > * Usecase 2: application facing network errors.
> > The application consumes messages on input topic, process them and
> > push them to an external system (ex: webservice). If there are
> > connectivity problem between my kafka consumer and the external
> > webservice, the lag of the application will grow. As I have alerting
> > rules on records-max-lag, I will be aware the backlog of the topic is
> > above a limit. I will take action as in the previous example, and I
> > will adjust retention time on the topic based on the processing rate.
> > If the processing rate is not constant, due to the network
> > connectivity problem, the retention time may not be enough and I will
> > lose messages.
> >
> > In both cases, I don't know if Ive lost messages or not. I suspect
> > that yes but I can not give an accurate number of messages lost, or
> > guarantee I have not lost any of them.
> >
> > I could solve both use cases setting up oversized retention time for
> > the topics, but in practice I'm limited by the hardware resources.
> >
> 

Re: [VOTE] KIP-221: Enhance KStream with Connecting Topic Creation and Repartition Hint

[Guozhang Wang]

Hello Levani,

Thanks for the KIP! Just got a quick question here about the scope: why do
we only want this for `KStream`, not `KTable#groupBy` for example?


Guozhang


On Tue, Jul 30, 2019 at 1:27 PM Bill Bejeck  wrote:

> Thanks for the KIP Levani.
>
> +1 (binding)
>
> -Bill
>
> On Tue, Jul 30, 2019 at 3:37 PM Levani Kokhreidze 
> wrote:
>
> > Hello,
> >
> > Still waiting for feedback on this KIP.
> > Please let me know if you have any concerns and/or questions.
> >
> > Regards,
> > Levani
> >
> >
> > > On Jul 24, 2019, at 8:20 PM, Sophie Blee-Goldman 
> > wrote:
> > >
> > > Looks good! Thanks Levani,
> > >
> > > +1 (non-binding)
> > >
> > > Sophie
> > >
> > > On Tue, Jul 23, 2019 at 2:16 PM Levani Kokhreidze <
> > levani.co...@gmail.com>
> > > wrote:
> > >
> > >> Hello,
> > >>
> > >> I’d like to initialize voting on KIP-221:
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-221%3A+Enhance+KStream+with+Connecting+Topic+Creation+and+Repartition+Hint
> > >> <
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-221:+Enhance+KStream+with+Connecting+Topic+Creation+and+Repartition+Hint
> > >>>
> > >> If there’re any more concerns about the KIP, happy to discuss further.
> > >>
> > >> Regards,
> > >> Levani
> >
> >
>


-- 
-- Guozhang

Build failed in Jenkins: kafka-trunk-jdk11 #721

[Apache Jenkins Server]

See 


Changes:

[jason] KAFKA-8640; Use generated classes in OffsetFetch request and response

[jason] KAFKA-8442; Include ISR in Metadata response even if there is no leader

[github] KAFKA-8717; Reuse cached offset metadata when reading from log (#7081)

--
[...truncated 2.57 MB...]

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnIsOpen 
PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldDeleteAndReturnPlainValue STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldDeleteAndReturnPlainValue PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnName 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnName 
PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutAllWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutAllWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldReturnIsPersistent STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldReturnIsPersistent PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutIfAbsentWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutIfAbsentWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldForwardClose 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldForwardClose 
PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldForwardFlush 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldForwardFlush 
PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldForwardInit 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldForwardInit 
PASSED

> Task :streams:streams-scala:test

org.apache.kafka.streams.scala.WordCountTest > testShouldCountWordsMaterialized 
STARTED

org.apache.kafka.streams.scala.WordCountTest > testShouldCountWordsMaterialized 
PASSED

org.apache.kafka.streams.scala.WordCountTest > testShouldCountWordsJava STARTED

org.apache.kafka.streams.scala.WordCountTest > testShouldCountWordsJava PASSED

org.apache.kafka.streams.scala.WordCountTest > testShouldCountWords STARTED

org.apache.kafka.streams.scala.WordCountTest > testShouldCountWords PASSED

org.apache.kafka.streams.scala.TopologyTest > 
shouldBuildIdenticalTopologyInJavaNScalaJoin STARTED

org.apache.kafka.streams.scala.TopologyTest > 
shouldBuildIdenticalTopologyInJavaNScalaJoin PASSED

org.apache.kafka.streams.scala.TopologyTest > 
shouldBuildIdenticalTopologyInJavaNScalaSimple STARTED

org.apache.kafka.streams.scala.TopologyTest > 
shouldBuildIdenticalTopologyInJavaNScalaSimple PASSED

org.apache.kafka.streams.scala.TopologyTest > 
shouldBuildIdenticalTopologyInJavaNScalaAggregate STARTED

org.apache.kafka.streams.scala.TopologyTest > 
shouldBuildIdenticalTopologyInJavaNScalaAggregate PASSED

org.apache.kafka.streams.scala.TopologyTest > 
shouldBuildIdenticalTopologyInJavaNScalaProperties STARTED

org.apache.kafka.streams.scala.TopologyTest > 
shouldBuildIdenticalTopologyInJavaNScalaProperties PASSED

org.apache.kafka.streams.scala.TopologyTest > 
shouldBuildIdenticalTopologyInJavaNScalaTransform STARTED

org.apache.kafka.streams.scala.TopologyTest > 
shouldBuildIdenticalTopologyInJavaNScalaTransform PASSED

org.apache.kafka.streams.scala.StreamToTableJoinScalaIntegrationTestImplicitSerdes
 > testShouldCountClicksPerRegionWithNamedRepartitionTopic STARTED

org.apache.kafka.streams.scala.StreamToTableJoinScalaIntegrationTestImplicitSerdes
 > testShouldCountClicksPerRegionWithNamedRepartitionTopic PASSED

org.apache.kafka.streams.scala.StreamToTableJoinScalaIntegrationTestImplicitSerdes
 > testShouldCountClicksPerRegionJava STARTED

org.apache.kafka.streams.scala.StreamToTableJoinScalaIntegrationTestImplicitSerdes
 > testShouldCountClicksPerRegionJava PASSED

org.apache.kafka.streams.scala.StreamToTableJoinScalaIntegrationTestImplicitSerdes
 > testShouldCountClicksPerRegion STARTED

org.apache.kafka.streams.scala.StreamToTableJoinScalaIntegrationTestImplicitSerdes
 > testShouldCountClicksPerRegion PASSED

org.apache.kafka.streams.scala.kstream.ConsumedTest > Create a Consumed should 
create a Consumed with Serdes STARTED

org.apache.kafka.streams.scala.kstream.ConsumedTest > Create a Consumed should 
create a Consumed with Serdes PASSED

org.apache.kafka.streams.scala.kstream.ConsumedTest > Create a Consumed with 
timestampExtractor and resetPolicy should create a Consumed with Serdes,

[jira] [Created] (KAFKA-8736) Performance: ThreadCache uses size() for empty cache check

[Matthew Jarvie (JIRA)]

Matthew Jarvie created KAFKA-8736:
-

 Summary: Performance: ThreadCache uses size() for empty cache check
 Key: KAFKA-8736
 URL: https://issues.apache.org/jira/browse/KAFKA-8736
 Project: Kafka
  Issue Type: Improvement
  Components: streams
Affects Versions: 2.3.0
Reporter: Matthew Jarvie
 Attachments: size.patch

While load testing Kafka Streams in 2.3.0, we stumbled across a potential 
performance improvement. The test showed we were spending 80% of CPU time in 
ConcurrentSkipListMap.size():

 
{noformat}
100% org.apache.kafka.streams.processor.internals.StreamThread.run():774
100% org.apache.kafka.streams.processor.internals.StreamThread.runLoop():805
96.84% org.apache.kafka.streams.processor.internals.StreamThread.runOnce():890
96.84% 
org.apache.kafka.streams.processor.internals.TaskManager.process(long):420
96.83% 
org.apache.kafka.streams.processor.internals.AssignedStreamsTasks.process(long):199
96.4% org.apache.kafka.streams.processor.internals.StreamTask.process():366
96.3% 
org.apache.kafka.streams.processor.internals.SourceNode.process(java.lang.Object,
 java.lang.Object):87
96.3% 
org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(java.lang.Object,
 java.lang.Object):133
96.3% 
org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(java.lang.Object,
 java.lang.Object, org.apache.kafka.streams.processor.To):180
96.3% 
org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(org.apache.kafka.streams.processor.internals.ProcessorNode,
 java.lang.Object, java.lang.Object):201
96.23% 
org.apache.kafka.streams.processor.internals.ProcessorNode.process(java.lang.Object,
 java.lang.Object):117
96.12% 
org.apache.kafka.streams.kstream.internals.KStreamFilter$KStreamFilterProcessor.process(java.lang.Object,
 java.lang.Object):43
96.12% 
org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(java.lang.Object,
 java.lang.Object):133
96.12% 
org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(java.lang.Object,
 java.lang.Object, org.apache.kafka.streams.processor.To):180
96.12% 
org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(org.apache.kafka.streams.processor.internals.ProcessorNode,
 java.lang.Object, java.lang.Object):201
96.08% 
org.apache.kafka.streams.processor.internals.ProcessorNode.process(java.lang.Object,
 java.lang.Object):117
82.78% 
org.apache.kafka.streams.kstream.internals.KStreamSessionWindowAggregate$KStreamSessionWindowAggregateProcessor.process(java.lang.Object,
 java.lang.Object):169
82.78% 
org.apache.kafka.streams.processor.internals.ProcessorContextImpl$SessionStoreReadWriteDecorator.put(org.apache.kafka.streams.kstream.Windowed,
 java.lang.Object):612
82.59% 
org.apache.kafka.streams.state.internals.MeteredSessionStore.put(org.apache.kafka.streams.kstream.Windowed,
 java.lang.Object):127
81.11% 
org.apache.kafka.streams.state.internals.CachingSessionStore.put(org.apache.kafka.streams.kstream.Windowed,
 java.lang.Object):35
81.09% 
org.apache.kafka.streams.state.internals.CachingSessionStore.put(org.apache.kafka.streams.kstream.Windowed,
 byte[]):131
81.09% 
org.apache.kafka.streams.state.internals.ThreadCache.put(java.lang.String, 
org.apache.kafka.common.utils.Bytes, 
org.apache.kafka.streams.state.internals.LRUCacheEntry):151
80.53% 
org.apache.kafka.streams.state.internals.ThreadCache.maybeEvict(java.lang.String):238
80.53% org.apache.kafka.streams.state.internals.NamedCache.size():266
80.53% java.util.concurrent.ConcurrentSkipListMap.size():1639{noformat}
According to 
[https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/ConcurrentSkipListMap.html#size--],
 the size method has to traverse all elements to get a count. It looks like the 
count is being compared against 0 to determine if the map is empty; In this 
case, we don't need a full count. Instead, the isEmpty() method should be used, 
which just looks for one node. We patched this and gained about 25% max 
throughput, and this method disappeared from thread dumps as a hot spot.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Request to be a contributor

[Michael Carter]

Hi dev,

Can I please be added to the contributor list?
My JIRA username is michael_carter

Cheers,
Michael

[DISCUSS] KIP-499 - Unify connection name flag for command line tool

[Mitchell]

Hello,
I have written a proposal to add the command line argument
`--bootstrap-server` to 5 of the existing command line tools that do not
currently use `--broker-list` for passing cluster connection information.

https://cwiki.apache.org/confluence/display/KAFKA/KIP-499+-+Unify+connection+name+flag+for+command+line+tool

Please take a look and let me know what you think.
Thanks,
-Mitch

[jira] [Created] (KAFKA-8737) TaskMigrated Exception while rebalancing kafka streams

[KUMAR (JIRA)]

KUMAR created KAFKA-8737:


 Summary: TaskMigrated Exception while rebalancing kafka streams
 Key: KAFKA-8737
 URL: https://issues.apache.org/jira/browse/KAFKA-8737
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 1.0.1, 1.0.0
 Environment: 20 partitions 
1 topic 
8 Streamer service 
topic-region-1 9  7841726 8236017 
394291 
streams-subscriberstopic-region-1-29d615ed-4243-4b9d-90b7-9c517aa0f2e3-StreamThread-1-consumer-0276e83d-40b5-4b44-b764-7d29e0dab663/
 
streams-subscriberstopic-region-1-29d615ed-4243-4b9d-90b7-9c517aa0f2e3-StreamThread-1-consumer
topic-region-1 15 7421710 7467666 45956 
 
streams-subscriberstopic-region-1-29d615ed-4243-4b9d-90b7-9c517aa0f2e3-StreamThread-1-consumer-0276e83d-40b5-4b44-b764-7d29e0dab663/
 
streams-subscriberstopic-region-1-29d615ed-4243-4b9d-90b7-9c517aa0f2e3-StreamThread-1-consumer
topic-region-1 19 7737360 8120611 
383251 
streams-subscriberstopic-region-1-29d615ed-4243-4b9d-90b7-9c517aa0f2e3-StreamThread-1-consumer-0276e83d-40b5-4b44-b764-7d29e0dab663/

streams-subscriberstopic-region-1-29d615ed-4243-4b9d-90b7-9c517aa0f2e3-StreamThread-1-consumer
topic-region-1
Reporter: KUMAR


Kafka  streams throws following exception while restart of a stream client 
service - 

o.a.k.s.p.internals.StreamThread.? - stream-thread 
[streams-subscriberstopic-region-1-32d968e3-f892-4772-a7a4-6f684d7e43c9-StreamThread-1]
 Detected a task that got migrated to another thread. This implies that this 
thread missed a rebalance and dropped out of the consumer group. Trying to 
rejoin the consumer group now.
org.apache.kafka.streams.errors.TaskMigratedException: Log end offset of 
topic-region-1-12 should not change while restoring: old end offset 6286727, 
current offset 6380997

 

Kafka version is 1.0.0 and we have back merged the fix for KIP-6269-

[https://github.com/apache/kafka/pull/4300/files#|https://github.com/apache/kafka/pull/4300/files]

However we observe that there seems to be an issue in rebalance when 
"auto.offset.reset" is configured as "latest". Based on log analysis we see 
following behavior - 
 # StreamThread starts a restore consumer 
 # While Fetching it gets offset out of range                               
o.a.k.c.consumer.internals.Fetcher.? - [Consumer 
clientId=streams-subscriberstopic-region-1-11b2d7fb-11ce-4b0b-a40a-388d3c7b6bc9-StreamThread-1-restore-
 consumer, groupId=] Fetch READ_UNCOMMITTED at offset 246431 for partition 
topic-region-1-12 returned fetch data (error=OFFSET_OUT_OF_RANGE, 
highWaterMark=-1, lastStableOffset = -1, logStartOffset = -1,
 abortedTransactions = null, recordsSizeInBytes=0) 
 # Fetcher tries to reset the offset 
 # While reset the offset it appears it is changing the offset position and 
causing TaskMigrated exception

Above test repeated with "auto.offset.reset" is configured as "earliest" does 
not throw any TaskMigrated exception as in earliest case we are not reseting 
the restore consumer position.

 

Please let us know if this is possible and if a fix would be needed for the 
offset reset piece when set to latest.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (KAFKA-8738) Cleaning thread blocked when more than one ALTER_REPLICA_LOG_DIRS requests sent

[dingsainan (JIRA)]

dingsainan created KAFKA-8738:
-

 Summary: Cleaning thread blocked  when more than one 
ALTER_REPLICA_LOG_DIRS requests sent
 Key: KAFKA-8738
 URL: https://issues.apache.org/jira/browse/KAFKA-8738
 Project: Kafka
  Issue Type: Bug
Affects Versions: 2.1.1
Reporter: dingsainan


Hi,
 
I am experiencing one situation  that the log cleaner dose not work  for the 
related topic-partition when using --kafka-reassign-partitions.sh tool for 
V2.1.1 for more than one time frequently.
 
My operation:
submitting one task for migration replica in one same broker first,  when the 
previous task still in progress, we submit one new task for the same 
topic-partition.
 
My search:
Kafka executes abortAndPauseCleaning() once task is submitted, shortly, another 
task is submitted for the same topic-partition, so the clean thread status is 
{color:#ff}LogCleaningPaused(2){color} currently. When the second task 
completed, the clean thread will be resumed for this topic-partition once. In 
my case, the previous task is killed directly, no resumeClean() is executed for 
the first task, so when the second task is completed, the clean status for the 
topic-partition is still {color:#ff}LogCleaningPaused(1){color}, which 
blocks the clean thread for the topic-partition.
 
_That's all my search, please confirm._
 
_Thanks_
_Nora_



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Jenkins build is back to normal : kafka-trunk-jdk8 #3819

[Apache Jenkins Server]

See 

Build failed in Jenkins: kafka-trunk-jdk11 #722

[Apache Jenkins Server]

See 


Changes:

[github] MINOR: Refactor abstractConfig#configuredInstance (#7129)

--
[...truncated 2.57 MB...]

org.apache.kafka.streams.StreamsConfigTest > 
shouldOverrideStreamsDefaultProducerConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldOverrideStreamsDefaultProducerConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedPropertiesThatAreNotPartOfProducerConfig STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedPropertiesThatAreNotPartOfProducerConfig PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedGlobalConsumerConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedGlobalConsumerConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldBeSupportNonPrefixedConsumerConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldBeSupportNonPrefixedConsumerConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > 
consumerConfigShouldContainAdminClientConfigsForRetriesAndRetryBackOffMsWithAdminPrefix
 STARTED

org.apache.kafka.streams.StreamsConfigTest > 
consumerConfigShouldContainAdminClientConfigsForRetriesAndRetryBackOffMsWithAdminPrefix
 PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldOverrideStreamsDefaultConsumerConifgsOnRestoreConsumer STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldOverrideStreamsDefaultConsumerConifgsOnRestoreConsumer PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedPropertiesThatAreNotPartOfGlobalConsumerConfig STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedPropertiesThatAreNotPartOfGlobalConsumerConfig PASSED

org.apache.kafka.streams.StreamsConfigTest > shouldAcceptExactlyOnce STARTED

org.apache.kafka.streams.StreamsConfigTest > shouldAcceptExactlyOnce PASSED

org.apache.kafka.streams.StreamsConfigTest > 
testGetGlobalConsumerConfigsWithGlobalConsumerOverridenPrefix STARTED

org.apache.kafka.streams.StreamsConfigTest > 
testGetGlobalConsumerConfigsWithGlobalConsumerOverridenPrefix PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldThrowExceptionIfMaxInFlightRequestsGreaterThanFiveIfEosEnabled STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldThrowExceptionIfMaxInFlightRequestsGreaterThanFiveIfEosEnabled PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSetInternalLeaveGroupOnCloseConfigToFalseInConsumer STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSetInternalLeaveGroupOnCloseConfigToFalseInConsumer PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedPropertiesThatAreNotPartOfConsumerConfig STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedPropertiesThatAreNotPartOfConsumerConfig PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldForwardCustomConfigsWithNoPrefixToAllClients STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldForwardCustomConfigsWithNoPrefixToAllClients PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldResetToDefaultIfRestoreConsumerAutoCommitIsOverridden STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldResetToDefaultIfRestoreConsumerAutoCommitIsOverridden PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSpecifyCorrectValueSerdeClassOnError STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSpecifyCorrectValueSerdeClassOnError PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldAllowToSpecifyMaxInFlightRequestsPerConnectionAsStringIfEosEnabled STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldAllowToSpecifyMaxInFlightRequestsPerConnectionAsStringIfEosEnabled PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldResetToDefaultIfGlobalConsumerAutoCommitIsOverridden STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldResetToDefaultIfGlobalConsumerAutoCommitIsOverridden PASSED

org.apache.kafka.streams.StreamsConfigTest > testGetGroupInstanceIdConfigs 
STARTED

org.apache.kafka.streams.StreamsConfigTest > testGetGroupInstanceIdConfigs 
PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldResetToDefaultIfConsumerIsolationLevelIsOverriddenIfEosEnabled STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldResetToDefaultIfConsumerIsolationLevelIsOverriddenIfEosEnabled PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldBeSupportNonPrefixedGlobalConsumerConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldBeSupportNonPrefixedGlobalConsumerConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportNonPrefixedAdminConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportNonPrefixedAdminConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldNotOverrideUserConfigCommitIntervalMsIfExactlyOnceEnabled STARTED

org.apache.kafka.streams.StreamsConfigTest > 
s

Re: [DISCUSS] KIP-499 - Unify connection name flag for command line tool

[Jason Gustafson]

Hey Mitch, thanks for the KIP! This command line inconsistency frustrates
me almost every day. I'm definitely +1 on this.

One minor nitpick. The compatibility section mentions there will be no
deprecations, but it sounds like we are planning on deprecating the old
arguments?

Thanks,
Jason

On Tue, Jul 30, 2019 at 5:25 PM Mitchell  wrote:

> Hello,
> I have written a proposal to add the command line argument
> `--bootstrap-server` to 5 of the existing command line tools that do not
> currently use `--broker-list` for passing cluster connection information.
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-499+-+Unify+connection+name+flag+for+command+line+tool
>
> Please take a look and let me know what you think.
> Thanks,
> -Mitch
>

Re: [VOTE] KIP-221: Enhance KStream with Connecting Topic Creation and Repartition Hint

[Levani Kokhreidze]

Hello Guozhang,

Thanks for the feedback. That’s an interesting point. To be honest, I totally 
missed it. I wasn’t aware that there’s `groupBy` possibility on KTable. 
I don’t see any reasons why not to add same functionality to KTable interface.

I’ve updated the KIP: 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-221%3A+Enhance+DSL+with+Connecting+Topic+Creation+and+Repartition+Hint
 

Please let me know if you have any other questions and/or concerns.

Regards,
Levani

> On Jul 31, 2019, at 1:24 AM, Guozhang Wang  wrote:
> 
> Hello Levani,
> 
> Thanks for the KIP! Just got a quick question here about the scope: why do
> we only want this for `KStream`, not `KTable#groupBy` for example?
> 
> 
> Guozhang
> 
> 
> On Tue, Jul 30, 2019 at 1:27 PM Bill Bejeck  wrote:
> 
>> Thanks for the KIP Levani.
>> 
>> +1 (binding)
>> 
>> -Bill
>> 
>> On Tue, Jul 30, 2019 at 3:37 PM Levani Kokhreidze 
>> wrote:
>> 
>>> Hello,
>>> 
>>> Still waiting for feedback on this KIP.
>>> Please let me know if you have any concerns and/or questions.
>>> 
>>> Regards,
>>> Levani
>>> 
>>> 
 On Jul 24, 2019, at 8:20 PM, Sophie Blee-Goldman 
>>> wrote:
 
 Looks good! Thanks Levani,
 
 +1 (non-binding)
 
 Sophie
 
 On Tue, Jul 23, 2019 at 2:16 PM Levani Kokhreidze <
>>> levani.co...@gmail.com>
 wrote:
 
> Hello,
> 
> I’d like to initialize voting on KIP-221:
> 
>>> 
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-221%3A+Enhance+KStream+with+Connecting+Topic+Creation+and+Repartition+Hint
> <
> 
>>> 
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-221:+Enhance+KStream+with+Connecting+Topic+Creation+and+Repartition+Hint
>> 
> If there’re any more concerns about the KIP, happy to discuss further.
> 
> Regards,
> Levani
>>> 
>>> 
>> 
> 
> 
> -- 
> -- Guozhang

Re: Request to be a contributor

[Matthias J. Sax]

Done.

On 7/30/19 4:30 PM, Michael Carter wrote:
> Hi dev,
> 
> Can I please be added to the contributor list?
> My JIRA username is michael_carter
> 
> Cheers,
> Michael
> 



signature.asc
Description: OpenPGP digital signature