Build failed in Jenkins: kafka-trunk-jdk11 #1196

[Apache Jenkins Server]

See 


Changes:

[github] MINOR: Remove tag from metric to measure process-rate on source nodes


--
[...truncated 1.71 MB...]
kafka.controller.ControllerChannelManagerTest > testLeaderAndIsrRequestIsNew 
PASSED

kafka.controller.ControllerChannelManagerTest > 
testStopReplicaRequestsWhileTopicQueuedForDeletion STARTED

kafka.controller.ControllerChannelManagerTest > 
testStopReplicaRequestsWhileTopicQueuedForDeletion PASSED

kafka.controller.ControllerChannelManagerTest > 
testLeaderAndIsrRequestSentToLiveOrShuttingDownBrokers STARTED

kafka.controller.ControllerChannelManagerTest > 
testLeaderAndIsrRequestSentToLiveOrShuttingDownBrokers PASSED

kafka.controller.ControllerChannelManagerTest > 
testStopReplicaInterBrokerProtocolVersion STARTED

kafka.controller.ControllerChannelManagerTest > 
testStopReplicaInterBrokerProtocolVersion PASSED

kafka.controller.ControllerChannelManagerTest > 
testStopReplicaSentOnlyToLiveAndShuttingDownBrokers STARTED

kafka.controller.ControllerChannelManagerTest > 
testStopReplicaSentOnlyToLiveAndShuttingDownBrokers PASSED

kafka.controller.ControllerChannelManagerTest > testStopReplicaGroupsByBroker 
STARTED

kafka.controller.ControllerChannelManagerTest > testStopReplicaGroupsByBroker 
PASSED

kafka.controller.ControllerChannelManagerTest > 
testUpdateMetadataDoesNotIncludePartitionsWithoutLeaderAndIsr STARTED

kafka.controller.ControllerChannelManagerTest > 
testUpdateMetadataDoesNotIncludePartitionsWithoutLeaderAndIsr PASSED

kafka.controller.ControllerChannelManagerTest > 
testMixedDeleteAndNotDeleteStopReplicaRequests STARTED

kafka.controller.ControllerChannelManagerTest > 
testMixedDeleteAndNotDeleteStopReplicaRequests PASSED

kafka.controller.ControllerChannelManagerTest > 
testLeaderAndIsrInterBrokerProtocolVersion STARTED

kafka.controller.ControllerChannelManagerTest > 
testLeaderAndIsrInterBrokerProtocolVersion PASSED

kafka.controller.ControllerChannelManagerTest > testUpdateMetadataRequestSent 
STARTED

kafka.controller.ControllerChannelManagerTest > testUpdateMetadataRequestSent 
PASSED

kafka.controller.ControllerChannelManagerTest > 
testUpdateMetadataRequestDuringTopicDeletion STARTED

kafka.controller.ControllerChannelManagerTest > 
testUpdateMetadataRequestDuringTopicDeletion PASSED

kafka.controller.ControllerChannelManagerTest > 
testUpdateMetadataIncludesLiveOrShuttingDownBrokers STARTED

kafka.controller.ControllerChannelManagerTest > 
testUpdateMetadataIncludesLiveOrShuttingDownBrokers PASSED

kafka.controller.ControllerChannelManagerTest > testStopReplicaRequestSent 
STARTED

kafka.controller.ControllerChannelManagerTest > testStopReplicaRequestSent 
PASSED

kafka.controller.ControllerChannelManagerTest > 
testStopReplicaRequestsWhileTopicDeletionStarted STARTED

kafka.controller.ControllerChannelManagerTest > 
testStopReplicaRequestsWhileTopicDeletionStarted PASSED

kafka.controller.ControllerChannelManagerTest > testLeaderAndIsrRequestSent 
STARTED

kafka.controller.ControllerChannelManagerTest > testLeaderAndIsrRequestSent 
PASSED

unit.kafka.cluster.AssignmentStateTest > testPartitionAssignmentStatus[0] 
STARTED

unit.kafka.cluster.AssignmentStateTest > testPartitionAssignmentStatus[0] PASSED

unit.kafka.cluster.AssignmentStateTest > testPartitionAssignmentStatus[1] 
STARTED

unit.kafka.cluster.AssignmentStateTest > testPartitionAssignmentStatus[1] PASSED

unit.kafka.cluster.AssignmentStateTest > testPartitionAssignmentStatus[2] 
STARTED

unit.kafka.cluster.AssignmentStateTest > testPartitionAssignmentStatus[2] PASSED

unit.kafka.cluster.AssignmentStateTest > testPartitionAssignmentStatus[3] 
STARTED

unit.kafka.cluster.AssignmentStateTest > testPartitionAssignmentStatus[3] PASSED

unit.kafka.cluster.AssignmentStateTest > testPartitionAssignmentStatus[4] 
STARTED

unit.kafka.cluster.AssignmentStateTest > testPartitionAssignmentStatus[4] PASSED

unit.kafka.cluster.AssignmentStateTest > testPartitionAssignmentStatus[5] 
STARTED

unit.kafka.cluster.AssignmentStateTest > testPartitionAssignmentStatus[5] PASSED

unit.kafka.cluster.AssignmentStateTest > testPartitionAssignmentStatus[6] 
STARTED

unit.kafka.cluster.AssignmentStateTest > testPartitionAssignmentStatus[6] PASSED

unit.kafka.cluster.AssignmentStateTest > testPartitionAssignmentStatus[7] 
STARTED

unit.kafka.cluster.AssignmentStateTest > testPartitionAssignmentStatus[7] PASSED

unit.kafka.cluster.AssignmentStateTest > testPartitionAssignmentStatus[8] 
STARTED

unit.kafka.cluster.AssignmentStateTest > testPartitionAssignmentStatus[8] PASSED
ERROR: No tool found matching GRADLE_4_10_2_HOME
ERROR: Could not install GRADLE_4_10_3_HOME
java.lang.NullPointerException
at 
hudson.plugins.toolenv.ToolEnvBuildWrapper$1.buildEnvVars(ToolEnvBuildWrapper.java:46)
at hudson.model.AbstractBuild.getEnvironment(AbstractBuild.java:873)
at hudson.plugins.git.Git

Build failed in Jenkins: kafka-2.5-jdk8 #46

[Apache Jenkins Server]

See 


Changes:

[rhauch] KAFKA-9601: Stop logging raw connector config values (#8165)


--
[...truncated 5.86 MB...]
org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfInMemoryBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfInMemoryBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourcesThatMatchMultiplePattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourcesThatMatchMultiplePattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldPopulateGlobalStore[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldPopulateGlobalStore[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfPersistentBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfPersistentBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldAllowPrePopulatingStatesStoresWithCachingEnabled[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldAllowPrePopulatingStatesStoresWithCachingEnabled[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldRespectTaskIdling[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldRespectTaskIdling[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldApplyGlobalUpdatesCorrectlyInRecursiveTopologies[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldApplyGlobalUpdatesCorrectlyInRecursiveTopologies[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabl

Build failed in Jenkins: kafka-trunk-jdk8 #4278

[Apache Jenkins Server]

See 


Changes:

[github] HOTFIX: fix failing test


--
[...truncated 5.82 MB...]

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfTimestampIsDifferentForCompareValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfTimestampIsDifferentForCompareValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualWithNullForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualWithNullForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestampWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordWithExpectedRecordForCompareValueTimestamp 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordWithExpectedRecordForCompareValueTimestamp 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullExpectedRecordForCompareValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullExpectedRecordForCompareValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareKeyValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareKeyValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestampWithProducerRecord

[jira] [Resolved] (KAFKA-9609) Memory Leak in Kafka Producer

[Jira]

 [ 
https://issues.apache.org/jira/browse/KAFKA-9609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sönke Liebau resolved KAFKA-9609.
-
  Assignee: Sönke Liebau
Resolution: Not A Bug

As discussed in the comments I do not believe that this is a bug. Yes, memory 
usage may increase over time with this usage pattern, but it is not a memory 
leak, as the information needs to be kept in order to report correct metrics.

> Memory Leak in Kafka Producer
> -
>
> Key: KAFKA-9609
> URL: https://issues.apache.org/jira/browse/KAFKA-9609
> Project: Kafka
>  Issue Type: Bug
>  Components: clients, producer 
>Affects Versions: 2.4.0
>Reporter: Satish
>Assignee: Sönke Liebau
>Priority: Major
>
> org.apache.kafka.clients.producer.internals.Sender adds Topic Metrics for 
> every topic that we are writing messages to but it never been cleaned up 
> until we close the producer.
> This is an issue if we use single producer and have more number of Dynamic 
> topics (eg: ~ 500 topics per hour) and writing messages to them.  As this 
> Metrics map is getting accumulated for every topic, over a period of time we 
> notice the memory usage gets increased gradually. 
> It can be easily reproducible by writing messages to the more # of dynamic 
> topics using the same KafkaProducer from apache kafka client libraries or 
> KafkaTemplate from Spring.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [ANNOUNCE] New committer: Konstantine Karantasis

[Mickael Maison]

Congratulations Konstantine! Well deserved

On Thu, Feb 27, 2020 at 7:11 AM Satish Duggana  wrote:
>
> Congrats Konstantine!!
>
> On Thu, Feb 27, 2020 at 12:35 PM Tom Bentley  wrote:
> >
> > Congratulations!
> >
> > On Thu, Feb 27, 2020 at 6:43 AM David Jacot  wrote:
> >
> > > Congrats!
> > >
> > > Le jeu. 27 févr. 2020 à 06:58, Vahid Hashemian 
> > > a écrit :
> > >
> > > > Congratulations Konstantine!
> > > >
> > > > Regards,
> > > > --Vahid
> > > >
> > > > On Wed, Feb 26, 2020 at 6:49 PM Boyang Chen 
> > > > wrote:
> > > >
> > > > > Congrats Konstantine！
> > > > >
> > > > > On Wed, Feb 26, 2020 at 6:32 PM Manikumar 
> > > > > wrote:
> > > > >
> > > > > > Congrats Konstantine!
> > > > >
> > > > >
> > > > > > On Thu, Feb 27, 2020 at 7:46 AM Matthias J. Sax 
> > > > > wrote:
> > > > > >
> > > > > > > -BEGIN PGP SIGNED MESSAGE-
> > > > > > > Hash: SHA512
> > > > > > >
> > > > > > > Congrats!
> > > > > > >
> > > > > > > On 2/27/20 2:21 AM, Jeremy Custenborder wrote:
> > > > > > > > Congrats Konstantine!
> > > > > > > >
> > > > > > > > On Wed, Feb 26, 2020 at 2:39 PM Bill Bejeck 
> > > > > > > > wrote:
> > > > > > > >>
> > > > > > > >> Congratulations Konstantine! Well deserved.
> > > > > > > >>
> > > > > > > >> -Bill
> > > > > > > >>
> > > > > > > >> On Wed, Feb 26, 2020 at 5:37 PM Jason Gustafson
> > > > > > > >>  wrote:
> > > > > > > >>
> > > > > > > >>> The PMC for Apache Kafka has invited Konstantine Karantasis as
> > > > > > > >>> a committer and we are pleased to announce that he has
> > > > > > > >>> accepted!
> > > > > > > >>>
> > > > > > > >>> Konstantine has contributed 56 patches and helped to review
> > > > > > > >>> even more. His recent work includes a major overhaul of the
> > > > > > > >>> Connect task management system in order to support incremental
> > > > > > > >>> rebalancing. In addition to code contributions, Konstantine
> > > > > > > >>> helps the community in many other ways including talks at
> > > > > > > >>> meetups and at Kafka Summit and answering questions on
> > > > > > > >>> stackoverflow. He consistently shows good judgement in design
> > > > > > > >>> and a careful attention to details when it comes to code.
> > > > > > > >>>
> > > > > > > >>> Thanks for all the contributions and looking forward to more!
> > > > > > > >>>
> > > > > > > >>> Jason, on behalf of the Apache Kafka PMC
> > > > > > > >>>
> > > > > > > -BEGIN PGP SIGNATURE-
> > > > > > >
> > > > > > > iQIzBAEBCgAdFiEEI8mthP+5zxXZZdDSO4miYXKq/OgFAl5XJl0ACgkQO4miYXKq
> > > > > > > /OjEERAAp08DioD903r9aqGCJ3oHbkUi2ZwEr0yeu1veFD3rGFflhni7sm4J87/Z
> > > > > > > NArUdpHQ7+99YATtm+HY8gHN+eQeVY1+NV4lX77vPNLEONx09aIxbcSnc4Ih7JnX
> > > > > > > XxcHHBeR6N9EJkMQ82nOUk5cQaDiIdkOvbOv8+NoOcZC6RMy9PKsFgwd8NTMKL8l
> > > > > > > nqG8KwnV6WYNOJ05BszFSpTwJBJBg8zhZlRcmAF7iRD0cLzM4BReOmEVCcoas7ar
> > > > > > > RK38okIAALDRkC9JNYpQ/s0si4V+OwP4igp0MAjM+Y2NVLhC6kK6uqNzfeD21M6U
> > > > > > > mWm7nE9Tbh/K+8hgZbqfprN6vw6+NfU8dwPD0iaEOfisXwbavCfDeonwSWK0BoHo
> > > > > > > zEeHRGEx7e2FHWp8KyC6XgfFWmkWJP6tCWiTtCFEScxSTzZUC+cG+a5PF1n6hIHo
> > > > > > > /CH3Oml2ZGDxoEl1zt8Hs5AgKW8X4PQCsfA4LWqA4GgR6PPFPn6g8mX3/AR3wkyn
> > > > > > > 8Dmlh3k8ZtsW8wX26IYBywm/yyjbnlSzRVSgAHAbpaIqe5PoMG+5fdc1tnO/2Tuf
> > > > > > > jd1BjbgAD6u5BksmIGBeZXADblQ/qqfp5Q+WRTJSYLItf8HMNAZoqJFp0bRy/QXP
> > > > > > > 5EZzI9J+ngJG8MYr08UcWQMZt0ytwBVTX/+FVC8Rx5r0D0fqizo=
> > > > > > > =68IH
> > > > > > > -END PGP SIGNATURE-
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Thanks!
> > > > --Vahid
> > > >
> > >

Re: [ANNOUNCE] New committer: Konstantine Karantasis

[Rajini Sivaram]

Congratulations Konstantine!

On Thu, Feb 27, 2020 at 9:57 AM Mickael Maison 
wrote:

> Congratulations Konstantine! Well deserved
>
> On Thu, Feb 27, 2020 at 7:11 AM Satish Duggana 
> wrote:
> >
> > Congrats Konstantine!!
> >
> > On Thu, Feb 27, 2020 at 12:35 PM Tom Bentley 
> wrote:
> > >
> > > Congratulations!
> > >
> > > On Thu, Feb 27, 2020 at 6:43 AM David Jacot 
> wrote:
> > >
> > > > Congrats!
> > > >
> > > > Le jeu. 27 févr. 2020 à 06:58, Vahid Hashemian <
> vahid.hashem...@gmail.com>
> > > > a écrit :
> > > >
> > > > > Congratulations Konstantine!
> > > > >
> > > > > Regards,
> > > > > --Vahid
> > > > >
> > > > > On Wed, Feb 26, 2020 at 6:49 PM Boyang Chen <
> reluctanthero...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Congrats Konstantine！
> > > > > >
> > > > > > On Wed, Feb 26, 2020 at 6:32 PM Manikumar <
> manikumar.re...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Congrats Konstantine!
> > > > > >
> > > > > >
> > > > > > > On Thu, Feb 27, 2020 at 7:46 AM Matthias J. Sax <
> mj...@apache.org>
> > > > > > wrote:
> > > > > > >
> > > > > > > > -BEGIN PGP SIGNED MESSAGE-
> > > > > > > > Hash: SHA512
> > > > > > > >
> > > > > > > > Congrats!
> > > > > > > >
> > > > > > > > On 2/27/20 2:21 AM, Jeremy Custenborder wrote:
> > > > > > > > > Congrats Konstantine!
> > > > > > > > >
> > > > > > > > > On Wed, Feb 26, 2020 at 2:39 PM Bill Bejeck <
> b...@confluent.io>
> > > > > > > > > wrote:
> > > > > > > > >>
> > > > > > > > >> Congratulations Konstantine! Well deserved.
> > > > > > > > >>
> > > > > > > > >> -Bill
> > > > > > > > >>
> > > > > > > > >> On Wed, Feb 26, 2020 at 5:37 PM Jason Gustafson
> > > > > > > > >>  wrote:
> > > > > > > > >>
> > > > > > > > >>> The PMC for Apache Kafka has invited Konstantine
> Karantasis as
> > > > > > > > >>> a committer and we are pleased to announce that he has
> > > > > > > > >>> accepted!
> > > > > > > > >>>
> > > > > > > > >>> Konstantine has contributed 56 patches and helped to
> review
> > > > > > > > >>> even more. His recent work includes a major overhaul of
> the
> > > > > > > > >>> Connect task management system in order to support
> incremental
> > > > > > > > >>> rebalancing. In addition to code contributions,
> Konstantine
> > > > > > > > >>> helps the community in many other ways including talks at
> > > > > > > > >>> meetups and at Kafka Summit and answering questions on
> > > > > > > > >>> stackoverflow. He consistently shows good judgement in
> design
> > > > > > > > >>> and a careful attention to details when it comes to code.
> > > > > > > > >>>
> > > > > > > > >>> Thanks for all the contributions and looking forward to
> more!
> > > > > > > > >>>
> > > > > > > > >>> Jason, on behalf of the Apache Kafka PMC
> > > > > > > > >>>
> > > > > > > > -BEGIN PGP SIGNATURE-
> > > > > > > >
> > > > > > > >
> iQIzBAEBCgAdFiEEI8mthP+5zxXZZdDSO4miYXKq/OgFAl5XJl0ACgkQO4miYXKq
> > > > > > > >
> /OjEERAAp08DioD903r9aqGCJ3oHbkUi2ZwEr0yeu1veFD3rGFflhni7sm4J87/Z
> > > > > > > >
> NArUdpHQ7+99YATtm+HY8gHN+eQeVY1+NV4lX77vPNLEONx09aIxbcSnc4Ih7JnX
> > > > > > > >
> XxcHHBeR6N9EJkMQ82nOUk5cQaDiIdkOvbOv8+NoOcZC6RMy9PKsFgwd8NTMKL8l
> > > > > > > >
> nqG8KwnV6WYNOJ05BszFSpTwJBJBg8zhZlRcmAF7iRD0cLzM4BReOmEVCcoas7ar
> > > > > > > >
> RK38okIAALDRkC9JNYpQ/s0si4V+OwP4igp0MAjM+Y2NVLhC6kK6uqNzfeD21M6U
> > > > > > > >
> mWm7nE9Tbh/K+8hgZbqfprN6vw6+NfU8dwPD0iaEOfisXwbavCfDeonwSWK0BoHo
> > > > > > > >
> zEeHRGEx7e2FHWp8KyC6XgfFWmkWJP6tCWiTtCFEScxSTzZUC+cG+a5PF1n6hIHo
> > > > > > > >
> /CH3Oml2ZGDxoEl1zt8Hs5AgKW8X4PQCsfA4LWqA4GgR6PPFPn6g8mX3/AR3wkyn
> > > > > > > >
> 8Dmlh3k8ZtsW8wX26IYBywm/yyjbnlSzRVSgAHAbpaIqe5PoMG+5fdc1tnO/2Tuf
> > > > > > > >
> jd1BjbgAD6u5BksmIGBeZXADblQ/qqfp5Q+WRTJSYLItf8HMNAZoqJFp0bRy/QXP
> > > > > > > > 5EZzI9J+ngJG8MYr08UcWQMZt0ytwBVTX/+FVC8Rx5r0D0fqizo=
> > > > > > > > =68IH
> > > > > > > > -END PGP SIGNATURE-
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > > Thanks!
> > > > > --Vahid
> > > > >
> > > >
>

Re: [ANNOUNCE] New committer: Konstantine Karantasis

[Bruno Cadonna]

Congrats, Konstantine! Excellent!

Best,
Bruno

On Thu, Feb 27, 2020 at 11:28 AM Rajini Sivaram  wrote:
>
> Congratulations Konstantine!
>
> On Thu, Feb 27, 2020 at 9:57 AM Mickael Maison 
> wrote:
>
> > Congratulations Konstantine! Well deserved
> >
> > On Thu, Feb 27, 2020 at 7:11 AM Satish Duggana 
> > wrote:
> > >
> > > Congrats Konstantine!!
> > >
> > > On Thu, Feb 27, 2020 at 12:35 PM Tom Bentley 
> > wrote:
> > > >
> > > > Congratulations!
> > > >
> > > > On Thu, Feb 27, 2020 at 6:43 AM David Jacot 
> > wrote:
> > > >
> > > > > Congrats!
> > > > >
> > > > > Le jeu. 27 févr. 2020 à 06:58, Vahid Hashemian <
> > vahid.hashem...@gmail.com>
> > > > > a écrit :
> > > > >
> > > > > > Congratulations Konstantine!
> > > > > >
> > > > > > Regards,
> > > > > > --Vahid
> > > > > >
> > > > > > On Wed, Feb 26, 2020 at 6:49 PM Boyang Chen <
> > reluctanthero...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Congrats Konstantine！
> > > > > > >
> > > > > > > On Wed, Feb 26, 2020 at 6:32 PM Manikumar <
> > manikumar.re...@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Congrats Konstantine!
> > > > > > >
> > > > > > >
> > > > > > > > On Thu, Feb 27, 2020 at 7:46 AM Matthias J. Sax <
> > mj...@apache.org>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > -BEGIN PGP SIGNED MESSAGE-
> > > > > > > > > Hash: SHA512
> > > > > > > > >
> > > > > > > > > Congrats!
> > > > > > > > >
> > > > > > > > > On 2/27/20 2:21 AM, Jeremy Custenborder wrote:
> > > > > > > > > > Congrats Konstantine!
> > > > > > > > > >
> > > > > > > > > > On Wed, Feb 26, 2020 at 2:39 PM Bill Bejeck <
> > b...@confluent.io>
> > > > > > > > > > wrote:
> > > > > > > > > >>
> > > > > > > > > >> Congratulations Konstantine! Well deserved.
> > > > > > > > > >>
> > > > > > > > > >> -Bill
> > > > > > > > > >>
> > > > > > > > > >> On Wed, Feb 26, 2020 at 5:37 PM Jason Gustafson
> > > > > > > > > >>  wrote:
> > > > > > > > > >>
> > > > > > > > > >>> The PMC for Apache Kafka has invited Konstantine
> > Karantasis as
> > > > > > > > > >>> a committer and we are pleased to announce that he has
> > > > > > > > > >>> accepted!
> > > > > > > > > >>>
> > > > > > > > > >>> Konstantine has contributed 56 patches and helped to
> > review
> > > > > > > > > >>> even more. His recent work includes a major overhaul of
> > the
> > > > > > > > > >>> Connect task management system in order to support
> > incremental
> > > > > > > > > >>> rebalancing. In addition to code contributions,
> > Konstantine
> > > > > > > > > >>> helps the community in many other ways including talks at
> > > > > > > > > >>> meetups and at Kafka Summit and answering questions on
> > > > > > > > > >>> stackoverflow. He consistently shows good judgement in
> > design
> > > > > > > > > >>> and a careful attention to details when it comes to code.
> > > > > > > > > >>>
> > > > > > > > > >>> Thanks for all the contributions and looking forward to
> > more!
> > > > > > > > > >>>
> > > > > > > > > >>> Jason, on behalf of the Apache Kafka PMC
> > > > > > > > > >>>
> > > > > > > > > -BEGIN PGP SIGNATURE-
> > > > > > > > >
> > > > > > > > >
> > iQIzBAEBCgAdFiEEI8mthP+5zxXZZdDSO4miYXKq/OgFAl5XJl0ACgkQO4miYXKq
> > > > > > > > >
> > /OjEERAAp08DioD903r9aqGCJ3oHbkUi2ZwEr0yeu1veFD3rGFflhni7sm4J87/Z
> > > > > > > > >
> > NArUdpHQ7+99YATtm+HY8gHN+eQeVY1+NV4lX77vPNLEONx09aIxbcSnc4Ih7JnX
> > > > > > > > >
> > XxcHHBeR6N9EJkMQ82nOUk5cQaDiIdkOvbOv8+NoOcZC6RMy9PKsFgwd8NTMKL8l
> > > > > > > > >
> > nqG8KwnV6WYNOJ05BszFSpTwJBJBg8zhZlRcmAF7iRD0cLzM4BReOmEVCcoas7ar
> > > > > > > > >
> > RK38okIAALDRkC9JNYpQ/s0si4V+OwP4igp0MAjM+Y2NVLhC6kK6uqNzfeD21M6U
> > > > > > > > >
> > mWm7nE9Tbh/K+8hgZbqfprN6vw6+NfU8dwPD0iaEOfisXwbavCfDeonwSWK0BoHo
> > > > > > > > >
> > zEeHRGEx7e2FHWp8KyC6XgfFWmkWJP6tCWiTtCFEScxSTzZUC+cG+a5PF1n6hIHo
> > > > > > > > >
> > /CH3Oml2ZGDxoEl1zt8Hs5AgKW8X4PQCsfA4LWqA4GgR6PPFPn6g8mX3/AR3wkyn
> > > > > > > > >
> > 8Dmlh3k8ZtsW8wX26IYBywm/yyjbnlSzRVSgAHAbpaIqe5PoMG+5fdc1tnO/2Tuf
> > > > > > > > >
> > jd1BjbgAD6u5BksmIGBeZXADblQ/qqfp5Q+WRTJSYLItf8HMNAZoqJFp0bRy/QXP
> > > > > > > > > 5EZzI9J+ngJG8MYr08UcWQMZt0ytwBVTX/+FVC8Rx5r0D0fqizo=
> > > > > > > > > =68IH
> > > > > > > > > -END PGP SIGNATURE-
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > >
> > > > > > Thanks!
> > > > > > --Vahid
> > > > > >
> > > > >
> >

Build failed in Jenkins: kafka-trunk-jdk11 #1197

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-9607: Do not clear partition queues during close (#8168)

[github] HOTFIX: fix failing test


--
[...truncated 5.85 MB...]
org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
PASSED

org.apache.kafka.streams.TestTopicsTest > testNonUsedOutputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonUsedOutputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testEmptyTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testEmptyTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testStartTimestamp STARTED

org.apache.kafka.streams.TestTopicsTest > testStartTimestamp PASSED

org.apache.kafka.streams.TestTopicsTest > testNegativeAdvance STARTED

org.apache.kafka.streams.TestTopicsTest > testNegativeAdvance PASSED

org.apache.kafka.streams.TestTopicsTest > shouldNotAllowToCreateWithNullDriver 
STARTED

org.apache.kafka.streams.TestTopicsTest > shouldNotAllowToCreateWithNullDriver 
PASSED

org.apache.kafka.streams.TestTopicsTest > testDuration STARTED

org.apache.kafka.streams.TestTopicsTest > testDuration PASSED

org.apache.kafka.streams.TestTopicsTest > testOutputToString STARTED

org.apache.kafka.streams.TestTopicsTest > testOutputToString PASSED

org.apache.kafka.streams.TestTopicsTest > testValue STARTED

org.apache.kafka.streams.TestTopicsTest > testValue PASSED

org.apache.kafka.streams.TestTopicsTest > testTimestampAutoAdvance STARTED

org.apache.kafka.streams.TestTopicsTest > testTimestampAutoAdvance PASSED

org.apache.kafka.streams.TestTopicsTest > testOutputWrongSerde STARTED

org.apache.kafka.streams.TestTopicsTest > testOutputWrongSerde PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputTopicWithNullTopicName STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputTopicWithNullTopicName PASSED

org.apache.kafka.streams.TestTopicsTest > testWrongSerde STARTED

org.apache.kafka.streams.TestTopicsTest > testWrongSerde PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMapWithNull STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMapWithNull PASSED

org.apache.kafka.streams.TestTopicsTest > testNonExistingOutputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonExistingOutputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testMultipleTopics STARTED

org.apache.kafka.streams.TestTopicsTest > testMultipleTopics PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValueList STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValueList PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputWithNullDriver STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputWithNullDriver PASSED

org.apache.kafka.streams.TestTopicsTest > testValueList STARTED

org.apache.kafka.streams.TestTopicsTest > testValueList PASSED

org.apache.kafka.streams.TestTopicsTest > testRecordList STARTED

org.apache.kafka.streams.TestTopicsTest > testRecordList PASSED

org.apache.kafka.streams.TestTopicsTest > testNonExistingInputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonExistingInputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMap STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMap PASSED

org.apache.kafka.streams.TestTopicsTest > testRecordsToList STARTED

org.apache.kafka.streams.TestTopicsTest > testRecordsToList PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValueListDuration STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValueListDuration PASSED

org.apache.kafka.streams.TestTopicsTest > testInputToString STARTED

org.apache.kafka.streams.TestTopicsTest > testInputToString PASSED

org.apache.kafka.streams.TestTopicsTest > testTimestamp STARTED

org.apache.kafka.streams.TestTopicsTest > testTimestamp PASSED

org.apache.kafka.streams.TestTopicsTest > testWithHeaders STARTED

org.apache.kafka.streams.TestTopicsTest > testWithHeaders PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValue STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValue PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateTopicWithNullTopicName STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateTopicWithNullTopicName PASSED

> Task :streams:upgrade-system-tests-0100:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0100:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:classes UP-TO-DATE
> Task :streams:upgrade-system-tests-0100:checkstyleMain NO-SOURCE
> Task :streams:upgrade-system-tests-0100:compileTestJava
> Task :streams:upgrade-system-tests-0100:processTestReso

Re: [KAFKA-557] Add emit on change support for Kafka Streams

[Bruno Cadonna]

Hi John,

I agree with you. It is better to measure the metric on processor node
level. The users can do the rollup to task-level by themselves.

Best,
Bruno

On Thu, Feb 27, 2020 at 12:09 AM John Roesler  wrote:
>
> Hi Richard,
>
> I've been making a final pass over the KIP.
>
> Re: Proposed Behavior Change:
>
> I think this point is controversial and probably doesn't need to be there at 
> all:
> > 2.b. In certain situations where there is a high volume of idempotent
> > updates throughout the Streams DAG, it will be recommended practice
> > to materialize all operations to reduce traffic overall across the entire
> >  network of nodes.
>
> Re-reading all the points, it seems like we can sum them up in a way that's
> a little more straight to the point, and gives us the right amount of 
> flexibility:
>
> > Proposed Behavior Changes
> >
> > Definition: "idempotent update" is one in which the new result and prior
> > result,  when serialized, are identical byte arrays
> >
> > Note: an "update" is a concept that only applies to Table operations, so
> > the concept of an "idempotent update" also only applies to Table operations.
> > See 
> > https://kafka.apache.org/documentation/streams/developer-guide/dsl-api.html#streams_concepts_ktable
> > for more information.
> >
> > Given that definition, we propose for Streams to drop idempotent updates
> > in any situation where it's possible and convenient to do so. For example,
> > any time we already have both the prior and new results serialized, we
> > may compare them, and drop the update if it is idempotent.
> >
> > Note that under this proposal, we can implement idempotence checking
> > in the following situations:
> > 1. Any aggregation (for example, KGroupedStream, KGroupedTable,
> > TimeWindowedKStream, and SessionWindowedKStream operations)
> > 2. Any Materialized KTable operation
> > 3. Repartition operations, when we need to send both prior and new results
>
> Notice that in my proposed wording, we neither limit ourselves to just the
> situations enumerated, nor promise to implement the optimization in every
> possible situation. IMHO, this is the best way to propose such a feature.
> That way, we have the flexibility to implement it in stages, and also to add
> on to the implementation in the future.
>
>
> Re: Metrics
>
> I agree with Bruno, although, I think it might just be a confusing statement.
> It might be clearer to drop all the "discussion", and just say: "We will add a
> metric to count the number of idempotent updates that we have dropped".
>
> Also, with respect to the metric, I'm wondering if the metric should be task-
> level or processor-node-level. Since the interesting action takes place inside
> individual processor nodes, I _think_ it would be higher leverage to just
> measure it at the node level. WDYT?
>
> Re: Design Reasoning
>
> This section seems to be a little bit outdated. I also just noticed a 
> "surprise"
> configuration "timestamp.aggregation.selection.policy" hidden in point 1.a.
> Is that part of the proposal? We haven't discussed it, and I think we were
> talking about this KIP being "configuration free".
>
> There is also some discussion of discarded alternative in the Design Reasoning
> section, which is confusing. Finally, there was a point there I didn't 
> understand
> at all, about stateless operators not being intended to load prior results.
> This statement doesn't seem to be true, but it also doesn't seem to be 
> relevant,
> so maybe we can just drop it.
>
> Overall, it might help if you make a pass on this section, and just discuss as
> briefly as possible the justification for the proposed behavior change, and
> not adding a configuration. Try to avoid talking about things that we are not
> proposing, since that will just lead to confusion.
>
> Similarly, I'd just completely remove the "Implementation [discarded]" 
> section.
> It was good to have this as part of the discussion initially, but as we move
> toward a vote, it's better to just streamline the KIP document as much as
> possible. Keeping a "discarded" section in the document will just make it
> harder for new people to understand the proposal. We did the same thing
> with KIP-441, where there were two prior drafts included at the end of the
> document, and we just deleted them for clarity.
>
> I liked the "Compatibility" and "Rejected Alternatives" section. Very clear
> and to the point.
>
> Thanks again for the contribution! I think once the KIP document is cleaned
> up, we'll be in good shape to finalize the discussion.
> -John
>
>
> On Wed, Feb 26, 2020, at 07:27, Bruno Cadonna wrote:
> > Hi Richard,
> >
> > 1. Could you change "idempotent update operations will only be dropped
> > from KTables, not from other classes." -> idempotent update operations
> > will only be dropped from materialized KTables? For non-materialized
> > KTables -- as they can occur after optimization of the topology -- we
> > cannot drop idempotent updates.
> >
> > 2

Jira access

[Paolo Moriello]

Hello,

Can you please add me as a contributor to Jira?

My Jira username: paolomoriello

Thanks,
Paolo

[jira] [Created] (KAFKA-9617) Replica Fetcher can mark partition as failed when max.message.bytes is changed

[Stanislav Kozlovski (Jira)]

Stanislav Kozlovski created KAFKA-9617:
--

 Summary: Replica Fetcher can mark partition as failed when 
max.message.bytes is changed
 Key: KAFKA-9617
 URL: https://issues.apache.org/jira/browse/KAFKA-9617
 Project: Kafka
  Issue Type: Bug
Reporter: Stanislav Kozlovski
Assignee: Stanislav Kozlovski


There exists a race condition when changing the dynamic max.message.bytes 
config for a topic. A follower replica can replicate a message that is over 
that size after it processes the config change. When this happens, the replica 
fetcher catches the unexpected exception, marks the partition as failed and 
stops replicating it.
{code:java}
06:38:46.596Processing override for entityPath: topics/partition-1 with 
config: Map(max.message.bytes -> 512)

06:38:46.597 [ReplicaFetcher replicaId=1, leaderId=3, fetcherId=0] 
Unexpected error occurred while processing data for partition partition-1 at 
offset 20964
org.apache.kafka.common.errors.RecordTooLargeException: The record batch size 
in the append to partition-1 is 3349 bytes which exceeds the maximum configured 
value of 512.
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [ANNOUNCE] New committer: Konstantine Karantasis

[Dongjin Lee]

Congratulations, Konstantine!

Cheers,
Dongjin

On Thu, Feb 27, 2020 at 8:26 PM Bruno Cadonna  wrote:

> Congrats, Konstantine! Excellent!
>
> Best,
> Bruno
>
> On Thu, Feb 27, 2020 at 11:28 AM Rajini Sivaram 
> wrote:
> >
> > Congratulations Konstantine!
> >
> > On Thu, Feb 27, 2020 at 9:57 AM Mickael Maison  >
> > wrote:
> >
> > > Congratulations Konstantine! Well deserved
> > >
> > > On Thu, Feb 27, 2020 at 7:11 AM Satish Duggana <
> satish.dugg...@gmail.com>
> > > wrote:
> > > >
> > > > Congrats Konstantine!!
> > > >
> > > > On Thu, Feb 27, 2020 at 12:35 PM Tom Bentley 
> > > wrote:
> > > > >
> > > > > Congratulations!
> > > > >
> > > > > On Thu, Feb 27, 2020 at 6:43 AM David Jacot 
> > > wrote:
> > > > >
> > > > > > Congrats!
> > > > > >
> > > > > > Le jeu. 27 févr. 2020 à 06:58, Vahid Hashemian <
> > > vahid.hashem...@gmail.com>
> > > > > > a écrit :
> > > > > >
> > > > > > > Congratulations Konstantine!
> > > > > > >
> > > > > > > Regards,
> > > > > > > --Vahid
> > > > > > >
> > > > > > > On Wed, Feb 26, 2020 at 6:49 PM Boyang Chen <
> > > reluctanthero...@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Congrats Konstantine！
> > > > > > > >
> > > > > > > > On Wed, Feb 26, 2020 at 6:32 PM Manikumar <
> > > manikumar.re...@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Congrats Konstantine!
> > > > > > > >
> > > > > > > >
> > > > > > > > > On Thu, Feb 27, 2020 at 7:46 AM Matthias J. Sax <
> > > mj...@apache.org>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > -BEGIN PGP SIGNED MESSAGE-
> > > > > > > > > > Hash: SHA512
> > > > > > > > > >
> > > > > > > > > > Congrats!
> > > > > > > > > >
> > > > > > > > > > On 2/27/20 2:21 AM, Jeremy Custenborder wrote:
> > > > > > > > > > > Congrats Konstantine!
> > > > > > > > > > >
> > > > > > > > > > > On Wed, Feb 26, 2020 at 2:39 PM Bill Bejeck <
> > > b...@confluent.io>
> > > > > > > > > > > wrote:
> > > > > > > > > > >>
> > > > > > > > > > >> Congratulations Konstantine! Well deserved.
> > > > > > > > > > >>
> > > > > > > > > > >> -Bill
> > > > > > > > > > >>
> > > > > > > > > > >> On Wed, Feb 26, 2020 at 5:37 PM Jason Gustafson
> > > > > > > > > > >>  wrote:
> > > > > > > > > > >>
> > > > > > > > > > >>> The PMC for Apache Kafka has invited Konstantine
> > > Karantasis as
> > > > > > > > > > >>> a committer and we are pleased to announce that he
> has
> > > > > > > > > > >>> accepted!
> > > > > > > > > > >>>
> > > > > > > > > > >>> Konstantine has contributed 56 patches and helped to
> > > review
> > > > > > > > > > >>> even more. His recent work includes a major overhaul
> of
> > > the
> > > > > > > > > > >>> Connect task management system in order to support
> > > incremental
> > > > > > > > > > >>> rebalancing. In addition to code contributions,
> > > Konstantine
> > > > > > > > > > >>> helps the community in many other ways including
> talks at
> > > > > > > > > > >>> meetups and at Kafka Summit and answering questions
> on
> > > > > > > > > > >>> stackoverflow. He consistently shows good judgement
> in
> > > design
> > > > > > > > > > >>> and a careful attention to details when it comes to
> code.
> > > > > > > > > > >>>
> > > > > > > > > > >>> Thanks for all the contributions and looking forward
> to
> > > more!
> > > > > > > > > > >>>
> > > > > > > > > > >>> Jason, on behalf of the Apache Kafka PMC
> > > > > > > > > > >>>
> > > > > > > > > > -BEGIN PGP SIGNATURE-
> > > > > > > > > >
> > > > > > > > > >
> > > iQIzBAEBCgAdFiEEI8mthP+5zxXZZdDSO4miYXKq/OgFAl5XJl0ACgkQO4miYXKq
> > > > > > > > > >
> > > /OjEERAAp08DioD903r9aqGCJ3oHbkUi2ZwEr0yeu1veFD3rGFflhni7sm4J87/Z
> > > > > > > > > >
> > > NArUdpHQ7+99YATtm+HY8gHN+eQeVY1+NV4lX77vPNLEONx09aIxbcSnc4Ih7JnX
> > > > > > > > > >
> > > XxcHHBeR6N9EJkMQ82nOUk5cQaDiIdkOvbOv8+NoOcZC6RMy9PKsFgwd8NTMKL8l
> > > > > > > > > >
> > > nqG8KwnV6WYNOJ05BszFSpTwJBJBg8zhZlRcmAF7iRD0cLzM4BReOmEVCcoas7ar
> > > > > > > > > >
> > > RK38okIAALDRkC9JNYpQ/s0si4V+OwP4igp0MAjM+Y2NVLhC6kK6uqNzfeD21M6U
> > > > > > > > > >
> > > mWm7nE9Tbh/K+8hgZbqfprN6vw6+NfU8dwPD0iaEOfisXwbavCfDeonwSWK0BoHo
> > > > > > > > > >
> > > zEeHRGEx7e2FHWp8KyC6XgfFWmkWJP6tCWiTtCFEScxSTzZUC+cG+a5PF1n6hIHo
> > > > > > > > > >
> > > /CH3Oml2ZGDxoEl1zt8Hs5AgKW8X4PQCsfA4LWqA4GgR6PPFPn6g8mX3/AR3wkyn
> > > > > > > > > >
> > > 8Dmlh3k8ZtsW8wX26IYBywm/yyjbnlSzRVSgAHAbpaIqe5PoMG+5fdc1tnO/2Tuf
> > > > > > > > > >
> > > jd1BjbgAD6u5BksmIGBeZXADblQ/qqfp5Q+WRTJSYLItf8HMNAZoqJFp0bRy/QXP
> > > > > > > > > > 5EZzI9J+ngJG8MYr08UcWQMZt0ytwBVTX/+FVC8Rx5r0D0fqizo=
> > > > > > > > > > =68IH
> > > > > > > > > > -END PGP SIGNATURE-
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > >
> > > > > > > Thanks!
> > > > > > > --Vahid
> > > > > > >
> > > > > >
> > >
>
-- 
*Dongjin Lee*

*A hitchhiker in the mathematical world.*
*github:  github.com/dongjinleekr
linkedin: kr.linkedin.com/in/dongjinleekr

subscribe kafka dev mail

[Walker Xia]

subscribe kafka dev mail

Re: [DISCUSS] KIP-508: Make Suppression State Queriable - rebooted.

[John Roesler]

Hi Dongjin,

No problem; glad we got it sorted out.

Thanks again for picking this up!
-John

On Wed, Feb 26, 2020, at 09:24, Dongjin Lee wrote:
> > I was under the impression that you wanted to expand the scope of the KIP
> to additionally allow querying the internal buffer, not just the result.
> Can you clarify whether you are proposing to allow querying the state of
> the internal buffer, the result, or both?
> 
> Sorry for the confusion. As we already talked with, we only need to query
> the suppressed output, not the internal buffer. The current implementation
> is wrong. After refining the KIP and implementation accordingly I will
> notify you - I must be confused, also.
> 
> Thanks,
> Dongjin
> 
> On Tue, Feb 25, 2020 at 12:17 AM John Roesler  wrote:
> 
> > Hi Dongjin,
> >
> > Ah, I think I may have been confused. I 100% agree that we need a
> > materialized variant for suppress(). Then, you could do:
> > ...suppress(..., Materialized.as(“final-count”))
> >
> > If that’s your proposal, then we are on the same page.
> >
> > I was under the impression that you wanted to expand the scope of the KIP
> > to additionally allow querying the internal buffer, not just the result.
> > Can you clarify whether you are proposing to allow querying the state of
> > the internal buffer, the result, or both?
> >
> > Thanks,
> > John
> >
> > On Thu, Feb 20, 2020, at 08:41, Dongjin Lee wrote:
> > > Hi John,
> > > Thanks for your kind explanation with an example.
> > >
> > > > But it feels like you're saying you're trying to do something different
> > > than just query the windowed key and get back the current count?
> > >
> > > Yes, for example, what if we need to retrieve the (all or range) keys
> > with
> > > a closed window? In this example, let's imagine we need to retrieve only
> > > (key=A, window=10), not (key=A, window=20).
> > >
> > > Of course, the value accompanied by a flushed key is exactly the same to
> > > the one in the upstream KTable; However, if our intention is not pointing
> > > out a specific key but retrieving a group of unspecified keys, we stuck
> > in
> > > trouble - since we can't be sure which key is flushed out beforehand.
> > >
> > > One workaround would be materializing it with `suppressed.filter(e ->
> > true,
> > > Materialized.as("final-count"))`. But I think providing a materialized
> > > variant for suppress method is better than this workaround.
> > >
> > > Thanks,
> > > Dongjin
> > >
> > > On Thu, Feb 20, 2020 at 1:26 AM John Roesler 
> > wrote:
> > >
> > > > Thanks for the response, Dongjin,
> > > >
> > > > I'm sorry, but I'm still not following. It seems like the view you
> > would
> > > > get on the "current state of the buffer" would always be equivalent to
> > > > the view of the upstream table.
> > > >
> > > > Let me try an example, and maybe you can point out the flaw in my
> > > > reasoning.
> > > >
> > > > Let's say we're doing 10 ms windows with a grace period of zero.
> > > > Let's also say we're computing a windowed count, and that we have
> > > > a "final results" suppression after the count. Let's  materialize the
> > > > count as "Count" and the suppressed result as "Final Count".
> > > >
> > > > Suppose we get an input event:
> > > > (time=10, key=A, value=...)
> > > >
> > > > Then, Count will look like:
> > > >
> > > > | window | key | value |
> > > > | 10 | A   | 1 |
> > > >
> > > > The (internal) suppression buffer will contain:
> > > >
> > > > | window | key | value |
> > > > | 10 | A   | 1 |
> > > >
> > > > The record is still buffered because the window isn't closed yet.
> > > > Final Count is an empty table:
> > > >
> > > > | window | key | value |
> > > >
> > > > ---
> > > >
> > > > Now, we get a second event:
> > > > (time=15, key=A, value=...)
> > > >
> > > > Then, Count will look like:
> > > >
> > > > | window | key | value |
> > > > | 10 | A   | 2 |
> > > >
> > > > The (internal) suppression buffer will contain:
> > > >
> > > > | window | key | value |
> > > > | 10 | A   | 2 |
> > > >
> > > > The record is still buffered because the window isn't closed yet.
> > > > Final Count is an empty table:
> > > >
> > > > | window | key | value |
> > > >
> > > >
> > > > ---
> > > >
> > > > Finally, we get a third event:
> > > > (time=20, key=A, value=...)
> > > >
> > > > Then, Count will look like:
> > > >
> > > > | window | key | value |
> > > > | 10 | A   | 2 |
> > > > | 20 | A   | 1 |
> > > >
> > > > The (internal) suppression buffer will contain:
> > > >
> > > > | window | key | value |
> > > > | 20 | A   | 1 |
> > > >
> > > > Note that window 10 has been flushed out, because it's now closed.
> > > > And window 20 is buffered because it isn't closed yet.
> > > > Final Count is now:
> > > >
> > > > | window | key | value |
> > > > | 10 | A   | 2 |
> > > >
> > > >
> > > > ---
> > > >
> > > > Reading your email, I can't figure out what value there is in querying
>

Re: Jira access

[Guozhang Wang]

Hello Paolo,

I've added you to Kafka JIRA.

Guozhang

On Thu, Feb 27, 2020 at 6:29 AM Paolo Moriello 
wrote:

> Hello,
>
> Can you please add me as a contributor to Jira?
>
> My Jira username: paolomoriello
>
> Thanks,
> Paolo
>


-- 
-- Guozhang

Re: [ANNOUNCE] New committer: Konstantine Karantasis

[Konstantine Karantasis]

Thank you all!
It's an honor and a great responsibility. Really looking forward to being
helpful to the Apache Kafka community on this new role.

Warm regards!
Konstantine



On Thu, Feb 27, 2020 at 8:03 AM Dongjin Lee  wrote:

> Congratulations, Konstantine!
>
> Cheers,
> Dongjin
>
> On Thu, Feb 27, 2020 at 8:26 PM Bruno Cadonna  wrote:
>
> > Congrats, Konstantine! Excellent!
> >
> > Best,
> > Bruno
> >
> > On Thu, Feb 27, 2020 at 11:28 AM Rajini Sivaram  >
> > wrote:
> > >
> > > Congratulations Konstantine!
> > >
> > > On Thu, Feb 27, 2020 at 9:57 AM Mickael Maison <
> mickael.mai...@gmail.com
> > >
> > > wrote:
> > >
> > > > Congratulations Konstantine! Well deserved
> > > >
> > > > On Thu, Feb 27, 2020 at 7:11 AM Satish Duggana <
> > satish.dugg...@gmail.com>
> > > > wrote:
> > > > >
> > > > > Congrats Konstantine!!
> > > > >
> > > > > On Thu, Feb 27, 2020 at 12:35 PM Tom Bentley 
> > > > wrote:
> > > > > >
> > > > > > Congratulations!
> > > > > >
> > > > > > On Thu, Feb 27, 2020 at 6:43 AM David Jacot  >
> > > > wrote:
> > > > > >
> > > > > > > Congrats!
> > > > > > >
> > > > > > > Le jeu. 27 févr. 2020 à 06:58, Vahid Hashemian <
> > > > vahid.hashem...@gmail.com>
> > > > > > > a écrit :
> > > > > > >
> > > > > > > > Congratulations Konstantine!
> > > > > > > >
> > > > > > > > Regards,
> > > > > > > > --Vahid
> > > > > > > >
> > > > > > > > On Wed, Feb 26, 2020 at 6:49 PM Boyang Chen <
> > > > reluctanthero...@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Congrats Konstantine！
> > > > > > > > >
> > > > > > > > > On Wed, Feb 26, 2020 at 6:32 PM Manikumar <
> > > > manikumar.re...@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Congrats Konstantine!
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > > On Thu, Feb 27, 2020 at 7:46 AM Matthias J. Sax <
> > > > mj...@apache.org>
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > -BEGIN PGP SIGNED MESSAGE-
> > > > > > > > > > > Hash: SHA512
> > > > > > > > > > >
> > > > > > > > > > > Congrats!
> > > > > > > > > > >
> > > > > > > > > > > On 2/27/20 2:21 AM, Jeremy Custenborder wrote:
> > > > > > > > > > > > Congrats Konstantine!
> > > > > > > > > > > >
> > > > > > > > > > > > On Wed, Feb 26, 2020 at 2:39 PM Bill Bejeck <
> > > > b...@confluent.io>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >>
> > > > > > > > > > > >> Congratulations Konstantine! Well deserved.
> > > > > > > > > > > >>
> > > > > > > > > > > >> -Bill
> > > > > > > > > > > >>
> > > > > > > > > > > >> On Wed, Feb 26, 2020 at 5:37 PM Jason Gustafson
> > > > > > > > > > > >>  wrote:
> > > > > > > > > > > >>
> > > > > > > > > > > >>> The PMC for Apache Kafka has invited Konstantine
> > > > Karantasis as
> > > > > > > > > > > >>> a committer and we are pleased to announce that he
> > has
> > > > > > > > > > > >>> accepted!
> > > > > > > > > > > >>>
> > > > > > > > > > > >>> Konstantine has contributed 56 patches and helped
> to
> > > > review
> > > > > > > > > > > >>> even more. His recent work includes a major
> overhaul
> > of
> > > > the
> > > > > > > > > > > >>> Connect task management system in order to support
> > > > incremental
> > > > > > > > > > > >>> rebalancing. In addition to code contributions,
> > > > Konstantine
> > > > > > > > > > > >>> helps the community in many other ways including
> > talks at
> > > > > > > > > > > >>> meetups and at Kafka Summit and answering questions
> > on
> > > > > > > > > > > >>> stackoverflow. He consistently shows good judgement
> > in
> > > > design
> > > > > > > > > > > >>> and a careful attention to details when it comes to
> > code.
> > > > > > > > > > > >>>
> > > > > > > > > > > >>> Thanks for all the contributions and looking
> forward
> > to
> > > > more!
> > > > > > > > > > > >>>
> > > > > > > > > > > >>> Jason, on behalf of the Apache Kafka PMC
> > > > > > > > > > > >>>
> > > > > > > > > > > -BEGIN PGP SIGNATURE-
> > > > > > > > > > >
> > > > > > > > > > >
> > > > iQIzBAEBCgAdFiEEI8mthP+5zxXZZdDSO4miYXKq/OgFAl5XJl0ACgkQO4miYXKq
> > > > > > > > > > >
> > > > /OjEERAAp08DioD903r9aqGCJ3oHbkUi2ZwEr0yeu1veFD3rGFflhni7sm4J87/Z
> > > > > > > > > > >
> > > > NArUdpHQ7+99YATtm+HY8gHN+eQeVY1+NV4lX77vPNLEONx09aIxbcSnc4Ih7JnX
> > > > > > > > > > >
> > > > XxcHHBeR6N9EJkMQ82nOUk5cQaDiIdkOvbOv8+NoOcZC6RMy9PKsFgwd8NTMKL8l
> > > > > > > > > > >
> > > > nqG8KwnV6WYNOJ05BszFSpTwJBJBg8zhZlRcmAF7iRD0cLzM4BReOmEVCcoas7ar
> > > > > > > > > > >
> > > > RK38okIAALDRkC9JNYpQ/s0si4V+OwP4igp0MAjM+Y2NVLhC6kK6uqNzfeD21M6U
> > > > > > > > > > >
> > > > mWm7nE9Tbh/K+8hgZbqfprN6vw6+NfU8dwPD0iaEOfisXwbavCfDeonwSWK0BoHo
> > > > > > > > > > >
> > > > zEeHRGEx7e2FHWp8KyC6XgfFWmkWJP6tCWiTtCFEScxSTzZUC+cG+a5PF1n6hIHo
> > > > > > > > > > >
> > > > /CH3Oml2ZGDxoEl1zt8Hs5AgKW8X4PQCsfA4LWqA4GgR6PPFPn6g8mX3/AR3wkyn
> > > > > > > > > > >
> > > > 8Dmlh3k8ZtsW8wX26IYBywm/yyjbnlSzRVSgAHAbpaIqe5PoMG+5fdc1tnO/2Tuf
> > > > > > > > > > >
> > > > jd1BjbgAD6u5BksmIGBeZXADblQ/qqfp5Q+WRTJSYLItf8HMNAZoqJFp0bR

Re: 回复：[Discuss] KIP-571: Add option to force remove members in StreamsResetter

[John Roesler]

Hey Feyman,

Thanks for starting the vote. While reviewing the discussion I saw
one thing that should be in the KIP:
> If it is used upon the older clusters like 2.3, UnsupportedVersionException 
> will be thrown.

I'll cast my vote now.

Thanks,
-John

On Wed, Feb 26, 2020, at 19:40, feyman2009 wrote:
> Hi, Sophie
> Thanks a lot!
> I have initiated a vote 
> 
> Thanks!
> Feyman
> 
> 
> --
> 发件人：Sophie Blee-Goldman 
> 发送时间：2020年2月27日(星期四) 08:04
> 收件人：feyman2009 
> 主　题：Re: [Discuss] KIP-571: Add option to force remove members in 
> StreamsResetter
> 
> Hi guys,
> 
> Just to clarify, I meant a batch API on the admin not for the 
> StreamsResetter, to avoid
> extra round trips and a simpler API. But I suppose it might be useful 
> to be able to
> remove individual (dynamic) members and not the whole group for other 
> use cases
> that could then benefit from this as well.
> 
> Anyways, I'm fine with the current plan if that makes sense to you. 
> Feel free to call
> for a vote if the KIP is ready
> 
> Cheers,
> Sophie
> On Mon, Feb 24, 2020 at 4:16 AM feyman2009  wrote:
> 
> Hi, Boyang
> Thanks! I have updated the KIP :)
> If Sophie also thinks it's ok, I will start a vote soon.
> 
> Thanks!
> Feyman
> 
> --
> 发件人：Boyang Chen 
> 发送时间：2020年2月24日(星期一) 00:42
> 收件人：dev 
> 主　题：Re: [Discuss] KIP-571: Add option to force remove members in 
> StreamsResetter
> 
> Hey Feyman,
> 
> thanks a lot for the update, the KIP LGTM now. Will let Sophie take a look
> again, also a minor API change:
> s/setGroupInstanceId/withGroupInstanceId, and similar to setMemberId, as
> usually setters are not expected to return an actual object.
> 
> Boyang
> 
> On Sat, Feb 22, 2020 at 11:05 PM feyman2009  wrote:
> 
> > Hi, Boyang
> > Thanks for your review, I have updated the KIP page :)
> >
> > Hi, Sophie
> > Thanks for your suggestions!
> > 1)  Did you consider an API that just removes *all* remaining members
> > from a group?
> > We plan to implement the batch removal in StreamsResetter as below:
> > 1) adminClient#describeConsumerGroups to get members in each
> > group, this part needs no change.
> > 2) adminClient#removeMembersFromConsumerGroup to remove all the
> > members got from the above call (This involves API change to support the
> > dynamic member removal)
> > I think your suggestion is feasible but maybe not necessary currently
> > since it is a subset of the combination of the above two APIs. Looking at
> > the APIs in KafkaAdminClient, the adminClient.deleteXXX always takes a
> > collection as the input parameter and the caller does the "query and
> > delete" if "delete all" is needed, this leaves more burden on the caller
> > side but increases flexibility. Since the KafkaAdminClient's API is still
> > evolving, I think it would be reasonable to follow the convention and not
> > adding a "removal all members" API.
> >
> > 2) Thanks to Boyang's correction, broker version >= 2.4 is needed
> > since batch members removal is introduced since then(please check KIP-345
> > 
> >  for
> > details).
> > If it is used upon the older clusters like 2.3, 
> > *UnsupportedVersionException
> > *will be thrown.
> >
> > Thanks!
> > Haoran
> >
> > --
> > 发件人：Boyang Chen 
> > 发送时间：2020年2月19日(星期三) 11:57
> > 收件人：dev 
> > 主 题：Re: [Discuss] KIP-571: Add option to force remove members in
> > StreamsResetter
> >
> > Also Feyman, there is one thing I forget which is that the leave group
> > change was introduced in 2.4 broker instead of 2.3. Feel free to correct it
> > on the KIP.
> >
> > On Tue, Feb 18, 2020 at 5:44 PM Sophie Blee-Goldman 
> > wrote:
> >
> > > Hey Feyman,
> > >
> > > Thanks for the KIP! I had two high-level questions:
> > >
> >
> > > It seems like, in the specific case motivating this KIP, we would only 
> > > ever
> > > want to remove *all* the members remaining in the group (and never just a
> > > single member at a time). As you mention there is already an admin API to
> >
> > > remove static members, but we'd still need something new to handle dynamic
> > > ones. Did you consider an API that just removes *all* remaining members
> > > from a group, rather than requiring the caller to determine and then
> > > specify the
> > > group.id (static) or member.id (dynamic) for each one? This way we can
> > > just
> >
> > > have a single API exposed that will handle what we need to do regardless 
> > > of
> > > whether static membership is used or not.
> > >
> >
> > > My other question is, will this new option only work for clusters that are
> > > on 2.3
> > > or higher? Do you have any thoughts about whether it would be possible to
> > > implement this fea

Re: 回复：[Vote] KIP-571: Add option to force remove members in StreamsResetter

[John Roesler]

Thanks for the proposal!

I'm +1 (binding)
-John

On Wed, Feb 26, 2020, at 19:41, feyman2009 wrote:
> Updated with the KIP link: 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-571%3A+Add+option+to+force+remove+members+in+StreamsResetter
> 
> 
> --
> 发件人：feyman2009 
> 发送时间：2020年2月27日(星期四) 09:38
> 收件人：dev 
> 主　题：[Vote] KIP-571: Add option to force remove members in StreamsResetter
> 
> 
> Hi, all
> I would like to start a vote on KIP-571: Add option to force remove 
> members in StreamsResetter .
> 
> Thanks!
> Feyman
> 
>

[jira] [Created] (KAFKA-9618) Failed state store deletion could lead to task file not found

[Boyang Chen (Jira)]

Boyang Chen created KAFKA-9618:
--

 Summary: Failed state store deletion could lead to task file not 
found
 Key: KAFKA-9618
 URL: https://issues.apache.org/jira/browse/KAFKA-9618
 Project: Kafka
  Issue Type: Bug
  Components: streams
Reporter: Boyang Chen
Assignee: Boyang Chen


A failed deletion of a stream task directory could later lead to the impression 
that the task state is still there, thus causing file not found exception as 
the directory was partially deleted.
{code:java}
[2020-02-26T22:08:05-08:00] 
(streams-soak-trunk-eos_soak_i-04ebd21fd0e0da9bf_streamslog) [2020-02-27 
06:08:04,394] WARN 
[stream-soak-test-b26adb53-07e2-4013-933a-0f4bcac84c04-StreamThread-2] 
stream-thread 
[stream-soak-test-b26adb53-07e2-4013-933a-0f4bcac84c04-StreamThread-2] task 
[2_2] Failed to wiping state stores for task 2_2 
(org.apache.kafka.streams.processor.internals.StreamTask) 
[2020-02-26T22:08:05-08:00] 
(streams-soak-trunk-eos_soak_i-04ebd21fd0e0da9bf_streamslog) [2020-02-27 
06:08:04,394] INFO 
[stream-soak-test-b26adb53-07e2-4013-933a-0f4bcac84c04-StreamThread-2] 
[Producer 
clientId=stream-soak-test-b26adb53-07e2-4013-933a-0f4bcac84c04-StreamThread-2-2_2-producer,
 transactionalId=stream-soak-test-2_2] Closing the Kafka producer with 
timeoutMillis = 9223372036854775807 ms. 
(org.apache.kafka.clients.producer.KafkaProducer)
[2020-02-26T22:08:05-08:00] 
(streams-soak-trunk-eos_soak_i-04ebd21fd0e0da9bf_streamslog) [2020-02-27 
06:08:04,411] ERROR 
[stream-soak-test-b26adb53-07e2-4013-933a-0f4bcac84c04-StreamThread-1] 
stream-thread 
[stream-soak-test-b26adb53-07e2-4013-933a-0f4bcac84c04-StreamThread-1] 
Encountered the following exception during processing and the thread is going 
to shut down:  (org.apache.kafka.streams.processor.internals.StreamThread) 
[2020-02-26T22:08:05-08:00] 
(streams-soak-trunk-eos_soak_i-04ebd21fd0e0da9bf_streamslog) 
org.apache.kafka.streams.errors.ProcessorStateException: Error opening store 
KSTREAM-AGGREGATE-STATE-STORE-40 at location 
/mnt/run/streams/state/stream-soak-test/2_2/rocksdb/KSTREAM-AGGREGATE-STATE-STORE-40
         at 
org.apache.kafka.streams.state.internals.RocksDBTimestampedStore.openRocksDB(RocksDBTimestampedStore.java:87)
         at 
org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:191)
         at 
org.apache.kafka.streams.state.internals.RocksDBStore.init(RocksDBStore.java:230)
         at 
org.apache.kafka.streams.state.internals.WrappedStateStore.init(WrappedStateStore.java:48)
         at 
org.apache.kafka.streams.state.internals.ChangeLoggingKeyValueBytesStore.init(ChangeLoggingKeyValueBytesStore.java:44)
         at 
org.apache.kafka.streams.state.internals.WrappedStateStore.init(WrappedStateStore.java:48)
         at 
org.apache.kafka.streams.state.internals.CachingKeyValueStore.init(CachingKeyValueStore.java:58)
         at 
org.apache.kafka.streams.state.internals.WrappedStateStore.init(WrappedStateStore.java:48)
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: 回复：[Vote] KIP-571: Add option to force remove members in StreamsResetter

[Boyang Chen]

Thanks Feyman, +1 (non-binding)

On Thu, Feb 27, 2020 at 9:25 AM John Roesler  wrote:

> Thanks for the proposal!
>
> I'm +1 (binding)
> -John
>
> On Wed, Feb 26, 2020, at 19:41, feyman2009 wrote:
> > Updated with the KIP link:
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-571%3A+Add+option+to+force+remove+members+in+StreamsResetter
> >
> >
> > --
> > 发件人：feyman2009 
> > 发送时间：2020年2月27日(星期四) 09:38
> > 收件人：dev 
> > 主 题：[Vote] KIP-571: Add option to force remove members in StreamsResetter
> >
> >
> > Hi, all
> > I would like to start a vote on KIP-571: Add option to force remove
> > members in StreamsResetter .
> >
> > Thanks!
> > Feyman
> >
> >
>

[jira] [Created] (KAFKA-9619) Receiving duplicates when application is configured for exactly once

[Cristian Manoliu (Jira)]

Cristian Manoliu created KAFKA-9619:
---

 Summary: Receiving duplicates when application is configured for 
exactly once
 Key: KAFKA-9619
 URL: https://issues.apache.org/jira/browse/KAFKA-9619
 Project: Kafka
  Issue Type: Bug
  Components: consumer, producer 
Affects Versions: 2.0.1
 Environment: Red Hat Enterprise Linux Server release 6.10 (Santiago)
Reporter: Cristian Manoliu
 Attachments: log

Hi. There are cases (very rarely, but there are) when I receive duplicates, 
even if everything is configured for high durability and we use exactly once 
configuration.

 

Please check below the application context and test scenario that causes this 
issue.
h2. Kafka Cluster Setup

3 x Kafka Brokers (1 on *host1*, 2 on *host2* and 3 on *host3*)

3 x Zookeeper instances (1 on *host1*, 2 on *host2* and 3 on *host3*)
h3. Kafka configuration
broker.id=1,2,3
num.network.threads=2
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/home/kafka/logs/kafka
min.insync.replicas=3
transaction.state.log.min.isr=3
default.replication.factor=3
log.retention.minutes=600
log.segment.bytes=1073741824
log.retention.check.interval.ms=30
zookeeper.connect=host1:2181,host2:2181,host3:2181
zookeeper.connection.timeout.ms=6000
group.initial.rebalance.delay.ms=1000
log.message.timestamp.type=LogAppendTime
delete.topic.enable=true
auto.create.topics.enable=false
unclean.leader.election.enable=false
h3. ZooKeeper configuration
tickTime=2000
dataDir=/home/kafka/logs/zk
clientPort=2181
maxClientCnxns=0
initLimit=5
syncLimit=2
server.1=host1:2888:3888
server.2=host2:2888:3888
server.3=host3:2888:3888
autopurge.snapRetainCount=3
autopurge.purgeInterval=24
h2. Kafka internal topics description
Topic:__transaction_state       PartitionCount:50       ReplicationFactor:3     
Configs:segment.bytes=104857600,unclean.leader.election.enable=false,compression.type=uncompressed,cleanup.policy=compact,min.insync.replicas=3
        Topic: __transaction_state      Partition: 0    Leader: 1       
Replicas: 3,2,1 Isr: 1,2,3
​
Topic:__consumer_offsets        PartitionCount:50       ReplicationFactor:3     
Configs:segment.bytes=104857600,unclean.leader.election.enable=false,min.insync.replicas=3,cleanup.policy=compact,compression.type=producer
        Topic: __consumer_offsets       Partition: 0    Leader: 1       
Replicas: 3,2,1 Isr: 1,2,3
h2. Application topics
h3. Topic input-event
Topic:input-event      PartitionCount:3        ReplicationFactor:3   
Configs:retention.ms=2881,unclean.leader.election.enable=false,min.insync.replicas=3,message.timestamp.difference.max.ms=2880
        Topic: input-event     Partition: 0    Leader: 1       Replicas: 1,2,3 
Isr: 1,2,3
        Topic: input-event     Partition: 1    Leader: 2       Replicas: 2,3,1 
Isr: 1,2,3
        Topic: input-event     Partition: 2    Leader: 3       Replicas: 3,1,2 
Isr: 1,2,3
h3. Topic output-event
Topic:output-event        PartitionCount:3        ReplicationFactor:3    
Configs:retention.ms=2881,unclean.leader.election.enable=false,min.insync.replicas=3,message.timestamp.difference.max.ms=2880
        Topic: output-event       Partition: 0    Leader: 2       Replicas: 
2,3,1 Isr: 1,2,3
        Topic: output-event       Partition: 1    Leader: 3       Replicas: 
3,1,2 Isr: 1,2,3
        Topic: output-event       Partition: 2    Leader: 1       Replicas: 
1,2,3 Isr: 1,2,3
h2. Application consumer properties
o.a.k.clients.consumer.ConsumerConfig : ConsumerConfig values: 
                auto.commit.interval.ms = 5000
                auto.offset.reset = earliest
                bootstrap.servers = [host1:9092, host2:9092, host3:9092]
                check.crcs = true
                client.id = 
                connections.max.idle.ms = 54
                default.api.timeout.ms = 6
                enable.auto.commit = false
                exclude.internal.topics = true
                fetch.max.bytes = 134217728
                fetch.max.wait.ms = 500
                fetch.min.bytes = 1
                group.id = groupId
                heartbeat.interval.ms = 3000
                interceptor.classes = []
                internal.leave.group.on.close = true
                isolation.level = read_committed
                key.deserializer = class 
org.apache.kafka.common.serialization.ByteArrayDeserializer
                max.partition.fetch.bytes = 134217728
                max.poll.interval.ms = 30
                max.poll.records = 1
                metadata.max.age.ms = 30
                metric.reporters = []
                metrics.num.samples = 2
                metrics.recording.level = INFO
                metrics.sample.window.ms = 3
                partition.assignment.strategy = [class 
org.apache.kafka.clients.consumer.Ran

Re: [DISCUSS] KIP-572: Improve timeouts and retires in Kafka Streams

[John Roesler]

Hi Matthias,

Thanks for the proposal! I think this will be a wonderful improvement
to Streams. In particular, thanks for the motivation. It would indeed
be nice not to have to set long timeout configs and block individual 
client requests in order to cope with transient slow responses.

I'm very well aware that this might make me sound like a crazy person,
but one alternative I'd like to consider is not bounding the retries at all.
Instead, Streams would just skip over timed-out tasks and try again
on the next iteration, as you proposed, but would continue to do so
indefinitely. Clearly, we shouldn't do such a thing silently, so I'd further
propose to log a warning every time a task times out and also maintain
a new metric indicating task timeouts.

To see why this might be attractive, let me pose a hypothetical installation
which has thousands of Streams instances, maybe as part of hundreds of
applications belonging to dozens of teams. Let's also assume there is a
single broker cluster serving all these instances. Such an environment has
a number of transient failure modes:
* A single broker instance may become slow to respond due to hardware
failures (e.g., a bad NIC) or other environmental causes (CPU competition
with co-resident processes, long JVM GC pauses, etc.). Single-broker
unavailability could cause some tasks to time out while others can proceed
in an individual Streams instance.
* The entire broker cluster could become temporarily unavailable (consider:
a faulty firewall configuration gets deployed, severing all Streams instances
from the brokers).
* A faulty security configuration may temporarily sever whole application from
the brokers.
* Any number of causes could likewise sever a single instance in a single
application from all brokers.
* Finally, networking problems can disconnect arbitrary pairs of Streams
instances and Broker instances.

This is not an accounting of all possible failure modes, obviously, but the
point is that, in a large, decentralized organization, you can experience
lots of transient failures that have some features in common:
F1. It's someone else's fault, and someone else must take action to fix it.
F2. It will take "human time" to fix it. I.e., hours, not milliseconds.
F3. A single failure can affect "everyone" (e.g., one broker with long GCs
can cause timeouts in all thousands of instances over all dozens of teams).

As an operator belonging to one team, whether we bound retries or not,
I would need to be alerted when the app stops making progress, I'd need
to investigate, and in the above cases, I'd need to escalate to the network
and/or broker infrastructure teams.

Clearly, I can be alerted either by threads dying or by non-progress metrics.
As a responsible operator, I'd have alerts on _both_, so we shouldn't consider
either signal to be "louder" or more reliable than the other.

A side observation: in a lot of the failure modes, a specific task won't be able
to make progress no matter which thread or instance it's on (i.e., if the
transaction coordinator for one of its output partitions is slow or 
unresponsive).
Therefore, killing the thread with a bounded retry config would only result
in a cascade of thread deaths across all my instances until either I run out of
threads or the incident is resolved.

The key questions to me are:
Q1. Do we want to continue trying to make what progress we can while
the incident is being investigated and remediated?
Q2. Should I (the operator for a single team) have to take any action once
the infrastructure failures are resolved?

We can paraphrase these as, "do you want your business to grind to a halt
due to a single failure?", and "do you want everyone to stay up all night
waiting for a fix so they can all restart their applications?"

Just from the operator/business perspective, it seems like we want:
Q1:yes and Q2:no, which in combination with F1,2,3 above indicates
to me that it would be better for Streams to just keep retrying indefinitely.

There is one point I think you've mentioned to me in the past that it
may not be _safe_ to just quit working on one specific task while
progressing on others. If we have a repartition topic sourced by
two tasks T1 and T2, and feeding a windowed aggregation (for example),
then failing to process T1 while continuing on T2 for a long time
would cause a lot of timestamp skew, and could ultimately result in all
those delayed records in T1 being out of grace period by the time they
get processed. Arguably, this is a completely normal and expected
situation in a distributed system, which is why we have grace period to
begin with, but since the cause of this particular skew is inside of
Streams, it would be possible and nice to detect and avoid the situation.

However, we should note that killing a single thread that hosts T1 will
_not_ deterministically halt processing on T2, nor will stopping the
single instance that hosts T1, since T2 might be on another instance.
We would need a clus

[DISCUSS] KAFKA-4680: min.insync.replica can be set > replication factor

[Paolo Moriello]

Hello,

I'd like to take up this Jira ticket
. This is an old ticket,
marked as a Kafka bug.

Before moving forward, I'd like to open a discussion on what would be the
best approach to take on when doing the validation, as well as discuss
about possible edge cases we should consider.

Basically, at the moment, it is possible to specify min.insync.replicas >
replication factor. When this happens, it is not possible to produce on a
topic when acks=all as client callback returns NOT_ENOUGH_REPLICAS, and the
broker logs error messages on each request. As suggested in the Jira, the
validation should happen much earlier in the process, eg. at topic
creation/configuration setup.

Regarding the approach to use on validating the configuration; do we want,
for instance, to:
1. print a WARN about the mismatch in the configuration
2. make the request FAIL
3. or print a more specific message on produce

Options 1 and 2 anticipate the validation on topic creation / configuration
change. These require to validate the configuration in more than one place:
at topic creation, at configuration setup/update (both for
min.insync.replicas and the default.replication.factor), at partition
reassignment (when reducing replication factor). Don't know about
consequences
Option 3 is simpler; it does not anticipate the validation, but at least
improves the visibility over the issue on the client side.

I'd be in favor of a softer approach, which might include both printing a
warning on topic creation/configuration-update and eventually a more
specific message when producing on the topic. On the other end, this does
not solve the problem, as we would allow anyway the mismatch in the
configuration. Option 2 would solve the problem with an harder validation
(eg blocking topic creation or configuration setup), but this requires to
validate any possible edge case (eg. how do we prevent a change in
min.insync.replicas if we have already created topic with lower replication
factor?).

Let me know what's your opinion on this, and if there is any other scenario
we should consider for the validation (for instance, what's the impact on
internal topics?).

Thanks,
Paolo

Re: [DISCUSS] KIP-569: DescribeConfigsResponse - Update the schema to include datatype of the field

[Shailesh Panwar]

Thanks Colin for the inputs. Regarding

*I think most users of the DescribeConfigs API do not want to get help text
or configuration schema information.  *
There is already a precedence of including this information as part of
Connect Config - both type and documentation - and we have found it quite
useful in the Config client forms.

*It would be better to create a new, separate API for getting this
configuration schema information, if that's what we really want.  This API
should probably also allow configuration management systems to list all the
possible configurations for topics, brokers, etc., which is something that
I think many of them would want.> *
Correct. I believe AdminClient.describeConfig already provides us that
today. From the docs - "Get the configuration for the specified resources
with the default options."
This api gives us back all the configuration information for the specified
resources(s) - broker or  topic.

*We also need to consider compatibility.  One example is, what if we later
add a new type of configuration key, such as UUID.  What would the
hypothetical DescribeConfigurationSchema API return in this case, for older
clients?  We probably need an UNKNOWN enum value to be used to indicate
that the server knows about configuration key types that the client does
not.> *
I believe we already handle backward compatibility via versioning on the
message schema. For example 'Synonyms' is only available from 1+ version
and treated nullable for previous ones. The expectation with the new fields
is similar.

Thanks
Shailesh

On 2020/02/26 22:17:39, "Colin McCabe"  wrote:
> Hi Shailesh,>
>
> I think most users of the DescribeConfigs API do not want to get help
text or configuration schema information.  So, it would be inefficient to
always include this information as part of the DescribeConfigs response.
It would be better to create a new, separate API for getting this
configuration schema information, if that's what we really want.  This API
should probably also allow configuration management systems to list all the
possible configurations for topics, brokers, etc., which is something that
I think many of them would want.>
>
> We also need to consider compatibility.  One example is, what if we later
add a new type of configuration key, such as UUID.  What would the
hypothetical DescribeConfigurationSchema API return in this case, for older
clients?  We probably need an UNKNOWN enum value to be used to indicate
that the server knows about configuration key types that the client does
not.>
>
> best,>
> Colin>
>
>
> On Wed, Feb 19, 2020, at 09:13, Shailesh Panwar wrote:>
> > Bump.>
> > >
> > Thanks>
> > Shailesh>
> > >
> > On Tue, Feb 11, 2020 at 1:00 PM Shailesh Panwar >
> > wrote:>
> > >
> > > Hi all,>
> > > We would like to extend the DescribeConfigsResponse schema to include
the>
> > > data-type of the fields.>
> > >>
> > > The KIP can be found here:>
> > >>
> > >
https://cwiki.apache.org/confluence/display/KAFKA/KIP-569%3A+DescribeConfigsResponse+-+Update+the+schema+to+include+datatype+of+the+field>

> > >>
> > > Thanks>
> > > Shailesh>
> > >>
> >>
>

Re: [KAFKA-557] Add emit on change support for Kafka Streams

[Richard Yu]

Hi Bruno, Hi John,

Thanks for your comments! I updated the KIP accordingly, and it looks like
for quite a few points. I was doing some beating around the bush which
could've been avoided.

Looks like we can reduce the metric to Level 1 (per processor node) then.

I've cleaned up most of the unnecessary info, and we should be fairly close.
I will start working on a PR soon for this KIP. (although we might split
that up into stages)

Cheers,
Richard

On Thu, Feb 27, 2020 at 6:06 AM Bruno Cadonna  wrote:

> Hi John,
>
> I agree with you. It is better to measure the metric on processor node
> level. The users can do the rollup to task-level by themselves.
>
> Best,
> Bruno
>
> On Thu, Feb 27, 2020 at 12:09 AM John Roesler  wrote:
> >
> > Hi Richard,
> >
> > I've been making a final pass over the KIP.
> >
> > Re: Proposed Behavior Change:
> >
> > I think this point is controversial and probably doesn't need to be
> there at all:
> > > 2.b. In certain situations where there is a high volume of idempotent
> > > updates throughout the Streams DAG, it will be recommended practice
> > > to materialize all operations to reduce traffic overall across the
> entire
> > >  network of nodes.
> >
> > Re-reading all the points, it seems like we can sum them up in a way
> that's
> > a little more straight to the point, and gives us the right amount of
> flexibility:
> >
> > > Proposed Behavior Changes
> > >
> > > Definition: "idempotent update" is one in which the new result and
> prior
> > > result,  when serialized, are identical byte arrays
> > >
> > > Note: an "update" is a concept that only applies to Table operations,
> so
> > > the concept of an "idempotent update" also only applies to Table
> operations.
> > > See
> https://kafka.apache.org/documentation/streams/developer-guide/dsl-api.html#streams_concepts_ktable
> > > for more information.
> > >
> > > Given that definition, we propose for Streams to drop idempotent
> updates
> > > in any situation where it's possible and convenient to do so. For
> example,
> > > any time we already have both the prior and new results serialized, we
> > > may compare them, and drop the update if it is idempotent.
> > >
> > > Note that under this proposal, we can implement idempotence checking
> > > in the following situations:
> > > 1. Any aggregation (for example, KGroupedStream, KGroupedTable,
> > > TimeWindowedKStream, and SessionWindowedKStream operations)
> > > 2. Any Materialized KTable operation
> > > 3. Repartition operations, when we need to send both prior and new
> results
> >
> > Notice that in my proposed wording, we neither limit ourselves to just
> the
> > situations enumerated, nor promise to implement the optimization in every
> > possible situation. IMHO, this is the best way to propose such a feature.
> > That way, we have the flexibility to implement it in stages, and also to
> add
> > on to the implementation in the future.
> >
> >
> > Re: Metrics
> >
> > I agree with Bruno, although, I think it might just be a confusing
> statement.
> > It might be clearer to drop all the "discussion", and just say: "We will
> add a
> > metric to count the number of idempotent updates that we have dropped".
> >
> > Also, with respect to the metric, I'm wondering if the metric should be
> task-
> > level or processor-node-level. Since the interesting action takes place
> inside
> > individual processor nodes, I _think_ it would be higher leverage to just
> > measure it at the node level. WDYT?
> >
> > Re: Design Reasoning
> >
> > This section seems to be a little bit outdated. I also just noticed a
> "surprise"
> > configuration "timestamp.aggregation.selection.policy" hidden in point
> 1.a.
> > Is that part of the proposal? We haven't discussed it, and I think we
> were
> > talking about this KIP being "configuration free".
> >
> > There is also some discussion of discarded alternative in the Design
> Reasoning
> > section, which is confusing. Finally, there was a point there I didn't
> understand
> > at all, about stateless operators not being intended to load prior
> results.
> > This statement doesn't seem to be true, but it also doesn't seem to be
> relevant,
> > so maybe we can just drop it.
> >
> > Overall, it might help if you make a pass on this section, and just
> discuss as
> > briefly as possible the justification for the proposed behavior change,
> and
> > not adding a configuration. Try to avoid talking about things that we
> are not
> > proposing, since that will just lead to confusion.
> >
> > Similarly, I'd just completely remove the "Implementation [discarded]"
> section.
> > It was good to have this as part of the discussion initially, but as we
> move
> > toward a vote, it's better to just streamline the KIP document as much as
> > possible. Keeping a "discarded" section in the document will just make it
> > harder for new people to understand the proposal. We did the same thing
> > with KIP-441, where there were two prior drafts included at the end of
>

Re: [KAFKA-557] Add emit on change support for Kafka Streams

[Richard Yu]

Hi all,

I might've made a minor mistake. The processor node level is level 3, not
level 1.
I will correct the KIP accordingly.

After looking over things, I decided to start the voting thread this
afternoon.

Cheers,
Richard

On Thu, Feb 27, 2020 at 12:29 PM Richard Yu 
wrote:

> Hi Bruno, Hi John,
>
> Thanks for your comments! I updated the KIP accordingly, and it looks like
> for quite a few points. I was doing some beating around the bush which
> could've been avoided.
>
> Looks like we can reduce the metric to Level 1 (per processor node) then.
>
> I've cleaned up most of the unnecessary info, and we should be fairly
> close.
> I will start working on a PR soon for this KIP. (although we might split
> that up into stages)
>
> Cheers,
> Richard
>
> On Thu, Feb 27, 2020 at 6:06 AM Bruno Cadonna  wrote:
>
>> Hi John,
>>
>> I agree with you. It is better to measure the metric on processor node
>> level. The users can do the rollup to task-level by themselves.
>>
>> Best,
>> Bruno
>>
>> On Thu, Feb 27, 2020 at 12:09 AM John Roesler 
>> wrote:
>> >
>> > Hi Richard,
>> >
>> > I've been making a final pass over the KIP.
>> >
>> > Re: Proposed Behavior Change:
>> >
>> > I think this point is controversial and probably doesn't need to be
>> there at all:
>> > > 2.b. In certain situations where there is a high volume of idempotent
>> > > updates throughout the Streams DAG, it will be recommended practice
>> > > to materialize all operations to reduce traffic overall across the
>> entire
>> > >  network of nodes.
>> >
>> > Re-reading all the points, it seems like we can sum them up in a way
>> that's
>> > a little more straight to the point, and gives us the right amount of
>> flexibility:
>> >
>> > > Proposed Behavior Changes
>> > >
>> > > Definition: "idempotent update" is one in which the new result and
>> prior
>> > > result,  when serialized, are identical byte arrays
>> > >
>> > > Note: an "update" is a concept that only applies to Table operations,
>> so
>> > > the concept of an "idempotent update" also only applies to Table
>> operations.
>> > > See
>> https://kafka.apache.org/documentation/streams/developer-guide/dsl-api.html#streams_concepts_ktable
>> > > for more information.
>> > >
>> > > Given that definition, we propose for Streams to drop idempotent
>> updates
>> > > in any situation where it's possible and convenient to do so. For
>> example,
>> > > any time we already have both the prior and new results serialized, we
>> > > may compare them, and drop the update if it is idempotent.
>> > >
>> > > Note that under this proposal, we can implement idempotence checking
>> > > in the following situations:
>> > > 1. Any aggregation (for example, KGroupedStream, KGroupedTable,
>> > > TimeWindowedKStream, and SessionWindowedKStream operations)
>> > > 2. Any Materialized KTable operation
>> > > 3. Repartition operations, when we need to send both prior and new
>> results
>> >
>> > Notice that in my proposed wording, we neither limit ourselves to just
>> the
>> > situations enumerated, nor promise to implement the optimization in
>> every
>> > possible situation. IMHO, this is the best way to propose such a
>> feature.
>> > That way, we have the flexibility to implement it in stages, and also
>> to add
>> > on to the implementation in the future.
>> >
>> >
>> > Re: Metrics
>> >
>> > I agree with Bruno, although, I think it might just be a confusing
>> statement.
>> > It might be clearer to drop all the "discussion", and just say: "We
>> will add a
>> > metric to count the number of idempotent updates that we have dropped".
>> >
>> > Also, with respect to the metric, I'm wondering if the metric should be
>> task-
>> > level or processor-node-level. Since the interesting action takes place
>> inside
>> > individual processor nodes, I _think_ it would be higher leverage to
>> just
>> > measure it at the node level. WDYT?
>> >
>> > Re: Design Reasoning
>> >
>> > This section seems to be a little bit outdated. I also just noticed a
>> "surprise"
>> > configuration "timestamp.aggregation.selection.policy" hidden in point
>> 1.a.
>> > Is that part of the proposal? We haven't discussed it, and I think we
>> were
>> > talking about this KIP being "configuration free".
>> >
>> > There is also some discussion of discarded alternative in the Design
>> Reasoning
>> > section, which is confusing. Finally, there was a point there I didn't
>> understand
>> > at all, about stateless operators not being intended to load prior
>> results.
>> > This statement doesn't seem to be true, but it also doesn't seem to be
>> relevant,
>> > so maybe we can just drop it.
>> >
>> > Overall, it might help if you make a pass on this section, and just
>> discuss as
>> > briefly as possible the justification for the proposed behavior change,
>> and
>> > not adding a configuration. Try to avoid talking about things that we
>> are not
>> > proposing, since that will just lead to confusion.
>> >
>> > Similarly, I'd just completely remove th

[VOTE] KIP-557: Add emit on change support for Kafka Streams

[Richard Yu]

Hi all,

I am proposing a new optimization to Kafka Streams which would greatly
reduce the number of idempotent updates (or no-ops) in the Kafka Streams
DAG.
A number of users have been interested in this feature, so it would be nice
to pass this one in.

For information, the KIP is described below:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-557%3A+Add+emit+on+change+support+for+Kafka+Streams

We aim to make Kafka Streams more efficient by adopting the "emit on
change" reporting strategy.

Please cast your vote!

Best,
Richard

[jira] [Created] (KAFKA-9620) Task revocation failure could introduce remaining unclean tasks

[Boyang Chen (Jira)]

Boyang Chen created KAFKA-9620:
--

 Summary: Task revocation failure could introduce remaining unclean 
tasks
 Key: KAFKA-9620
 URL: https://issues.apache.org/jira/browse/KAFKA-9620
 Project: Kafka
  Issue Type: Bug
Reporter: Boyang Chen
Assignee: Boyang Chen


The task revocation call should enforce the close of a task, otherwise we could 
potentially hit the exception during `handleAssignment`.

During revoke we failed:

 
{code:java}
[2020-02-27T11:05:48-08:00] 
(streams-soak-trunk-eos_soak_i-099dc04bf946ce2f0_streamslog) [2020-02-27 
19:05:47,321] ERROR 
[stream-soak-test-d1c291a8-ee54-4058-ac9c-7cd46d5484de-StreamThread-1] 
[Consumer 
clientId=stream-soak-test-d1c291a8-ee54-4058-ac9c-7cd46d5484de-StreamThread-1-consumer,
 groupId=stream-soak-test] User provided listener 
org.apache.kafka.streams.processor.internals.StreamsRebalanceListener failed on 
invocation of onPartitionsRevoked for partitions [logs.json.kafka-2, 
logs.json.zookeeper-2, node-name-repartition-1, logs.kubernetes-2, 
windowed-node-counts-1, logs.operator-2, logs.syslog-2] 
(org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
[2020-02-27T11:05:48-08:00] 
(streams-soak-trunk-eos_soak_i-099dc04bf946ce2f0_streamslog) 
org.apache.kafka.streams.errors.TaskMigratedException: Producer get fenced 
trying to commit a transaction; it means all tasks belonging to this thread 
should be migrated.
        at 
org.apache.kafka.streams.processor.internals.StreamsProducer.commitTransaction(StreamsProducer.java:172)
        at 
org.apache.kafka.streams.processor.internals.RecordCollectorImpl.commit(RecordCollectorImpl.java:226)
        at 
org.apache.kafka.streams.processor.internals.StreamTask.commitState(StreamTask.java:368)
        at 
org.apache.kafka.streams.processor.internals.StreamTask.suspend(StreamTask.java:242)
        at 
org.apache.kafka.streams.processor.internals.TaskManager.handleRevocation(TaskManager.java:314)
        at 
org.apache.kafka.streams.processor.internals.StreamsRebalanceListener.onPartitionsRevoked(StreamsRebalanceListener.java:72)
        at 
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.invokePartitionsRevoked(ConsumerCoordinator.java:297)
        at 
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:383)
        at 
org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:439)
        at 
org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:358)
        at 
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:477)
        at 
org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1277)
        at 
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1243)
        at 
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1218)
        at 
org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:920)
        at 
org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:800)
        at 
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:749)
        at 
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:725)
[2020-02-27T11:05:48-08:00] 
(streams-soak-trunk-eos_soak_i-099dc04bf946ce2f0_streamslog) Caused by: 
org.apache.kafka.common.errors.ProducerFencedException: Producer attempted an 
operation with an old epoch. Either there is a newer producer with the same 
transactionalId, or the producer's transaction has been expired by the broker.
{code}
During assignment we are checking the cleanness of task close and throw fatal:
{code:java}
[2020-02-27T11:05:48-08:00] 
(streams-soak-trunk-eos_soak_i-099dc04bf946ce2f0_streamslog) [2020-02-27 
19:05:48,032] ERROR 
[stream-soak-test-d1c291a8-ee54-4058-ac9c-7cd46d5484de-StreamThread-1] 
stream-thread 
[stream-soak-test-d1c291a8-ee54-4058-ac9c-7cd46d5484de-StreamThread-1] 
Encountered the following exception during processing and the thread is going 
to shut down:  (org.apache.kafka.streams.processor.internals.StreamThread) 
[2020-02-27T11:05:48-08:00] 
(streams-soak-trunk-eos_soak_i-099dc04bf946ce2f0_streamslog) 
java.lang.RuntimeException: Unexpected failure to close 1 task(s) [[0_2]]. 
First exception (for task 0_2) follows.         at 
org.apache.kafka.streams.processor.internals.TaskManager.handleAssignment(TaskManager.java:205)
         at 
org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor.onAssignment(StreamsPartitionAssignor.java:1176)
         at 
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:397)
         at 
org.apache.kafka.clients.consumer.internals.

Jenkins build is back to normal : kafka-trunk-jdk8 #4279

[Apache Jenkins Server]

See 

Jenkins build is back to normal : kafka-trunk-jdk11 #1198

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-572: Improve timeouts and retires in Kafka Streams

[Guozhang Wang]

Hello John,

I'll make note that you owe me a beer now :)

I think I'm leaning towards your approach as well based on my observations
on previously reported timeout exceptions in the past. I once left some
thoughts on Matthias' PR here
https://github.com/apache/kafka/pull/8122#pullrequestreview-360749510 and I
think I can better summarize my thoughts in this thread:

1) First of all, we need to think from user's perspective, what they'd
really want to be notified:

a. "If my application cannot make progress temporarily due to various
transient issues (John had listed several examples already), just handle
that internally and I do not wanna be notified and worry about how to tune
my timeout configs at all".
b. "If my application cannot make progress for a long time, which is likely
due to a bad config, a human error, network issues, etc such that I should
be involved in the loop of trouble shooting, let me know sooner than later".

and what they'd not preferred but may happen today:

c. "one transient error cause a thread to die, but then after tasks
migrated everything goes to normal; so the application silently lost a
thread without letting me know"; in fact, in such cases even a cascading
exception that eventually kills all thread may be better since at least the
users would be notified.

Based on that, instead of retrying immediately at the granularity each
blocking call, it should be sufficient to only consider the handling logic
at the thread level. That is, within an iteration of the thread, it would
try to:

* initialized some created tasks;
* restored some restoring tasks;
* processed some running tasks who have buffered records that are
processable;
* committed some tasks.

In each of these steps, we may need to make some blocking calls in the
underlying embedded clients, and if either of them timed out, we would not
be able to make progress partially or not being able to make any progress
at all. If we still want to set a configured value of "retries", I think a
better idea would be to say "if we cannot make progress for consecutive N
iterations of a thread, the user should be notified".

---

2) Second, let's consider what's a good way to notify the user. Today our
way of notification is simple: throw the exception all the way up to user's
uncaught-exception-handler (if it's registered) and let the thread die. I'm
wondering if we could instead educate users to watch on some key metrics
for "progress indicate" than relying on the exception handler though. Some
candidates in mind:

* consumer-lag: this is for both source topics and for repartition topics,
it indicates if one or more of the tasks within each sub-topology is
lagging or not; in the case where *some or all* of the threads cannot make
progress. E.g. if a downstream task's thread is blocked somehow while its
upstream task's thread is not, then the consumer-lag on the repartition
topic would keep growing.

* *idle* state: this is an idea we discussed in
https://issues.apache.org/jira/browse/KAFKA-6520, i.e. to introduce an
instance-level new state, if all threads of the instance cannot make
progress (primarily for the reason that it cannot talk to the brokers).

* process-rate: this is at thread-level. However if some tasks cannot make
progress while others can still make progress within a thread, its
process-rate would now drop to zero and it's a bit hard to indicate
compared with comsumer-lag.

If we feel that relying on metrics is better than throwing the exception
and let the thread die, then we would not need to have the "retry" config
as well.

---

3) This maybe semi-related to the timeout itself, but as I mentioned today
one common issue we would need to resolve is to lose a thread BUT not
losing the whole instance. In other words, we should consider when we have
to throw an exception from a thread (not due to timeouts, but say due to
some fatal error), should we just kill the corresponding thread or should
we be more brutal and just kill the whole instance instead. I'm happy to
defer this to another discussion thread but just bring this up here.



Guozhang


On Thu, Feb 27, 2020 at 10:40 AM John Roesler  wrote:

> Hi Matthias,
>
> Thanks for the proposal! I think this will be a wonderful improvement
> to Streams. In particular, thanks for the motivation. It would indeed
> be nice not to have to set long timeout configs and block individual
> client requests in order to cope with transient slow responses.
>
> I'm very well aware that this might make me sound like a crazy person,
> but one alternative I'd like to consider is not bounding the retries at
> all.
> Instead, Streams would just skip over timed-out tasks and try again
> on the next iteration, as you proposed, but would continue to do so
> indefinitely. Clearly, we shouldn't do such a thing silently, so I'd
> further
> propose to log a warning every time a task times out and also maintain
> a new metric indicating task timeouts.
>
> To see why thi

[jira] [Resolved] (KAFKA-9091) KIP-538: Add a metric tracking the number of open connections with a given SSL cipher type

[David Arthur (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-9091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Arthur resolved KAFKA-9091.
-
Resolution: Fixed

> KIP-538: Add a metric tracking the number of open connections with a given 
> SSL cipher type
> --
>
> Key: KAFKA-9091
> URL: https://issues.apache.org/jira/browse/KAFKA-9091
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 2.5.0
>Reporter: Colin McCabe
>Assignee: Colin McCabe
>Priority: Major
> Fix For: 2.5.0
>
>
> KIP-538: Add a metric tracking the number of open connections with a given 
> SSL cipher type



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (KAFKA-7932) Streams needs to handle new Producer exceptions

[John Roesler (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-7932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Roesler resolved KAFKA-7932.
-
Resolution: Duplicate

Subsumed by https://issues.apache.org/jira/browse/KAFKA-9274

> Streams needs to handle new Producer exceptions
> ---
>
> Key: KAFKA-7932
> URL: https://issues.apache.org/jira/browse/KAFKA-7932
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: John Roesler
>Assignee: John Roesler
>Priority: Critical
> Fix For: 2.5.0
>
>
> Following on KAFKA-7763, Streams needs to handle the new behavior.
> See also [https://github.com/apache/kafka/pull/6066]
> Streams code (StreamTask.java) needs to be modified to handle the new 
> exception.
> From the upstream change, `commtit/abort Transaction` can also throw 
> TimeoutException now: default `MAX_BLOCK_MS_CONFIG` in producer is 60 
> seconds, so I think just wrapping it as StreamsException should be 
> reasonable, similar to what we do for `producer#send`'s TimeoutException 
> ([https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/processor/internals/RecordCollectorImpl.java#L220-L225]
>  ).
>  
> See also [https://github.com/apache/kafka/pull/6066#issuecomment-464403448]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-9621) AdminClient listOffsets operation does not respect retries and backoff

[Sanjana Kaundinya (Jira)]

Sanjana Kaundinya created KAFKA-9621:


 Summary: AdminClient listOffsets operation does not respect 
retries and backoff
 Key: KAFKA-9621
 URL: https://issues.apache.org/jira/browse/KAFKA-9621
 Project: Kafka
  Issue Type: Bug
  Components: admin
Reporter: Sanjana Kaundinya
Assignee: Cheng Tan
 Fix For: 2.6.0


Similar to https://issues.apache.org/jira/browse/KAFKA-9047, currently the 
listOffsets operation doesn't respect the configured retries and backoff for a 
given call. 

For example, the code path could go like so:
1) Make a metadata request and schedule subsequent list offsets calls
2) Metadata error comes back with `InvalidMetadataException`

3) Go back to 1

The problem here is that the state is not preserved across calls. We loose the 
information regarding how many tries the call has been tried and how far out we 
should schedule the call to try again. This could lead to a tight retry loop 
and put pressure on the brokers.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-9622) Improve tests to detect post-processing failures in Jetty request handling.

[Rigel Bezerra de Melo (Jira)]

Rigel Bezerra de Melo created KAFKA-9622:


 Summary: Improve tests to detect post-processing failures in Jetty 
request handling.
 Key: KAFKA-9622
 URL: https://issues.apache.org/jira/browse/KAFKA-9622
 Project: Kafka
  Issue Type: Test
  Components: streams
Reporter: Rigel Bezerra de Melo


There was a recent jetty-server version bump to 
[9.4.26.v20200117|https://mvnrepository.com/artifact/org.eclipse.jetty/jetty-server/9.4.26.v20200117],
 that caused errors in request post-processing. Jetty version 
[9.4.25.v20191220|https://mvnrepository.com/artifact/org.eclipse.jetty/jetty-server/9.4.25.v20191220]
 deleted a method which was still required by the Jersey version in use. 

That particular error did not surface in any tests because it only happens on 
request post-processing, i.e. after the response has already been sent to the 
client. From the test's point of view, the request behaved as expected. 
Internally on Jetty though, post-processing crashes and is aborted. That would 
include server bookkeeping, like freeing resources, etc. 

It would be ideal to have a way to verify that request handling completed 
successfully in the tests, after the response is validated.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] KIP-574: CLI Dynamic Configuration with file input

[Aneel Nazareth]

I've created a PR for a potential implementation of this:
https://github.com/apache/kafka/pull/8184 if we decide to go ahead with
this KIP.

On Wed, Feb 26, 2020 at 12:36 PM Aneel Nazareth  wrote:

> Hi,
>
> I'd like to discuss adding a new argument to kafka-configs.sh
> (ConfigCommand.scala).
>
> Recently I've been working on some things that require complex
> configurations. I've chosen to represent them as JSON strings in my
> server.properties. This works well, and I'm able to update the
> configurations by editing server.properties and restarting the broker. I've
> added the ability to dynamically configure them, and that works well using
> the AdminClient. However, when I try to update these configurations using
> kafka-configs.sh, I run into a problem. My configurations contain commas,
> and kafka-configs.sh tries to break them up into key/value pairs at the
> comma boundary.
>
> I'd like to enable setting these configurations from the command line, so
> I'm proposing that we add a new option to kafka-configs.sh that takes a
> properties file.
>
> I've created a KIP for this idea:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-574%3A+CLI+Dynamic+Configuration+with+file+input
> And a JIRA: https://issues.apache.org/jira/browse/KAFKA-9612
>
> I'd appreciate your feedback on the proposal.
>
> Thanks,
> Aneel
>

Re: [KAFKA-557] Add emit on change support for Kafka Streams

[John Roesler]

Hi Richard,

Thanks for the update!

I read it over, and overall it looks good!

I have only a minor concern about the rate metric definition:
> The rate option indicates the ratio of records dropped to actual volume of 
> records passing through the task
That's not the definition of a "rate". It should be something more like
"the average number of dropped idempotent updates per second".

Incidentally, I mentioned this KIP to Guozhang, and he brought up an
interesting concern I'd like to share with the list. He noted that if we filter
out idempotent table updates, stream time will not advance with every
input event anymore. He asked if this would have a negative impact on
operations that depend on stream time.

I think this is a valid concern. For example, you can use Suppress to
buffer KTable updates until a certain amount of stream time passes. 
Specifically,
the Suppress processor itself advances stream time as it receives new records
to its `process` method. In the pathological case, all updates are idempotent,
get dropped, and the Suppress operator never emits anything, even though to
an outside observer, stream time should have advanced.

Example scenario:
> inputTable = builder.table(input)
> suppressed = inputTable.suppress(untilTimeLimit(10))
> 
> input: key=A, timestamp=0, value=X
> inputTable: key=A, timestamp=0, value=X
> suppressed buffer: key=A, timestamp=0, value=X (observed stream time = 0)
> output: (nothing)
> 
> input: key=A, timestamp=11, value=X
> // update is idempotent, so it gets dropped
> inputTable: key=A, timestamp=0, value=X
> suppressed buffer: key=A, timestamp=0, value=X (observed stream time = 0)
> output: (nothing)

Note that even though stream time has technically advanced beyond the
suppression config of 10, the suppression buffer doesn't see it because the
KTable source already dropped the idempotent update.

I'm thinking that this situation is indeed concerning, and we should be aware
and make a note of it. However, I don't think that it should change our 
proposal.
To understand why, I'll enumerate all the usages of stream time in Kafka 
Streams:

windowed operations
-
KStreamSessionWindowAggregate & KStreamWindowAggregate:
  - used to determine if an incoming record belongs to a window that is already 
closed
Suppressed.untilWindowCloses():
  - used to flush out results for windows, once they are closed
AbstractRocksDBSegmentedBytesStore & InMemorySessionStore & InMemoryWindowStore:
  - used to create new segments and drop old ones that are out of retention

non-windowed operations
---
Suppressed.untilTimeLimit
  - used to flush out prior updates that have passed the configured age, in 
stream time

Note that most of the usages are in windowed operations. One interesting thing
about this context is that records with higher timestamps belong to different 
windows.
In order to advance stream time far enough to close a window or push it out of 
retention, the new records must have timestamps that in later windows, which 
means
that they are updates to _new_ keys, which means they would not be suppressed as
idempotent. 

Updates within a window could still be suppressed, though:
For example, if the window size is 10, and the grace period is 5, and we get 
updates all
for the same key with timestamps 0, 11, and 16, today, we would emit the record 
for
the [0,10) window as soon as we got the update at time 16 (since stream time is 
now
past the window end + grace period time of 15). But if we drop the time=16 
update, we
wouldn't emit the [0,10) window results until we get a record with time >= 20. 

You can see that the maximum amount of (stream) time that dropping idempotent
updates could delay updates is one window. This might be surprising, but it 
doesn't
seem too bad, especially since Suppress explicitly does _not_ promise to emit 
the
results at the earliest possible time, just at some time after the window 
closes.

Even better, though, all the windowed aggregations are _stream_ aggregations
that _produce_ a KTable, and we already decided that (at least for now), we 
would
include the timestamp in the idempotence check for stream aggregation results.
With this strategy, we would actually not suppress _any_ update to a stream 
aggregation (windowed or otherwise) that advances stream time.

So there's no problem at all with windowed operations.

This only leaves non-windowed operations, of which there's only one. I have to 
admit
that I regret Suppressed.untilTimeLimit. It turns out that everyone I've heard 
of who
used this API actually wanted it to be wall-clock based, not stream-time based. 
So,
I'm not sure in practice if this sharp edge will actually cut anyone.

I think a "proper" fix would be to introduce some kind of control message to 
advance
stream time independently of records. We've already talked about this a little 
in
KAFKA-8769, and it would also be necessary for global consistency markers. 
B

Build failed in Jenkins: kafka-2.5-jdk8 #47

[Apache Jenkins Server]

See 


Changes:

[mumrah] Update year in NOTICE

[wangguoz] MINOR: Remove tag from metric to measure process-rate on source nodes


--
[...truncated 5.86 MB...]
org.apache.kafka.streams.MockProcessorContextTest > shouldCapturePunctuator 
PASSED

org.apache.kafka.streams.TestTopicsTest > testNonUsedOutputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonUsedOutputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testEmptyTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testEmptyTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testStartTimestamp STARTED

org.apache.kafka.streams.TestTopicsTest > testStartTimestamp PASSED

org.apache.kafka.streams.TestTopicsTest > testNegativeAdvance STARTED

org.apache.kafka.streams.TestTopicsTest > testNegativeAdvance PASSED

org.apache.kafka.streams.TestTopicsTest > shouldNotAllowToCreateWithNullDriver 
STARTED

org.apache.kafka.streams.TestTopicsTest > shouldNotAllowToCreateWithNullDriver 
PASSED

org.apache.kafka.streams.TestTopicsTest > testDuration STARTED

org.apache.kafka.streams.TestTopicsTest > testDuration PASSED

org.apache.kafka.streams.TestTopicsTest > testOutputToString STARTED

org.apache.kafka.streams.TestTopicsTest > testOutputToString PASSED

org.apache.kafka.streams.TestTopicsTest > testValue STARTED

org.apache.kafka.streams.TestTopicsTest > testValue PASSED

org.apache.kafka.streams.TestTopicsTest > testTimestampAutoAdvance STARTED

org.apache.kafka.streams.TestTopicsTest > testTimestampAutoAdvance PASSED

org.apache.kafka.streams.TestTopicsTest > testOutputWrongSerde STARTED

org.apache.kafka.streams.TestTopicsTest > testOutputWrongSerde PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputTopicWithNullTopicName STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputTopicWithNullTopicName PASSED

org.apache.kafka.streams.TestTopicsTest > testWrongSerde STARTED

org.apache.kafka.streams.TestTopicsTest > testWrongSerde PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMapWithNull STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMapWithNull PASSED

org.apache.kafka.streams.TestTopicsTest > testNonExistingOutputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonExistingOutputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testMultipleTopics STARTED

org.apache.kafka.streams.TestTopicsTest > testMultipleTopics PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValueList STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValueList PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputWithNullDriver STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputWithNullDriver PASSED

org.apache.kafka.streams.TestTopicsTest > testValueList STARTED

org.apache.kafka.streams.TestTopicsTest > testValueList PASSED

org.apache.kafka.streams.TestTopicsTest > testRecordList STARTED

org.apache.kafka.streams.TestTopicsTest > testRecordList PASSED

org.apache.kafka.streams.TestTopicsTest > testNonExistingInputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonExistingInputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMap STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMap PASSED

org.apache.kafka.streams.TestTopicsTest > testRecordsToList STARTED

org.apache.kafka.streams.TestTopicsTest > testRecordsToList PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValueListDuration STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValueListDuration PASSED

org.apache.kafka.streams.TestTopicsTest > testInputToString STARTED

org.apache.kafka.streams.TestTopicsTest > testInputToString PASSED

org.apache.kafka.streams.TestTopicsTest > testTimestamp STARTED

org.apache.kafka.streams.TestTopicsTest > testTimestamp PASSED

org.apache.kafka.streams.TestTopicsTest > testWithHeaders STARTED

org.apache.kafka.streams.TestTopicsTest > testWithHeaders PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValue STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValue PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateTopicWithNullTopicName STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateTopicWithNullTopicName PASSED

> Task :streams:upgrade-system-tests-0100:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0100:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:classes UP-TO-DATE
> Task :streams:upgrade-system-tests-0100:checkstyleMain NO-SOURCE
> Task :streams:upgrade-system-tests-0100:compileTestJava
> Task :streams:upgrade-system-tests-0100:processTestResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:testClasses
> Task :streams:upgrade-system-tests-0100:checkstyleTest
> Task :streams:upgrade-system-tests-0100:spotb

Build failed in Jenkins: kafka-trunk-jdk8 #4280

[Apache Jenkins Server]

See 


Changes:

[github] Adapt docs about metrics of Streams according to KIP-444 (#8171)


--
[...truncated 5.82 MB...]

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForSmallerValue[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = false] PASSED

org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis STARTED

org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis PASSED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime STARTED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime PASSED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep PASSED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWindowStartTimestampWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWindowStartTimestampWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldReturnIsPersistent STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldReturnIsPersistent PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardClose 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardClose 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
PASSED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord PASSED

org.apache.kafka.streams.test.TestRecordTest > testToString STARTED

org.apache.kafka.streams.test.TestRecordTest > testToString PASSED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords STARTED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords PASSED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
STARTED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
PASSED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher STARTED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher PASSED

org.apache.kafka.streams.test.TestRecordTest > testFields STARTED

org.apache.kafka.streams.test.TestRecordTest > testFields PASSED

org.apache.kafka.streams.test.TestRecordTest > testProducerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testProducerRecord PASSED

org.apache.kafka.streams.test.TestRecordTest > testEqualsAndHashCode STARTED

org.apache.kafka.streams.test.TestRecordTest > testEqualsAndHashCode PASSED

> Task :streams:upgrade-system-tests-0100:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0100:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:classes UP-TO-DATE
> Task :streams:upgrade-system-tests-0100:checkstyleMain NO-SOURCE
> Task :streams:upgrade-system-tests-0100:compileTestJava
> Task :streams:upgrade-system-tests-0100:processTestResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:testClasses
> Task :streams:upgrade-system-tests-0100:checkstyleTest
> Task :streams:upgrade-system-tests-0100:spotbugsMain NO-SOURCE
> Task :streams:upgrade-system-tests-0100:test
> Task :streams:upgrade-system-tests-0101:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0101:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0101:classes UP-TO-DATE
> Task :streams:upgrade-system-tests-0101:checkstyleMain NO-SOURCE
> T

Build failed in Jenkins: kafka-trunk-jdk11 #1199

[Apache Jenkins Server]

See 


Changes:

[github] Adapt docs about metrics of Streams according to KIP-444 (#8171)

[github] MINOR: Revert Jetty to 9.4.25 (#8183)


--
[...truncated 5.85 MB...]
org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecord PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairs 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicNameAndTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicNameAndTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairsAndCustomTimestamps
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairsAndCustomTimestamps
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithCustomTimestampAndIncrementsAndNotAdvanceTime
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithCustomTimestampAndIncrementsAndNotAdvanceTime
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithTimestampWithTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithTimestampWithTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKeyAndDefaultTimestamp
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKeyAndDefaultTimestamp
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithTimestampAndIncrements STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithTimestampAndIncrements PASSED

org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis STARTED

org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis PASSED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime STARTED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime PASSED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep PASSED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnIsOpen 
STARTED

org.apache.kafka.streams.internals.KeyValueStor

[jira] [Created] (KAFKA-9623) Streams will attempt to commit during shutdown if rebalance is in progress

[Guozhang Wang (Jira)]

Guozhang Wang created KAFKA-9623:


 Summary: Streams will attempt to commit during shutdown if 
rebalance is in progress
 Key: KAFKA-9623
 URL: https://issues.apache.org/jira/browse/KAFKA-9623
 Project: Kafka
  Issue Type: Bug
  Components: streams
Reporter: Guozhang Wang
Assignee: Guozhang Wang


This will throw a retriable `RebalanceInProgressException` which Streams does 
not currently expect. 

A possible fix is to change the condition of while(isRunning()) inside runLoop 
to sth. like isRunning() || !taskManager.rebalanceInProgress(), and within an 
iteration after we’ve added the records we will check isRunning() again and if 
false we would skip processing any records anyways.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Build failed in Jenkins: kafka-2.5-jdk8 #48

[Apache Jenkins Server]

See 


Changes:

[wangguoz] Adapt docs about metrics of Streams according to KIP-444 (#8171)

[ismael] MINOR: Revert Jetty to 9.4.25 (#8183)

[bill] throttle consumer timeout increase (#8188)


--
[...truncated 2.89 MB...]
org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourcesThatMatchMultiplePattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourcesThatMatchMultiplePattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldPopulateGlobalStore[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldPopulateGlobalStore[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfPersistentBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfPersistentBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldAllowPrePopulatingStatesStoresWithCachingEnabled[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldAllowPrePopulatingStatesStoresWithCachingEnabled[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldRespectTaskIdling[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldRespectTaskIdling[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldApplyGlobalUpdatesCorrectlyInRecursiveTopologies[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldApplyGlobalUpdatesCorrectlyInRecursiveTopologies[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForLargerValue[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverT

Re: 回复：[Vote] KIP-571: Add option to force remove members in StreamsResetter

[Sophie Blee-Goldman]

Thanks for the KIP, +1 (non-binding)

On Thu, Feb 27, 2020 at 12:40 PM Boyang Chen 
wrote:

> Thanks Feyman, +1 (non-binding)
>
> On Thu, Feb 27, 2020 at 9:25 AM John Roesler  wrote:
>
> > Thanks for the proposal!
> >
> > I'm +1 (binding)
> > -John
> >
> > On Wed, Feb 26, 2020, at 19:41, feyman2009 wrote:
> > > Updated with the KIP link:
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-571%3A+Add+option+to+force+remove+members+in+StreamsResetter
> > >
> > >
> > > --
> > > 发件人：feyman2009 
> > > 发送时间：2020年2月27日(星期四) 09:38
> > > 收件人：dev 
> > > 主 题：[Vote] KIP-571: Add option to force remove members in
> StreamsResetter
> > >
> > >
> > > Hi, all
> > > I would like to start a vote on KIP-571: Add option to force remove
> > > members in StreamsResetter .
> > >
> > > Thanks!
> > > Feyman
> > >
> > >
> >
>

Re: [KAFKA-557] Add emit on change support for Kafka Streams

[Richard Yu]

Hi all,

@John Will add some notes accordingly.

To all: Thanks for all your input!
It looks like we can wrap up this discussion thread then.

I've started a vote thread, so please feel free to cast your vote there!

We should be pretty close. :)

Cheers,
Richard

On Thu, Feb 27, 2020 at 2:34 PM John Roesler  wrote:

> Hi Richard,
>
> Thanks for the update!
>
> I read it over, and overall it looks good!
>
> I have only a minor concern about the rate metric definition:
> > The rate option indicates the ratio of records dropped to actual volume
> of records passing through the task
> That's not the definition of a "rate". It should be something more like
> "the average number of dropped idempotent updates per second".
>
> Incidentally, I mentioned this KIP to Guozhang, and he brought up an
> interesting concern I'd like to share with the list. He noted that if we
> filter
> out idempotent table updates, stream time will not advance with every
> input event anymore. He asked if this would have a negative impact on
> operations that depend on stream time.
>
> I think this is a valid concern. For example, you can use Suppress to
> buffer KTable updates until a certain amount of stream time passes.
> Specifically,
> the Suppress processor itself advances stream time as it receives new
> records
> to its `process` method. In the pathological case, all updates are
> idempotent,
> get dropped, and the Suppress operator never emits anything, even though to
> an outside observer, stream time should have advanced.
>
> Example scenario:
> > inputTable = builder.table(input)
> > suppressed = inputTable.suppress(untilTimeLimit(10))
> >
> > input: key=A, timestamp=0, value=X
> > inputTable: key=A, timestamp=0, value=X
> > suppressed buffer: key=A, timestamp=0, value=X (observed stream time = 0)
> > output: (nothing)
> >
> > input: key=A, timestamp=11, value=X
> > // update is idempotent, so it gets dropped
> > inputTable: key=A, timestamp=0, value=X
> > suppressed buffer: key=A, timestamp=0, value=X (observed stream time = 0)
> > output: (nothing)
>
> Note that even though stream time has technically advanced beyond the
> suppression config of 10, the suppression buffer doesn't see it because the
> KTable source already dropped the idempotent update.
>
> I'm thinking that this situation is indeed concerning, and we should be
> aware
> and make a note of it. However, I don't think that it should change our
> proposal.
> To understand why, I'll enumerate all the usages of stream time in Kafka
> Streams:
>
> windowed operations
> -
> KStreamSessionWindowAggregate & KStreamWindowAggregate:
>   - used to determine if an incoming record belongs to a window that is
> already closed
> Suppressed.untilWindowCloses():
>   - used to flush out results for windows, once they are closed
> AbstractRocksDBSegmentedBytesStore & InMemorySessionStore &
> InMemoryWindowStore:
>   - used to create new segments and drop old ones that are out of retention
>
> non-windowed operations
> ---
> Suppressed.untilTimeLimit
>   - used to flush out prior updates that have passed the configured age,
> in stream time
>
> Note that most of the usages are in windowed operations. One interesting
> thing
> about this context is that records with higher timestamps belong to
> different windows.
> In order to advance stream time far enough to close a window or push it
> out of retention, the new records must have timestamps that in later
> windows, which means
> that they are updates to _new_ keys, which means they would not be
> suppressed as
> idempotent.
>
> Updates within a window could still be suppressed, though:
> For example, if the window size is 10, and the grace period is 5, and we
> get updates all
> for the same key with timestamps 0, 11, and 16, today, we would emit the
> record for
> the [0,10) window as soon as we got the update at time 16 (since stream
> time is now
> past the window end + grace period time of 15). But if we drop the time=16
> update, we
> wouldn't emit the [0,10) window results until we get a record with time >=
> 20.
>
> You can see that the maximum amount of (stream) time that dropping
> idempotent
> updates could delay updates is one window. This might be surprising, but
> it doesn't
> seem too bad, especially since Suppress explicitly does _not_ promise to
> emit the
> results at the earliest possible time, just at some time after the window
> closes.
>
> Even better, though, all the windowed aggregations are _stream_
> aggregations
> that _produce_ a KTable, and we already decided that (at least for now),
> we would
> include the timestamp in the idempotence check for stream aggregation
> results.
> With this strategy, we would actually not suppress _any_ update to a
> stream aggregation (windowed or otherwise) that advances stream time.
>
> So there's no problem at all with windowed operations.
>
> This only leaves non-windowed operations, of which there's only 

Build failed in Jenkins: kafka-trunk-jdk11 #1200

[Apache Jenkins Server]

See 


Changes:

[github] throttle consumer timeout increase (#8188)


--
[...truncated 2.89 MB...]
org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecord PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairs 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicNameAndTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicNameAndTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairsAndCustomTimestamps
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairsAndCustomTimestamps
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithCustomTimestampAndIncrementsAndNotAdvanceTime
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithCustomTimestampAndIncrementsAndNotAdvanceTime
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithTimestampWithTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithTimestampWithTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKeyAndDefaultTimestamp
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKeyAndDefaultTimestamp
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithTimestampAndIncrements STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithTimestampAndIncrements PASSED

org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis STARTED

org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis PASSED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime STARTED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime PASSED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep PASSED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnIsOpen 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnIsOpen 
PASSED

org.apache.kafka.streams.int

Jenkins build is back to normal : kafka-trunk-jdk8 #4281

[Apache Jenkins Server]

See 

[jira] [Reopened] (KAFKA-8460) Flaky Test PlaintextConsumerTest#testLowMaxFetchSizeForRequestAndPartition

[Boyang Chen (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-8460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Boyang Chen reopened KAFKA-8460:


> Flaky Test  PlaintextConsumerTest#testLowMaxFetchSizeForRequestAndPartition
> ---
>
> Key: KAFKA-8460
> URL: https://issues.apache.org/jira/browse/KAFKA-8460
> Project: Kafka
>  Issue Type: Bug
>Reporter: Boyang Chen
>Priority: Major
>
> [https://builds.apache.org/job/kafka-pr-jdk11-scala2.12/5168/consoleFull] 
>  *16:17:04* kafka.api.PlaintextConsumerTest > 
> testLowMaxFetchSizeForRequestAndPartition FAILED*16:17:04* 
> org.scalatest.exceptions.TestFailedException: Timed out before consuming 
> expected 2700 records. The number consumed was 1980.*16:17:04* at 
> org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:530)*16:17:04*
>  at 
> org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:529)*16:17:04*
>  at 
> org.scalatest.Assertions$.newAssertionFailedException(Assertions.scala:1389)*16:17:04*
>  at org.scalatest.Assertions.fail(Assertions.scala:1091)*16:17:04* at 
> org.scalatest.Assertions.fail$(Assertions.scala:1087)*16:17:04* at 
> org.scalatest.Assertions$.fail(Assertions.scala:1389)*16:17:04* at 
> kafka.utils.TestUtils$.waitUntilTrue(TestUtils.scala:789)*16:17:04* at 
> kafka.utils.TestUtils$.pollRecordsUntilTrue(TestUtils.scala:765)*16:17:04* at 
> kafka.api.AbstractConsumerTest.consumeRecords(AbstractConsumerTest.scala:156)*16:17:04*
>  at 
> kafka.api.PlaintextConsumerTest.testLowMaxFetchSizeForRequestAndPartition(PlaintextConsumerTest.scala:801)*16:17:04*



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Build failed in Jenkins: kafka-2.4-jdk8 #158

[Apache Jenkins Server]

See 


Changes:

[bill] throttle consumer timeout increase (#8188)

[github] MINOR: Update upgrade guide for ZK (#8182)


--
[...truncated 5.57 MB...]
org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFeedStoreFromGlobalKTable[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFeedStoreFromGlobalKTable[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCleanUpPersistentStateStoresOnClose[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCleanUpPersistentStateStoresOnClose[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowForUnknownTopicDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowForUnknownTopicDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowPatternNotValidForTopicNameException[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowPatternNotValidForTopicNameException[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldEnqueueLaterOutputsAfterEarlierOnes[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldEnqueueLaterOutputsAfterEarlierOnes[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializersDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializersDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfEvenTimeAdvances[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfEvenTimeAdvances[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldInitProcessor[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldInitProcessor[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowForUnknownTopic[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowForUnknownTopic[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnStreamsTime[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnStreamsTime[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfInMemoryBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfInMemoryBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourcesThatMatchMultiplePattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourcesThatMatchMultiplePattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldPopulateGlobalStore[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldPopulateGlobalStore[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfPersistentBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfPersistentBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldAllowPrePopulatingStatesStoresWithCachingEnabled[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldAllowPrePopulatingStatesStoresWithCachingEnabled[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldApplyGlobalUpdatesCorrectlyInRecursiveTopologies[Eos enabled = false] 
STARTED

Re: [DISCUSS] KIP-565: Using AclCommand,avoid call the global method loadcache in SimpleAclAuthorizer

[Steven Lu]

Thanks for your replay,
this switch same not the best.I have changed another way to solve this 
problom,can you help me review the pr:  
https://github.com/apache/kafka/pull/7706/files

On 2020/01/21 09:48:00, Rajini Sivaram  wrote: 
> Hi Steven,
> 
> Thanks for the KIP. A few questions/comments:
> 
> 1) The command line option for AclCommand makes it the user's
> responsibility to determine whether cache should be loaded. That doesn't
> feel like a good idea. If you are listing ACLs, you need the cache. More
> importantly, you need the cache for some code paths in delete and that
> could be authorizer-dependent. It feels dangerous to make that a choice
> when the result of not doing so would potentially retain ACLs that you
> didn't intend to.
> 
> 2) Even though the KIP talks about the deprecated SimpleAclAuthorizer, I
> guess you also mean the new AclAuthorizer since the PR updates the new one.
> We should clarify in the KIP.
> 
> 3) The recommended way to update ACLs is using --bootstrap-server option
> for AclCommand which uses the Kafka protocol to talk to brokers and the
> update is performed by brokers which already have all ACLs loaded into
> their cache. In case you have found issues with this approach, it will be
> good to understand what the issues are so that we can improve this path.
> 
> On Tue, Jan 21, 2020 at 1:50 AM Steven Lu  wrote:
> 
> > Hello all,
> >
> > In the class Named AclCommand,configure SimpleAclAuthorizer,but no need
> > call loadCache.
> > now we have 20,000 topics in kafka cluster,everytime I run AclCommand,all
> > these topics's Alcs need to be authed, it will be very slow.
> > The purpose of this optimization is：we can choose to not load the acl of
> > all topics into memory, mainly for adding and deleting permissions.
> >
> > PR Available here: https://github.com/apache/kafka/pull/7706
> > KIP Available here:
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-565%3A+Using+AclCommand%2Cavoid+call+the+global+method+loadcache+in+SimpleAclAuthorizer
> > Issue Available here: https://issues.apache.org/jira/browse/KAFKA-9424
> >
> > mainly for adding and deleting permissions,we can choose to not load the
> > acl of all topics into memory,then we can add two args "--load-acl-cache"
> > "false" in AclCommand.main;else you don't add these args, it will load the
> > acl cache defaultly.
> >
> > we can choose improve the running time from minutes to less than one
> > second.
> >
> > Thanks,
> > Steven
> >
>