[GitHub] kafka pull request #3786: MINOR: preparing for bug fix release

[dguy]

Github user dguy closed the pull request at:

https://github.com/apache/kafka/pull/3786


---

[jira] [Created] (KAFKA-5835) CommitFailedException message is misleading and cause is swallowed

[Stevo Slavic (JIRA)]

Stevo Slavic created KAFKA-5835:
---

 Summary: CommitFailedException message is misleading and cause is 
swallowed
 Key: KAFKA-5835
 URL: https://issues.apache.org/jira/browse/KAFKA-5835
 Project: Kafka
  Issue Type: Bug
  Components: clients
Affects Versions: 0.11.0.0
Reporter: Stevo Slavic
Priority: Trivial


{{CommitFailedException}}'s message suggests that it can only be thrown as 
consequence of rebalancing. JavaDoc of the {{CommitFailedException}} suggests 
differently that in general it can be thrown for any kind of unrecoverable 
failure from {{KafkaConsumer#commitSync()}} call (e.g. if offset being 
committed is invalid / outside of range).

{{CommitFailedException}}'s message is misleading in a way that one can just 
see the message in logs, and without consulting JavaDoc or source code one can 
assume that message is correct and that rebalancing is the only potential 
cause, so one can wast time proceeding with the debugging in wrong direction.

Additionally, since {{CommitFailedException}} can be thrown for different 
reasons, cause should not be swallowed. This makes impossible to handle each 
potential cause in a specific way. If the cause is another exception please 
pass it as cause, or construct appropriate exception hierarchy with specific 
exception for every failure cause and make {{CommitFailedException}} abstract.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] kafka pull request #3788: MINOR: update docs

[omkreddy]

GitHub user omkreddy opened a pull request:

https://github.com/apache/kafka/pull/3788

MINOR: update docs



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/omkreddy/kafka RULE-DOC

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3788.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3788


commit c37c4f78fcf49efefe5bbb5b55f442f0fd020886
Author: Manikumar Reddy 
Date:   2017-09-05T08:59:00Z

MINOR: update docs




---

[GitHub] kafka pull request #3789: MINOR: Fix a couple of ftp related issues in relea...

[ijuma]

GitHub user ijuma opened a pull request:

https://github.com/apache/kafka/pull/3789

MINOR: Fix a couple of ftp related issues in release.py



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijuma/kafka ftp-release-py-fixes

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3789.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3789


commit 11d7afeeb36b694ed4df06a0f48ed501b2c54edb
Author: Ismael Juma 
Date:   2017-09-05T09:17:56Z

MINOR: Fix a couple of ftp related issues in release.py




---

[GitHub] kafka pull request #3789: MINOR: Fix sftp_mkdir in release.py

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3789


---

[jira] [Created] (KAFKA-5836) Kafka Streams - API for specifying internal stream name on join

[JIRA]

Lovro Pandžić created KAFKA-5836:


 Summary: Kafka Streams - API for specifying internal stream name 
on join
 Key: KAFKA-5836
 URL: https://issues.apache.org/jira/browse/KAFKA-5836
 Project: Kafka
  Issue Type: New Feature
Reporter: Lovro Pandžić


Automatic topic name can be problematic in case of streams operation 
change/migration.
I'd like to be able to specify name of an internal topic so I can avoid 
creation of new stream and data "loss" when changing the Stream building.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is back to normal : kafka-trunk-jdk7 #2706

[Apache Jenkins Server]

See 

[GitHub] kafka pull request #3790: MINOR: fix scalaVersion variable in templateData.j...

[dguy]

GitHub user dguy opened a pull request:

https://github.com/apache/kafka/pull/3790

MINOR: fix scalaVersion variable in templateData.js



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dguy/kafka fix-js-tempate

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3790.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3790


commit 7d58ffc1e9f58208cf6601fe54fe5717140549ff
Author: Damian Guy 
Date:   2017-09-05T10:41:03Z

fix templateData.js




---

Re: [DISCUSS] KIP-170: Enhanced TopicCreatePolicy and introduction of TopicDeletePolicy

[Ismael Juma]

Hi Edoardo,

Do you intend to update the KIP to avoid the introduction of another
interface?

Ismael

On Fri, Jun 23, 2017 at 5:47 PM, Edoardo Comar  wrote:

> Thanks for the thoughts Ismael
>
> > 1. Have you considered extending RequestMetadata with the additional
> > information you need? We could add Cluster to it, which has topic
> > assignment information, for example. This way, there would be no need
> for a
> > V2 interface.
>
> 1. I hadn't thought of it - but this seems a feasible alternative.
>
> The XXXPolicy.RequestMetadata could be enriched to include more
> information -
> for backward compatibility with existing Policies we would only add
> methods to these classes.
>
> Presumably the number of Admin Request is not huge so we should not be too
> worried
> about the overhead of populating a org.apache.kafka.common.Cluster for
> every admin request , right ?
>
> The only feature I can't see how to easily support with the enriched
> XXXPolicy.RequestMetadata is how to check
> for a topic being marked for deletion using the Cluster information
>
> This check is useful for us in excluding such outliers when counting the
> number of partitions.
>
>
> > 2. Something else that could be useful is passing an instance of
> `Session`
> > so that one can provide custom behaviour depending on the logged in
> user.
> > Would this be useful?
> 2. Definitely I would expect that a general solution includes the Session
> or the KafkaPrincipal associated with it
> (the latter may be a simpler dependency for the XXXPolicy.RequestMetadata
>
>
> > 3. For the delete case, we may consider passing a class instead of just
> a
> > string to the validate method so that we have options if we need to
> extend
> > it.
>
> 3. Agree, we should have the DeletePolicy define its RequestMetadata
> class, too
>
>
> > 4. Do we want to enhance the AlterConfigs policy as well?
>
> 4. I don't see why not :-)
>
>
>
> thanks
> Edo
> --
>
> Edoardo Comar
>
> IBM Message Hub
>
>
> isma...@gmail.com wrote on 22/06/2017 15:05:06:
>
> > From: Ismael Juma 
> > To: dev@kafka.apache.org
> > Date: 22/06/2017 15:05
> > Subject: Re: [DISCUSS] KIP-170: Enhanced TopicCreatePolicy and
> > introduction of TopicDeletePolicy
> > Sent by: isma...@gmail.com
> >
> > Thanks for the KIP, Edoardo. A few comments:
> >
>
> >
>
> >
>
> >
>
> >
> > Ismael
> >
> > On Thu, Jun 22, 2017 at 2:41 PM, Edoardo Comar 
> wrote:
> >
> > > Hi all,
> > >
> > > We've drafted "KIP-170: Enhanced TopicCreatePolicy and introduction of
> > > TopicDeletePolicy" for discussion:
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-170%3A+Enhanced+
> > > TopicCreatePolicy+and+introduction+of+TopicDeletePolicy
> > >
> > > Please take a look. Your feedback is welcome and much needed.
> > >
> > > Thanks,
> > > Edoardo
> > > --
> > > Edoardo Comar
> > > IBM Message Hub
> > > eco...@uk.ibm.com
> > > IBM UK Ltd, Hursley Park, SO21 2JN
> > > Unless stated otherwise above:
> > > IBM United Kingdom Limited - Registered in England and Wales with
> number
> > > 741598.
> > > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6
> 3AU
> > >
>
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>

[GitHub] kafka pull request #3790: MINOR: fix scalaVersion variable in templateData.j...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3790


---

Build failed in Jenkins: kafka-0.11.0-jdk7 #295

[Apache Jenkins Server]

See 


Changes:

[damian.guy] MINOR: Fix sftp_mkdir in release.py

--
[...truncated 2.44 MB...]
org.apache.kafka.streams.state.internals.ChangeLoggingKeyValueBytesStoreTest > 
shouldNotWriteToChangeLogOnPutIfAbsentWhenValueForKeyExists PASSED

org.apache.kafka.streams.state.internals.ChangeLoggingKeyValueBytesStoreTest > 
shouldReturnNullOnPutIfAbsentWhenNoPreviousValue STARTED

org.apache.kafka.streams.state.internals.ChangeLoggingKeyValueBytesStoreTest > 
shouldReturnNullOnPutIfAbsentWhenNoPreviousValue PASSED

org.apache.kafka.streams.state.internals.ChangeLoggingKeyValueBytesStoreTest > 
shouldLogKeyNullOnDelete STARTED

org.apache.kafka.streams.state.internals.ChangeLoggingKeyValueBytesStoreTest > 
shouldLogKeyNullOnDelete PASSED

org.apache.kafka.streams.state.internals.ChangeLoggingKeyValueBytesStoreTest > 
shouldReturnNullOnGetWhenDoesntExist STARTED

org.apache.kafka.streams.state.internals.ChangeLoggingKeyValueBytesStoreTest > 
shouldReturnNullOnGetWhenDoesntExist PASSED

org.apache.kafka.streams.state.internals.ChangeLoggingKeyValueBytesStoreTest > 
shouldReturnOldValueOnDelete STARTED

org.apache.kafka.streams.state.internals.ChangeLoggingKeyValueBytesStoreTest > 
shouldReturnOldValueOnDelete PASSED

org.apache.kafka.streams.state.internals.ChangeLoggingKeyValueBytesStoreTest > 
shouldNotWriteToInnerOnPutIfAbsentWhenValueForKeyExists STARTED

org.apache.kafka.streams.state.internals.ChangeLoggingKeyValueBytesStoreTest > 
shouldNotWriteToInnerOnPutIfAbsentWhenValueForKeyExists PASSED

org.apache.kafka.streams.state.internals.ChangeLoggingKeyValueBytesStoreTest > 
shouldReturnValueOnGetWhenExists STARTED

org.apache.kafka.streams.state.internals.ChangeLoggingKeyValueBytesStoreTest > 
shouldReturnValueOnGetWhenExists PASSED

org.apache.kafka.streams.state.internals.RocksDBSegmentedBytesStoreTest > 
shouldRemove STARTED

org.apache.kafka.streams.state.internals.RocksDBSegmentedBytesStoreTest > 
shouldRemove PASSED

org.apache.kafka.streams.state.internals.RocksDBSegmentedBytesStoreTest > 
shouldPutAndFetch STARTED

org.apache.kafka.streams.state.internals.RocksDBSegmentedBytesStoreTest > 
shouldPutAndFetch PASSED

org.apache.kafka.streams.state.internals.RocksDBSegmentedBytesStoreTest > 
shouldRollSegments STARTED

org.apache.kafka.streams.state.internals.RocksDBSegmentedBytesStoreTest > 
shouldRollSegments PASSED

org.apache.kafka.streams.state.internals.RocksDBSegmentedBytesStoreTest > 
shouldFindValuesWithinRange STARTED

org.apache.kafka.streams.state.internals.RocksDBSegmentedBytesStoreTest > 
shouldFindValuesWithinRange PASSED

org.apache.kafka.streams.StreamsConfigTest > shouldUseNewConfigsWhenPresent 
STARTED

org.apache.kafka.streams.StreamsConfigTest > shouldUseNewConfigsWhenPresent 
PASSED

org.apache.kafka.streams.StreamsConfigTest > testGetConsumerConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > testGetConsumerConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > shouldAcceptAtLeastOnce STARTED

org.apache.kafka.streams.StreamsConfigTest > shouldAcceptAtLeastOnce PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldUseCorrectDefaultsWhenNoneSpecified STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldUseCorrectDefaultsWhenNoneSpecified PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldAllowSettingProducerEnableIdempotenceIfEosDisabled STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldAllowSettingProducerEnableIdempotenceIfEosDisabled PASSED

org.apache.kafka.streams.StreamsConfigTest > defaultSerdeShouldBeConfigured 
STARTED

org.apache.kafka.streams.StreamsConfigTest > defaultSerdeShouldBeConfigured 
PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSetDifferentDefaultsIfEosEnabled STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSetDifferentDefaultsIfEosEnabled PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldNotOverrideUserConfigRetriesIfExactlyOnceEnabled STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldNotOverrideUserConfigRetriesIfExactlyOnceEnabled PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedConsumerConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedConsumerConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldOverrideStreamsDefaultConsumerConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldOverrideStreamsDefaultConsumerConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > testGetProducerConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > testGetProducerConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldAllowSettingProducerMaxInFlightRequestPerConnectionsWhenEosDisabled 
STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldAllowSettingProducerMaxInFlightRequestPerConnectionsWhenEosDisabled PASSED

org.apache.kafka.streams.

Build failed in Jenkins: kafka-trunk-jdk7 #2707

[Apache Jenkins Server]

See 

--
[...truncated 922.92 KB...]
kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldUpdateSavedOffsetWhenOffsetToClearToIsBetweenEpochs PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotResetEpochHistoryTailIfUndefinedPassed STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotResetEpochHistoryTailIfUndefinedPassed PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldReturnUnsupportedIfNoEpochRecorded STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldReturnUnsupportedIfNoEpochRecorded PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldRetainLatestEpochOnClearAllEarliest STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldRetainLatestEpochOnClearAllEarliest PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldPersistEpochsBetweenInstances STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldPersistEpochsBetweenInstances PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotClearAnythingIfOffsetToFirstOffset STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotClearAnythingIfOffsetToFirstOffset PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotLetOffsetsGoBackwardsEvenIfEpochsProgress STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotLetOffsetsGoBackwardsEvenIfEpochsProgress PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldGetFirstOffsetOfSubsequentEpochWhenOffsetRequestedForPreviousEpoch STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldGetFirstOffsetOfSubsequentEpochWhenOffsetRequestedForPreviousEpoch PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldUpdateOffsetBetweenEpochBoundariesOnClearEarliest2 STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldUpdateOffsetBetweenEpochBoundariesOnClearEarliest2 PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearEarliestOnEmptyCache 
STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearEarliestOnEmptyCache 
PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldPreserveResetOffsetOnClearEarliestIfOneExists STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldPreserveResetOffsetOnClearEarliestIfOneExists PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldUpdateOffsetBetweenEpochBoundariesOnClearEarliest STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldUpdateOffsetBetweenEpochBoundariesOnClearEarliest PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldReturnInvalidOffsetIfEpochIsRequestedWhichIsNotCurrentlyTracked STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldReturnInvalidOffsetIfEpochIsRequestedWhichIsNotCurrentlyTracked PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldFetchEndOffsetOfEmptyCache 
STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldFetchEndOffsetOfEmptyCache 
PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldRetainLatestEpochOnClearAllEarliestAndUpdateItsOffset STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldRetainLatestEpochOnClearAllEarliestAndUpdateItsOffset PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearAllEntries STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearAllEntries PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearLatestOnEmptyCache 
STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearLatestOnEmptyCache 
PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotResetEpochHistoryHeadIfUndefinedPassed STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotResetEpochHistoryHeadIfUndefinedPassed PASSED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldIncreaseLeaderEpochBetweenLeaderRestarts STARTED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldIncreaseLeaderEpochBetweenLeaderRestarts PASSED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldAddCurrentLeaderEpochToMessagesAsTheyAreWrittenToLeader STARTED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldAddCurrentLeaderEpochToMessagesAsTheyAreWrittenToLeader PASSED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldSendLeaderEpochRequestAndGetAResponse STARTED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldSendLeaderEpochRequestAndGetAResponse PASSED

kafka.server.epoch.OffsetsForLeaderEpochTest > shouldGetEpochsFromReplica 
STARTED

kafka.server.epoch.OffsetsForLeaderEpochTest > shouldGetEpochsFromReplica PASSED

kafka.server.epoch.OffsetsForLeaderEpochTest > 
shouldReturnUnknownTopicOrPartitionIfThrown STARTED

kafka.server.epoch.OffsetsForLeaderEpochTest > 
shouldReturnUnknownTopicOrPartitionIfThrown PASSED

kafka.server.epoch.OffsetsForLeaderEpochTest > 
shouldReturnNoLeaderForPartitionIfThrown STARTED

kafka.server.epoch.OffsetsForLeaderEpochTest > 
shouldReturnNoLeaderForPartitionIfThrown PASSED

kafka.server.epoch.EpochDrivenReplicationProtocolAc

Build failed in Jenkins: kafka-trunk-jdk7 #2708

[Apache Jenkins Server]

See 


Changes:

[damian.guy] MINOR: fix scalaVersion variable in templateData.js

--
[...truncated 922.02 KB...]

kafka.server.ServerGenerateBrokerIdTest > testMultipleLogDirsMetaProps STARTED

kafka.server.ServerGenerateBrokerIdTest > testMultipleLogDirsMetaProps PASSED

kafka.server.ServerGenerateBrokerIdTest > testDisableGeneratedBrokerId STARTED

kafka.server.ServerGenerateBrokerIdTest > testDisableGeneratedBrokerId PASSED

kafka.server.ServerGenerateBrokerIdTest > testUserConfigAndGeneratedBrokerId 
STARTED

kafka.server.ServerGenerateBrokerIdTest > testUserConfigAndGeneratedBrokerId 
PASSED

kafka.server.ServerGenerateBrokerIdTest > 
testConsistentBrokerIdFromUserConfigAndMetaProps STARTED

kafka.server.ServerGenerateBrokerIdTest > 
testConsistentBrokerIdFromUserConfigAndMetaProps PASSED

kafka.server.DelayedOperationTest > testRequestPurge STARTED

kafka.server.DelayedOperationTest > testRequestPurge PASSED

kafka.server.DelayedOperationTest > testRequestExpiry STARTED

kafka.server.DelayedOperationTest > testRequestExpiry PASSED

kafka.server.DelayedOperationTest > 
shouldReturnNilOperationsOnCancelForKeyWhenKeyDoesntExist STARTED

kafka.server.DelayedOperationTest > 
shouldReturnNilOperationsOnCancelForKeyWhenKeyDoesntExist PASSED

kafka.server.DelayedOperationTest > 
shouldCancelForKeyReturningCancelledOperations STARTED

kafka.server.DelayedOperationTest > 
shouldCancelForKeyReturningCancelledOperations PASSED

kafka.server.DelayedOperationTest > testRequestSatisfaction STARTED

kafka.server.DelayedOperationTest > testRequestSatisfaction PASSED

kafka.server.MultipleListenersWithDefaultJaasContextTest > testProduceConsume 
STARTED

kafka.server.MultipleListenersWithDefaultJaasContextTest > testProduceConsume 
PASSED

kafka.server.ThrottledResponseExpirationTest > testThrottledRequest STARTED

kafka.server.ThrottledResponseExpirationTest > testThrottledRequest PASSED

kafka.server.ThrottledResponseExpirationTest > testExpire STARTED

kafka.server.ThrottledResponseExpirationTest > testExpire PASSED

kafka.server.ReplicaManagerQuotasTest > shouldGetBothMessagesIfQuotasAllow 
STARTED

kafka.server.ReplicaManagerQuotasTest > shouldGetBothMessagesIfQuotasAllow 
PASSED

kafka.server.ReplicaManagerQuotasTest > 
shouldExcludeSubsequentThrottledPartitions STARTED

kafka.server.ReplicaManagerQuotasTest > 
shouldExcludeSubsequentThrottledPartitions PASSED

kafka.server.ReplicaManagerQuotasTest > 
shouldGetNoMessagesIfQuotasExceededOnSubsequentPartitions STARTED

kafka.server.ReplicaManagerQuotasTest > 
shouldGetNoMessagesIfQuotasExceededOnSubsequentPartitions PASSED

kafka.server.ReplicaManagerQuotasTest > shouldIncludeInSyncThrottledReplicas 
STARTED

kafka.server.ReplicaManagerQuotasTest > shouldIncludeInSyncThrottledReplicas 
PASSED

kafka.server.SaslApiVersionsRequestTest > 
testApiVersionsRequestWithUnsupportedVersion STARTED

kafka.server.SaslApiVersionsRequestTest > 
testApiVersionsRequestWithUnsupportedVersion PASSED

kafka.server.SaslApiVersionsRequestTest > 
testApiVersionsRequestBeforeSaslHandshakeRequest STARTED

kafka.server.SaslApiVersionsRequestTest > 
testApiVersionsRequestBeforeSaslHandshakeRequest PASSED

kafka.server.SaslApiVersionsRequestTest > 
testApiVersionsRequestAfterSaslHandshakeRequest STARTED

kafka.server.SaslApiVersionsRequestTest > 
testApiVersionsRequestAfterSaslHandshakeRequest PASSED

kafka.server.ControlledShutdownLeaderSelectorTest > testSelectLeader STARTED

kafka.server.ControlledShutdownLeaderSelectorTest > testSelectLeader PASSED

kafka.server.MetadataCacheTest > 
getTopicMetadataWithNonSupportedSecurityProtocol STARTED

kafka.server.MetadataCacheTest > 
getTopicMetadataWithNonSupportedSecurityProtocol PASSED

kafka.server.MetadataCacheTest > getTopicMetadataIsrNotAvailable STARTED

kafka.server.MetadataCacheTest > getTopicMetadataIsrNotAvailable PASSED

kafka.server.MetadataCacheTest > getTopicMetadata STARTED

kafka.server.MetadataCacheTest > getTopicMetadata PASSED

kafka.server.MetadataCacheTest > getTopicMetadataReplicaNotAvailable STARTED

kafka.server.MetadataCacheTest > getTopicMetadataReplicaNotAvailable PASSED

kafka.server.MetadataCacheTest > getTopicMetadataPartitionLeaderNotAvailable 
STARTED

kafka.server.MetadataCacheTest > getTopicMetadataPartitionLeaderNotAvailable 
PASSED

kafka.server.MetadataCacheTest > getAliveBrokersShouldNotBeMutatedByUpdateCache 
STARTED

kafka.server.MetadataCacheTest > getAliveBrokersShouldNotBeMutatedByUpdateCache 
PASSED

kafka.server.MetadataCacheTest > getTopicMetadataNonExistingTopics STARTED

kafka.server.MetadataCacheTest > getTopicMetadataNonExistingTopics PASSED

kafka.server.FetchRequestTest > testBrokerRespectsPartitionsOrderAndSizeLimits 
STARTED

kafka.server.FetchRequestTest > testBrokerRespectsPartitionsOrderAndSizeLimits 
PASSED

kafka.server.FetchRequestTest > 
testDownConversionFromBatc

Jenkins build is back to normal : kafka-0.11.0-jdk7 #296

[Apache Jenkins Server]

See 

Build failed in Jenkins: kafka-trunk-jdk8 #1972

[Apache Jenkins Server]

See 


Changes:

[damian.guy] MINOR: fix scalaVersion variable in templateData.js

--
[...truncated 2.65 MB...]

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRestartConnectorRedirectToOwner STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRestartConnectorRedirectToOwner PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRestartUnknownTask STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRestartUnknownTask PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRequestProcessingOrder STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRequestProcessingOrder PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRestartTaskRedirectToLeader STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRestartTaskRedirectToLeader PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRestartTaskRedirectToOwner STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRestartTaskRedirectToOwner PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testConnectorConfigAdded STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testConnectorConfigAdded PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testConnectorConfigUpdate STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testConnectorConfigUpdate PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testConnectorPaused STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testConnectorPaused PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testConnectorResumed STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testConnectorResumed PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testUnknownConnectorPaused STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testUnknownConnectorPaused PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testConnectorPausedRunningTaskOnly STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testConnectorPausedRunningTaskOnly PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testConnectorResumedRunningTaskOnly STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testConnectorResumedRunningTaskOnly PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testTaskConfigAdded STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testTaskConfigAdded PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testJoinLeaderCatchUpFails STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testJoinLeaderCatchUpFails PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testInconsistentConfigs STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testInconsistentConfigs PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testDestroyConnector STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testDestroyConnector PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testCreateConnectorAlreadyExists STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testCreateConnectorAlreadyExists PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRestartTask STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRestartTask PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testCreateConnector STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testCreateConnector PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRestartConnector STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRestartConnector PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testCreateConnectorFailedBasicValidation STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testCreateConnectorFailedBasicValidation PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testAccessors STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testAccessors PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testCreateConnectorFailedCustomValidation STARTED

org.apache.kafka.connect.runtime.dis

[GitHub] kafka pull request #3791: MINOR: Update dependencies for 1.0.0 release

[andrasbeni]

GitHub user andrasbeni opened a pull request:

https://github.com/apache/kafka/pull/3791

MINOR: Update dependencies for 1.0.0 release

Updates dependencies to their latest versions.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/andrasbeni/kafka dependency-upgrade

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3791.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3791


commit 7ac4e2250ca6b48aae9bc755119771acc535cdc6
Author: Andras Beni 
Date:   2017-09-05T09:45:41Z

MINOR: Update dependencies for 1.0.0 release
Updates dependencies to their latest versions.




---

Re: [DISCUSS] KIP-131 : Add access to OffsetStorageReader from SourceConnector

[Randall Hauch]

Thanks for taking into account my suggestions/concerns. I had a few very
minor suggestions on the PR regarding documentation, but overall everything
looks great to me.

I'd encourage anyone else to review
https://cwiki.apache.org/confluence/display/KAFKA/KIP-131+-+Add+access+to+OffsetStorageReader+from+SourceConnector
and the PR, and to then provide feedback here. If we hear nothing, we'll
submit for a vote.

Best regards,

Randall

On Thu, Aug 24, 2017 at 12:38 PM, Florian Hussonnois 
wrote:

> Hi Randall,
>
> Thank you for your answer.
>
> I will update the KIP and the PR with your last approach which sounds
> better.
>
> Thanks.
>
> Le 16 août 2017 00:53, "Randall Hauch"  a écrit :
>
> Sorry it's taken me so long to come back to this.
>
> Have you considered creating a `SourceConnectorContext` interface that
> extends `ConnectorContext` and that adds the method to access the offset
> storage? This would very closely match the existing `SourceTaskContext`.
>
> `SourceConnector` implementations could always cast the `context` field in
> the superclass to `SourceConnectorContext`, but perhaps a slightly better
> way to do this is to add the following method to the `SourceConnector`
> class:
>
>
> public SourceConnectorContext context() {
> return (SourceConnectorContext)context;
> }
>
>
> Now, `SourceConnector` implementations can either cast themselves or use
> this additional method to obtain the correctly cast context.
>
> In fact, it might be good to do this for `SinkConnector` as well, and we
> could even add a `context()` method in the `Connector` interface, since
> subinterfaces can change the return type to be a subtype of that returned
> by the interface:
>
> ConnectorContext context();
>
> One advantage of this approach is that `SourceConnectorContext` and
> `SinkConnectorContext` remain interfaces. Another is not adding new method
> to `SourceConnector` that implementers may need to learn that they should
> not override or implement them. A third is that now we have a
> `SourceConnectorContext` and `SinkConnectorContext` to which we can add
> more methods if needed, and they are very similar to `SourceTaskContext`
> and `SinkTaskContext`.
>
> Thoughts?
>
> On Wed, Apr 5, 2017 at 3:59 PM, Florian Hussonnois 
> wrote:
>
> > Hi All,
> >
> > Is there any feedback regarding that KIP ?
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 131+-+Add+access+to+
> > OffsetStorageReader+from+SourceConnector
> >
> > Thanks,
> >
> > 2017-03-14 22:51 GMT+01:00 Florian Hussonnois :
> >
> > > Hi Matthias,
> > >
> > > Sorry I didn't know this page. Ths KIP has been added to it.
> > >
> > > Thanks,
> > >
> > > 2017-03-13 21:30 GMT+01:00 Matthias J. Sax :
> > >
> > >> Can you please add the KIP to this table:
> > >>
> > >> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+
> > >> Improvement+Proposals#KafkaImprovementProposals-KIPsunderdiscussion
> > >>
> > >> Thanks,
> > >>
> > >>  Matthias
> > >>
> > >>
> > >> On 3/7/17 1:24 PM, Florian Hussonnois wrote:
> > >> > Hi all,
> > >> >
> > >> > I've created a new KIP to add access to OffsetStorageReader from
> > >> > SourceConnector
> > >> >
> > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-131+-+
> > >> Add+access+to+OffsetStorageReader+from+SourceConnector
> > >> >
> > >> > Thanks.
> > >> >
> > >>
> > >>
> > >
> > >
> > > --
> > > Florian HUSSONNOIS
> > >
> >
> >
> >
> > --
> > Florian HUSSONNOIS
> >
>

[jira] [Created] (KAFKA-5837) ReassignPartitionsCommand fails if default throttle/timeout used

[Rajini Sivaram (JIRA)]

Rajini Sivaram created KAFKA-5837:
-

 Summary: ReassignPartitionsCommand fails if default 
throttle/timeout used
 Key: KAFKA-5837
 URL: https://issues.apache.org/jira/browse/KAFKA-5837
 Project: Kafka
  Issue Type: Bug
  Components: admin
Affects Versions: 1.0.0
Reporter: Rajini Sivaram
Assignee: Rajini Sivaram
 Fix For: 1.0.0


System ReassignPartitionsTest.test_reassign_partitions failed with:

{quote}
Partitions reassignment failed due to java.lang.String cannot be cast to 
java.lang.Long
java.lang.ClassCastException: java.lang.String cannot be cast to java.lang.Long
  at scala.runtime.BoxesRunTime.unboxToLong(BoxesRunTime.java:105)
  at 
kafka.admin.ReassignPartitionsCommand$.executeAssignment(ReassignPartitionsCommand.scala:188)
  at 
kafka.admin.ReassignPartitionsCommand$.main(ReassignPartitionsCommand.scala:61)
  at kafka.admin.ReassignPartitionsCommand.main(ReassignPartitionsCommand.scala)
{quote}

This is because default throttle is being set as a String rather than a Long. 
Default throttle was never being set properly in the command opts, but it 
didn't matter earlier because code used to set it explicitly:
{quote}
val throttle = if (opts.options.has(opts.throttleOpt)) 
opts.options.valueOf(opts.throttleOpt) else -1
{quote}
But the commit 
https://github.com/apache/kafka/commit/adefc8ea076354e07839f0319fee1fba52343b91 
started using the default directly (and also added a timeout with default set 
in the same way).




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [VOTE] KIP-188 - Add new metrics to support health checks

[Ismael Juma]

Thanks Rajini, +1 (binding) from me. Just a few minor comments:

1. FetchDownConversionsPerSec should probably be MessageConversionsPerSec
with a request tag for consistency with MessageConversionsTimeMs. The text
in that paragraph should also be updated to talk about message conversions
instead of down conversions only.

2. What will TemporaryMemorySize represent for requests other than
`ProduceRequest`?

Ismael

On Mon, Sep 4, 2017 at 2:09 PM, Rajini Sivaram 
wrote:

> All the suggestions on the discuss thread have been incorporated into the
> KIP. Please let me know if you have any more concerns or else can we
> proceed with voting for this KIP?
>
> Thank you,
>
> Rajini
>
> On Thu, Aug 24, 2017 at 6:50 PM, Rajini Sivaram 
> wrote:
>
> > Hi all,
> >
> > I would like to start the vote on KIP-188 that adds additional metrics to
> > support health checks for Kafka Ops. Details are here:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 188+-+Add+new+metrics+to+support+health+checks
> >
> > Thank you,
> >
> > Rajini
> >
> >
>

[GitHub] kafka pull request #3792: KAFKA-5837: Set defaults for ReassignPartitionsCom...

[rajinisivaram]

GitHub user rajinisivaram opened a pull request:

https://github.com/apache/kafka/pull/3792

KAFKA-5837: Set defaults for ReassignPartitionsCommand correctly



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rajinisivaram/kafka KAFKA-5837

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3792.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3792


commit ea852fc052bcb1bcabde249a315c006e7ef16d27
Author: Rajini Sivaram 
Date:   2017-09-05T15:51:32Z

KAFKA-5837: Set defaults for ReassignPartitionsCommand correctly




---

[GitHub] kafka pull request #3793: MINOR: add mvn-pgp-plugin to sign streams quicksta...

[dguy]

GitHub user dguy opened a pull request:

https://github.com/apache/kafka/pull/3793

MINOR: add mvn-pgp-plugin to sign streams quickstart jars



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dguy/kafka sign-mvn-jars

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3793.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3793


commit daecefcd46a9afb657bae86cdf7303260f037fea
Author: Damian Guy 
Date:   2017-09-05T16:03:25Z

add mvn-pgp-plugin




---

[GitHub] kafka-site pull request #73: Add coding guidelines for Streams API

[mjsax]

Github user mjsax commented on a diff in the pull request:

https://github.com/apache/kafka-site/pull/73#discussion_r137037867
  
--- Diff: coding-guide.html ---
@@ -102,6 +102,47 @@
We should attempt to maintain API compatibility when 
possible, though at this point in the project's lifecycle it is more important 
to make things good rather than avoid breakage.

 
+   Streams API
+   Kafka's Streams API (aka Kafka Streams) uses a few more 
additional coding guidelines.
+   All contributors should follow those the get a high 
quality and uniform code base.
+   Some rules help to simplify PR reviews and thus make 
the live of all contributors easier.
+   
+   Use final when possible.
+This holds for all class members, local 
variables, loop variables, and method parameters.
+   Write modular and thus testable code. Refactor if 
necessary!
+   Avoid large PRs (recommended is not more the 500 
lines per PR).
+Many JIRAs requires larger code changes; thus, 
split the work in multiple PRs and create according sub-task on the JIRA to 
track the work.
+   All public APIs must have JavaDocs.
+   Verify if JavaDocs are still up to date or if they 
need to be updated.
+   JavaDocs: Write grammatically correct sentences and 
use punctuation marks correctly.
+   Use proper markup (e.g., {@code 
null}).
+   Update the documentation on the Kafka webpage 
(i.e., within folder docs/.
+Doc changes are not additional work (i.e. no 
follow up PRs) but part of the actuall PR (can also be a sub-tasks).
--- End diff --

Thanks @Kamal15 @Sharon2017 !


---

Re: [DISCUSS] KIP-179: Change ReassignPartitionsCommand to use AdminClient

[Tom Bentley]

I've revised this KIP again:

* Change the alterPartitionCounts() API to support passing an optional
assignment for the new partitions (which is already supported by
kafka-topics.sh). At the same time I didn't want the API to suggest it was
possible to change the existing assignments in the same call (which isn't
supported today).

* Removed alterReplicationFactor(), since it's really just a special case
of reassignPartitions(): In both cases it is necessary to give a complete
partition assignment and both can be long running due to data movement.


Outstanding questions:

1. Is the proposed alterInterReplicaThrottle() API really better than
changing the throttle via alterConfigs()? It wouldn't be necessary to
support alterConfigs() for all broker configs, just DynamicConfigs (which
can't be set via the config file any way). Would it be a problem that
triggering the reassignment required ClusterAction on the Cluster, but
throttling the assignment required Alter on the Topic? What if a user had
the former permission, but not the latter?

2. Is reassignPartitions() really the best name? I find the distinction
between reassigning and just assigning to be of limited value, and
"reassign" is misleading when the APIs now used for changing the
replication factor too. Since the API is asynchronous/long running, it
might be better to indicate that in the name some how. What about
startPartitionAssignment()?

I am, of course, interested in any other questions or comments people have.




On 30 August 2017 at 16:17, Tom Bentley  wrote:

> Hi all,
>
> I've updated the KIP as follows:
>
> * remove the APIs supporting progress reporting in favour of the APIs
> being implemented in KIP-113.
> * added some APIs to cover the existing functionality around throttling
> inter-broker transfers, which was previously a TODO.
>
> To respond to Colin's suggestion:
>
> > TopicAlterationResult alterTopics(TopicAlterationOptions options,
> >   Map alters)
> >
> > PartitionCountAlteration(int numPartitions) implements TopicAlteration
> >
> > ReplicationFactorAlteration(int repl) implements TopicAlteration
> >
> > ReassignPartitionsAlteration(...) implements TopicAlteration
>
> That is a workable alternative to providing 3 separate methods on the
> AdminClient, but I don't see why it is objectively better. I don't see
> clients commonly needing to do a mixture of alterations, and it assumes
> that the options make sense for all alterations.
>
> Cheers,
>
> Tom
>
> On 22 August 2017 at 23:49, Colin McCabe  wrote:
>
>> On Wed, Aug 9, 2017, at 07:17, Tom Bentley wrote:
>> > Hi Dong and Jun,
>> >
>> > Thanks again for your input in this discussion and on KIP-113. It's
>> > difficult that discussion is split between this thread and the one for
>> > KIP-113, but I'll try to respond on this thread to questions asked on
>> > this
>> > thread.
>> >
>> > It seems there is some consensus that the alterTopic() API is the wrong
>> > thing, and it would make more sense to separate the different kinds of
>> > alteration into separate APIs. It seems to me there is then a choice.
>> >
>> > 1. Have separate alterPartitionCount(), alterReplicationFactor() and
>> > reassignPartitions() methods. This would most closely match the
>> > facilities
>> > currently offered by kafka-alter-topics and kafka-reassign-partitions.
>> > 2. Just provide a reassignPartitions() method and infer from the shape
>> of
>> > the data passed to that that a change in replication factor and/or
>> > partition count is required, as suggested by Dong.
>> >
>> > The choice we make here is also relevant to KIP-113 and KIP-178. By
>> > choosing (1) we can change the replication factor or partition count
>> > without providing an assignment and therefore are necessarily requiring
>> > the
>> > controller to make a decision for us about which broker (and, for 113,
>> > which log directory and thus which disk) the replica should be reside.
>> > There is then the matter of what criteria the controller should use to
>> > make
>> > that decision (a decision is also required on topic creation of course,
>> > but
>> > I'll try not to go there right now).
>>
>> Hmm.  One approach would be to have something like
>>
>> > TopicAlterationResult alterTopics(TopicAlterationOptions options,
>> >   Map alters)
>> >
>> > PartitionCountAlteration(int numPartitions) implements TopicAlteration
>> >
>> > ReplicationFactorAlteration(int repl) implements TopicAlteration
>> >
>> > ReassignPartitionsAlteration(...) implements TopicAlteration
>>
>>
>> >
>> > Choosing (2), on the other hand, forces us to make an assignment, and
>> > currently the AdminClient doesn't provide the APIs necessary to make a
>> > very
>> > informed decision. To do the job properly we'd need APIs to enumerate
>> the
>> > permissible log directories on each broker, know the current disk usage
>> > etc. These just don't exist today, and I think it would be a lot of work
>> > 

Re: [VOTE] KIP-188 - Add new metrics to support health checks

[Rajini Sivaram]

Hi Ismael,

1. At the moment FetchDownConversionsPerSec is a topic metric while
MessageConversionTimeMs is a request metric which indicates Produce/Fetch
as a tag. Are you suggesting that we should convert
FetchDownConversionsPerSec to a request metric called MessageConversionsPerSec
for fetch requests?

2. TemporaryMessageSize for Produce/Fetch would indicate the space
allocated for conversions. For other requests, this metric will not be
created (unless we find a request where the size is large and this
information is useful).

Thank you,

Rajini


On Tue, Sep 5, 2017 at 4:55 PM, Ismael Juma  wrote:

> Thanks Rajini, +1 (binding) from me. Just a few minor comments:
>
> 1. FetchDownConversionsPerSec should probably be MessageConversionsPerSec
> with a request tag for consistency with MessageConversionsTimeMs. The text
> in that paragraph should also be updated to talk about message conversions
> instead of down conversions only.
>
> 2. What will TemporaryMemorySize represent for requests other than
> `ProduceRequest`?
>
> Ismael
>
> On Mon, Sep 4, 2017 at 2:09 PM, Rajini Sivaram 
> wrote:
>
> > All the suggestions on the discuss thread have been incorporated into the
> > KIP. Please let me know if you have any more concerns or else can we
> > proceed with voting for this KIP?
> >
> > Thank you,
> >
> > Rajini
> >
> > On Thu, Aug 24, 2017 at 6:50 PM, Rajini Sivaram  >
> > wrote:
> >
> > > Hi all,
> > >
> > > I would like to start the vote on KIP-188 that adds additional metrics
> to
> > > support health checks for Kafka Ops. Details are here:
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 188+-+Add+new+metrics+to+support+health+checks
> > >
> > > Thank you,
> > >
> > > Rajini
> > >
> > >
> >
>

Re: [DISCUSS] KIP-179: Change ReassignPartitionsCommand to use AdminClient

[Ted Yu]

bq. What about startPartitionAssignment() ?

Make sense.
startPartitionReassignment() seems to be better since the API deals with
reassignment.

Cheers

On Tue, Sep 5, 2017 at 9:39 AM, Tom Bentley  wrote:

> I've revised this KIP again:
>
> * Change the alterPartitionCounts() API to support passing an optional
> assignment for the new partitions (which is already supported by
> kafka-topics.sh). At the same time I didn't want the API to suggest it was
> possible to change the existing assignments in the same call (which isn't
> supported today).
>
> * Removed alterReplicationFactor(), since it's really just a special case
> of reassignPartitions(): In both cases it is necessary to give a complete
> partition assignment and both can be long running due to data movement.
>
>
> Outstanding questions:
>
> 1. Is the proposed alterInterReplicaThrottle() API really better than
> changing the throttle via alterConfigs()? It wouldn't be necessary to
> support alterConfigs() for all broker configs, just DynamicConfigs (which
> can't be set via the config file any way). Would it be a problem that
> triggering the reassignment required ClusterAction on the Cluster, but
> throttling the assignment required Alter on the Topic? What if a user had
> the former permission, but not the latter?
>
> 2. Is reassignPartitions() really the best name? I find the distinction
> between reassigning and just assigning to be of limited value, and
> "reassign" is misleading when the APIs now used for changing the
> replication factor too. Since the API is asynchronous/long running, it
> might be better to indicate that in the name some how. What about
> startPartitionAssignment()?
>
> I am, of course, interested in any other questions or comments people have.
>
>
>
>
> On 30 August 2017 at 16:17, Tom Bentley  wrote:
>
> > Hi all,
> >
> > I've updated the KIP as follows:
> >
> > * remove the APIs supporting progress reporting in favour of the APIs
> > being implemented in KIP-113.
> > * added some APIs to cover the existing functionality around throttling
> > inter-broker transfers, which was previously a TODO.
> >
> > To respond to Colin's suggestion:
> >
> > > TopicAlterationResult alterTopics(TopicAlterationOptions options,
> > >   Map alters)
> > >
> > > PartitionCountAlteration(int numPartitions) implements TopicAlteration
> > >
> > > ReplicationFactorAlteration(int repl) implements TopicAlteration
> > >
> > > ReassignPartitionsAlteration(...) implements TopicAlteration
> >
> > That is a workable alternative to providing 3 separate methods on the
> > AdminClient, but I don't see why it is objectively better. I don't see
> > clients commonly needing to do a mixture of alterations, and it assumes
> > that the options make sense for all alterations.
> >
> > Cheers,
> >
> > Tom
> >
> > On 22 August 2017 at 23:49, Colin McCabe  wrote:
> >
> >> On Wed, Aug 9, 2017, at 07:17, Tom Bentley wrote:
> >> > Hi Dong and Jun,
> >> >
> >> > Thanks again for your input in this discussion and on KIP-113. It's
> >> > difficult that discussion is split between this thread and the one for
> >> > KIP-113, but I'll try to respond on this thread to questions asked on
> >> > this
> >> > thread.
> >> >
> >> > It seems there is some consensus that the alterTopic() API is the
> wrong
> >> > thing, and it would make more sense to separate the different kinds of
> >> > alteration into separate APIs. It seems to me there is then a choice.
> >> >
> >> > 1. Have separate alterPartitionCount(), alterReplicationFactor() and
> >> > reassignPartitions() methods. This would most closely match the
> >> > facilities
> >> > currently offered by kafka-alter-topics and kafka-reassign-partitions.
> >> > 2. Just provide a reassignPartitions() method and infer from the shape
> >> of
> >> > the data passed to that that a change in replication factor and/or
> >> > partition count is required, as suggested by Dong.
> >> >
> >> > The choice we make here is also relevant to KIP-113 and KIP-178. By
> >> > choosing (1) we can change the replication factor or partition count
> >> > without providing an assignment and therefore are necessarily
> requiring
> >> > the
> >> > controller to make a decision for us about which broker (and, for 113,
> >> > which log directory and thus which disk) the replica should be reside.
> >> > There is then the matter of what criteria the controller should use to
> >> > make
> >> > that decision (a decision is also required on topic creation of
> course,
> >> > but
> >> > I'll try not to go there right now).
> >>
> >> Hmm.  One approach would be to have something like
> >>
> >> > TopicAlterationResult alterTopics(TopicAlterationOptions options,
> >> >   Map alters)
> >> >
> >> > PartitionCountAlteration(int numPartitions) implements TopicAlteration
> >> >
> >> > ReplicationFactorAlteration(int repl) implements TopicAlteration
> >> >
> >> > ReassignPartitionsAlteration(...) implements TopicA

Re: [VOTE] KIP-188 - Add new metrics to support health checks

[Ismael Juma]

Thanks Rajini.

1. I meant a topic metric, but we could have one for fetch and one for
produce differentiated by the additional tag. The advantage is that the
name would be consistent with the request metric for message conversions.
However, on closer inspection, this would make the name inconsistent with
the broker topic metrics:

val totalProduceRequestRate =
newMeter(BrokerTopicStats.TotalProduceRequestsPerSec, "requests",
TimeUnit.SECONDS, tags)
val totalFetchRequestRate =
newMeter(BrokerTopicStats.TotalFetchRequestsPerSec, "requests",
TimeUnit.SECONDS, tags)

So, we maybe we can instead go for FetchMessageConversionsPerSecond and
ProduceMessageConversionsPerSec.

2. Sounds good.

Ismael

On Tue, Sep 5, 2017 at 5:46 PM, Rajini Sivaram 
wrote:

> Hi Ismael,
>
> 1. At the moment FetchDownConversionsPerSec is a topic metric while
> MessageConversionTimeMs is a request metric which indicates Produce/Fetch
> as a tag. Are you suggesting that we should convert
> FetchDownConversionsPerSec to a request metric called
> MessageConversionsPerSec
> for fetch requests?
>
> 2. TemporaryMessageSize for Produce/Fetch would indicate the space
> allocated for conversions. For other requests, this metric will not be
> created (unless we find a request where the size is large and this
> information is useful).
>
> Thank you,
>
> Rajini
>
>
> On Tue, Sep 5, 2017 at 4:55 PM, Ismael Juma  wrote:
>
> > Thanks Rajini, +1 (binding) from me. Just a few minor comments:
> >
> > 1. FetchDownConversionsPerSec should probably be MessageConversionsPerSec
> > with a request tag for consistency with MessageConversionsTimeMs. The
> text
> > in that paragraph should also be updated to talk about message
> conversions
> > instead of down conversions only.
> >
> > 2. What will TemporaryMemorySize represent for requests other than
> > `ProduceRequest`?
> >
> > Ismael
> >
> > On Mon, Sep 4, 2017 at 2:09 PM, Rajini Sivaram 
> > wrote:
> >
> > > All the suggestions on the discuss thread have been incorporated into
> the
> > > KIP. Please let me know if you have any more concerns or else can we
> > > proceed with voting for this KIP?
> > >
> > > Thank you,
> > >
> > > Rajini
> > >
> > > On Thu, Aug 24, 2017 at 6:50 PM, Rajini Sivaram <
> rajinisiva...@gmail.com
> > >
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I would like to start the vote on KIP-188 that adds additional
> metrics
> > to
> > > > support health checks for Kafka Ops. Details are here:
> > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > 188+-+Add+new+metrics+to+support+health+checks
> > > >
> > > > Thank you,
> > > >
> > > > Rajini
> > > >
> > > >
> > >
> >
>

Re: [VOTE] KIP-189 - Improve principal builder interface and add support for SASL

[Rajini Sivaram]

+1 (binding)

Thank you for the KIP, Jason.

On Thu, Aug 31, 2017 at 8:33 PM, Mickael Maison 
wrote:

> +1 (non binding)
>
> Thanks for the KIP
>
> On Thu, Aug 31, 2017 at 10:10 AM, Jun Rao  wrote:
> > That's also fine as long as it's documented in the interface somehow. The
> > only thing with java Closeable is that it throws IOException. In most of
> > our usage, we use close() with no exception since it's not clear what you
> > will do when an exception is thrown in close().
> >
> > Thanks,
> >
> > Jun
> >
> > On Thu, Aug 31, 2017 at 10:02 AM, Jason Gustafson 
> > wrote:
> >
> >> Hey Jun,
> >>
> >> I will add a note to the KIP, but my plan was to dynamically check
> whether
> >> the KafkaPrincipalBuilder implemented Configurable or Closeable. That
> makes
> >> implementing close() and configure() optional. Does that seem
> reasonable?
> >>
> >> -Jason
> >>
> >> On Thu, Aug 31, 2017 at 9:53 AM, Jun Rao  wrote:
> >>
> >> > Hi, Jason,
> >> >
> >> > Thanks for the KIP. +1. Just one minor comment. It seems that the new
> >> > KafkaPrincipalBuilder
> >> > interface should support Configurable and close() as the existing
> >> > PrincipalBuilder?
> >> >
> >> > Jun
> >> >
> >> > On Wed, Aug 30, 2017 at 8:51 AM, Jason Gustafson 
> >> > wrote:
> >> >
> >> > > I'd like to open the vote for KIP-189:
> >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >> > > 189%3A+Improve+principal+builder+interface+and+add+
> support+for+SASL.
> >> > > Thanks to everyone who helped review.
> >> > >
> >> > > -Jason
> >> > >
> >> >
> >>
>

Re: [DISCUSS] KIP-179: Change ReassignPartitionsCommand to use AdminClient

[Colin McCabe]

On Tue, Sep 5, 2017, at 09:39, Tom Bentley wrote:
> I've revised this KIP again:
> 
> * Change the alterPartitionCounts() API to support passing an optional
> assignment for the new partitions (which is already supported by
> kafka-topics.sh). At the same time I didn't want the API to suggest it
> was
> possible to change the existing assignments in the same call (which isn't
> supported today).
> 
> * Removed alterReplicationFactor(), since it's really just a special case
> of reassignPartitions(): In both cases it is necessary to give a complete
> partition assignment and both can be long running due to data movement.

Thanks, Tom.

Do we expect that reducing the number of partitions will ever be
supported by this API?  It seems like decreasing would require a
different API-- one which supported data movement, had a "check status
of this operation" feature, etc. etc.  If this API is only ever going to
be used to increase the number of partitions, I think we should just
call it "increasePartitionCount" to avoid confusion.

>
> Outstanding questions:
> 
> 1. Is the proposed alterInterReplicaThrottle() API really better than
> changing the throttle via alterConfigs()?

That's a good point.  I would argue that we should just use alterConfigs
to set the broker configuration, rather than having a special RPC just
for this.  

> It wouldn't be necessary to
> support alterConfigs() for all broker configs, just DynamicConfigs (which
> can't be set via the config file any way).

Hmm.  For this purpose, it's not even necessary to support alterConfigs
for all dynamic configurations.  You could just support alterConfigs on
this particular dynamic configuration key.  It might be better to be
conservative here, since exposing all of our internals could lead to
compatibility problems later on.

> Would it be a problem that
> triggering the reassignment required ClusterAction on the Cluster, but
> throttling the assignment required Alter on the Topic? What if a user had
> the former permission, but not the latter?

We've been trying to reserve ClusterAction on Cluster for
broker-initiated operations.  Alter on Cluster is the ACL for "root
stuff" and I would argue that it should be what we use here.

For reconfiguring the broker, I think we should follow KIP-133 and use
AlterConfigs on the Broker resource.  (Of course, if you just use the
existing alterConfigs call, you get this without any special effort.)

> 
> 2. Is reassignPartitions() really the best name? I find the distinction
> between reassigning and just assigning to be of limited value, and
> "reassign" is misleading when the APIs now used for changing the
> replication factor too. Since the API is asynchronous/long running, it
> might be better to indicate that in the name some how. What about
> startPartitionAssignment()?

Good idea -- I like the idea of using "start" or "initiate" to indicate
that this is kicking off a long-running operation.

"reassign" seemed like a natural choice to me since this is changing an
existing assignment.  It was assigned (when the topic it was created)--
now it's being re-assigned.  If you change it to just "assign" then it
feels confusing to me.  A new user might ask if "assigning partitions"
is a step that they should take after creating a topic, similar to how
subscribing to topics is a step you take after creating a consumer.

Basically, "reassign" makes it clear that it is changing an assignment
that already exists.  "assign" leaves open the possibility that no
assignment exists (which of course we know is not true, but it would be
nice if the name reflected that.)

> On 30 August 2017 at 16:17, Tom Bentley  wrote:
> 
> > Hi all,
> >
> > I've updated the KIP as follows:
> >
> > * remove the APIs supporting progress reporting in favour of the APIs
> > being implemented in KIP-113.
> > * added some APIs to cover the existing functionality around throttling
> > inter-broker transfers, which was previously a TODO.
> >
> > To respond to Colin's suggestion:
> >
> > > TopicAlterationResult alterTopics(TopicAlterationOptions options,
> > >   Map alters)
> > >
> > > PartitionCountAlteration(int numPartitions) implements TopicAlteration
> > >
> > > ReplicationFactorAlteration(int repl) implements TopicAlteration
> > >
> > > ReassignPartitionsAlteration(...) implements TopicAlteration
> >
> > That is a workable alternative to providing 3 separate methods on the
> > AdminClient, but I don't see why it is objectively better. I don't see
> > clients commonly needing to do a mixture of alterations, and it assumes
> > that the options make sense for all alterations.

Yeah-- it was just an idea I had for an alternate API that grouped
things a little bit more.  I agree that it does constrain us to using
only options that make sense for all the operations, which is not ideal.
 So your existing split into separate functions probably makes more
sense.

cheers,
Colin

> >
> > Cheers,
> >
> > Tom
> >
> > On 22 Aug

Re: [VOTE] KIP-188 - Add new metrics to support health checks

[Rajini Sivaram]

Hi Ismael,

1. Yes, that makes sense. Updated the KIP to use FetchMessageConversionsPerSec
and ProduceMessageConversionsPerSec.

Thank you,

Rajini

On Tue, Sep 5, 2017 at 6:20 PM, Ismael Juma  wrote:

> Thanks Rajini.
>
> 1. I meant a topic metric, but we could have one for fetch and one for
> produce differentiated by the additional tag. The advantage is that the
> name would be consistent with the request metric for message conversions.
> However, on closer inspection, this would make the name inconsistent with
> the broker topic metrics:
>
> val totalProduceRequestRate =
> newMeter(BrokerTopicStats.TotalProduceRequestsPerSec, "requests",
> TimeUnit.SECONDS, tags)
> val totalFetchRequestRate =
> newMeter(BrokerTopicStats.TotalFetchRequestsPerSec, "requests",
> TimeUnit.SECONDS, tags)
>
> So, we maybe we can instead go for FetchMessageConversionsPerSecond and
> ProduceMessageConversionsPerSec.
>
> 2. Sounds good.
>
> Ismael
>
> On Tue, Sep 5, 2017 at 5:46 PM, Rajini Sivaram 
> wrote:
>
> > Hi Ismael,
> >
> > 1. At the moment FetchDownConversionsPerSec is a topic metric while
> > MessageConversionTimeMs is a request metric which indicates Produce/Fetch
> > as a tag. Are you suggesting that we should convert
> > FetchDownConversionsPerSec to a request metric called
> > MessageConversionsPerSec
> > for fetch requests?
> >
> > 2. TemporaryMessageSize for Produce/Fetch would indicate the space
> > allocated for conversions. For other requests, this metric will not be
> > created (unless we find a request where the size is large and this
> > information is useful).
> >
> > Thank you,
> >
> > Rajini
> >
> >
> > On Tue, Sep 5, 2017 at 4:55 PM, Ismael Juma  wrote:
> >
> > > Thanks Rajini, +1 (binding) from me. Just a few minor comments:
> > >
> > > 1. FetchDownConversionsPerSec should probably be
> MessageConversionsPerSec
> > > with a request tag for consistency with MessageConversionsTimeMs. The
> > text
> > > in that paragraph should also be updated to talk about message
> > conversions
> > > instead of down conversions only.
> > >
> > > 2. What will TemporaryMemorySize represent for requests other than
> > > `ProduceRequest`?
> > >
> > > Ismael
> > >
> > > On Mon, Sep 4, 2017 at 2:09 PM, Rajini Sivaram <
> rajinisiva...@gmail.com>
> > > wrote:
> > >
> > > > All the suggestions on the discuss thread have been incorporated into
> > the
> > > > KIP. Please let me know if you have any more concerns or else can we
> > > > proceed with voting for this KIP?
> > > >
> > > > Thank you,
> > > >
> > > > Rajini
> > > >
> > > > On Thu, Aug 24, 2017 at 6:50 PM, Rajini Sivaram <
> > rajinisiva...@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I would like to start the vote on KIP-188 that adds additional
> > metrics
> > > to
> > > > > support health checks for Kafka Ops. Details are here:
> > > > >
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > 188+-+Add+new+metrics+to+support+health+checks
> > > > >
> > > > > Thank you,
> > > > >
> > > > > Rajini
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-170: Enhanced TopicCreatePolicy and introduction of TopicDeletePolicy

[Edoardo Comar]

Hi Ismael,
I was on leave for a long while. I will update the KIP.

Edo

On 5 September 2017 at 11:42, Ismael Juma  wrote:

> Hi Edoardo,
>
> Do you intend to update the KIP to avoid the introduction of another
> interface?
>
> Ismael
>
> On Fri, Jun 23, 2017 at 5:47 PM, Edoardo Comar  wrote:
>
> > Thanks for the thoughts Ismael
> >
> > > 1. Have you considered extending RequestMetadata with the additional
> > > information you need? We could add Cluster to it, which has topic
> > > assignment information, for example. This way, there would be no need
> > for a
> > > V2 interface.
> >
> > 1. I hadn't thought of it - but this seems a feasible alternative.
> >
> > The XXXPolicy.RequestMetadata could be enriched to include more
> > information -
> > for backward compatibility with existing Policies we would only add
> > methods to these classes.
> >
> > Presumably the number of Admin Request is not huge so we should not be
> too
> > worried
> > about the overhead of populating a org.apache.kafka.common.Cluster for
> > every admin request , right ?
> >
> > The only feature I can't see how to easily support with the enriched
> > XXXPolicy.RequestMetadata is how to check
> > for a topic being marked for deletion using the Cluster information
> >
> > This check is useful for us in excluding such outliers when counting the
> > number of partitions.
> >
> >
> > > 2. Something else that could be useful is passing an instance of
> > `Session`
> > > so that one can provide custom behaviour depending on the logged in
> > user.
> > > Would this be useful?
> > 2. Definitely I would expect that a general solution includes the Session
> > or the KafkaPrincipal associated with it
> > (the latter may be a simpler dependency for the XXXPolicy.RequestMetadata
> >
> >
> > > 3. For the delete case, we may consider passing a class instead of just
> > a
> > > string to the validate method so that we have options if we need to
> > extend
> > > it.
> >
> > 3. Agree, we should have the DeletePolicy define its RequestMetadata
> > class, too
> >
> >
> > > 4. Do we want to enhance the AlterConfigs policy as well?
> >
> > 4. I don't see why not :-)
> >
> >
> >
> > thanks
> > Edo
> > --
> >
> > Edoardo Comar
> >
> > IBM Message Hub
> >
> >
> > isma...@gmail.com wrote on 22/06/2017 15:05:06:
> >
> > > From: Ismael Juma 
> > > To: dev@kafka.apache.org
> > > Date: 22/06/2017 15:05
> > > Subject: Re: [DISCUSS] KIP-170: Enhanced TopicCreatePolicy and
> > > introduction of TopicDeletePolicy
> > > Sent by: isma...@gmail.com
> > >
> > > Thanks for the KIP, Edoardo. A few comments:
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> > > Ismael
> > >
> > > On Thu, Jun 22, 2017 at 2:41 PM, Edoardo Comar 
> > wrote:
> > >
> > > > Hi all,
> > > >
> > > > We've drafted "KIP-170: Enhanced TopicCreatePolicy and introduction
> of
> > > > TopicDeletePolicy" for discussion:
> > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 170%3A+Enhanced+
> > > > TopicCreatePolicy+and+introduction+of+TopicDeletePolicy
> > > >
> > > > Please take a look. Your feedback is welcome and much needed.
> > > >
> > > > Thanks,
> > > > Edoardo
> > > > --
> > > > Edoardo Comar
> > > > IBM Message Hub
> > > > eco...@uk.ibm.com
> > > > IBM UK Ltd, Hursley Park, SO21 2JN
> > > > Unless stated otherwise above:
> > > > IBM United Kingdom Limited - Registered in England and Wales with
> > number
> > > > 741598.
> > > > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire
> PO6
> > 3AU
> > > >
> >
> > Unless stated otherwise above:
> > IBM United Kingdom Limited - Registered in England and Wales with number
> > 741598.
> > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6
> 3AU
> >
>



-- 
"When the people fear their government, there is tyranny; when the
government fears the people, there is liberty." [Thomas Jefferson]

Re: [VOTE] KIP-188 - Add new metrics to support health checks

[Jason Gustafson]

+1 Lots of good stuff in here.

One minor nit: in the name `FetchDownConversionsPerSec`, it's implicit that
down-conversion is for messages. Could we do the same for
`MessageConversionsTimeMs` and drop the `Message`? Then we don't have to
decide if it should be 'Record' instead.

On Tue, Sep 5, 2017 at 10:20 AM, Ismael Juma  wrote:

> Thanks Rajini.
>
> 1. I meant a topic metric, but we could have one for fetch and one for
> produce differentiated by the additional tag. The advantage is that the
> name would be consistent with the request metric for message conversions.
> However, on closer inspection, this would make the name inconsistent with
> the broker topic metrics:
>
> val totalProduceRequestRate =
> newMeter(BrokerTopicStats.TotalProduceRequestsPerSec, "requests",
> TimeUnit.SECONDS, tags)
> val totalFetchRequestRate =
> newMeter(BrokerTopicStats.TotalFetchRequestsPerSec, "requests",
> TimeUnit.SECONDS, tags)
>
> So, we maybe we can instead go for FetchMessageConversionsPerSecond and
> ProduceMessageConversionsPerSec.
>
> 2. Sounds good.
>
> Ismael
>
> On Tue, Sep 5, 2017 at 5:46 PM, Rajini Sivaram 
> wrote:
>
> > Hi Ismael,
> >
> > 1. At the moment FetchDownConversionsPerSec is a topic metric while
> > MessageConversionTimeMs is a request metric which indicates Produce/Fetch
> > as a tag. Are you suggesting that we should convert
> > FetchDownConversionsPerSec to a request metric called
> > MessageConversionsPerSec
> > for fetch requests?
> >
> > 2. TemporaryMessageSize for Produce/Fetch would indicate the space
> > allocated for conversions. For other requests, this metric will not be
> > created (unless we find a request where the size is large and this
> > information is useful).
> >
> > Thank you,
> >
> > Rajini
> >
> >
> > On Tue, Sep 5, 2017 at 4:55 PM, Ismael Juma  wrote:
> >
> > > Thanks Rajini, +1 (binding) from me. Just a few minor comments:
> > >
> > > 1. FetchDownConversionsPerSec should probably be
> MessageConversionsPerSec
> > > with a request tag for consistency with MessageConversionsTimeMs. The
> > text
> > > in that paragraph should also be updated to talk about message
> > conversions
> > > instead of down conversions only.
> > >
> > > 2. What will TemporaryMemorySize represent for requests other than
> > > `ProduceRequest`?
> > >
> > > Ismael
> > >
> > > On Mon, Sep 4, 2017 at 2:09 PM, Rajini Sivaram <
> rajinisiva...@gmail.com>
> > > wrote:
> > >
> > > > All the suggestions on the discuss thread have been incorporated into
> > the
> > > > KIP. Please let me know if you have any more concerns or else can we
> > > > proceed with voting for this KIP?
> > > >
> > > > Thank you,
> > > >
> > > > Rajini
> > > >
> > > > On Thu, Aug 24, 2017 at 6:50 PM, Rajini Sivaram <
> > rajinisiva...@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I would like to start the vote on KIP-188 that adds additional
> > metrics
> > > to
> > > > > support health checks for Kafka Ops. Details are here:
> > > > >
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > 188+-+Add+new+metrics+to+support+health+checks
> > > > >
> > > > > Thank you,
> > > > >
> > > > > Rajini
> > > > >
> > > > >
> > > >
> > >
> >
>

[GitHub] kafka pull request #3793: MINOR: add mvn-pgp-plugin to sign streams quicksta...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3793


---

Re: [VOTE] KIP-188 - Add new metrics to support health checks

[Mickael Maison]

+1 (non binding)

On Tue, Sep 5, 2017 at 6:51 PM, Jason Gustafson  wrote:
> +1 Lots of good stuff in here.
>
> One minor nit: in the name `FetchDownConversionsPerSec`, it's implicit that
> down-conversion is for messages. Could we do the same for
> `MessageConversionsTimeMs` and drop the `Message`? Then we don't have to
> decide if it should be 'Record' instead.
>
> On Tue, Sep 5, 2017 at 10:20 AM, Ismael Juma  wrote:
>
>> Thanks Rajini.
>>
>> 1. I meant a topic metric, but we could have one for fetch and one for
>> produce differentiated by the additional tag. The advantage is that the
>> name would be consistent with the request metric for message conversions.
>> However, on closer inspection, this would make the name inconsistent with
>> the broker topic metrics:
>>
>> val totalProduceRequestRate =
>> newMeter(BrokerTopicStats.TotalProduceRequestsPerSec, "requests",
>> TimeUnit.SECONDS, tags)
>> val totalFetchRequestRate =
>> newMeter(BrokerTopicStats.TotalFetchRequestsPerSec, "requests",
>> TimeUnit.SECONDS, tags)
>>
>> So, we maybe we can instead go for FetchMessageConversionsPerSecond and
>> ProduceMessageConversionsPerSec.
>>
>> 2. Sounds good.
>>
>> Ismael
>>
>> On Tue, Sep 5, 2017 at 5:46 PM, Rajini Sivaram 
>> wrote:
>>
>> > Hi Ismael,
>> >
>> > 1. At the moment FetchDownConversionsPerSec is a topic metric while
>> > MessageConversionTimeMs is a request metric which indicates Produce/Fetch
>> > as a tag. Are you suggesting that we should convert
>> > FetchDownConversionsPerSec to a request metric called
>> > MessageConversionsPerSec
>> > for fetch requests?
>> >
>> > 2. TemporaryMessageSize for Produce/Fetch would indicate the space
>> > allocated for conversions. For other requests, this metric will not be
>> > created (unless we find a request where the size is large and this
>> > information is useful).
>> >
>> > Thank you,
>> >
>> > Rajini
>> >
>> >
>> > On Tue, Sep 5, 2017 at 4:55 PM, Ismael Juma  wrote:
>> >
>> > > Thanks Rajini, +1 (binding) from me. Just a few minor comments:
>> > >
>> > > 1. FetchDownConversionsPerSec should probably be
>> MessageConversionsPerSec
>> > > with a request tag for consistency with MessageConversionsTimeMs. The
>> > text
>> > > in that paragraph should also be updated to talk about message
>> > conversions
>> > > instead of down conversions only.
>> > >
>> > > 2. What will TemporaryMemorySize represent for requests other than
>> > > `ProduceRequest`?
>> > >
>> > > Ismael
>> > >
>> > > On Mon, Sep 4, 2017 at 2:09 PM, Rajini Sivaram <
>> rajinisiva...@gmail.com>
>> > > wrote:
>> > >
>> > > > All the suggestions on the discuss thread have been incorporated into
>> > the
>> > > > KIP. Please let me know if you have any more concerns or else can we
>> > > > proceed with voting for this KIP?
>> > > >
>> > > > Thank you,
>> > > >
>> > > > Rajini
>> > > >
>> > > > On Thu, Aug 24, 2017 at 6:50 PM, Rajini Sivaram <
>> > rajinisiva...@gmail.com
>> > > >
>> > > > wrote:
>> > > >
>> > > > > Hi all,
>> > > > >
>> > > > > I would like to start the vote on KIP-188 that adds additional
>> > metrics
>> > > to
>> > > > > support health checks for Kafka Ops. Details are here:
>> > > > >
>> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> > > > > 188+-+Add+new+metrics+to+support+health+checks
>> > > > >
>> > > > > Thank you,
>> > > > >
>> > > > > Rajini
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>>

Re: [DISCUSS] KIP-170: Enhanced TopicCreatePolicy and introduction of TopicDeletePolicy

[Tom Bentley]

Hi Edoardo,

KIP-179 will support changing topic configurations via the AdminClient and
this would open a loophole to avoid the current CreateTopicPolicy:

1. Create a topic that sticks to the policy
2. Modify the topic after creation to something which would have violated
the policy.

For this reason KIP-179 proposes to apply the create topic policy to topic
modification too. If there were going to be a new "V2" interface (as
opposed to changing the RequestMetadata) I think it would be beneficial if
the name reflected that it applied to updates too.
CreateOrAlterTopicPolicy? Though obviously that name doesn't reflect that
its heritage in the original CreateTopicPolicy.

Cheers,

Tom

On 5 September 2017 at 18:48, Edoardo Comar  wrote:

> Hi Ismael,
> I was on leave for a long while. I will update the KIP.
>
> Edo
>
> On 5 September 2017 at 11:42, Ismael Juma  wrote:
>
> > Hi Edoardo,
> >
> > Do you intend to update the KIP to avoid the introduction of another
> > interface?
> >
> > Ismael
> >
> > On Fri, Jun 23, 2017 at 5:47 PM, Edoardo Comar 
> wrote:
> >
> > > Thanks for the thoughts Ismael
> > >
> > > > 1. Have you considered extending RequestMetadata with the additional
> > > > information you need? We could add Cluster to it, which has topic
> > > > assignment information, for example. This way, there would be no need
> > > for a
> > > > V2 interface.
> > >
> > > 1. I hadn't thought of it - but this seems a feasible alternative.
> > >
> > > The XXXPolicy.RequestMetadata could be enriched to include more
> > > information -
> > > for backward compatibility with existing Policies we would only add
> > > methods to these classes.
> > >
> > > Presumably the number of Admin Request is not huge so we should not be
> > too
> > > worried
> > > about the overhead of populating a org.apache.kafka.common.Cluster for
> > > every admin request , right ?
> > >
> > > The only feature I can't see how to easily support with the enriched
> > > XXXPolicy.RequestMetadata is how to check
> > > for a topic being marked for deletion using the Cluster information
> > >
> > > This check is useful for us in excluding such outliers when counting
> the
> > > number of partitions.
> > >
> > >
> > > > 2. Something else that could be useful is passing an instance of
> > > `Session`
> > > > so that one can provide custom behaviour depending on the logged in
> > > user.
> > > > Would this be useful?
> > > 2. Definitely I would expect that a general solution includes the
> Session
> > > or the KafkaPrincipal associated with it
> > > (the latter may be a simpler dependency for the
> XXXPolicy.RequestMetadata
> > >
> > >
> > > > 3. For the delete case, we may consider passing a class instead of
> just
> > > a
> > > > string to the validate method so that we have options if we need to
> > > extend
> > > > it.
> > >
> > > 3. Agree, we should have the DeletePolicy define its RequestMetadata
> > > class, too
> > >
> > >
> > > > 4. Do we want to enhance the AlterConfigs policy as well?
> > >
> > > 4. I don't see why not :-)
> > >
> > >
> > >
> > > thanks
> > > Edo
> > > --
> > >
> > > Edoardo Comar
> > >
> > > IBM Message Hub
> > >
> > >
> > > isma...@gmail.com wrote on 22/06/2017 15:05:06:
> > >
> > > > From: Ismael Juma 
> > > > To: dev@kafka.apache.org
> > > > Date: 22/06/2017 15:05
> > > > Subject: Re: [DISCUSS] KIP-170: Enhanced TopicCreatePolicy and
> > > > introduction of TopicDeletePolicy
> > > > Sent by: isma...@gmail.com
> > > >
> > > > Thanks for the KIP, Edoardo. A few comments:
> > > >
> > >
> > > >
> > >
> > > >
> > >
> > > >
> > >
> > > >
> > > > Ismael
> > > >
> > > > On Thu, Jun 22, 2017 at 2:41 PM, Edoardo Comar 
> > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > We've drafted "KIP-170: Enhanced TopicCreatePolicy and introduction
> > of
> > > > > TopicDeletePolicy" for discussion:
> > > > >
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 170%3A+Enhanced+
> > > > > TopicCreatePolicy+and+introduction+of+TopicDeletePolicy
> > > > >
> > > > > Please take a look. Your feedback is welcome and much needed.
> > > > >
> > > > > Thanks,
> > > > > Edoardo
> > > > > --
> > > > > Edoardo Comar
> > > > > IBM Message Hub
> > > > > eco...@uk.ibm.com
> > > > > IBM UK Ltd, Hursley Park, SO21 2JN
> > > > > Unless stated otherwise above:
> > > > > IBM United Kingdom Limited - Registered in England and Wales with
> > > number
> > > > > 741598.
> > > > > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire
> > PO6
> > > 3AU
> > > > >
> > >
> > > Unless stated otherwise above:
> > > IBM United Kingdom Limited - Registered in England and Wales with
> number
> > > 741598.
> > > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6
> > 3AU
> > >
> >
>
>
>
> --
> "When the people fear their government, there is tyranny; when the
> government fears the people, there is liberty." [Thomas Jefferson]

Re: [VOTE] KIP-188 - Add new metrics to support health checks

[Ismael Juma]

I was wondering about the message versus record question. The fact that we
already have MessagesInPerSec seemed to favour the former. The other aspect
is that for produce requests, we can up convert as well, so it seemed
better to keep it generic.

Ismael

On Tue, Sep 5, 2017 at 6:51 PM, Jason Gustafson  wrote:

> +1 Lots of good stuff in here.
>
> One minor nit: in the name `FetchDownConversionsPerSec`, it's implicit that
> down-conversion is for messages. Could we do the same for
> `MessageConversionsTimeMs` and drop the `Message`? Then we don't have to
> decide if it should be 'Record' instead.
>
> On Tue, Sep 5, 2017 at 10:20 AM, Ismael Juma  wrote:
>
> > Thanks Rajini.
> >
> > 1. I meant a topic metric, but we could have one for fetch and one for
> > produce differentiated by the additional tag. The advantage is that the
> > name would be consistent with the request metric for message conversions.
> > However, on closer inspection, this would make the name inconsistent with
> > the broker topic metrics:
> >
> > val totalProduceRequestRate =
> > newMeter(BrokerTopicStats.TotalProduceRequestsPerSec, "requests",
> > TimeUnit.SECONDS, tags)
> > val totalFetchRequestRate =
> > newMeter(BrokerTopicStats.TotalFetchRequestsPerSec, "requests",
> > TimeUnit.SECONDS, tags)
> >
> > So, we maybe we can instead go for FetchMessageConversionsPerSecond and
> > ProduceMessageConversionsPerSec.
> >
> > 2. Sounds good.
> >
> > Ismael
> >
> > On Tue, Sep 5, 2017 at 5:46 PM, Rajini Sivaram 
> > wrote:
> >
> > > Hi Ismael,
> > >
> > > 1. At the moment FetchDownConversionsPerSec is a topic metric while
> > > MessageConversionTimeMs is a request metric which indicates
> Produce/Fetch
> > > as a tag. Are you suggesting that we should convert
> > > FetchDownConversionsPerSec to a request metric called
> > > MessageConversionsPerSec
> > > for fetch requests?
> > >
> > > 2. TemporaryMessageSize for Produce/Fetch would indicate the space
> > > allocated for conversions. For other requests, this metric will not be
> > > created (unless we find a request where the size is large and this
> > > information is useful).
> > >
> > > Thank you,
> > >
> > > Rajini
> > >
> > >
> > > On Tue, Sep 5, 2017 at 4:55 PM, Ismael Juma  wrote:
> > >
> > > > Thanks Rajini, +1 (binding) from me. Just a few minor comments:
> > > >
> > > > 1. FetchDownConversionsPerSec should probably be
> > MessageConversionsPerSec
> > > > with a request tag for consistency with MessageConversionsTimeMs. The
> > > text
> > > > in that paragraph should also be updated to talk about message
> > > conversions
> > > > instead of down conversions only.
> > > >
> > > > 2. What will TemporaryMemorySize represent for requests other than
> > > > `ProduceRequest`?
> > > >
> > > > Ismael
> > > >
> > > > On Mon, Sep 4, 2017 at 2:09 PM, Rajini Sivaram <
> > rajinisiva...@gmail.com>
> > > > wrote:
> > > >
> > > > > All the suggestions on the discuss thread have been incorporated
> into
> > > the
> > > > > KIP. Please let me know if you have any more concerns or else can
> we
> > > > > proceed with voting for this KIP?
> > > > >
> > > > > Thank you,
> > > > >
> > > > > Rajini
> > > > >
> > > > > On Thu, Aug 24, 2017 at 6:50 PM, Rajini Sivaram <
> > > rajinisiva...@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I would like to start the vote on KIP-188 that adds additional
> > > metrics
> > > > to
> > > > > > support health checks for Kafka Ops. Details are here:
> > > > > >
> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > > 188+-+Add+new+metrics+to+support+health+checks
> > > > > >
> > > > > > Thank you,
> > > > > >
> > > > > > Rajini
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] KIP-189 - Improve principal builder interface and add support for SASL

[Ismael Juma]

Thanks for the KIP, +1 (binding).

Ismael

On Wed, Aug 30, 2017 at 4:51 PM, Jason Gustafson  wrote:

> I'd like to open the vote for KIP-189:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 189%3A+Improve+principal+builder+interface+and+add+support+for+SASL.
> Thanks to everyone who helped review.
>
> -Jason
>

[jira] [Created] (KAFKA-5838) Speed up running system tests in docker a bit

[Colin P. McCabe (JIRA)]

Colin P. McCabe created KAFKA-5838:
--

 Summary: Speed up running system tests in docker a bit
 Key: KAFKA-5838
 URL: https://issues.apache.org/jira/browse/KAFKA-5838
 Project: Kafka
  Issue Type: Bug
Reporter: Colin P. McCabe
Assignee: Colin P. McCabe


Speed up running system tests in docker a bit by using optimized sshd options.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] kafka pull request #3794: KAFKA-5838. Speed up running system tests in docke...

[cmccabe]

GitHub user cmccabe opened a pull request:

https://github.com/apache/kafka/pull/3794

KAFKA-5838. Speed up running system tests in docker a bit with better…

… sshd options

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cmccabe/kafka KAFKA-5838

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3794.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3794


commit e610cf6386a6a273e2d0ca53f9c725807f213cb2
Author: Colin P. Mccabe 
Date:   2017-09-05T18:13:56Z

KAFKA-5838. Speed up running system tests in docker a bit with better sshd 
options




---

Re: [DISCUSS] KIP-192 - Provide cleaner semantics when idempotence is enabled

[Jason Gustafson]

The proposal looks good. Two minor comments:

1. Can we call out how we handle the duplicate case? This is a change in
behavior since we currently raise OutOfOrderSequence in this case.

2. Instead of passing through `idempotenceLevel` in the ProduceRequest, I
wonder if we should have a field for the minimum required message format.
When using enable.idempotence=required, we could set the minimum required
version to v2. For enable.idempotence=requested, we could use v0. The
advantage is that we may find other uses for a more general field in the
future. Alternatively, maybe we really should be returning the message
format version of each topic in the TopicMetadata response. A nice bonus of
doing so is that it gives the producer the ability to craft the right
format version ahead of time and avoid the need for conversion on the
broker.

Thanks,
Jason

On Tue, Aug 29, 2017 at 4:32 PM, Apurva Mehta  wrote:

> Hi,
>
> In the discussion of KIP-185 (enable idempotence by default), we discovered
> some shortcomings of the existing idempotent producer implementation.
> Fixing these issues requires changes to the ProduceRequest and
> ProduceResponse protocols as well as changes to the values of the
> 'enable.idempotence' producer config.
>
> Hence, I split off those changes into another KIP so as to decouple the two
> issues. Please have a look at my follow up KIP below:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 192+%3A+Provide+cleaner+semantics+when+idempotence+is+enabled
>
> KIP-185 depends on KIP-192, and I hope to make progress on the latter
> independently.
>
> Thanks,
> Apurva
>

Re: [VOTE] KIP-188 - Add new metrics to support health checks

[Jason Gustafson]

>
> I was wondering about the message versus record question. The fact that we
> already have MessagesInPerSec seemed to favour the former. The other aspect
> is that for produce requests, we can up convert as well, so it seemed
> better to keep it generic.


Yeah, so I thought maybe we could bypass the question and drop `Message`
from the names if they were already clear enough. I'm fine with either way.

On Tue, Sep 5, 2017 at 11:09 AM, Ismael Juma  wrote:

> I was wondering about the message versus record question. The fact that we
> already have MessagesInPerSec seemed to favour the former. The other aspect
> is that for produce requests, we can up convert as well, so it seemed
> better to keep it generic.
>
> Ismael
>
> On Tue, Sep 5, 2017 at 6:51 PM, Jason Gustafson 
> wrote:
>
> > +1 Lots of good stuff in here.
> >
> > One minor nit: in the name `FetchDownConversionsPerSec`, it's implicit
> that
> > down-conversion is for messages. Could we do the same for
> > `MessageConversionsTimeMs` and drop the `Message`? Then we don't have to
> > decide if it should be 'Record' instead.
> >
> > On Tue, Sep 5, 2017 at 10:20 AM, Ismael Juma  wrote:
> >
> > > Thanks Rajini.
> > >
> > > 1. I meant a topic metric, but we could have one for fetch and one for
> > > produce differentiated by the additional tag. The advantage is that the
> > > name would be consistent with the request metric for message
> conversions.
> > > However, on closer inspection, this would make the name inconsistent
> with
> > > the broker topic metrics:
> > >
> > > val totalProduceRequestRate =
> > > newMeter(BrokerTopicStats.TotalProduceRequestsPerSec, "requests",
> > > TimeUnit.SECONDS, tags)
> > > val totalFetchRequestRate =
> > > newMeter(BrokerTopicStats.TotalFetchRequestsPerSec, "requests",
> > > TimeUnit.SECONDS, tags)
> > >
> > > So, we maybe we can instead go for FetchMessageConversionsPerSecond
> and
> > > ProduceMessageConversionsPerSec.
> > >
> > > 2. Sounds good.
> > >
> > > Ismael
> > >
> > > On Tue, Sep 5, 2017 at 5:46 PM, Rajini Sivaram <
> rajinisiva...@gmail.com>
> > > wrote:
> > >
> > > > Hi Ismael,
> > > >
> > > > 1. At the moment FetchDownConversionsPerSec is a topic metric while
> > > > MessageConversionTimeMs is a request metric which indicates
> > Produce/Fetch
> > > > as a tag. Are you suggesting that we should convert
> > > > FetchDownConversionsPerSec to a request metric called
> > > > MessageConversionsPerSec
> > > > for fetch requests?
> > > >
> > > > 2. TemporaryMessageSize for Produce/Fetch would indicate the space
> > > > allocated for conversions. For other requests, this metric will not
> be
> > > > created (unless we find a request where the size is large and this
> > > > information is useful).
> > > >
> > > > Thank you,
> > > >
> > > > Rajini
> > > >
> > > >
> > > > On Tue, Sep 5, 2017 at 4:55 PM, Ismael Juma 
> wrote:
> > > >
> > > > > Thanks Rajini, +1 (binding) from me. Just a few minor comments:
> > > > >
> > > > > 1. FetchDownConversionsPerSec should probably be
> > > MessageConversionsPerSec
> > > > > with a request tag for consistency with MessageConversionsTimeMs.
> The
> > > > text
> > > > > in that paragraph should also be updated to talk about message
> > > > conversions
> > > > > instead of down conversions only.
> > > > >
> > > > > 2. What will TemporaryMemorySize represent for requests other than
> > > > > `ProduceRequest`?
> > > > >
> > > > > Ismael
> > > > >
> > > > > On Mon, Sep 4, 2017 at 2:09 PM, Rajini Sivaram <
> > > rajinisiva...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > All the suggestions on the discuss thread have been incorporated
> > into
> > > > the
> > > > > > KIP. Please let me know if you have any more concerns or else can
> > we
> > > > > > proceed with voting for this KIP?
> > > > > >
> > > > > > Thank you,
> > > > > >
> > > > > > Rajini
> > > > > >
> > > > > > On Thu, Aug 24, 2017 at 6:50 PM, Rajini Sivaram <
> > > > rajinisiva...@gmail.com
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > I would like to start the vote on KIP-188 that adds additional
> > > > metrics
> > > > > to
> > > > > > > support health checks for Kafka Ops. Details are here:
> > > > > > >
> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > > > 188+-+Add+new+metrics+to+support+health+checks
> > > > > > >
> > > > > > > Thank you,
> > > > > > >
> > > > > > > Rajini
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-192 - Provide cleaner semantics when idempotence is enabled

[Ismael Juma]

If we add the message format version (a topic config) in the response of
TopicMetadata, we should consider adding the max message bytes as well.
That would allow us to later improve the implementation of KIP-126 to split
the batch _before_ sending.

Ismael

On Tue, Sep 5, 2017 at 7:17 PM, Jason Gustafson  wrote:

> The proposal looks good. Two minor comments:
>
> 1. Can we call out how we handle the duplicate case? This is a change in
> behavior since we currently raise OutOfOrderSequence in this case.
>
> 2. Instead of passing through `idempotenceLevel` in the ProduceRequest, I
> wonder if we should have a field for the minimum required message format.
> When using enable.idempotence=required, we could set the minimum required
> version to v2. For enable.idempotence=requested, we could use v0. The
> advantage is that we may find other uses for a more general field in the
> future. Alternatively, maybe we really should be returning the message
> format version of each topic in the TopicMetadata response. A nice bonus of
> doing so is that it gives the producer the ability to craft the right
> format version ahead of time and avoid the need for conversion on the
> broker.
>
> Thanks,
> Jason
>
> On Tue, Aug 29, 2017 at 4:32 PM, Apurva Mehta  wrote:
>
> > Hi,
> >
> > In the discussion of KIP-185 (enable idempotence by default), we
> discovered
> > some shortcomings of the existing idempotent producer implementation.
> > Fixing these issues requires changes to the ProduceRequest and
> > ProduceResponse protocols as well as changes to the values of the
> > 'enable.idempotence' producer config.
> >
> > Hence, I split off those changes into another KIP so as to decouple the
> two
> > issues. Please have a look at my follow up KIP below:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 192+%3A+Provide+cleaner+semantics+when+idempotence+is+enabled
> >
> > KIP-185 depends on KIP-192, and I hope to make progress on the latter
> > independently.
> >
> > Thanks,
> > Apurva
> >
>

Build failed in Jenkins: kafka-trunk-jdk8 #1973

[Apache Jenkins Server]

See 


Changes:

[wangguoz] MINOR: add mvn-pgp-plugin to sign streams quickstart jars

--
[...truncated 892.47 KB...]

kafka.security.auth.SimpleAclAuthorizerTest > testAclInheritance PASSED

kafka.security.auth.SimpleAclAuthorizerTest > 
testDistributedConcurrentModificationOfResourceAcls STARTED

kafka.security.auth.SimpleAclAuthorizerTest > 
testDistributedConcurrentModificationOfResourceAcls PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testAclManagementAPIs STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testAclManagementAPIs PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testWildCardAcls STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testWildCardAcls PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testTopicAcl STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testTopicAcl PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testSuperUserHasAccess STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testSuperUserHasAccess PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testDenyTakesPrecedence STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testDenyTakesPrecedence PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testNoAclFoundOverride STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testNoAclFoundOverride PASSED

kafka.security.auth.SimpleAclAuthorizerTest > 
testHighConcurrencyModificationOfResourceAcls STARTED

kafka.security.auth.SimpleAclAuthorizerTest > 
testHighConcurrencyModificationOfResourceAcls PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testLoadCache STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testLoadCache PASSED

kafka.security.auth.OperationTest > testJavaConversions STARTED

kafka.security.auth.OperationTest > testJavaConversions PASSED

kafka.admin.AclCommandTest > testInvalidAuthorizerProperty STARTED

kafka.admin.AclCommandTest > testInvalidAuthorizerProperty PASSED

kafka.admin.AclCommandTest > testAclCli STARTED

kafka.admin.AclCommandTest > testAclCli PASSED

kafka.admin.AclCommandTest > testProducerConsumerCli STARTED

kafka.admin.AclCommandTest > testProducerConsumerCli PASSED

kafka.admin.ListConsumerGroupTest > testListGroupWithSomeGroups STARTED

kafka.admin.ListConsumerGroupTest > testListGroupWithSomeGroups PASSED

kafka.admin.ListConsumerGroupTest > testListGroupWithNoExistingGroup STARTED

kafka.admin.ListConsumerGroupTest > testListGroupWithNoExistingGroup PASSED

kafka.admin.ReassignPartitionsClusterTest > shouldExecuteThrottledReassignment 
STARTED

kafka.admin.ReassignPartitionsClusterTest > shouldExecuteThrottledReassignment 
PASSED

kafka.admin.ReassignPartitionsClusterTest > 
shouldFailIfProposedDoesNotMatchExisting STARTED

kafka.admin.ReassignPartitionsClusterTest > 
shouldFailIfProposedDoesNotMatchExisting PASSED

kafka.admin.ReassignPartitionsClusterTest > 
shouldFailIfProposedHasInconsistentReplicasAndLogDirs STARTED

kafka.admin.ReassignPartitionsClusterTest > 
shouldFailIfProposedHasInconsistentReplicasAndLogDirs PASSED

kafka.admin.ReassignPartitionsClusterTest > shouldOnlyThrottleMovingReplicas 
STARTED

kafka.admin.ReassignPartitionsClusterTest > shouldOnlyThrottleMovingReplicas 
PASSED

kafka.admin.ReassignPartitionsClusterTest > shouldExpandCluster STARTED

kafka.admin.ReassignPartitionsClusterTest > shouldExpandCluster PASSED

kafka.admin.ReassignPartitionsClusterTest > 
shouldFailIfProposedHasInvalidLogDir STARTED

kafka.admin.ReassignPartitionsClusterTest > 
shouldFailIfProposedHasInvalidLogDir PASSED

kafka.admin.ReassignPartitionsClusterTest > 
shouldFailIfProposedHasEmptyReplicaList STARTED

kafka.admin.ReassignPartitionsClusterTest > 
shouldFailIfProposedHasEmptyReplicaList PASSED

kafka.admin.ReassignPartitionsClusterTest > 
shouldFailIfProposedHasInvalidBrokerID STARTED

kafka.admin.ReassignPartitionsClusterTest > 
shouldFailIfProposedHasInvalidBrokerID PASSED

kafka.admin.ReassignPartitionsClusterTest > shouldMoveSinglePartition STARTED

kafka.admin.ReassignPartitionsClusterTest > shouldMoveSinglePartition PASSED

kafka.admin.ReassignPartitionsClusterTest > shouldShrinkCluster STARTED

kafka.admin.ReassignPartitionsClusterTest > shouldShrinkCluster PASSED

kafka.admin.ReassignPartitionsClusterTest > 
shouldFailIfProposedMoveReplicaWithinBroker STARTED

kafka.admin.ReassignPartitionsClusterTest > 
shouldFailIfProposedMoveReplicaWithinBroker PASSED

kafka.admin.ReassignPartitionsClusterTest > 
shouldPerformThrottledReassignmentOverVariousTopics STARTED

kafka.admin.ReassignPartitionsClusterTest > 
shouldPerformThrottledReassignmentOverVariousTopics PASSED

kafka.admin.ReassignPartitionsClusterTest > 
shouldChangeThrottleOnRerunAndRemoveOnVerify STARTED

kafka.admin.ReassignPartitionsClusterTest > 
shouldChangeThrottleOnRerunAndRemoveOnVerify PASSED

kafka.admin.ReassignPartitionsClusterTest > shouldMoveSubsetOfPartitions STARTED

kafka.ad

Build failed in Jenkins: kafka-trunk-jdk8 #1974

[Apache Jenkins Server]

See 

--
[...truncated 86.33 KB...]
  ^
:70:
 object OffsetRequest in package api is deprecated: This object has been 
deprecated and will be removed in a future release.
val request = OffsetRequest(
  ^
:99:
 object OffsetRequest in package api is deprecated: This object has been 
deprecated and will be removed in a future release.
val offsets = server.apis.fetchOffsets(logManager, new 
TopicPartition(topic, part), OffsetRequest.LatestTime, 15)

^
:105:
 object OffsetRequest in package api is deprecated: This object has been 
deprecated and will be removed in a future release.
  Map(topicAndPartition -> 
PartitionOffsetRequestInfo(OffsetRequest.LatestTime, 15)),
  ^
:104:
 object OffsetRequest in package api is deprecated: This object has been 
deprecated and will be removed in a future release.
val offsetRequest = OffsetRequest(
^
:130:
 object OffsetRequest in package api is deprecated: This object has been 
deprecated and will be removed in a future release.
val offsets = server.apis.fetchOffsets(logManager, new 
TopicPartition(topic, part), OffsetRequest.LatestTime, 15)

^
:136:
 object OffsetRequest in package api is deprecated: This object has been 
deprecated and will be removed in a future release.
  Map(topicAndPartition -> 
PartitionOffsetRequestInfo(OffsetRequest.LatestTime, 15)),
  ^
:135:
 object OffsetRequest in package api is deprecated: This object has been 
deprecated and will be removed in a future release.
val offsetRequest = OffsetRequest(
^
:144:
 class FetchRequestBuilder in package api is deprecated: This class has been 
deprecated and will be removed in a future release.
  new FetchRequestBuilder().addFetch(topic, 0, consumerOffsets.head, 300 * 
1024).build())
  ^
:164:
 object OffsetRequest in package api is deprecated: This object has been 
deprecated and will be removed in a future release.
OffsetRequest(Map(topicAndPartition -> 
PartitionOffsetRequestInfo(OffsetRequest.EarliestTime, 1)))
  ^
:164:
 object OffsetRequest in package api is deprecated: This object has been 
deprecated and will be removed in a future release.
OffsetRequest(Map(topicAndPartition -> 
PartitionOffsetRequestInfo(OffsetRequest.EarliestTime, 1)))
^
:198:
 object OffsetRequest in package api is deprecated: This object has been 
deprecated and will be removed in a future release.
val offsetRequest = OffsetRequest(Map(topicAndPartition -> 
PartitionOffsetRequestInfo(now, 15)), replicaId = 0)
^
:219:
 object OffsetRequest in package api is deprecated: This object has been 
deprecated and will be removed in a future release.
val offsets = server.apis.fetchOffsets(logManager, new 
TopicPartition(topic, part), OffsetRequest.EarliestTime, 10)

^
:226:
 object OffsetRequest in package api is deprecated: This object has been 
deprecated and will be removed in a futur

Jenkins build is back to normal : kafka-trunk-jdk7 #2709

[Apache Jenkins Server]

See 

Build failed in Jenkins: kafka-0.11.0-jdk7 #297

[Apache Jenkins Server]

See 


Changes:

[wangguoz] MINOR: add mvn-pgp-plugin to sign streams quickstart jars

--
[...truncated 971.43 KB...]
kafka.admin.ResetConsumerGroupOffsetTest > testResetOffsetsByDuration PASSED

kafka.admin.ResetConsumerGroupOffsetTest > testResetOffsetsToLocalDateTime 
STARTED

kafka.admin.ResetConsumerGroupOffsetTest > testResetOffsetsToLocalDateTime 
PASSED

kafka.admin.ResetConsumerGroupOffsetTest > 
testResetOffsetsToEarliestOnTopicsAndPartitions STARTED

kafka.admin.ResetConsumerGroupOffsetTest > 
testResetOffsetsToEarliestOnTopicsAndPartitions PASSED

kafka.admin.ResetConsumerGroupOffsetTest > testResetOffsetsToEarliestOnTopics 
STARTED

kafka.admin.ResetConsumerGroupOffsetTest > testResetOffsetsToEarliestOnTopics 
PASSED

kafka.controller.ControllerEventManagerTest > testEventThatThrowsException 
STARTED

kafka.controller.ControllerEventManagerTest > testEventThatThrowsException 
PASSED

kafka.controller.ControllerEventManagerTest > testSuccessfulEvent STARTED

kafka.controller.ControllerEventManagerTest > testSuccessfulEvent PASSED

kafka.controller.ControllerIntegrationTest > 
testPartitionReassignmentWithOfflineReplicaHaltingProgress STARTED

kafka.controller.ControllerIntegrationTest > 
testPartitionReassignmentWithOfflineReplicaHaltingProgress PASSED

kafka.controller.ControllerIntegrationTest > 
testControllerEpochPersistsWhenAllBrokersDown STARTED

kafka.controller.ControllerIntegrationTest > 
testControllerEpochPersistsWhenAllBrokersDown PASSED

kafka.controller.ControllerIntegrationTest > 
testTopicCreationWithOfflineReplica STARTED

kafka.controller.ControllerIntegrationTest > 
testTopicCreationWithOfflineReplica PASSED

kafka.controller.ControllerIntegrationTest > 
testPartitionReassignmentResumesAfterReplicaComesOnline STARTED

kafka.controller.ControllerIntegrationTest > 
testPartitionReassignmentResumesAfterReplicaComesOnline PASSED

kafka.controller.ControllerIntegrationTest > 
testLeaderAndIsrWhenEntireIsrOfflineAndUncleanLeaderElectionDisabled STARTED

kafka.controller.ControllerIntegrationTest > 
testLeaderAndIsrWhenEntireIsrOfflineAndUncleanLeaderElectionDisabled PASSED

kafka.controller.ControllerIntegrationTest > 
testTopicPartitionExpansionWithOfflineReplica STARTED

kafka.controller.ControllerIntegrationTest > 
testTopicPartitionExpansionWithOfflineReplica PASSED

kafka.controller.ControllerIntegrationTest > 
testPreferredReplicaLeaderElectionWithOfflinePreferredReplica STARTED

kafka.controller.ControllerIntegrationTest > 
testPreferredReplicaLeaderElectionWithOfflinePreferredReplica PASSED

kafka.controller.ControllerIntegrationTest > 
testAutoPreferredReplicaLeaderElection STARTED

kafka.controller.ControllerIntegrationTest > 
testAutoPreferredReplicaLeaderElection PASSED

kafka.controller.ControllerIntegrationTest > testTopicCreation STARTED

kafka.controller.ControllerIntegrationTest > testTopicCreation PASSED

kafka.controller.ControllerIntegrationTest > testPartitionReassignment STARTED

kafka.controller.ControllerIntegrationTest > testPartitionReassignment PASSED

kafka.controller.ControllerIntegrationTest > testTopicPartitionExpansion STARTED

kafka.controller.ControllerIntegrationTest > testTopicPartitionExpansion PASSED

kafka.controller.ControllerIntegrationTest > 
testControllerMoveIncrementsControllerEpoch STARTED

kafka.controller.ControllerIntegrationTest > 
testControllerMoveIncrementsControllerEpoch PASSED

kafka.controller.ControllerIntegrationTest > 
testLeaderAndIsrWhenEntireIsrOfflineAndUncleanLeaderElectionEnabled STARTED

kafka.controller.ControllerIntegrationTest > 
testLeaderAndIsrWhenEntireIsrOfflineAndUncleanLeaderElectionEnabled PASSED

kafka.controller.ControllerIntegrationTest > testEmptyCluster STARTED

kafka.controller.ControllerIntegrationTest > testEmptyCluster PASSED

kafka.controller.ControllerIntegrationTest > testPreferredReplicaLeaderElection 
STARTED

kafka.controller.ControllerIntegrationTest > testPreferredReplicaLeaderElection 
PASSED

kafka.controller.ControllerFailoverTest > testHandleIllegalStateException 
STARTED

kafka.controller.ControllerFailoverTest > testHandleIllegalStateException PASSED

kafka.tools.ConsoleProducerTest > testParseKeyProp STARTED

kafka.tools.ConsoleProducerTest > testParseKeyProp PASSED

kafka.tools.ConsoleProducerTest > testValidConfigsOldProducer STARTED

kafka.tools.ConsoleProducerTest > testValidConfigsOldProducer PASSED

kafka.tools.ConsoleProducerTest > testInvalidConfigs STARTED

kafka.tools.ConsoleProducerTest > testInvalidConfigs PASSED

kafka.tools.ConsoleProducerTest > testValidConfigsNewProducer STARTED

kafka.tools.ConsoleProducerTest > testValidConfigsNewProducer PASSED

kafka.tools.ConsoleConsumerTest > shouldLimitReadsToMaxMessageLimit STARTED

kafka.tools.ConsoleConsumerTest > shouldLimitReadsToMaxMessageLimit PASSED

kafka.tools.ConsoleConsumerTest > shouldParseVali

[GitHub] kafka pull request #3328: KAFKA-5559: Metrics should throw if two client reg...

[mjsax]

Github user mjsax closed the pull request at:

https://github.com/apache/kafka/pull/3328


---

Re: [DISCUSS] KIP-192 - Provide cleaner semantics when idempotence is enabled

[Ted Yu]

For enable.idempotence=safe, it seems giving user impression that idempotence
would be safe.

However, since it really means best effort, the 'safety' is debatable.

Why not just call the new mode besteffort ?

Cheers

On Tue, Sep 5, 2017 at 11:24 AM, Ismael Juma  wrote:

> If we add the message format version (a topic config) in the response of
> TopicMetadata, we should consider adding the max message bytes as well.
> That would allow us to later improve the implementation of KIP-126 to split
> the batch _before_ sending.
>
> Ismael
>
> On Tue, Sep 5, 2017 at 7:17 PM, Jason Gustafson 
> wrote:
>
> > The proposal looks good. Two minor comments:
> >
> > 1. Can we call out how we handle the duplicate case? This is a change in
> > behavior since we currently raise OutOfOrderSequence in this case.
> >
> > 2. Instead of passing through `idempotenceLevel` in the ProduceRequest, I
> > wonder if we should have a field for the minimum required message format.
> > When using enable.idempotence=required, we could set the minimum required
> > version to v2. For enable.idempotence=requested, we could use v0. The
> > advantage is that we may find other uses for a more general field in the
> > future. Alternatively, maybe we really should be returning the message
> > format version of each topic in the TopicMetadata response. A nice bonus
> of
> > doing so is that it gives the producer the ability to craft the right
> > format version ahead of time and avoid the need for conversion on the
> > broker.
> >
> > Thanks,
> > Jason
> >
> > On Tue, Aug 29, 2017 at 4:32 PM, Apurva Mehta 
> wrote:
> >
> > > Hi,
> > >
> > > In the discussion of KIP-185 (enable idempotence by default), we
> > discovered
> > > some shortcomings of the existing idempotent producer implementation.
> > > Fixing these issues requires changes to the ProduceRequest and
> > > ProduceResponse protocols as well as changes to the values of the
> > > 'enable.idempotence' producer config.
> > >
> > > Hence, I split off those changes into another KIP so as to decouple the
> > two
> > > issues. Please have a look at my follow up KIP below:
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 192+%3A+Provide+cleaner+semantics+when+idempotence+is+enabled
> > >
> > > KIP-185 depends on KIP-192, and I hope to make progress on the latter
> > > independently.
> > >
> > > Thanks,
> > > Apurva
> > >
> >
>

Jenkins build is back to normal : kafka-0.11.0-jdk7 #298

[Apache Jenkins Server]

See 

Re: [VOTE] KIP-182 - Reduce Streams DSL overloads and allow easier use of custom storage engines

[Guozhang Wang]

+1

On Fri, Sep 1, 2017 at 3:45 PM, Matthias J. Sax 
wrote:

> +1
>
> On 9/1/17 2:53 PM, Bill Bejeck wrote:
> > +1
> >
> > On Thu, Aug 31, 2017 at 10:20 AM, Damian Guy 
> wrote:
> >
> >> Thanks everyone for voting! Unfortunately i've had to make a bit of an
> >> update based on some issues found during implementation.
> >> The main changes are:
> >> BytesStoreSupplier -> StoreSupplier
> >> Addition of:
> >> WindowBytesStoreSupplier, KeyValueBytesStoreSupplier,
> >> SessionBytesStoreSupplier that will restrict store types to  byte[]>
> >> 3 new overloads added to Materialized to enable developers to create a
> >> Materialized of the appropriate type, i..e, WindowStore etc
> >> Update DSL where Materialized is used such that the stores have generic
> >> types of 
> >> Some minor changes to the arguments to Store#persistentWindowStore and
> >> Store#persistentSessionStore
> >>
> >> Please take a look and recast the votes.
> >>
> >> Thanks for your time,
> >> Damian
> >>
> >> On Fri, 25 Aug 2017 at 17:05 Matthias J. Sax 
> >> wrote:
> >>
> >>> Thanks Damian. Great KIP!
> >>>
> >>> +1
> >>>
> >>>
> >>> -Matthias
> >>>
> >>> On 8/25/17 6:45 AM, Damian Guy wrote:
>  Hi,
> 
>  I've just realised we need to add two methods to StateStoreBuilder or
> >> it
>  isn't going to work:
> 
>  Map logConfig();
>  boolean loggingEnabled();
> 
>  These are needed when we are building the topology and determining
>  changelog topic names and configs.
> 
> 
>  I've also update the KIP to add
> 
>  StreamBuilder#stream(String topic)
> 
>  StreamBuilder#stream(String topic, Consumed options)
> 
> 
>  Thanks
> 
> 
>  On Thu, 24 Aug 2017 at 22:11 Sriram Subramanian 
> >>> wrote:
> 
> > +1
> >
> > On Thu, Aug 24, 2017 at 10:20 AM, Guozhang Wang 
> > wrote:
> >
> >> +1. Thanks Damian!
> >>
> >> On Thu, Aug 24, 2017 at 9:47 AM, Bill Bejeck 
> >>> wrote:
> >>
> >>> Thanks for the KIP!
> >>>
> >>> +1
> >>>
> >>> Thanks,
> >>> Bill
> >>>
> >>> On Thu, Aug 24, 2017 at 12:25 PM, Damian Guy  >
> >> wrote:
> >>>
>  Hi,
> 
>  I'd like to kick off the voting thread for KIP-182:
>  https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>  182%3A+Reduce+Streams+DSL+overloads+and+allow+easier+
>  use+of+custom+storage+engines
> 
>  Thanks,
>  Damian
> 
> >>>
> >>
> >>
> >>
> >> --
> >> -- Guozhang
> >>
> >
> 
> >>>
> >>>
> >>
> >
>
>


-- 
-- Guozhang

[VOTE] 0.11.0.1 RC0

[Damian Guy]

Hello Kafka users, developers and client-developers,

This is the first candidate for release of Apache Kafka 0.11.0.1.

This is a bug fix release and it includes fixes and improvements from 49
JIRAs (including a few critical bugs).

Release notes for the 0.11.0.1 release:
http://home.apache.org/~damianguy/kafka-0.11.0.1-rc0/RELEASE_NOTES.html

*** Please download, test and vote by Saturday, September 9, 9am PT

Kafka's KEYS file containing PGP keys we use to sign the release:
http://kafka.apache.org/KEYS

* Release artifacts to be voted upon (source and binary):
http://home.apache.org/~damianguy/kafka-0.11.0.1-rc0/

* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/

* Javadoc:
http://home.apache.org/~damianguy/kafka-0.11.0.1-rc0/javadoc/

* Tag to be voted upon (off 0.11.0 branch) is the 0.11.0.1 tag:
https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=a8aa61266aedcf62e45b3595a2cf68c819ca1a6c


* Documentation:
Note the documentation can't be pushed live due to changes that will not go
live until the release. You can manually verify by downloading
http://home.apache.org/~damianguy/kafka-0.11.0.1-rc0/kafka_2.11-0.11.0.1-site-docs.tgz


* Protocol:
http://kafka.apache.org/0110/protocol.html

* Successful Jenkins builds for the 0.11.0 branch:
Unit/integration tests: https://builds.apache.org/job/kafka-0.11.0-jdk7/298

System tests:
http://confluent-kafka-0-11-0-system-test-results.s3-us-west-2.amazonaws.com/2017-09-05--001.1504612096--apache--0.11.0--7b6e5f9/report.html

/**

Thanks,
Damian

Re: [VOTE] KIP-138: Change punctuate semantics

[Guozhang Wang]

Thanks for your inputs. The main motivation is indeed to achieve
consistency as we use "wall-clock-time" in some existing classes already,
e.g.
http://kafka.apache.org/0101/javadoc/index.html?org/apache/kafka/streams/processor/WallclockTimestampExtractor.html
.

Since there is no objections I will go ahead and do the renaming now.

Guozhang


On Thu, Aug 31, 2017 at 1:24 AM, Michal Borowiecki <
michal.borowie...@openbet.com> wrote:

> +0 ambivalent about the naming but do agree that should be kept consistent
>
>
>
> On 31/08/17 00:43, Matthias J. Sax wrote:
>
>> +1
>>
>> On 8/30/17 12:00 PM, Bill Bejeck wrote:
>>
>>> +1
>>>
>>> On Wed, Aug 30, 2017 at 1:06 PM, Damian Guy 
>>> wrote:
>>>
>>> +1

 On Wed, 30 Aug 2017 at 17:49 Guozhang Wang  wrote:

 Hello Michal and community:
>
> While working on updating the web docs and java docs for this KIP, I
> felt
> that the term SYSTEM_TIME a bit confusing sometimes from a reader's
> perspective as we are actually talking about wall-clock time. I'd hence
> like to propose an minor addendum to this adopted KIP before the
> release
>
 to

> rename this enum:
>
> SYSTEM_TIME
>
> to
>
> WALL_CLOCK_TIME
>
>
> For people who have voted on this KIP, could you vote again for this
> addendum (detailed discussions can be found in this PR:
> https://github.com/apache/kafka/pull/3732#issuecomment-326043657)?
>
>
> Guozhang
>
>
>
>
> On Sat, May 13, 2017 at 8:13 AM, Michal Borowiecki <
> michal.borowie...@openbet.com> wrote:
>
> Thank you all!
>>
>> This KIP passed the vote with 3 binding and 5 non-binding +1s:
>>
>> +1 (binding) from Guozhang Wang, Ismael Juma and Ewen Cheslack-Postava
>>
>> +1 (non-binding) from Matthias J. Sax, Bill Bejeck, Eno Thereska, Arun
>> Mathew and Thomas Becker
>>
>>
>> Created KAFKA-5233 
>> created to track implementation.
>> It's been a fantastic experience for me working with this great
>>
> community

> to produce a KIP for the first time.
>> Big thank you to everyone who contributed!
>>
>> Cheers,
>> Michał
>>
>>
>> On 12/05/17 02:01, Ismael Juma wrote:
>>
>> Michal, you have enough votes, would you like to close the vote?
>>
>> Ismael
>>
>> On Thu, May 11, 2017 at 4:49 PM, Ewen Cheslack-Postava <
>>
> e...@confluent.io> 

> wrote:
>>
>>
>> +1 (binding)
>>
>> -Ewen
>>
>> On Thu, May 11, 2017 at 7:12 AM, Ismael Juma  <
>>
> ism...@juma.me.uk> wrote:

>
>> Thanks for the KIP, Michal. +1(binding) from me.
>>
>> Ismael
>>
>> On Sat, May 6, 2017 at 6:18 PM, Michal Borowiecki <
>>
> michal.borowie...@openbet.com> wrote:

>
>> Hi all,
>>
>> Given I'm not seeing any contentious issues remaining on the
>> discussion
>> thread, I'd like to initiate the vote for:
>>
>> KIP-138: Change punctuate semantics
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-138%
>> 3A+Change+punctuate+semantics
>>
>>
>> Thanks,
>> Michał
>> --  Michal
>> Borowiecki
>> Senior Software Engineer L4
>> T: +44 208 742 1600 <+44%2020%208742%201600> <020%208742%201600>
>>
>> +44 203 249 8448 <+44%2020%203249%208448> <020%203249%208448>
>>
>>
>>
>> E: michal.borowie...@openbet.com
>> W: www.openbet.com
>> OpenBet Ltd
>>
>> Chiswick Park Building 9
>>
>> 566 Chiswick High Rd
>>
>> London
>>
>> W4 5XT
>>
>> UK >
> email_promo>

> This message is confidential and intended only for the addressee. If
>> you
>> have received this message in error, please immediately notify
>>
> thepostmas...@openbet.com and delete it from your system as well as
 any

> copies. The content of e-mails as well as traffic data may be monitored
>>
>> by
>>
>> OpenBet for employment and security purposes. To protect the
>> environment
>> please do not print this e-mail unless necessary. OpenBet Ltd.
>>
>> Registered
>>
>> Office: Chiswick Park Building 9, 566 Chiswick High Road, London, W4
>>
>> 5XT,
>>
>> United Kingdom. A company registered in England and Wales. Registered
>>
>> no.
>>
>> 3134634. VAT no. GB927523612
>>
>>
>>
>> --
>>  Michal Borowiecki
>> Senior Software Engineer L4
>> T: +44 208 742 1600 <+44%2020%208742%201600>
>>
>>
>> +44 203 249 8448 <+44%2020%203249%208448>
>>
>>
>>
>> E: michal.borowie...@openbet.com
>> W: www.openbet.com
>> O

[GitHub] kafka-site pull request #73: Add coding guidelines for Streams API

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka-site/pull/73


---

[GitHub] kafka-site issue #73: Add coding guidelines for Streams API

[guozhangwang]

Github user guozhangwang commented on the issue:

https://github.com/apache/kafka-site/pull/73
  
LGTM. Merged to `asf-site`.


---

Re: [DISCUSS] KIP-192 - Provide cleaner semantics when idempotence is enabled

[Ismael Juma]

Hi Ted,

The current proposal has 3 options: requested, required, off ("safe" was in
an earlier proposal). I think these convey the meaning more clearly IMO.

Ismael

On 5 Sep 2017 9:22 pm, "Ted Yu"  wrote:

> For enable.idempotence=safe, it seems giving user impression that
> idempotence
> would be safe.
>
> However, since it really means best effort, the 'safety' is debatable.
>
> Why not just call the new mode besteffort ?
>
> Cheers
>
> On Tue, Sep 5, 2017 at 11:24 AM, Ismael Juma  wrote:
>
> > If we add the message format version (a topic config) in the response of
> > TopicMetadata, we should consider adding the max message bytes as well.
> > That would allow us to later improve the implementation of KIP-126 to
> split
> > the batch _before_ sending.
> >
> > Ismael
> >
> > On Tue, Sep 5, 2017 at 7:17 PM, Jason Gustafson 
> > wrote:
> >
> > > The proposal looks good. Two minor comments:
> > >
> > > 1. Can we call out how we handle the duplicate case? This is a change
> in
> > > behavior since we currently raise OutOfOrderSequence in this case.
> > >
> > > 2. Instead of passing through `idempotenceLevel` in the
> ProduceRequest, I
> > > wonder if we should have a field for the minimum required message
> format.
> > > When using enable.idempotence=required, we could set the minimum
> required
> > > version to v2. For enable.idempotence=requested, we could use v0. The
> > > advantage is that we may find other uses for a more general field in
> the
> > > future. Alternatively, maybe we really should be returning the message
> > > format version of each topic in the TopicMetadata response. A nice
> bonus
> > of
> > > doing so is that it gives the producer the ability to craft the right
> > > format version ahead of time and avoid the need for conversion on the
> > > broker.
> > >
> > > Thanks,
> > > Jason
> > >
> > > On Tue, Aug 29, 2017 at 4:32 PM, Apurva Mehta 
> > wrote:
> > >
> > > > Hi,
> > > >
> > > > In the discussion of KIP-185 (enable idempotence by default), we
> > > discovered
> > > > some shortcomings of the existing idempotent producer implementation.
> > > > Fixing these issues requires changes to the ProduceRequest and
> > > > ProduceResponse protocols as well as changes to the values of the
> > > > 'enable.idempotence' producer config.
> > > >
> > > > Hence, I split off those changes into another KIP so as to decouple
> the
> > > two
> > > > issues. Please have a look at my follow up KIP below:
> > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > 192+%3A+Provide+cleaner+semantics+when+idempotence+is+enabled
> > > >
> > > > KIP-185 depends on KIP-192, and I hope to make progress on the latter
> > > > independently.
> > > >
> > > > Thanks,
> > > > Apurva
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-192 - Provide cleaner semantics when idempotence is enabled

[Ted Yu]

Sounds good.

Please update the KIP and:

https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Exactly+Once+-+Dealing+with+older+message+formats+when+idempotence+is+enabled

On Tue, Sep 5, 2017 at 2:03 PM, Ismael Juma  wrote:

> Hi Ted,
>
> The current proposal has 3 options: requested, required, off ("safe" was in
> an earlier proposal). I think these convey the meaning more clearly IMO.
>
> Ismael
>
> On 5 Sep 2017 9:22 pm, "Ted Yu"  wrote:
>
> > For enable.idempotence=safe, it seems giving user impression that
> > idempotence
> > would be safe.
> >
> > However, since it really means best effort, the 'safety' is debatable.
> >
> > Why not just call the new mode besteffort ?
> >
> > Cheers
> >
> > On Tue, Sep 5, 2017 at 11:24 AM, Ismael Juma  wrote:
> >
> > > If we add the message format version (a topic config) in the response
> of
> > > TopicMetadata, we should consider adding the max message bytes as well.
> > > That would allow us to later improve the implementation of KIP-126 to
> > split
> > > the batch _before_ sending.
> > >
> > > Ismael
> > >
> > > On Tue, Sep 5, 2017 at 7:17 PM, Jason Gustafson 
> > > wrote:
> > >
> > > > The proposal looks good. Two minor comments:
> > > >
> > > > 1. Can we call out how we handle the duplicate case? This is a change
> > in
> > > > behavior since we currently raise OutOfOrderSequence in this case.
> > > >
> > > > 2. Instead of passing through `idempotenceLevel` in the
> > ProduceRequest, I
> > > > wonder if we should have a field for the minimum required message
> > format.
> > > > When using enable.idempotence=required, we could set the minimum
> > required
> > > > version to v2. For enable.idempotence=requested, we could use v0. The
> > > > advantage is that we may find other uses for a more general field in
> > the
> > > > future. Alternatively, maybe we really should be returning the
> message
> > > > format version of each topic in the TopicMetadata response. A nice
> > bonus
> > > of
> > > > doing so is that it gives the producer the ability to craft the right
> > > > format version ahead of time and avoid the need for conversion on the
> > > > broker.
> > > >
> > > > Thanks,
> > > > Jason
> > > >
> > > > On Tue, Aug 29, 2017 at 4:32 PM, Apurva Mehta 
> > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > In the discussion of KIP-185 (enable idempotence by default), we
> > > > discovered
> > > > > some shortcomings of the existing idempotent producer
> implementation.
> > > > > Fixing these issues requires changes to the ProduceRequest and
> > > > > ProduceResponse protocols as well as changes to the values of the
> > > > > 'enable.idempotence' producer config.
> > > > >
> > > > > Hence, I split off those changes into another KIP so as to decouple
> > the
> > > > two
> > > > > issues. Please have a look at my follow up KIP below:
> > > > >
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > 192+%3A+Provide+cleaner+semantics+when+idempotence+is+enabled
> > > > >
> > > > > KIP-185 depends on KIP-192, and I hope to make progress on the
> latter
> > > > > independently.
> > > > >
> > > > > Thanks,
> > > > > Apurva
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-192 - Provide cleaner semantics when idempotence is enabled

[Ismael Juma]

Agreed, we should make it consistent across the various pages.

On Tue, Sep 5, 2017 at 10:06 PM, Ted Yu  wrote:

> Sounds good.
>
> Please update the KIP and:
>
> https://cwiki.apache.org/confluence/display/KAFKA/
> Kafka+Exactly+Once+-+Dealing+with+older+message+formats+
> when+idempotence+is+enabled
>
> On Tue, Sep 5, 2017 at 2:03 PM, Ismael Juma  wrote:
>
> > Hi Ted,
> >
> > The current proposal has 3 options: requested, required, off ("safe" was
> in
> > an earlier proposal). I think these convey the meaning more clearly IMO.
> >
> > Ismael
> >
> > On 5 Sep 2017 9:22 pm, "Ted Yu"  wrote:
> >
> > > For enable.idempotence=safe, it seems giving user impression that
> > > idempotence
> > > would be safe.
> > >
> > > However, since it really means best effort, the 'safety' is debatable.
> > >
> > > Why not just call the new mode besteffort ?
> > >
> > > Cheers
> > >
> > > On Tue, Sep 5, 2017 at 11:24 AM, Ismael Juma 
> wrote:
> > >
> > > > If we add the message format version (a topic config) in the response
> > of
> > > > TopicMetadata, we should consider adding the max message bytes as
> well.
> > > > That would allow us to later improve the implementation of KIP-126 to
> > > split
> > > > the batch _before_ sending.
> > > >
> > > > Ismael
> > > >
> > > > On Tue, Sep 5, 2017 at 7:17 PM, Jason Gustafson 
> > > > wrote:
> > > >
> > > > > The proposal looks good. Two minor comments:
> > > > >
> > > > > 1. Can we call out how we handle the duplicate case? This is a
> change
> > > in
> > > > > behavior since we currently raise OutOfOrderSequence in this case.
> > > > >
> > > > > 2. Instead of passing through `idempotenceLevel` in the
> > > ProduceRequest, I
> > > > > wonder if we should have a field for the minimum required message
> > > format.
> > > > > When using enable.idempotence=required, we could set the minimum
> > > required
> > > > > version to v2. For enable.idempotence=requested, we could use v0.
> The
> > > > > advantage is that we may find other uses for a more general field
> in
> > > the
> > > > > future. Alternatively, maybe we really should be returning the
> > message
> > > > > format version of each topic in the TopicMetadata response. A nice
> > > bonus
> > > > of
> > > > > doing so is that it gives the producer the ability to craft the
> right
> > > > > format version ahead of time and avoid the need for conversion on
> the
> > > > > broker.
> > > > >
> > > > > Thanks,
> > > > > Jason
> > > > >
> > > > > On Tue, Aug 29, 2017 at 4:32 PM, Apurva Mehta  >
> > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > In the discussion of KIP-185 (enable idempotence by default), we
> > > > > discovered
> > > > > > some shortcomings of the existing idempotent producer
> > implementation.
> > > > > > Fixing these issues requires changes to the ProduceRequest and
> > > > > > ProduceResponse protocols as well as changes to the values of the
> > > > > > 'enable.idempotence' producer config.
> > > > > >
> > > > > > Hence, I split off those changes into another KIP so as to
> decouple
> > > the
> > > > > two
> > > > > > issues. Please have a look at my follow up KIP below:
> > > > > >
> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > > 192+%3A+Provide+cleaner+semantics+when+idempotence+is+enabled
> > > > > >
> > > > > > KIP-185 depends on KIP-192, and I hope to make progress on the
> > latter
> > > > > > independently.
> > > > > >
> > > > > > Thanks,
> > > > > > Apurva
> > > > > >
> > > > >
> > > >
> > >
> >
>

[GitHub] kafka pull request #2612: KAFKA-4819: Expose states for active tasks to publ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2612


---

Re: [DISCUSS] KIP-183 - Change PreferredReplicaLeaderElectionCommand to use AdminClient

[Colin McCabe]

On Mon, Sep 4, 2017, at 04:54, Tom Bentley wrote:
> The KIP has been adopted after a successful vote.

Thanks for working on this, Tom.  It's a nice improvement.

> 
> Unfortunately I've discovered that there's an annoying detail in the
> handling of the case that electPreferredLeaders() is called with a null
> partitions argument. As discussed with Ewen, this is supposed to mean
> "all
> partitions", but we don't know all the partitions in the AdminClient, yet
> we have to return a ElectPreferredLeadersResults instance, supposedly
> with
> the partitions as keys.
> 
> We could handle this by passing a KafkaFuture KafkaFuture>> to the ctor of ElectPreferredLeadersResults, instead
> of
> an extant Map> (the API of
> ElectPreferredLeadersResults would not change). In the case that the
> partitions argument was not null this future will already be completed.
> In
> the case where partitions argument was null this future will be completed
> when we have a response from which we discover the partitions; in the
> meantime the AdminClient can carry on being used for other calls. So in
> the
> normal case there's not really a problem.
> 
> The problem comes where there's an exception *before we get the
> response*,
> that means we still don't know the partitions to populate the map with.
> In
> practice this would mean that an exception could propagate out of
> ElectPreferredLeadersResults.values() rather than when the map was
> accessed
> element-wise. Since we control the API of ElectPreferredLeadersResults we
> could document that values() (and consequently all()) could throw,. We
> could even use checked exceptions, though since the exception would only
> happen in the case that the partitions argument was null that would feel
> rather heavy-handed to me.
> 
> Another alternative would be to block in
> AdminClient.electPreferredLeaders()
> in the case that the partitions argument was null, and if there was an
> error let the exception propagate out of electPreferredLeaders()
> directly.
> 
> Sorry about having to ask about this once people have already voted, but
> what do people think about these options?

I think we need to be very careful to keep the APIs asynchronous all the
time.  Having electPreferredLeaders() or values() sometimes block might
superficially seem reasonable, but it destroys the usefulness of the API
for true async programming.  Basically the async user is forced to put
the call into a thread pool in case it decides to block.  And they
cannot do nice things like chain Futures.

Why does there need to be a map at all in the API?  Why not just have
something like this:

  ElectPreferredLeadersResults {
/**
 * Get the result of the election for the given TopicPartition.
 * If there was not an election triggered for the given
 TopicPartition, the
 * future will complete with an error.
 */
public KafkaFuture partitionResult(TopicPartition
topicPartition);

/**
 * Get the topic partitions on which we attempted to trigger an
 election.
 * This tracks attempts, not successes.  A partition will appear in
 this result
 * even if the election was not successfully triggered.
 */
public KafkaFuture> partitions();

/**
 * Return a future which gives an error result if we fail for any
 partition.
 */
public KafkaFuture all();
  }

We can fill in all this information when we actually know it.  In some
cases that will be later than others.  But all the calls can immediately
return a KafkaFuture, not block or throw an exception.

best,
Colin


> 
> Thanks,
> 
> Tom
> 
> On 30 August 2017 at 16:55, Tom Bentley  wrote:
> 
> > I've updated in the KIP.
> >
> > Thanks,
> >
> > Tom
> >
> > On 30 August 2017 at 16:42, Ismael Juma  wrote:
> >
> >> If you agree with the change, yes, please rename. It's OK to make changes
> >> after the VOTE thread starts. In cases where some people have already
> >> voted, it's recommended to mention the changes in the VOTE thread as a
> >> heads up. Generally, we don't restart the vote unless the changes are
> >> significant.
> >>
> >> Ismael
> >>
> >> On Wed, Aug 30, 2017 at 4:26 PM, Tom Bentley 
> >> wrote:
> >>
> >> > Hi Ismael,
> >> >
> >> > I agree that `electPreferredReplicaLeader` is a mouthful and am happy to
> >> > change it to `electPreferredLeaders`. I'd rename the correspond request
> >> and
> >> > response similarly.
> >> >
> >> > Should I rename it in the KIP now, even though I initiated a VOTE thread
> >> > yesterday?
> >> >
> >> > Cheers,
> >> >
> >> > Tom
> >> >
> >> > On 30 August 2017 at 16:01, Ismael Juma  wrote:
> >> >
> >> > > Hi Tom,
> >> > >
> >> > > Thanks for the KIP, it's a useful one. I find the proposed method name
> >> > > `electPreferredReplicaLeader` a little hard to read. It seems that a
> >> > small
> >> > > change would make it clearer: `electPreferredReplicaAsLeader`. The
> >> next
> >> > > point is that this is a batch API, so it should ideally be plural like
> >

Re: [VOTE] KIP-182 - Reduce Streams DSL overloads and allow easier use of custom storage engines

[Sriram Subramanian]

+1

On Tue, Sep 5, 2017 at 1:33 PM, Guozhang Wang  wrote:

> +1
>
> On Fri, Sep 1, 2017 at 3:45 PM, Matthias J. Sax 
> wrote:
>
> > +1
> >
> > On 9/1/17 2:53 PM, Bill Bejeck wrote:
> > > +1
> > >
> > > On Thu, Aug 31, 2017 at 10:20 AM, Damian Guy 
> > wrote:
> > >
> > >> Thanks everyone for voting! Unfortunately i've had to make a bit of an
> > >> update based on some issues found during implementation.
> > >> The main changes are:
> > >> BytesStoreSupplier -> StoreSupplier
> > >> Addition of:
> > >> WindowBytesStoreSupplier, KeyValueBytesStoreSupplier,
> > >> SessionBytesStoreSupplier that will restrict store types to  > byte[]>
> > >> 3 new overloads added to Materialized to enable developers to create a
> > >> Materialized of the appropriate type, i..e, WindowStore etc
> > >> Update DSL where Materialized is used such that the stores have
> generic
> > >> types of 
> > >> Some minor changes to the arguments to Store#persistentWindowStore and
> > >> Store#persistentSessionStore
> > >>
> > >> Please take a look and recast the votes.
> > >>
> > >> Thanks for your time,
> > >> Damian
> > >>
> > >> On Fri, 25 Aug 2017 at 17:05 Matthias J. Sax 
> > >> wrote:
> > >>
> > >>> Thanks Damian. Great KIP!
> > >>>
> > >>> +1
> > >>>
> > >>>
> > >>> -Matthias
> > >>>
> > >>> On 8/25/17 6:45 AM, Damian Guy wrote:
> >  Hi,
> > 
> >  I've just realised we need to add two methods to StateStoreBuilder
> or
> > >> it
> >  isn't going to work:
> > 
> >  Map logConfig();
> >  boolean loggingEnabled();
> > 
> >  These are needed when we are building the topology and determining
> >  changelog topic names and configs.
> > 
> > 
> >  I've also update the KIP to add
> > 
> >  StreamBuilder#stream(String topic)
> > 
> >  StreamBuilder#stream(String topic, Consumed options)
> > 
> > 
> >  Thanks
> > 
> > 
> >  On Thu, 24 Aug 2017 at 22:11 Sriram Subramanian 
> > >>> wrote:
> > 
> > > +1
> > >
> > > On Thu, Aug 24, 2017 at 10:20 AM, Guozhang Wang <
> wangg...@gmail.com>
> > > wrote:
> > >
> > >> +1. Thanks Damian!
> > >>
> > >> On Thu, Aug 24, 2017 at 9:47 AM, Bill Bejeck 
> > >>> wrote:
> > >>
> > >>> Thanks for the KIP!
> > >>>
> > >>> +1
> > >>>
> > >>> Thanks,
> > >>> Bill
> > >>>
> > >>> On Thu, Aug 24, 2017 at 12:25 PM, Damian Guy <
> damian@gmail.com
> > >
> > >> wrote:
> > >>>
> >  Hi,
> > 
> >  I'd like to kick off the voting thread for KIP-182:
> >  https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >  182%3A+Reduce+Streams+DSL+overloads+and+allow+easier+
> >  use+of+custom+storage+engines
> > 
> >  Thanks,
> >  Damian
> > 
> > >>>
> > >>
> > >>
> > >>
> > >> --
> > >> -- Guozhang
> > >>
> > >
> > 
> > >>>
> > >>>
> > >>
> > >
> >
> >
>
>
> --
> -- Guozhang
>

[GitHub] kafka pull request #3795: KAFKA-5783: Add KafkaPrincipalBuilder with support...

[hachikuji]

GitHub user hachikuji opened a pull request:

https://github.com/apache/kafka/pull/3795

KAFKA-5783: Add KafkaPrincipalBuilder with support for SASL (KIP-189)



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hachikuji/kafka KAFKA-5783

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3795.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3795


commit 3c2f106802508bd3cf305d08b8031442cc868f8f
Author: Jason Gustafson 
Date:   2017-08-24T22:47:31Z

KAFKA-5783: Add KafkaPrincipalBuilder with support for SASL (KIP-189)




---

[GitHub] kafka-site pull request #74: Adding portoseguro, micronauticsresearch & cj l...

[manjuapu]

GitHub user manjuapu opened a pull request:

https://github.com/apache/kafka-site/pull/74

Adding portoseguro, micronauticsresearch & cj logos to powered by page



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/manjuapu/kafka-site asf-site

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka-site/pull/74.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #74


commit 548218756753afd49c5925898a194755a37e6c11
Author: Manjula K 
Date:   2017-09-05T21:37:53Z

Adding portoseguro, micronauticsresearch & cj logos to powered by page




---

[GitHub] kafka pull request #3719: KAFKA-5603: Don't abort TX for zombie tasks

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3719


---

[GitHub] kafka pull request #3771: MINOR: logging improvements

[guozhangwang]

Github user guozhangwang closed the pull request at:

https://github.com/apache/kafka/pull/3771


---

[GitHub] kafka pull request #2649: KAFKA-4860: Allow spaces in paths on windows

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2649


---

[jira] [Resolved] (KAFKA-5837) ReassignPartitionsCommand fails if default throttle/timeout used

[Rajini Sivaram (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajini Sivaram resolved KAFKA-5837.
---
Resolution: Fixed

Issue resolved by pull request 3792
[https://github.com/apache/kafka/pull/3792]

> ReassignPartitionsCommand fails if default throttle/timeout used
> 
>
> Key: KAFKA-5837
> URL: https://issues.apache.org/jira/browse/KAFKA-5837
> Project: Kafka
>  Issue Type: Bug
>  Components: admin
>Affects Versions: 1.0.0
>Reporter: Rajini Sivaram
>Assignee: Rajini Sivaram
> Fix For: 1.0.0
>
>
> System ReassignPartitionsTest.test_reassign_partitions failed with:
> {quote}
> Partitions reassignment failed due to java.lang.String cannot be cast to 
> java.lang.Long
> java.lang.ClassCastException: java.lang.String cannot be cast to 
> java.lang.Long
>   at scala.runtime.BoxesRunTime.unboxToLong(BoxesRunTime.java:105)
>   at 
> kafka.admin.ReassignPartitionsCommand$.executeAssignment(ReassignPartitionsCommand.scala:188)
>   at 
> kafka.admin.ReassignPartitionsCommand$.main(ReassignPartitionsCommand.scala:61)
>   at 
> kafka.admin.ReassignPartitionsCommand.main(ReassignPartitionsCommand.scala)
> {quote}
> This is because default throttle is being set as a String rather than a Long. 
> Default throttle was never being set properly in the command opts, but it 
> didn't matter earlier because code used to set it explicitly:
> {quote}
> val throttle = if (opts.options.has(opts.throttleOpt)) 
> opts.options.valueOf(opts.throttleOpt) else -1
> {quote}
> But the commit 
> https://github.com/apache/kafka/commit/adefc8ea076354e07839f0319fee1fba52343b91
>  started using the default directly (and also added a timeout with default 
> set in the same way).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] kafka pull request #3792: KAFKA-5837: Set defaults for ReassignPartitionsCom...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3792


---

Re: [VOTE] KIP-188 - Add new metrics to support health checks

[Rajini Sivaram]

I am ok with dropping 'Message'. So the names would be FetchConversionsPerSec,
ProduceConversionsPerSec and ConversionsTimeMs. The first two sound fine.
Not so sure about ConversionsTimeMs, but since it appears with
Produce/Fetch as the request tag, it should be ok. I haven't updated the
KIP yet. If there are no objections, I will update the KIP tomorrow.

Regards,

Rajini

On Tue, Sep 5, 2017 at 7:23 PM, Jason Gustafson  wrote:

> >
> > I was wondering about the message versus record question. The fact that
> we
> > already have MessagesInPerSec seemed to favour the former. The other
> aspect
> > is that for produce requests, we can up convert as well, so it seemed
> > better to keep it generic.
>
>
> Yeah, so I thought maybe we could bypass the question and drop `Message`
> from the names if they were already clear enough. I'm fine with either way.
>
> On Tue, Sep 5, 2017 at 11:09 AM, Ismael Juma  wrote:
>
> > I was wondering about the message versus record question. The fact that
> we
> > already have MessagesInPerSec seemed to favour the former. The other
> aspect
> > is that for produce requests, we can up convert as well, so it seemed
> > better to keep it generic.
> >
> > Ismael
> >
> > On Tue, Sep 5, 2017 at 6:51 PM, Jason Gustafson 
> > wrote:
> >
> > > +1 Lots of good stuff in here.
> > >
> > > One minor nit: in the name `FetchDownConversionsPerSec`, it's implicit
> > that
> > > down-conversion is for messages. Could we do the same for
> > > `MessageConversionsTimeMs` and drop the `Message`? Then we don't have
> to
> > > decide if it should be 'Record' instead.
> > >
> > > On Tue, Sep 5, 2017 at 10:20 AM, Ismael Juma 
> wrote:
> > >
> > > > Thanks Rajini.
> > > >
> > > > 1. I meant a topic metric, but we could have one for fetch and one
> for
> > > > produce differentiated by the additional tag. The advantage is that
> the
> > > > name would be consistent with the request metric for message
> > conversions.
> > > > However, on closer inspection, this would make the name inconsistent
> > with
> > > > the broker topic metrics:
> > > >
> > > > val totalProduceRequestRate =
> > > > newMeter(BrokerTopicStats.TotalProduceRequestsPerSec, "requests",
> > > > TimeUnit.SECONDS, tags)
> > > > val totalFetchRequestRate =
> > > > newMeter(BrokerTopicStats.TotalFetchRequestsPerSec, "requests",
> > > > TimeUnit.SECONDS, tags)
> > > >
> > > > So, we maybe we can instead go for FetchMessageConversionsPerSecond
> > and
> > > > ProduceMessageConversionsPerSec.
> > > >
> > > > 2. Sounds good.
> > > >
> > > > Ismael
> > > >
> > > > On Tue, Sep 5, 2017 at 5:46 PM, Rajini Sivaram <
> > rajinisiva...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Ismael,
> > > > >
> > > > > 1. At the moment FetchDownConversionsPerSec is a topic metric while
> > > > > MessageConversionTimeMs is a request metric which indicates
> > > Produce/Fetch
> > > > > as a tag. Are you suggesting that we should convert
> > > > > FetchDownConversionsPerSec to a request metric called
> > > > > MessageConversionsPerSec
> > > > > for fetch requests?
> > > > >
> > > > > 2. TemporaryMessageSize for Produce/Fetch would indicate the space
> > > > > allocated for conversions. For other requests, this metric will not
> > be
> > > > > created (unless we find a request where the size is large and this
> > > > > information is useful).
> > > > >
> > > > > Thank you,
> > > > >
> > > > > Rajini
> > > > >
> > > > >
> > > > > On Tue, Sep 5, 2017 at 4:55 PM, Ismael Juma 
> > wrote:
> > > > >
> > > > > > Thanks Rajini, +1 (binding) from me. Just a few minor comments:
> > > > > >
> > > > > > 1. FetchDownConversionsPerSec should probably be
> > > > MessageConversionsPerSec
> > > > > > with a request tag for consistency with MessageConversionsTimeMs.
> > The
> > > > > text
> > > > > > in that paragraph should also be updated to talk about message
> > > > > conversions
> > > > > > instead of down conversions only.
> > > > > >
> > > > > > 2. What will TemporaryMemorySize represent for requests other
> than
> > > > > > `ProduceRequest`?
> > > > > >
> > > > > > Ismael
> > > > > >
> > > > > > On Mon, Sep 4, 2017 at 2:09 PM, Rajini Sivaram <
> > > > rajinisiva...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > All the suggestions on the discuss thread have been
> incorporated
> > > into
> > > > > the
> > > > > > > KIP. Please let me know if you have any more concerns or else
> can
> > > we
> > > > > > > proceed with voting for this KIP?
> > > > > > >
> > > > > > > Thank you,
> > > > > > >
> > > > > > > Rajini
> > > > > > >
> > > > > > > On Thu, Aug 24, 2017 at 6:50 PM, Rajini Sivaram <
> > > > > rajinisiva...@gmail.com
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > I would like to start the vote on KIP-188 that adds
> additional
> > > > > metrics
> > > > > > to
> > > > > > > > support health checks for Kafka Ops. Details are here:
> > > > > > > >
> > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/

[jira] [Created] (KAFKA-5839) Upgrade Guide doc changes for KIP-130

[Guozhang Wang (JIRA)]

Guozhang Wang created KAFKA-5839:


 Summary: Upgrade Guide doc changes for KIP-130
 Key: KAFKA-5839
 URL: https://issues.apache.org/jira/browse/KAFKA-5839
 Project: Kafka
  Issue Type: Sub-task
Reporter: Guozhang Wang
Assignee: Florian Hussonnois


Related web docs:

1. developer guide.
2. upgrade guide.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (KAFKA-5840) TransactionsTest#testBasicTransactions hangs

[Ted Yu (JIRA)]

Ted Yu created KAFKA-5840:
-

 Summary: TransactionsTest#testBasicTransactions hangs
 Key: KAFKA-5840
 URL: https://issues.apache.org/jira/browse/KAFKA-5840
 Project: Kafka
  Issue Type: Test
Reporter: Ted Yu


Here is part of the stack trace:
{code}
"Test worker" #20 prio=5 os_prio=0 tid=0x7feb449fc000 nid=0x5f69 waiting on 
condition [0x7feb05f8c000]
   java.lang.Thread.State: WAITING (parking)
  at sun.misc.Unsafe.park(Native Method)
  - parking to wait for  <0x81272ec0> (a 
java.util.concurrent.CountDownLatch$Sync)
  at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
  at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
  at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
  at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
  at 
org.apache.kafka.clients.producer.internals.ProduceRequestResult.await(ProduceRequestResult.java:76)
  at 
org.apache.kafka.clients.producer.internals.RecordAccumulator.awaitFlushCompletion(RecordAccumulator.java:573)
  at 
org.apache.kafka.clients.producer.KafkaProducer.flush(KafkaProducer.java:948)
  at kafka.api.TransactionsTest.testBasicTransactions(TransactionsTest.scala:93)
{code}
{code}
Apache Maven 3.5.0 (ff8f5e7444045639af65f6095c62210b5713f426; 
2017-04-03T19:39:06Z)
Maven home: /apache-maven-3.5.0
Java version: 1.8.0_131, vendor: Oracle Corporation
Java home: /jdk1.8.0_131/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "3.10.0-327.28.3.el7.x86_64", arch: "amd64", family: 
"unix"
{code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] kafka pull request #3787: KAFKA-5823: Extend upgrade section for KIP-120

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3787


---

Jenkins build is back to normal : kafka-trunk-jdk8 #1975

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-91 Provide Intuitive User Timeouts in The Producer

[Sumant Tambe]

I've updated the kip-91 writeup

to capture some of the discussion here. Please confirm if it's sufficiently
accurate. Feel free to edit it if you think some explanation can be better
and has been agreed upon here.

How do you proceed from here?

-Sumant

On 30 August 2017 at 12:59, Jun Rao  wrote:

> Hi, Jiangjie,
>
> I mis-understood Jason's approach earlier. It does seem to be a good one.
> We still need to calculate the selector timeout based on the remaining
> delivery.timeout.ms to call the callback on time, but we can always wait
> for an inflight request based on request.timeout.ms.
>
> Thanks,
>
> Jun
>
> On Tue, Aug 29, 2017 at 5:16 PM, Becket Qin  wrote:
>
> > Yeah, I think expiring a batch but still wait for the response is
> probably
> > reasonable given the result is not guaranteed anyways.
> >
> > @Jun,
> >
> > I think the frequent PID reset may still be possible if we do not wait
> for
> > the in-flight response to return. Consider two partitions p0 and p1, the
> > deadline of the batches for p0 are T + 10, T + 30, T + 50... The deadline
> > of the batches for p1 are T + 20, T + 40, T + 60... Assuming each request
> > takes more than 10 ms to get the response. The following sequence may be
> > possible:
> >
> > T: PID0 send batch0_p0(PID0), batch0_p1(PID0)
> > T + 10: PID0 expires batch0_p0(PID0), without resetting PID, sends
> > batch1_p0(PID0) and batch0_p1(PID0, retry)
> > T + 20: PID0 expires batch0_p1(PID0, retry), resets the PID to PID1,
> sends
> > batch1_p0(PID0, retry) and batch1_p1(PID1)
> > T + 30: PID1 expires batch1_p0(PID0, retry), without resetting PID, sends
> > batch2_p0(PID1) and batch1_p1(PID1, retry)
> > T + 40: PID1 expires batch1_p1(PID1, retry), resets the PID to PID2,
> sends
> > batch2_p0(PID1, retry) and sends batch2_p1(PID2)
> > 
> >
> > In the above example, the producer will reset PID once every two
> requests.
> > The example did not take retry backoff into consideration, but it still
> > seems possible to encounter frequent PID reset if we do not wait for the
> > request to finish. Also, in this case we will have a lot of retries and
> > mixture of PIDs which seem to be pretty complicated.
> >
> > I think Jason's suggestion will address both concerns, i.e. we fire the
> > callback at exactly delivery.timeout.ms, but we will still wait for the
> > response to be returned before sending the next request.
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> >
> >
> > On Tue, Aug 29, 2017 at 4:00 PM, Jun Rao  wrote:
> >
> > > Hmm, I thought delivery.timeout.ms bounds the time from a message is
> in
> > > the
> > > accumulator (i.e., when send() returns) to the time when the callback
> is
> > > called. If we wait for request.timeout.ms for an inflight request and
> > the
> > > remaining delivery.timeout.ms is less than request.timeout.ms, the
> > > callback
> > > may be called later than delivery.timeout.ms, right?
> > >
> > > Jiangjie's concern on resetting the pid on every expired batch is
> > probably
> > > not an issue if we only reset the pid when the expired batch's pid is
> the
> > > same as the current pid, as Jason suggested.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Tue, Aug 29, 2017 at 3:09 PM, Jason Gustafson 
> > > wrote:
> > >
> > > > I think the semantics of delivery.timeout.ms need to allow for the
> > > > possibility that the record was actually written. Unless we can keep
> on
> > > > retrying indefinitely, there's really no way to know for sure whether
> > the
> > > > record was written or not. A delivery timeout just means that we
> cannot
> > > > guarantee that the record was delivered.
> > > >
> > > > -Jason
> > > >
> > > > On Tue, Aug 29, 2017 at 2:51 PM, Becket Qin 
> > > wrote:
> > > >
> > > > > Hi Jason,
> > > > >
> > > > > If we expire the batch from user's perspective but still waiting
> for
> > > the
> > > > > response, would that mean it is likely that the batch will be
> > > > successfully
> > > > > appended but the users will receive a TimeoutException? That seems
> a
> > > > little
> > > > > non-intuitive to the users. Arguably it maybe OK though because
> > > currently
> > > > > when TimeoutException is thrown, there is no guarantee whether the
> > > > messages
> > > > > are delivered or not.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jiangjie (Becket) Qin
> > > > >
> > > > > On Tue, Aug 29, 2017 at 12:33 PM, Jason Gustafson <
> > ja...@confluent.io>
> > > > > wrote:
> > > > >
> > > > > > I think I'm with Becket. We should wait for request.timeout.ms
> for
> > > > each
> > > > > > produce request we send. We can still await the response
> internally
> > > for
> > > > > > PID/sequence maintenance even if we expire the batch from the
> > user's
> > > > > > perspective. New sequence numbers would be assigned based on the
> > > > current
> > > > > > PID until the response returns and we find whether a PID reset is
> > 

[jira] [Created] (KAFKA-5841) Open old index files with read-only permission

[Jason Gustafson (JIRA)]

Jason Gustafson created KAFKA-5841:
--

 Summary: Open old index files with read-only permission
 Key: KAFKA-5841
 URL: https://issues.apache.org/jira/browse/KAFKA-5841
 Project: Kafka
  Issue Type: Improvement
Reporter: Jason Gustafson


Since old index files do not change, we may as well drop the write permission 
needed when opening them. From the doc comments in {{OffsetIndex}}, it sounds 
like we may have had this implemented at one point:

{code}
 * Index files can be opened in two ways: either as an empty, mutable index 
that allows appends or
 * an immutable read-only index file that has previously been populated. The 
makeReadOnly method will turn a mutable file into an 
 * immutable one and truncate off any extra bytes. This is done when the index 
file is rolled over.
{code}

So we should either support this or (if there is good reason not to) update the 
comment.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] kafka pull request #3796: MINOR: KIP-138 renaming of string names

[guozhangwang]

GitHub user guozhangwang opened a pull request:

https://github.com/apache/kafka/pull/3796

MINOR: KIP-138 renaming of string names



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/guozhangwang/kafka kip-138-minor-renames

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3796.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3796


commit 947ea33f11ec5b40b2f9669570c76310e2c00fbd
Author: Guozhang Wang 
Date:   2017-08-30T18:17:11Z

rename system-time to wall-clock-time

commit 87b4e7b4345f647bd8562e9552ba192ee3744c47
Author: Guozhang Wang 
Date:   2017-09-05T23:56:42Z

Merge branch 'trunk' of https://git-wip-us.apache.org/repos/asf/kafka into 
kip-138-minor-renames




---

Build failed in Jenkins: kafka-0.11.0-jdk7 #299

[Apache Jenkins Server]

See 


Changes:

[wangguoz] MINOR: logging improvements on StreamThread

--
[...truncated 2.09 MB...]

org.apache.kafka.clients.admin.KafkaAdminClientTest > testDescribeAcls PASSED

org.apache.kafka.clients.admin.KafkaAdminClientTest > testPrettyPrintException 
STARTED

org.apache.kafka.clients.admin.KafkaAdminClientTest > testPrettyPrintException 
PASSED

org.apache.kafka.clients.admin.KafkaAdminClientTest > 
testTimeoutWithoutMetadata STARTED

org.apache.kafka.clients.admin.KafkaAdminClientTest > 
testTimeoutWithoutMetadata PASSED

org.apache.kafka.clients.admin.KafkaAdminClientTest > testDescribeConfigs 
STARTED

org.apache.kafka.clients.admin.KafkaAdminClientTest > testDescribeConfigs PASSED

org.apache.kafka.clients.admin.KafkaAdminClientTest > testCreateAcls STARTED

org.apache.kafka.clients.admin.KafkaAdminClientTest > testCreateAcls PASSED

org.apache.kafka.clients.admin.KafkaAdminClientTest > testCloseAdminClient 
STARTED

org.apache.kafka.clients.admin.KafkaAdminClientTest > testCloseAdminClient 
PASSED

org.apache.kafka.clients.admin.KafkaAdminClientTest > testDeleteAcls STARTED

org.apache.kafka.clients.admin.KafkaAdminClientTest > testDeleteAcls PASSED

org.apache.kafka.clients.admin.KafkaAdminClientTest > testHandleTimeout STARTED

org.apache.kafka.clients.admin.KafkaAdminClientTest > testHandleTimeout PASSED

org.apache.kafka.clients.admin.KafkaAdminClientTest > testGenerateClientId 
STARTED

org.apache.kafka.clients.admin.KafkaAdminClientTest > testGenerateClientId 
PASSED

org.apache.kafka.clients.admin.KafkaAdminClientTest > 
testCalcTimeoutMsRemainingAsInt STARTED

org.apache.kafka.clients.admin.KafkaAdminClientTest > 
testCalcTimeoutMsRemainingAsInt PASSED

org.apache.kafka.clients.MetadataTest > testListenerCanUnregister STARTED

org.apache.kafka.clients.MetadataTest > testListenerCanUnregister PASSED

org.apache.kafka.clients.MetadataTest > testTopicExpiry STARTED

org.apache.kafka.clients.MetadataTest > testTopicExpiry PASSED

org.apache.kafka.clients.MetadataTest > testFailedUpdate STARTED

org.apache.kafka.clients.MetadataTest > testFailedUpdate PASSED

org.apache.kafka.clients.MetadataTest > testMetadataUpdateWaitTime STARTED

org.apache.kafka.clients.MetadataTest > testMetadataUpdateWaitTime PASSED

org.apache.kafka.clients.MetadataTest > testUpdateWithNeedMetadataForAllTopics 
STARTED

org.apache.kafka.clients.MetadataTest > testUpdateWithNeedMetadataForAllTopics 
PASSED

org.apache.kafka.clients.MetadataTest > testClusterListenerGetsNotifiedOfUpdate 
STARTED

org.apache.kafka.clients.MetadataTest > testClusterListenerGetsNotifiedOfUpdate 
PASSED

org.apache.kafka.clients.MetadataTest > testTimeToNextUpdate_RetryBackoff 
STARTED

org.apache.kafka.clients.MetadataTest > testTimeToNextUpdate_RetryBackoff PASSED

org.apache.kafka.clients.MetadataTest > testMetadata STARTED

org.apache.kafka.clients.MetadataTest > testMetadata PASSED

org.apache.kafka.clients.MetadataTest > testTimeToNextUpdate_OverwriteBackoff 
STARTED

org.apache.kafka.clients.MetadataTest > testTimeToNextUpdate_OverwriteBackoff 
PASSED

org.apache.kafka.clients.MetadataTest > testTimeToNextUpdate STARTED

org.apache.kafka.clients.MetadataTest > testTimeToNextUpdate PASSED

org.apache.kafka.clients.MetadataTest > testListenerGetsNotifiedOfUpdate STARTED

org.apache.kafka.clients.MetadataTest > testListenerGetsNotifiedOfUpdate PASSED

org.apache.kafka.clients.MetadataTest > testNonExpiringMetadata STARTED

org.apache.kafka.clients.MetadataTest > testNonExpiringMetadata PASSED
:clients:determineCommitId UP-TO-DATE
:clients:createVersionFile
:clients:jar UP-TO-DATE
:core:compileJava NO-SOURCE
:core:compileScala UP-TO-DATE
:core:processResources NO-SOURCE
:core:classes UP-TO-DATE
:core:copyDependantLibs
:core:jar
:examples:compileJava:19:
 warning: [deprecation] FetchRequest in kafka.api has been deprecated
import kafka.api.FetchRequest;
^
:20:
 warning: [deprecation] FetchRequestBuilder in kafka.api has been deprecated
import kafka.api.FetchRequestBuilder;
^
:22:
 warning: [deprecation] SimpleConsumer in kafka.javaapi.consumer has been 
deprecated
import kafka.javaapi.consumer.SimpleConsumer;
 ^
3 warnings

:examples:processResources NO-SOURCE
:examples:classes
:examples:checkstyleMain
:examples:compileTestJava NO-SOURCE
:examples:processTestResources NO-SOURCE
:examples:testClasses UP-TO-DATE
:examples:checkstyleTest NO-SOURCE
:examples:findbugsMain
Scanning archives (0 / 19)Scanning archives (1 / 19)

[jira] [Resolved] (KAFKA-3856) Cleanup Kafka Streams builder API

[Matthias J. Sax (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax resolved KAFKA-3856.

Resolution: Fixed

> Cleanup Kafka Streams builder API
> -
>
> Key: KAFKA-3856
> URL: https://issues.apache.org/jira/browse/KAFKA-3856
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Guozhang Wang
>Assignee: Matthias J. Sax
>  Labels: api, kip
> Fix For: 1.0.0
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-120%3A+Cleanup+Kafka+Streams+builder+API



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [VOTE] KIP-188 - Add new metrics to support health checks

[Ismael Juma]

I think I prefer the names with `Message` in them. For people less familiar
with Kafka, it makes it a bit clearer, I think.

Ismael

On Wed, Sep 6, 2017 at 12:39 AM, Rajini Sivaram 
wrote:

> I am ok with dropping 'Message'. So the names would be
> FetchConversionsPerSec,
> ProduceConversionsPerSec and ConversionsTimeMs. The first two sound fine.
> Not so sure about ConversionsTimeMs, but since it appears with
> Produce/Fetch as the request tag, it should be ok. I haven't updated the
> KIP yet. If there are no objections, I will update the KIP tomorrow.
>
> Regards,
>
> Rajini
>
> On Tue, Sep 5, 2017 at 7:23 PM, Jason Gustafson 
> wrote:
>
> > >
> > > I was wondering about the message versus record question. The fact that
> > we
> > > already have MessagesInPerSec seemed to favour the former. The other
> > aspect
> > > is that for produce requests, we can up convert as well, so it seemed
> > > better to keep it generic.
> >
> >
> > Yeah, so I thought maybe we could bypass the question and drop `Message`
> > from the names if they were already clear enough. I'm fine with either
> way.
> >
> > On Tue, Sep 5, 2017 at 11:09 AM, Ismael Juma  wrote:
> >
> > > I was wondering about the message versus record question. The fact that
> > we
> > > already have MessagesInPerSec seemed to favour the former. The other
> > aspect
> > > is that for produce requests, we can up convert as well, so it seemed
> > > better to keep it generic.
> > >
> > > Ismael
> > >
> > > On Tue, Sep 5, 2017 at 6:51 PM, Jason Gustafson 
> > > wrote:
> > >
> > > > +1 Lots of good stuff in here.
> > > >
> > > > One minor nit: in the name `FetchDownConversionsPerSec`, it's
> implicit
> > > that
> > > > down-conversion is for messages. Could we do the same for
> > > > `MessageConversionsTimeMs` and drop the `Message`? Then we don't have
> > to
> > > > decide if it should be 'Record' instead.
> > > >
> > > > On Tue, Sep 5, 2017 at 10:20 AM, Ismael Juma 
> > wrote:
> > > >
> > > > > Thanks Rajini.
> > > > >
> > > > > 1. I meant a topic metric, but we could have one for fetch and one
> > for
> > > > > produce differentiated by the additional tag. The advantage is that
> > the
> > > > > name would be consistent with the request metric for message
> > > conversions.
> > > > > However, on closer inspection, this would make the name
> inconsistent
> > > with
> > > > > the broker topic metrics:
> > > > >
> > > > > val totalProduceRequestRate =
> > > > > newMeter(BrokerTopicStats.TotalProduceRequestsPerSec, "requests",
> > > > > TimeUnit.SECONDS, tags)
> > > > > val totalFetchRequestRate =
> > > > > newMeter(BrokerTopicStats.TotalFetchRequestsPerSec, "requests",
> > > > > TimeUnit.SECONDS, tags)
> > > > >
> > > > > So, we maybe we can instead go for FetchMessageConversionsPerSeco
> nd
> > > and
> > > > > ProduceMessageConversionsPerSec.
> > > > >
> > > > > 2. Sounds good.
> > > > >
> > > > > Ismael
> > > > >
> > > > > On Tue, Sep 5, 2017 at 5:46 PM, Rajini Sivaram <
> > > rajinisiva...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi Ismael,
> > > > > >
> > > > > > 1. At the moment FetchDownConversionsPerSec is a topic metric
> while
> > > > > > MessageConversionTimeMs is a request metric which indicates
> > > > Produce/Fetch
> > > > > > as a tag. Are you suggesting that we should convert
> > > > > > FetchDownConversionsPerSec to a request metric called
> > > > > > MessageConversionsPerSec
> > > > > > for fetch requests?
> > > > > >
> > > > > > 2. TemporaryMessageSize for Produce/Fetch would indicate the
> space
> > > > > > allocated for conversions. For other requests, this metric will
> not
> > > be
> > > > > > created (unless we find a request where the size is large and
> this
> > > > > > information is useful).
> > > > > >
> > > > > > Thank you,
> > > > > >
> > > > > > Rajini
> > > > > >
> > > > > >
> > > > > > On Tue, Sep 5, 2017 at 4:55 PM, Ismael Juma 
> > > wrote:
> > > > > >
> > > > > > > Thanks Rajini, +1 (binding) from me. Just a few minor comments:
> > > > > > >
> > > > > > > 1. FetchDownConversionsPerSec should probably be
> > > > > MessageConversionsPerSec
> > > > > > > with a request tag for consistency with
> MessageConversionsTimeMs.
> > > The
> > > > > > text
> > > > > > > in that paragraph should also be updated to talk about message
> > > > > > conversions
> > > > > > > instead of down conversions only.
> > > > > > >
> > > > > > > 2. What will TemporaryMemorySize represent for requests other
> > than
> > > > > > > `ProduceRequest`?
> > > > > > >
> > > > > > > Ismael
> > > > > > >
> > > > > > > On Mon, Sep 4, 2017 at 2:09 PM, Rajini Sivaram <
> > > > > rajinisiva...@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > All the suggestions on the discuss thread have been
> > incorporated
> > > > into
> > > > > > the
> > > > > > > > KIP. Please let me know if you have any more concerns or else
> > can
> > > > we
> > > > > > > > proceed with voting for this KIP?
> > > > > > > >
> > > > > > > > Thank you,
> > > > > > > >
> > > 

Re: [VOTE] KIP-188 - Add new metrics to support health checks

[Jason Gustafson]

>
> I think I prefer the names with `Message` in them. For people less familiar
> with Kafka, it makes it a bit clearer, I think.


Works for me.

On Tue, Sep 5, 2017 at 5:19 PM, Ismael Juma  wrote:

> I think I prefer the names with `Message` in them. For people less familiar
> with Kafka, it makes it a bit clearer, I think.
>
> Ismael
>
> On Wed, Sep 6, 2017 at 12:39 AM, Rajini Sivaram 
> wrote:
>
> > I am ok with dropping 'Message'. So the names would be
> > FetchConversionsPerSec,
> > ProduceConversionsPerSec and ConversionsTimeMs. The first two sound fine.
> > Not so sure about ConversionsTimeMs, but since it appears with
> > Produce/Fetch as the request tag, it should be ok. I haven't updated the
> > KIP yet. If there are no objections, I will update the KIP tomorrow.
> >
> > Regards,
> >
> > Rajini
> >
> > On Tue, Sep 5, 2017 at 7:23 PM, Jason Gustafson 
> > wrote:
> >
> > > >
> > > > I was wondering about the message versus record question. The fact
> that
> > > we
> > > > already have MessagesInPerSec seemed to favour the former. The other
> > > aspect
> > > > is that for produce requests, we can up convert as well, so it seemed
> > > > better to keep it generic.
> > >
> > >
> > > Yeah, so I thought maybe we could bypass the question and drop
> `Message`
> > > from the names if they were already clear enough. I'm fine with either
> > way.
> > >
> > > On Tue, Sep 5, 2017 at 11:09 AM, Ismael Juma 
> wrote:
> > >
> > > > I was wondering about the message versus record question. The fact
> that
> > > we
> > > > already have MessagesInPerSec seemed to favour the former. The other
> > > aspect
> > > > is that for produce requests, we can up convert as well, so it seemed
> > > > better to keep it generic.
> > > >
> > > > Ismael
> > > >
> > > > On Tue, Sep 5, 2017 at 6:51 PM, Jason Gustafson 
> > > > wrote:
> > > >
> > > > > +1 Lots of good stuff in here.
> > > > >
> > > > > One minor nit: in the name `FetchDownConversionsPerSec`, it's
> > implicit
> > > > that
> > > > > down-conversion is for messages. Could we do the same for
> > > > > `MessageConversionsTimeMs` and drop the `Message`? Then we don't
> have
> > > to
> > > > > decide if it should be 'Record' instead.
> > > > >
> > > > > On Tue, Sep 5, 2017 at 10:20 AM, Ismael Juma 
> > > wrote:
> > > > >
> > > > > > Thanks Rajini.
> > > > > >
> > > > > > 1. I meant a topic metric, but we could have one for fetch and
> one
> > > for
> > > > > > produce differentiated by the additional tag. The advantage is
> that
> > > the
> > > > > > name would be consistent with the request metric for message
> > > > conversions.
> > > > > > However, on closer inspection, this would make the name
> > inconsistent
> > > > with
> > > > > > the broker topic metrics:
> > > > > >
> > > > > > val totalProduceRequestRate =
> > > > > > newMeter(BrokerTopicStats.TotalProduceRequestsPerSec,
> "requests",
> > > > > > TimeUnit.SECONDS, tags)
> > > > > > val totalFetchRequestRate =
> > > > > > newMeter(BrokerTopicStats.TotalFetchRequestsPerSec, "requests",
> > > > > > TimeUnit.SECONDS, tags)
> > > > > >
> > > > > > So, we maybe we can instead go for FetchMessageConversionsPerSeco
> > nd
> > > > and
> > > > > > ProduceMessageConversionsPerSec.
> > > > > >
> > > > > > 2. Sounds good.
> > > > > >
> > > > > > Ismael
> > > > > >
> > > > > > On Tue, Sep 5, 2017 at 5:46 PM, Rajini Sivaram <
> > > > rajinisiva...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Ismael,
> > > > > > >
> > > > > > > 1. At the moment FetchDownConversionsPerSec is a topic metric
> > while
> > > > > > > MessageConversionTimeMs is a request metric which indicates
> > > > > Produce/Fetch
> > > > > > > as a tag. Are you suggesting that we should convert
> > > > > > > FetchDownConversionsPerSec to a request metric called
> > > > > > > MessageConversionsPerSec
> > > > > > > for fetch requests?
> > > > > > >
> > > > > > > 2. TemporaryMessageSize for Produce/Fetch would indicate the
> > space
> > > > > > > allocated for conversions. For other requests, this metric will
> > not
> > > > be
> > > > > > > created (unless we find a request where the size is large and
> > this
> > > > > > > information is useful).
> > > > > > >
> > > > > > > Thank you,
> > > > > > >
> > > > > > > Rajini
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Sep 5, 2017 at 4:55 PM, Ismael Juma  >
> > > > wrote:
> > > > > > >
> > > > > > > > Thanks Rajini, +1 (binding) from me. Just a few minor
> comments:
> > > > > > > >
> > > > > > > > 1. FetchDownConversionsPerSec should probably be
> > > > > > MessageConversionsPerSec
> > > > > > > > with a request tag for consistency with
> > MessageConversionsTimeMs.
> > > > The
> > > > > > > text
> > > > > > > > in that paragraph should also be updated to talk about
> message
> > > > > > > conversions
> > > > > > > > instead of down conversions only.
> > > > > > > >
> > > > > > > > 2. What will TemporaryMemorySize represent for requests other
> > > than
> > > > > > > > `ProduceRequest`?
> > > > > > >

Re: [DISCUSS] KIP-91 Provide Intuitive User Timeouts in The Producer

[Ismael Juma]

Thanks for updating the KIP, Sumant. A couple of points:

1. I think the default for delivery.timeout.ms should be higher than 30
seconds given that we previously would reset the clock once the batch was
sent. The value should be large enough that batches are not expired due to
expected events like a new leader being elected due to broker failure.
Would it make sense to use a conservative value like 120 seconds?

2. The producer currently throws an exception for configuration
combinations that don't make sense. We should probably do the same here for
consistency (the KIP currently proposes a log warning).

3. We should mention that we will not cancel in flight requests until the
request timeout even though we'll expire the batch early if needed.

I think we should start the vote tomorrow so that we have a chance of
hitting the KIP freeze for 1.0.0.

Ismael

On Wed, Sep 6, 2017 at 1:03 AM, Sumant Tambe  wrote:

> I've updated the kip-91 writeup
>  91+Provide+Intuitive+User+Timeouts+in+The+Producer>
> to capture some of the discussion here. Please confirm if it's sufficiently
> accurate. Feel free to edit it if you think some explanation can be better
> and has been agreed upon here.
>
> How do you proceed from here?
>
> -Sumant
>
> On 30 August 2017 at 12:59, Jun Rao  wrote:
>
> > Hi, Jiangjie,
> >
> > I mis-understood Jason's approach earlier. It does seem to be a good one.
> > We still need to calculate the selector timeout based on the remaining
> > delivery.timeout.ms to call the callback on time, but we can always wait
> > for an inflight request based on request.timeout.ms.
> >
> > Thanks,
> >
> > Jun
> >
> > On Tue, Aug 29, 2017 at 5:16 PM, Becket Qin 
> wrote:
> >
> > > Yeah, I think expiring a batch but still wait for the response is
> > probably
> > > reasonable given the result is not guaranteed anyways.
> > >
> > > @Jun,
> > >
> > > I think the frequent PID reset may still be possible if we do not wait
> > for
> > > the in-flight response to return. Consider two partitions p0 and p1,
> the
> > > deadline of the batches for p0 are T + 10, T + 30, T + 50... The
> deadline
> > > of the batches for p1 are T + 20, T + 40, T + 60... Assuming each
> request
> > > takes more than 10 ms to get the response. The following sequence may
> be
> > > possible:
> > >
> > > T: PID0 send batch0_p0(PID0), batch0_p1(PID0)
> > > T + 10: PID0 expires batch0_p0(PID0), without resetting PID, sends
> > > batch1_p0(PID0) and batch0_p1(PID0, retry)
> > > T + 20: PID0 expires batch0_p1(PID0, retry), resets the PID to PID1,
> > sends
> > > batch1_p0(PID0, retry) and batch1_p1(PID1)
> > > T + 30: PID1 expires batch1_p0(PID0, retry), without resetting PID,
> sends
> > > batch2_p0(PID1) and batch1_p1(PID1, retry)
> > > T + 40: PID1 expires batch1_p1(PID1, retry), resets the PID to PID2,
> > sends
> > > batch2_p0(PID1, retry) and sends batch2_p1(PID2)
> > > 
> > >
> > > In the above example, the producer will reset PID once every two
> > requests.
> > > The example did not take retry backoff into consideration, but it still
> > > seems possible to encounter frequent PID reset if we do not wait for
> the
> > > request to finish. Also, in this case we will have a lot of retries and
> > > mixture of PIDs which seem to be pretty complicated.
> > >
> > > I think Jason's suggestion will address both concerns, i.e. we fire the
> > > callback at exactly delivery.timeout.ms, but we will still wait for
> the
> > > response to be returned before sending the next request.
> > >
> > > Thanks,
> > >
> > > Jiangjie (Becket) Qin
> > >
> > >
> > > On Tue, Aug 29, 2017 at 4:00 PM, Jun Rao  wrote:
> > >
> > > > Hmm, I thought delivery.timeout.ms bounds the time from a message is
> > in
> > > > the
> > > > accumulator (i.e., when send() returns) to the time when the callback
> > is
> > > > called. If we wait for request.timeout.ms for an inflight request
> and
> > > the
> > > > remaining delivery.timeout.ms is less than request.timeout.ms, the
> > > > callback
> > > > may be called later than delivery.timeout.ms, right?
> > > >
> > > > Jiangjie's concern on resetting the pid on every expired batch is
> > > probably
> > > > not an issue if we only reset the pid when the expired batch's pid is
> > the
> > > > same as the current pid, as Jason suggested.
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Tue, Aug 29, 2017 at 3:09 PM, Jason Gustafson  >
> > > > wrote:
> > > >
> > > > > I think the semantics of delivery.timeout.ms need to allow for the
> > > > > possibility that the record was actually written. Unless we can
> keep
> > on
> > > > > retrying indefinitely, there's really no way to know for sure
> whether
> > > the
> > > > > record was written or not. A delivery timeout just means that we
> > cannot
> > > > > guarantee that the record was delivered.
> > > > >
> > > > > -Jason
> > > > >
> > > > > On Tue, Aug 29, 2017 at 2:51 PM, Becket Qin 
> > > > wrote:
> > > > >
> 

[GitHub] kafka pull request #3535: KAFKA-5597: Autogenerate producer sender metrics.

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3535


---

Build failed in Jenkins: kafka-trunk-jdk7 #2711

[Apache Jenkins Server]

See 


Changes:

[wangguoz] KAFKA-4819: Expose states for active tasks to public API

--
[...truncated 689.33 KB...]

kafka.api.LegacyAdminClientTest > testListAllBrokerVersionInfo PASSED

kafka.api.LegacyAdminClientTest > testOffsetsForTimesWhenOffsetNotFound STARTED

kafka.api.LegacyAdminClientTest > testOffsetsForTimesWhenOffsetNotFound PASSED

kafka.api.LegacyAdminClientTest > testDescribeConsumerGroupForNonExistentGroup 
STARTED

kafka.api.LegacyAdminClientTest > testDescribeConsumerGroupForNonExistentGroup 
PASSED

kafka.api.LegacyAdminClientTest > testLogStartOffsetAfterDeleteRecords STARTED

kafka.api.LegacyAdminClientTest > testLogStartOffsetAfterDeleteRecords PASSED

kafka.api.LegacyAdminClientTest > testGetConsumerGroupSummary STARTED

kafka.api.LegacyAdminClientTest > testGetConsumerGroupSummary PASSED

kafka.api.SaslMultiMechanismConsumerTest > testMultipleBrokerMechanisms STARTED

kafka.api.SaslMultiMechanismConsumerTest > testMultipleBrokerMechanisms PASSED

kafka.api.SaslMultiMechanismConsumerTest > testCoordinatorFailover STARTED

kafka.api.SaslMultiMechanismConsumerTest > testCoordinatorFailover PASSED

kafka.api.SaslMultiMechanismConsumerTest > testSimpleConsumption STARTED

kafka.api.SaslMultiMechanismConsumerTest > testSimpleConsumption PASSED

kafka.api.AdminClientWithPoliciesIntegrationTest > testInvalidAlterConfigs 
STARTED

kafka.api.AdminClientWithPoliciesIntegrationTest > testInvalidAlterConfigs 
PASSED

kafka.api.AdminClientWithPoliciesIntegrationTest > testValidAlterConfigs STARTED

kafka.api.AdminClientWithPoliciesIntegrationTest > testValidAlterConfigs PASSED

kafka.api.AdminClientWithPoliciesIntegrationTest > 
testInvalidAlterConfigsDueToPolicy STARTED

kafka.api.AdminClientWithPoliciesIntegrationTest > 
testInvalidAlterConfigsDueToPolicy PASSED

kafka.api.GroupCoordinatorIntegrationTest > 
testGroupCoordinatorPropagatesOfffsetsTopicCompressionCodec STARTED

kafka.api.GroupCoordinatorIntegrationTest > 
testGroupCoordinatorPropagatesOfffsetsTopicCompressionCodec PASSED

kafka.api.SslConsumerTest > testCoordinatorFailover STARTED

kafka.api.SslConsumerTest > testCoordinatorFailover PASSED

kafka.api.SslConsumerTest > testSimpleConsumption STARTED

kafka.api.SslConsumerTest > testSimpleConsumption PASSED

kafka.api.SslEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaSubscribe STARTED

kafka.api.SslEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaSubscribe PASSED

kafka.api.SslEndToEndAuthorizationTest > testProduceConsumeViaAssign STARTED

kafka.api.SslEndToEndAuthorizationTest > testProduceConsumeViaAssign PASSED

kafka.api.SslEndToEndAuthorizationTest > testNoConsumeWithDescribeAclViaAssign 
STARTED

kafka.api.SslEndToEndAuthorizationTest > testNoConsumeWithDescribeAclViaAssign 
PASSED

kafka.api.SslEndToEndAuthorizationTest > 
testNoConsumeWithDescribeAclViaSubscribe STARTED

kafka.api.SslEndToEndAuthorizationTest > 
testNoConsumeWithDescribeAclViaSubscribe PASSED

kafka.api.SslEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaAssign STARTED

kafka.api.SslEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaAssign PASSED

kafka.api.SslEndToEndAuthorizationTest > testNoGroupAcl STARTED

kafka.api.SslEndToEndAuthorizationTest > testNoGroupAcl PASSED

kafka.api.SslEndToEndAuthorizationTest > testNoProduceWithDescribeAcl STARTED

kafka.api.SslEndToEndAuthorizationTest > testNoProduceWithDescribeAcl PASSED

kafka.api.SslEndToEndAuthorizationTest > testProduceConsumeViaSubscribe STARTED

kafka.api.SslEndToEndAuthorizationTest > testProduceConsumeViaSubscribe PASSED

kafka.api.SslEndToEndAuthorizationTest > testNoProduceWithoutDescribeAcl STARTED

kafka.api.SslEndToEndAuthorizationTest > testNoProduceWithoutDescribeAcl PASSED

kafka.api.FetchRequestTest > testShuffleWithSingleTopic STARTED

kafka.api.FetchRequestTest > testShuffleWithSingleTopic PASSED

kafka.api.FetchRequestTest > testShuffle STARTED

kafka.api.FetchRequestTest > testShuffle PASSED

kafka.api.SaslSslConsumerTest > testCoordinatorFailover STARTED

kafka.api.SaslSslConsumerTest > testCoordinatorFailover PASSED

kafka.api.SaslSslConsumerTest > testSimpleConsumption STARTED

kafka.api.SaslSslConsumerTest > testSimpleConsumption PASSED

kafka.api.test.ProducerCompressionTest > testCompression[0 compressionType = 
none] STARTED

kafka.api.test.ProducerCompressionTest > testCompression[0 compressionType = 
none] PASSED

kafka.api.test.ProducerCompressionTest > testCompression[1 compressionType = 
gzip] STARTED

kafka.api.test.ProducerCompressionTest > testCompression[1 compressionType = 
gzip] PASSED

kafka.api.test.ProducerCompressionTest > testCompression[2 compressionType = 
snappy] STARTED

kafka.api.test.ProducerCompressionTest > testCompression[2 compressionType = 
snappy] PASSED

kafka.api.test.ProducerCompressionTest > testCompression[3 compr

Build failed in Jenkins: kafka-0.11.0-jdk7 #300

[Apache Jenkins Server]

See 


Changes:

[wangguoz] HOTFIX: remove unused imports

--
[...truncated 2.08 MB...]

org.apache.kafka.clients.producer.MockProducerTest > 
shouldThrowOnAbortTransactionIfProducerGotFenced STARTED

org.apache.kafka.clients.producer.MockProducerTest > 
shouldThrowOnAbortTransactionIfProducerGotFenced PASSED

org.apache.kafka.clients.producer.MockProducerTest > testPartitioner STARTED

org.apache.kafka.clients.producer.MockProducerTest > testPartitioner PASSED

org.apache.kafka.clients.producer.MockProducerTest > 
shouldThrowOnFenceProducerIfProducerIsClosed STARTED

org.apache.kafka.clients.producer.MockProducerTest > 
shouldThrowOnFenceProducerIfProducerIsClosed PASSED

org.apache.kafka.clients.producer.MockProducerTest > 
shouldThrowOnCommitTransactionIfProducerIsClosed STARTED

org.apache.kafka.clients.producer.MockProducerTest > 
shouldThrowOnCommitTransactionIfProducerIsClosed PASSED

org.apache.kafka.clients.producer.MockProducerTest > 
shouldThrowFenceProducerIfTransactionsNotInitialized STARTED

org.apache.kafka.clients.producer.MockProducerTest > 
shouldThrowFenceProducerIfTransactionsNotInitialized PASSED

org.apache.kafka.clients.producer.MockProducerTest > 
shouldThrowOnFlushIfProducerGotFenced STARTED

org.apache.kafka.clients.producer.MockProducerTest > 
shouldThrowOnFlushIfProducerGotFenced PASSED

org.apache.kafka.clients.producer.MockProducerTest > 
shouldCountCommittedTransaction STARTED

org.apache.kafka.clients.producer.MockProducerTest > 
shouldCountCommittedTransaction PASSED

org.apache.kafka.clients.producer.MockProducerTest > 
shouldPreserveCommittedConsumerGroupsOffsetsOnAbortIfTransactionsAreEnabled 
STARTED

org.apache.kafka.clients.producer.MockProducerTest > 
shouldPreserveCommittedConsumerGroupsOffsetsOnAbortIfTransactionsAreEnabled 
PASSED

org.apache.kafka.clients.producer.MockProducerTest > 
shouldBeFlushedIfNoBufferedRecords STARTED

org.apache.kafka.clients.producer.MockProducerTest > 
shouldBeFlushedIfNoBufferedRecords PASSED

org.apache.kafka.clients.producer.MockProducerTest > 
shouldPublishMessagesOnlyAfterCommitIfTransactionsAreEnabled STARTED

org.apache.kafka.clients.producer.MockProducerTest > 
shouldPublishMessagesOnlyAfterCommitIfTransactionsAreEnabled PASSED

org.apache.kafka.clients.producer.MockProducerTest > 
shouldFlushOnCommitForNonAutoCompleteIfTransactionsAreEnabled STARTED

org.apache.kafka.clients.producer.MockProducerTest > 
shouldFlushOnCommitForNonAutoCompleteIfTransactionsAreEnabled PASSED

org.apache.kafka.clients.producer.MockProducerTest > 
shouldThrowOnAbortTransactionIfNoTransactionGotStarted STARTED

org.apache.kafka.clients.producer.MockProducerTest > 
shouldThrowOnAbortTransactionIfNoTransactionGotStarted PASSED

org.apache.kafka.clients.producer.MockProducerTest > 
shouldThrowOnBeginTransactionIfProducerIsClosed STARTED

org.apache.kafka.clients.producer.MockProducerTest > 
shouldThrowOnBeginTransactionIfProducerIsClosed PASSED

org.apache.kafka.clients.producer.MockProducerTest > 
shouldDropMessagesOnAbortIfTransactionsAreEnabled STARTED

org.apache.kafka.clients.producer.MockProducerTest > 
shouldDropMessagesOnAbortIfTransactionsAreEnabled PASSED

org.apache.kafka.clients.producer.MockProducerTest > 
shouldThrowOnSendIfProducerIsClosed STARTED

org.apache.kafka.clients.producer.MockProducerTest > 
shouldThrowOnSendIfProducerIsClosed PASSED

org.apache.kafka.clients.producer.MockProducerTest > 
shouldThrowOnCommitIfTransactionsNotInitialized STARTED

org.apache.kafka.clients.producer.MockProducerTest > 
shouldThrowOnCommitIfTransactionsNotInitialized PASSED

org.apache.kafka.clients.producer.MockProducerTest > 
shouldThrowOnBeginTransactionsIfProducerGotFenced STARTED

org.apache.kafka.clients.producer.MockProducerTest > 
shouldThrowOnBeginTransactionsIfProducerGotFenced PASSED

org.apache.kafka.clients.producer.MockProducerTest > 
shouldThrowOnAbortIfTransactionsNotInitialized STARTED

org.apache.kafka.clients.producer.MockProducerTest > 
shouldThrowOnAbortIfTransactionsNotInitialized PASSED

org.apache.kafka.clients.producer.MockProducerTest > shouldBeginTransactions 
STARTED

org.apache.kafka.clients.producer.MockProducerTest > shouldBeginTransactions 
PASSED

org.apache.kafka.clients.producer.MockProducerTest > 
shouldThrowOnSendOffsetsToTransactionIfTransactionsNotInitialized STARTED

org.apache.kafka.clients.producer.MockProducerTest > 
shouldThrowOnSendOffsetsToTransactionIfTransactionsNotInitialized PASSED

org.apache.kafka.clients.producer.MockProducerTest > 
shouldThrowOnCommitTransactionIfProducerGotFenced STARTED

org.apache.kafka.clients.producer.MockProducerTest > 
shouldThrowOnCommitTransactionIfProducerGotFenced PASSED

org.apache.kafka.clients.producer.MockProducerTest > 
shouldAbortEmptyTransaction STARTED

org.apache.kafka.clients.producer.MockProducerTest > 
shouldAbortEmptyTransaction PASSED

o

integration between pull request and JIRA

[Ted Yu]

Hi,
Currently the conversations on pull request are not posted back to JIRA.

Is there technical hurdle preventing this from being done ?

Other Apache projects, such as Flink, establish automatic post from pull
request to JIRA.

Cheers

[jira] [Created] (KAFKA-5842) QueryableStateIntegrationTest may fail with JDK 7

[Ted Yu (JIRA)]

Ted Yu created KAFKA-5842:
-

 Summary: QueryableStateIntegrationTest may fail with JDK 7
 Key: KAFKA-5842
 URL: https://issues.apache.org/jira/browse/KAFKA-5842
 Project: Kafka
  Issue Type: Test
Reporter: Ted Yu
Priority: Minor


Found the following when running test suite for 0.11.0.1 RC0 :
{code}
org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
concurrentAccesses FAILED
java.lang.AssertionError: Key not found one
at org.junit.Assert.fail(Assert.java:88)
at 
org.apache.kafka.streams.integration.QueryableStateIntegrationTest.verifyGreaterOrEqual(QueryableStateIntegrationTest.java:893)
at 
org.apache.kafka.streams.integration.QueryableStateIntegrationTest.concurrentAccesses(QueryableStateIntegrationTest.java:399)
{code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is back to normal : kafka-trunk-jdk7 #2712

[Apache Jenkins Server]

See 

Re: integration between pull request and JIRA

[Matthias J. Sax]

This integration was never set up for Kafka.

I personally don't see any advantage in this, as it just duplicates
everything and does not add value IMHO. The PRs are linked and one can
go to the PR to read the discussion if interested.

Or what do you think the value would be?


-Matthias


On 9/5/17 6:16 PM, Ted Yu wrote:
> Hi,
> Currently the conversations on pull request are not posted back to JIRA.
> 
> Is there technical hurdle preventing this from being done ?
> 
> Other Apache projects, such as Flink, establish automatic post from pull
> request to JIRA.
> 
> Cheers
> 



signature.asc
Description: OpenPGP digital signature

Re: integration between pull request and JIRA

[Ted Yu]

If a person watches github PR, that person watches conversations on all
PRs, not just the one he / she intends to pay attention to.

Quite often this leads to ton of emails in his / her inbox which is
distracting.

If the conversation is posted from PR to JIRA, watcher is per PR / JIRA.
This is much focused.

Cheers

On Tue, Sep 5, 2017 at 7:20 PM, Matthias J. Sax 
wrote:

> This integration was never set up for Kafka.
>
> I personally don't see any advantage in this, as it just duplicates
> everything and does not add value IMHO. The PRs are linked and one can
> go to the PR to read the discussion if interested.
>
> Or what do you think the value would be?
>
>
> -Matthias
>
>
> On 9/5/17 6:16 PM, Ted Yu wrote:
> > Hi,
> > Currently the conversations on pull request are not posted back to JIRA.
> >
> > Is there technical hurdle preventing this from being done ?
> >
> > Other Apache projects, such as Flink, establish automatic post from pull
> > request to JIRA.
> >
> > Cheers
> >
>
>

[GitHub] kafka pull request #3188: KAFKA-5358: Consumer perf tool should count rebala...

[huxihx]

GitHub user huxihx reopened a pull request:

https://github.com/apache/kafka/pull/3188

KAFKA-5358: Consumer perf tool should count rebalance time.

Added 'join.group.ms' for new consumer to calculate the time of joining 
group. 

@hachikuji  Please review the PR. Thanks.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/huxihx/kafka KAKFA-5358

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3188.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3188


commit be43bf3a1257ca5f058e38af3e185ba775749614
Author: amethystic 
Date:   2017-06-01T09:15:15Z

KAFKA-5358: Consumer perf tool should count rebalance time.

Added 'join.group.ms' for new consumer to calculate the time of joining 
group.

commit cbdf6c10d12e1b0528dc45c078e61bb6ee1b0d2f
Author: amethystic 
Date:   2017-06-02T07:11:03Z

1. Refined the name to `total.rebalance.time`
2. Added `total.fetch.time`
3. Add support to count total time for multiple rebalances

commit 7600e6b994c251422444c905f5d4c718bf6f9935
Author: huxihx 
Date:   2017-06-06T00:42:29Z

Refined time counting for both fetcher and rebalance as per hachikuji's 
comments.

commit 08ff452fcb41136b54cfc2850540b18d737e909d
Author: huxihx 
Date:   2017-06-07T06:27:00Z

Correct the counting for total fetch time.

commit 8c80f1376dba7ac715fa0887e128293690a7014c
Author: huxihx 
Date:   2017-06-08T01:31:49Z

1. Split `MB.sec` into two parts: `total.MB.sec` and `fetch.MB.sec`
2. Ditto for `nMsg.sec`
3. Refined output format

commit edf1d0888728bc5c39b6ee59d23d3eed108243c6
Author: huxihx 
Date:   2017-06-21T01:24:28Z

returned back to the original output format for new consumer

commit 722e16df1b96275a04bd2e6a5e3deec476b028a6
Author: huxihx 
Date:   2017-06-27T02:03:55Z

As per hackikuji's comments, refined code to print out rebalance time even 
when  is set.

commit 57bd0e44723b681a191a12135e3cc60188c18e9d
Author: huxihx 
Date:   2017-08-04T02:27:51Z

resovled conflicts with trunk

commit 16cc2cab52c394ed4688fe377496cd98d1e2eb4b
Author: huxi 
Date:   2017-08-04T03:42:17Z

Merge branch 'trunk' into KAKFA-5358

commit 37982726a6fd09ef202eb428a2dd364aded1b929
Author: huxihx 
Date:   2017-08-04T04:06:25Z

KAFKA-5358: Did not show newly-created headers if `--show-detailed-stats` 
is set since rebalance time does not change during most of the consuming rounds.

commit bbba9b00a6538e69bc4261ed456a31993e42650e
Author: huxihx 
Date:   2017-08-09T02:15:56Z

correct printHeader invoking to have testHeaderMatchBody passed

commit 3a1ce0f8897577608182ffe1a8ce2f9bfbe41f55
Author: huxihx 
Date:   2017-08-17T02:44:23Z

Added newly-created fields for detailed views.

commit c51781586646664bb9514d8ab03aa974f9e6a942
Author: huxihx 
Date:   2017-08-21T01:18:24Z

1. Added parameter name when invoking testDetailedHeaderMatchBody; 2. 
Removed useless initialization for

commit 9adda74004c531894a31b7653529b32336682316
Author: huxihx 
Date:   2017-09-06T02:41:42Z

added a field that tracks periodic join time.




---

[GitHub] kafka pull request #3723: KAFKA-5358: Consumer perf tool should count rebala...

[huxihx]

Github user huxihx closed the pull request at:

https://github.com/apache/kafka/pull/3723


---

Re: integration between pull request and JIRA

[Matthias J. Sax]

>> If a person watches github PR, that person watches conversations on all
>> PRs,

One can just "not watch" Kafka's Github repo. I don't watch it either
and thus I get emails for only those PRs I did get tagged or I did
comment on etc.

Would this not work for you?


-Matthias

On 9/5/17 7:31 PM, Ted Yu wrote:
> If a person watches github PR, that person watches conversations on all
> PRs, not just the one he / she intends to pay attention to.
> 
> Quite often this leads to ton of emails in his / her inbox which is
> distracting.
> 
> If the conversation is posted from PR to JIRA, watcher is per PR / JIRA.
> This is much focused.
> 
> Cheers
> 
> On Tue, Sep 5, 2017 at 7:20 PM, Matthias J. Sax 
> wrote:
> 
>> This integration was never set up for Kafka.
>>
>> I personally don't see any advantage in this, as it just duplicates
>> everything and does not add value IMHO. The PRs are linked and one can
>> go to the PR to read the discussion if interested.
>>
>> Or what do you think the value would be?
>>
>>
>> -Matthias
>>
>>
>> On 9/5/17 6:16 PM, Ted Yu wrote:
>>> Hi,
>>> Currently the conversations on pull request are not posted back to JIRA.
>>>
>>> Is there technical hurdle preventing this from being done ?
>>>
>>> Other Apache projects, such as Flink, establish automatic post from pull
>>> request to JIRA.
>>>
>>> Cheers
>>>
>>
>>
> 



signature.asc
Description: OpenPGP digital signature

Re: integration between pull request and JIRA

[Ted Yu]

bq. I did get tagged or I did comment on etc.

What if nobody tags me on the PR and I don't comment on it ?

Cheers

On Tue, Sep 5, 2017 at 8:55 PM, Matthias J. Sax 
wrote:

> >> If a person watches github PR, that person watches conversations on all
> >> PRs,
>
> One can just "not watch" Kafka's Github repo. I don't watch it either
> and thus I get emails for only those PRs I did get tagged or I did
> comment on etc.
>
> Would this not work for you?
>
>
> -Matthias
>
> On 9/5/17 7:31 PM, Ted Yu wrote:
> > If a person watches github PR, that person watches conversations on all
> > PRs, not just the one he / she intends to pay attention to.
> >
> > Quite often this leads to ton of emails in his / her inbox which is
> > distracting.
> >
> > If the conversation is posted from PR to JIRA, watcher is per PR / JIRA.
> > This is much focused.
> >
> > Cheers
> >
> > On Tue, Sep 5, 2017 at 7:20 PM, Matthias J. Sax 
> > wrote:
> >
> >> This integration was never set up for Kafka.
> >>
> >> I personally don't see any advantage in this, as it just duplicates
> >> everything and does not add value IMHO. The PRs are linked and one can
> >> go to the PR to read the discussion if interested.
> >>
> >> Or what do you think the value would be?
> >>
> >>
> >> -Matthias
> >>
> >>
> >> On 9/5/17 6:16 PM, Ted Yu wrote:
> >>> Hi,
> >>> Currently the conversations on pull request are not posted back to
> JIRA.
> >>>
> >>> Is there technical hurdle preventing this from being done ?
> >>>
> >>> Other Apache projects, such as Flink, establish automatic post from
> pull
> >>> request to JIRA.
> >>>
> >>> Cheers
> >>>
> >>
> >>
> >
>
>

Permissions to create a KIP?

[Matthias Margush]

Hi,

Can I be set up with access to author a KIP (
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals)?
My ID is matthias.margush.

Thanks,

Matthias
-- 

*Matthias Margush*

*matthias.marg...@fundingcircle.com * |
415.813.5245 (d) | 415.236.2884. (m)

747 Front St, 4th Fl | San Francisco, CA 94111

-- 


Our Mission: To build a better financial world

Unless specifically indicated, this e-mail is not an offer to sell or a 
solicitation of any investment products or other financial product or 
service, an official confirmation of any transaction, or an official 
statement of Funding Circle USA.  This e-mail is meant only for the 
intended recipient of this transmission, and contains trade secret 
and strictly confidential information belonging to the sender. It is 
unlawful for unauthorized individuals to review, use, copy, disclose, or 
disseminate confidential information. If you have received this e-mail in 
error, please notify the sender immediately by telephone at (415) 
813-5245 or by return email and promptly delete this message from your 
system.

Re: Permissions to create a KIP?

[Guozhang Wang]

Hi Matthias,

I have granted you the permission. Cheers.


Guozhang


On Tue, Sep 5, 2017 at 2:21 PM, Matthias Margush <
matthias.marg...@fundingcircle.com> wrote:

> Hi,
>
> Can I be set up with access to author a KIP (
> https://cwiki.apache.org/confluence/display/KAFKA/
> Kafka+Improvement+Proposals)?
> My ID is matthias.margush.
>
> Thanks,
>
> Matthias
> --
>
> *Matthias Margush*
>
> *matthias.marg...@fundingcircle.com * |
> 415.813.5245 (d) | 415.236.2884. (m)
>
> 747 Front St, 4th Fl | San Francisco, CA 94111
>
> --
>
>
> Our Mission: To build a better financial world
>
> Unless specifically indicated, this e-mail is not an offer to sell or a
> solicitation of any investment products or other financial product or
> service, an official confirmation of any transaction, or an official
> statement of Funding Circle USA.  This e-mail is meant only for the
> intended recipient of this transmission, and contains trade secret
> and strictly confidential information belonging to the sender. It is
> unlawful for unauthorized individuals to review, use, copy, disclose, or
> disseminate confidential information. If you have received this e-mail in
> error, please notify the sender immediately by telephone at (415)
> 813-5245 or by return email and promptly delete this message from your
> system.
>



-- 
-- Guozhang

[jira] [Created] (KAFKA-5843) Mx4jLoader.maybeLoad should only be executed if kafka_mx4jenable is set to true

[Dong Lin (JIRA)]

Dong Lin created KAFKA-5843:
---

 Summary: Mx4jLoader.maybeLoad should only be executed if 
kafka_mx4jenable is set to true
 Key: KAFKA-5843
 URL: https://issues.apache.org/jira/browse/KAFKA-5843
 Project: Kafka
  Issue Type: Bug
Reporter: Dong Lin
Assignee: Dong Lin






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] kafka pull request #3797: KAFKA-5843; Mx4jLoader.maybeLoad should only be ex...

[lindong28]

GitHub user lindong28 opened a pull request:

https://github.com/apache/kafka/pull/3797

KAFKA-5843; Mx4jLoader.maybeLoad should only be executed if 
kafka_mx4jenable is set to true



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lindong28/kafka KAFKA-5843

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3797.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3797


commit 5c1bd1c09d975dfde7f1831bf5cd091b4440bfa1
Author: Dong Lin 
Date:   2017-09-06T05:11:46Z

KAFKA-5843; Mx4jLoader.maybeLoad should only be executed if 
kafka_mx4jenable is set to true




---

[GitHub] kafka pull request #3798: KAFKA-5841: AbstractIndex should offer `makeReadOn...

[huxihx]

GitHub user huxihx opened a pull request:

https://github.com/apache/kafka/pull/3798

KAFKA-5841: AbstractIndex should offer `makeReadOnly` method

AbstractIndex should offer `makeReadOnly` method that changed the 
underlying MappedByteBuffer read-only.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/huxihx/kafka KAFKA-5841

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3798.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3798


commit a2d97ff6c814368ac7e7eadc63569de36d3965af
Author: huxihx 
Date:   2017-09-06T06:48:54Z

KAFKA-5841: AbstractIndex should offer `makeReadOnly` method as mentioned 
in comments




---