Build failed in Jenkins: kafka-trunk-jdk8 #4396

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-6145: Pt. 5 Implement high availability assignment (#8337)

[github] KAFKA-9778: Add methods to validate and assert connector configurations

[github] KAFKA-9809: Shrink transaction timeout for streams (#8407)

[github] KAFKA-9776: Downgrade TxnCommit API v3 when broker doesn't support


--
[...truncated 2.99 MB...]
org.apache.kafka.streams.TopologyTestDriverTest > shouldRespectTaskIdling[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldApplyGlobalUpdatesCorrectlyInRecursiveTopologies[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldApplyGlobalUpdatesCorrectlyInRecursiveTopologies[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForLargerValue[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForLargerValue[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectInMemoryStoreTypeOnly[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectInMemoryStoreTypeOnly[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldThrowForMissingTime[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldThrowForMissingTime[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTimeDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTimeDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessRecordForTopic[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessRecordForTopic[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldForwardRecordsFromSubtopologyToSubtopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldForwardRecordsFromSubtopologyToSubtopology[Eos enabled = false] PASSED

org.apach

Re: [DISCUSS] KIP-584: Versioning scheme for features

[Kowshik Prakasam]

Hey Boyang,

Great point! You are right, thanks for the suggestion!
Yes, we can just use ZK watches to propagate finalized features
information. I have updated the KIP write up with this change.
As a result, I feel the design is simpler as we have also eliminated
the changes to UpdateMetadataRequest.

You are right, after exploring/discussing KIP-500 further, we have now
realized that taking a ZK dependency here in this KIP just for reads is OK.
The future migration path off ZK (in post ZK world) will simply involve
reading the finalized features from the controller quorum via the new
MetadataFetch API that's proposed in KIP-500.

Also note that in the latest KIP write-up, the features metadata epoch
is just the ZK node version (as suggested by Jun).

Hey Colin,

Please feel free to let us know if you have any questions or concerns
on the above.


Cheers,
Kowshik

On Thu, Apr 2, 2020 at 10:39 AM Boyang Chen 
wrote:

> Thanks for the reply. The only remaining question is the propagation path.
> KIP-500  only restricts
> `write access` to the controller, in a sense that
> brokers in the pre-KIP-500 
> world could still listen to Zookeeper
> notifications. Thus, we are open to discuss the engineering effort to go
> through Zookeeper vs UpdateMetadata routing. What's your opinion on this
> matter? Will either path significantly simpler than another?
>
> Boyang
>
> On Wed, Apr 1, 2020 at 12:10 AM Kowshik Prakasam 
> wrote:
>
> > Hey Boyang,
> >
> > Thanks for the feedback! Please find below my response to your latest
> > comments.
> > I have modified the KIP wherever possible to address the comments.
> >
> > > My point is that during a bootstrapping stage of a cluster, we could
> not
> > > pick the desired feature version as no controller is actively handling
> > our
> > > request.
> >
> > (Kowshik): Note that just deploying the latest broker binary does not
> > always mean that the
> > new version of a certain feature will be automatically activated.
> Enabling
> > the effects of the
> > actual feature version is still left to the discretion of the
> > implementation logic for  the feature.
> > For example, for safety reasons, the feature can still be gated behind a
> > dynamic config
> > and later activated when the time comes.
> >
> > > Feature changes should be roughly the same frequency as config changes.
> > > Today, the dynamic configuration changes are propagated via Zookeeper.
> > > So I guess propagating through UpdateMetadata doesn't get us more
> > benefits,
> > > while going through ZK notification should be a simpler solution.
> >
> > (Kowshik): Maybe I'm missing something, but were you suggesting we should
> > have these
> > notifications delivered to the brokers directly via ZK? Note that with
> > KIP-500  (where we are
> replacing ZK),
> > for the bridge release we prefer that we will perform all access to ZK in
> > the controller,
> > rather than in other brokers, clients, or tools. Therefore, although ZK
> > will still be
> > required for the bridge release, it will be a well-isolated dependency.
> > Please read
> > this section of KIP-500 :
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum#KIP-500:ReplaceZooKeeperwithaSelf-ManagedMetadataQuorum-BridgeRelease
> > .
> >
> > Therefore, the existing approach in the KIP is future proof with regards
> to
> > the above requirement.
> > We deliver the ZK notification only via the controller's
> > `UpdateMetadataRequest` to the brokers.
> > We also access ZK only always via the controller.
> >
> > > Understood, I don't feel strong about deprecation, but does the current
> > KIP
> > > keep the door open for future improvements if
> > > someone has a need for feature deprecation? Could we briefly discuss
> > about
> > > it in the future work section?
> >
> > (Kowshik): Done. Please refer to the 'Future work' section:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-584%3A+Versioning+scheme+for+features#KIP-584:Versioningschemeforfeatures-Futurework
> >
> >
> > Cheers,
> > Kowshik
> >
> >
> > On Tue, Mar 31, 2020 at 9:12 PM Boyang Chen 
> > wrote:
> >
> > > Thanks Kowshik, the answers are making sense. Some follow-ups:
> > >
> > > On Tue, Mar 31, 2020 at 6:51 PM Jun Rao  wrote:
> > >
> > > > Hi, Kowshik,
> > > >
> > > > Thanks for the KIP. Looks good overall. A few comments below.
> > > >
> > > > 100. UpdateFeaturesRequest/UpdateFeaturesResponse
> > > > 100.1 Since this request waits for responses from brokers, should we
> > add
> > > a
> > > > timeout in the request (like createTopicRequest)?
> > > > 100.2 The response schema is a bit weird. Typically, the response
> just
> > > > shows an error code and an error message, instead of echoing the
> > request.
> > > > 100.3 Should we

Re: [DISCUSS] KIP-584: Versioning scheme for features

[Kowshik Prakasam]

Hey Jun,

Thanks a lot for the great feedback! Please note that the design
has changed a little bit on the KIP, and we now propagate the finalized
features metadata only via ZK watches (instead of UpdateMetadataRequest
from the controller).

Please find below my response to your questions/feedback, with the prefix
"(Kowshik):".

> 100. UpdateFeaturesRequest/UpdateFeaturesResponse
> 100.1 Since this request waits for responses from brokers, should we add a
> timeout in the request (like createTopicRequest)?

(Kowshik): Great point! Done. I have added a timeout field. Note: we no
longer
wait for responses from brokers, since the design has been changed so that
the
features information is propagated via ZK. Nevertheless, it is right to
have a timeout
for the request.

> 100.2 The response schema is a bit weird. Typically, the response just
> shows an error code and an error message, instead of echoing the request.

(Kowshik): Great point! Yeah, I have modified it to just return an error
code and a message.
Previously it was not echoing the "request", rather it was returning the
latest set of
cluster-wide finalized features (after applying the updates). But you are
right,
the additional info is not required, so I have removed it from the response
schema.

> 100.3 Should we add a separate request to list/describe the existing
> features?

(Kowshik): This is already present in the KIP via the 'DescribeFeatures'
Admin API,
which, underneath covers uses the ApiVersionsRequest to list/describe the
existing features. Please read the 'Tooling support' section.

> 100.4 We are mixing ADD_OR_UPDATE and DELETE in a single request. For
> DELETE, the version field doesn't make sense. So, I guess the broker just
> ignores this? An alternative way is to have a separate
DeleteFeaturesRequest

(Kowshik): Great point! I have modified the KIP now to have 2 separate
controller APIs
serving these different purposes:
1. updateFeatures
2. deleteFeatures

> 100.5 In UpdateFeaturesResponse, we have "The monotonically increasing
> version of the metadata for finalized features." I am wondering why the
> ordering is important?

(Kowshik): In the latest KIP write-up, it is called epoch (instead of
version), and
it is just the ZK node version. Basically, this is the epoch for the
cluster-wide
finalized feature version metadata. This metadata is served to clients via
the
ApiVersionsResponse (for reads). We propagate updates from the '/features'
ZK node
to all brokers, via ZK watches setup by each broker on the '/features' node.

Now here is why the ordering is important:
ZK watches don't propagate at the same time. As a result, the
ApiVersionsResponse
is eventually consistent across brokers. This can introduce cases
where clients see an older lower epoch of the features metadata, after a
more recent
higher epoch was returned at a previous point in time. We expect clients
to always employ the rule that the latest received higher epoch of metadata
always trumps an older smaller epoch. Those clients that are external to
Kafka should strongly consider discovering the latest metadata once during
startup from the brokers, and if required refresh the metadata periodically
(to get the latest metadata).

> 100.6 Could you specify the required ACL for this new request?

(Kowshik): What is ACL, and how could I find out which one to specify?
Please could you provide me some pointers? I'll be glad to update the
KIP once I know the next steps.

> 101. For the broker registration ZK node, should we bump up the version in
the json?

(Kowshik): Great point! Done. I've increased the version in the broker json
by 1.

> 102. For the /features ZK node, not sure if we need the epoch field. Each
> ZK node has an internal version field that is incremented on every update.

(Kowshik): Great point! Done. I'm using the ZK node version now, instead of
explicitly
incremented epoch.

> 103. "Enabling the actual semantics of a feature version cluster-wide is
> left to the discretion of the logic implementing the feature (ex: can be
> done via dynamic broker config)." Does that mean the broker registration
ZK
> node will be updated dynamically when this happens?

(Kowshik): Not really. The text was just conveying that a broker could
"know" of
a new feature version, but it does not mean the broker should have also
activated the effects of the feature version. Knowing vs activation are 2
separate things,
and the latter can be achieved by dynamic config. I have reworded the text
to
make this clear to the reader.


> 104. UpdateMetadataRequest
> 104.1 It would be useful to describe when the feature metadata is included
> in the request. My understanding is that it's only included if (1) there
is
> a change to the finalized feature; (2) broker restart; (3) controller
> failover.
> 104.2 The new fields have the following versions. Why are the versions 3+
> when the top version is bumped to 6?
>   "fields":  [
> {"name": "Name", "type":  "string", "versions":  "3+"

[jira] [Created] (KAFKA-9813) __consumer_offsets loaded with a very long time

[leibo (Jira)]

leibo created KAFKA-9813:


 Summary: __consumer_offsets loaded with a very long time
 Key: KAFKA-9813
 URL: https://issues.apache.org/jira/browse/KAFKA-9813
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 2.1.1
Reporter: leibo


__consumer_offsets loaded with a long time.

 

After I restart kafka(3 HA), one of __consumer_offsets loaded with  41256909 ms 
(about 11 hours), and many consumer can not commit offsets.

restart time:  2020-04-02 19:06

__consumer_offsets-14 loaded time: 2020-04-03 06:32,  41256909 ms

other info: 

     there are 72408 consumer group in kafka cluster, and most of it (70498) 
are empty.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Build failed in Jenkins: kafka-trunk-jdk11 #1317

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-9778: Add methods to validate and assert connector configurations


--
[...truncated 2.37 MB...]

org.apache.kafka.streams.state.internals.CachingKeyValueStoreTest > 
shouldThrowNullPointerExceptionOnGetNullKey STARTED

org.apache.kafka.streams.state.internals.CachingKeyValueStoreTest > 
shouldThrowNullPointerExceptionOnGetNullKey PASSED

org.apache.kafka.streams.state.internals.CachingKeyValueStoreTest > 
shouldThrowNullPointerExceptionOnRangeNullFromKey STARTED

org.apache.kafka.streams.state.internals.CachingKeyValueStoreTest > 
shouldThrowNullPointerExceptionOnRangeNullFromKey PASSED

org.apache.kafka.streams.state.internals.CachingKeyValueStoreTest > 
shouldThrowNullPointerExceptionOnDeleteNullKey STARTED

org.apache.kafka.streams.state.internals.CachingKeyValueStoreTest > 
shouldThrowNullPointerExceptionOnDeleteNullKey PASSED

org.apache.kafka.streams.state.internals.CachingKeyValueStoreTest > 
shouldReturnSameResultsForGetAndRangeWithEqualKeys STARTED

org.apache.kafka.streams.state.internals.CachingKeyValueStoreTest > 
shouldReturnSameResultsForGetAndRangeWithEqualKeys PASSED

org.apache.kafka.streams.state.internals.CachingKeyValueStoreTest > 
testPutGetRangeWithDefaultSerdes STARTED

org.apache.kafka.streams.state.internals.CachingKeyValueStoreTest > 
testPutGetRangeWithDefaultSerdes PASSED

org.apache.kafka.streams.state.internals.MergedSortedCacheKeyValueBytesStoreIteratorTest
 > shouldNotHaveNextIfOnlyCacheItemsAndAllDeleted STARTED

org.apache.kafka.streams.state.internals.MergedSortedCacheKeyValueBytesStoreIteratorTest
 > shouldNotHaveNextIfOnlyCacheItemsAndAllDeleted PASSED

org.apache.kafka.streams.state.internals.MergedSortedCacheKeyValueBytesStoreIteratorTest
 > shouldNotHaveNextIfAllCachedItemsDeleted STARTED

org.apache.kafka.streams.state.internals.MergedSortedCacheKeyValueBytesStoreIteratorTest
 > shouldNotHaveNextIfAllCachedItemsDeleted PASSED

org.apache.kafka.streams.state.internals.MergedSortedCacheKeyValueBytesStoreIteratorTest
 > shouldIterateOverRange STARTED

org.apache.kafka.streams.state.internals.MergedSortedCacheKeyValueBytesStoreIteratorTest
 > shouldIterateOverRange PASSED

org.apache.kafka.streams.state.internals.MergedSortedCacheKeyValueBytesStoreIteratorTest
 > shouldIgnoreIfDeletedInCacheButExistsInStore STARTED

org.apache.kafka.streams.state.internals.MergedSortedCacheKeyValueBytesStoreIteratorTest
 > shouldIgnoreIfDeletedInCacheButExistsInStore PASSED

org.apache.kafka.streams.state.internals.MergedSortedCacheKeyValueBytesStoreIteratorTest
 > shouldSkipAllDeletedFromCache STARTED

org.apache.kafka.streams.state.internals.MergedSortedCacheKeyValueBytesStoreIteratorTest
 > shouldSkipAllDeletedFromCache PASSED

org.apache.kafka.streams.state.internals.MergedSortedCacheKeyValueBytesStoreIteratorTest
 > shouldSkipLargerDeletedCacheValue STARTED

org.apache.kafka.streams.state.internals.MergedSortedCacheKeyValueBytesStoreIteratorTest
 > shouldSkipLargerDeletedCacheValue PASSED

org.apache.kafka.streams.state.internals.MergedSortedCacheKeyValueBytesStoreIteratorTest
 > shouldSkipSmallerDeletedCachedValue STARTED

org.apache.kafka.streams.state.internals.MergedSortedCacheKeyValueBytesStoreIteratorTest
 > shouldSkipSmallerDeletedCachedValue PASSED

org.apache.kafka.streams.state.internals.MergedSortedCacheKeyValueBytesStoreIteratorTest
 > shouldPeekNextKey STARTED

org.apache.kafka.streams.state.internals.MergedSortedCacheKeyValueBytesStoreIteratorTest
 > shouldPeekNextKey PASSED

org.apache.kafka.streams.state.HostInfoTest > 
shouldReturnNullHostInfoForNullEndPoint STARTED

org.apache.kafka.streams.state.HostInfoTest > 
shouldReturnNullHostInfoForNullEndPoint PASSED

org.apache.kafka.streams.state.HostInfoTest > 
shouldReturnNullHostInfoForEmptyEndPoint STARTED

org.apache.kafka.streams.state.HostInfoTest > 
shouldReturnNullHostInfoForEmptyEndPoint PASSED

org.apache.kafka.streams.state.HostInfoTest > shouldCreateHostInfo STARTED

org.apache.kafka.streams.state.HostInfoTest > shouldCreateHostInfo PASSED

org.apache.kafka.streams.state.HostInfoTest > 
shouldThrowConfigExceptionForNonsenseEndPoint STARTED

org.apache.kafka.streams.state.HostInfoTest > 
shouldThrowConfigExceptionForNonsenseEndPoint PASSED

org.apache.kafka.streams.state.StreamsMetadataTest > 
shouldNotAllowModificationOfInternalStateViaGetters STARTED

org.apache.kafka.streams.state.StreamsMetadataTest > 
shouldNotAllowModificationOfInternalStateViaGetters PASSED
streams-6: SMOKE-TEST-CLIENT-CLOSED
streams-4: SMOKE-TEST-CLIENT-CLOSED
streams-5: SMOKE-TEST-CLIENT-CLOSED
streams-0: SMOKE-TEST-CLIENT-CLOSED
streams-2: SMOKE-TEST-CLIENT-CLOSED
streams-3: SMOKE-TEST-CLIENT-CLOSED
streams-1: SMOKE-TEST-CLIENT-CLOSED
ERROR: No tool found matching GRADLE_4_10_2_HOME
ERROR: Could not install GRADLE_4_10_3_HOME
java.lang.NullPointerException
at 
hudson.plu

Re: [DISCUSS] KIP-584: Versioning scheme for features

[Kowshik Prakasam]

Hi all,

Any other feedback on this KIP before we start the vote?


Cheers,
Kowshik

On Fri, Apr 3, 2020 at 1:27 AM Kowshik Prakasam 
wrote:

> Hey Jun,
>
> Thanks a lot for the great feedback! Please note that the design
> has changed a little bit on the KIP, and we now propagate the finalized
> features metadata only via ZK watches (instead of UpdateMetadataRequest
> from the controller).
>
> Please find below my response to your questions/feedback, with the prefix
> "(Kowshik):".
>
> > 100. UpdateFeaturesRequest/UpdateFeaturesResponse
> > 100.1 Since this request waits for responses from brokers, should we add
> a
> > timeout in the request (like createTopicRequest)?
>
> (Kowshik): Great point! Done. I have added a timeout field. Note: we no
> longer
> wait for responses from brokers, since the design has been changed so that
> the
> features information is propagated via ZK. Nevertheless, it is right to
> have a timeout
> for the request.
>
> > 100.2 The response schema is a bit weird. Typically, the response just
> > shows an error code and an error message, instead of echoing the request.
>
> (Kowshik): Great point! Yeah, I have modified it to just return an error
> code and a message.
> Previously it was not echoing the "request", rather it was returning the
> latest set of
> cluster-wide finalized features (after applying the updates). But you are
> right,
> the additional info is not required, so I have removed it from the
> response schema.
>
> > 100.3 Should we add a separate request to list/describe the existing
> > features?
>
> (Kowshik): This is already present in the KIP via the 'DescribeFeatures'
> Admin API,
> which, underneath covers uses the ApiVersionsRequest to list/describe the
> existing features. Please read the 'Tooling support' section.
>
> > 100.4 We are mixing ADD_OR_UPDATE and DELETE in a single request. For
> > DELETE, the version field doesn't make sense. So, I guess the broker just
> > ignores this? An alternative way is to have a separate
> DeleteFeaturesRequest
>
> (Kowshik): Great point! I have modified the KIP now to have 2 separate
> controller APIs
> serving these different purposes:
> 1. updateFeatures
> 2. deleteFeatures
>
> > 100.5 In UpdateFeaturesResponse, we have "The monotonically increasing
> > version of the metadata for finalized features." I am wondering why the
> > ordering is important?
>
> (Kowshik): In the latest KIP write-up, it is called epoch (instead of
> version), and
> it is just the ZK node version. Basically, this is the epoch for the
> cluster-wide
> finalized feature version metadata. This metadata is served to clients via
> the
> ApiVersionsResponse (for reads). We propagate updates from the '/features'
> ZK node
> to all brokers, via ZK watches setup by each broker on the '/features'
> node.
>
> Now here is why the ordering is important:
> ZK watches don't propagate at the same time. As a result, the
> ApiVersionsResponse
> is eventually consistent across brokers. This can introduce cases
> where clients see an older lower epoch of the features metadata, after a
> more recent
> higher epoch was returned at a previous point in time. We expect clients
> to always employ the rule that the latest received higher epoch of metadata
> always trumps an older smaller epoch. Those clients that are external to
> Kafka should strongly consider discovering the latest metadata once during
> startup from the brokers, and if required refresh the metadata periodically
> (to get the latest metadata).
>
> > 100.6 Could you specify the required ACL for this new request?
>
> (Kowshik): What is ACL, and how could I find out which one to specify?
> Please could you provide me some pointers? I'll be glad to update the
> KIP once I know the next steps.
>
> > 101. For the broker registration ZK node, should we bump up the version
> in
> the json?
>
> (Kowshik): Great point! Done. I've increased the version in the broker
> json by 1.
>
> > 102. For the /features ZK node, not sure if we need the epoch field. Each
> > ZK node has an internal version field that is incremented on every
> update.
>
> (Kowshik): Great point! Done. I'm using the ZK node version now, instead
> of explicitly
> incremented epoch.
>
> > 103. "Enabling the actual semantics of a feature version cluster-wide is
> > left to the discretion of the logic implementing the feature (ex: can be
> > done via dynamic broker config)." Does that mean the broker registration
> ZK
> > node will be updated dynamically when this happens?
>
> (Kowshik): Not really. The text was just conveying that a broker could
> "know" of
> a new feature version, but it does not mean the broker should have also
> activated the effects of the feature version. Knowing vs activation are 2
> separate things,
> and the latter can be achieved by dynamic config. I have reworded the text
> to
> make this clear to the reader.
>
>
> > 104. UpdateMetadataRequest
> > 104.1 It would be useful to describe when the feature metadata is
> include

Build failed in Jenkins: kafka-trunk-jdk11 #1318

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-9809: Shrink transaction timeout for streams (#8407)

[github] KAFKA-9776: Downgrade TxnCommit API v3 when broker doesn't support


--
[...truncated 4.76 MB...]
kafka.admin.ConfigCommandTest > shouldFailIfShortBrokerEntityTypeIsNotAnInteger 
PASSED

kafka.admin.ConfigCommandTest > testDescribeAllBrokerConfig STARTED

kafka.admin.ConfigCommandTest > testDescribeAllBrokerConfig PASSED

kafka.admin.ConfigCommandTest > shouldExitWithNonZeroStatusOnBrokerCommandError 
STARTED

kafka.admin.ConfigCommandTest > shouldExitWithNonZeroStatusOnBrokerCommandError 
PASSED

kafka.admin.ConfigCommandTest > shouldParseArgumentsForBrokersEntityType STARTED

kafka.admin.ConfigCommandTest > shouldParseArgumentsForBrokersEntityType PASSED

kafka.admin.ConfigCommandTest > testDynamicBrokerConfigUpdateUsingZooKeeper 
STARTED

kafka.admin.ConfigCommandTest > testDynamicBrokerConfigUpdateUsingZooKeeper 
PASSED

kafka.admin.ConfigCommandTest > 
testNoSpecifiedEntityOptionWithDescribeBrokersInZKIsAllowed STARTED

kafka.admin.ConfigCommandTest > 
testNoSpecifiedEntityOptionWithDescribeBrokersInZKIsAllowed PASSED

kafka.admin.ConfigCommandTest > testQuotaDescribeEntities STARTED

kafka.admin.ConfigCommandTest > testQuotaDescribeEntities PASSED

kafka.admin.ConfigCommandTest > shouldParseArgumentsForClientsEntityType STARTED

kafka.admin.ConfigCommandTest > shouldParseArgumentsForClientsEntityType PASSED

kafka.admin.ConfigCommandTest > 
testDescribeAllBrokerConfigBootstrapServerRequired STARTED

kafka.admin.ConfigCommandTest > 
testDescribeAllBrokerConfigBootstrapServerRequired PASSED

kafka.admin.ConfigCommandTest > shouldExitWithNonZeroStatusOnArgError STARTED

kafka.admin.ConfigCommandTest > shouldExitWithNonZeroStatusOnArgError PASSED

kafka.admin.ConfigCommandTest > shouldAddTopicConfigUsingZookeeper STARTED

kafka.admin.ConfigCommandTest > shouldAddTopicConfigUsingZookeeper PASSED

kafka.admin.DeleteConsumerGroupsTest > testDeleteNonEmptyGroup STARTED

kafka.api.PlaintextConsumerTest > testMaxPollIntervalMsDelayInRevocation PASSED

kafka.api.PlaintextConsumerTest > testPerPartitionLagMetricsCleanUpWithAssign 
STARTED

kafka.admin.DeleteConsumerGroupsTest > testDeleteNonEmptyGroup PASSED

kafka.admin.DeleteConsumerGroupsTest > testDeleteCmdWithMixOfSuccessAndError 
STARTED

kafka.admin.DeleteConsumerGroupsTest > testDeleteCmdWithMixOfSuccessAndError 
PASSED

kafka.admin.DeleteConsumerGroupsTest > testDeleteWithTopicOption STARTED

kafka.api.PlaintextConsumerTest > testPerPartitionLagMetricsCleanUpWithAssign 
PASSED

kafka.api.PlaintextConsumerTest > testPartitionsForInvalidTopic STARTED

kafka.admin.DeleteConsumerGroupsTest > testDeleteWithTopicOption PASSED

kafka.admin.DeleteConsumerGroupsTest > testDeleteNonExistingGroup STARTED

kafka.api.PlaintextConsumerTest > testPartitionsForInvalidTopic PASSED

kafka.api.PlaintextConsumerTest > testMultiConsumerStickyAssignment STARTED

kafka.admin.DeleteConsumerGroupsTest > testDeleteNonExistingGroup PASSED

kafka.admin.DeleteConsumerGroupsTest > testDeleteCmdNonEmptyGroup STARTED

kafka.admin.DeleteConsumerGroupsTest > testDeleteCmdNonEmptyGroup PASSED

kafka.admin.DeleteConsumerGroupsTest > testDeleteCmdNonExistingGroup STARTED

kafka.admin.DeleteConsumerGroupsTest > testDeleteCmdNonExistingGroup PASSED

kafka.admin.DeleteConsumerGroupsTest > testDeleteEmptyGroup STARTED

kafka.admin.DeleteConsumerGroupsTest > testDeleteEmptyGroup PASSED

kafka.admin.DeleteConsumerGroupsTest > testDeleteWithMixOfSuccessAndError 
STARTED

kafka.admin.DeleteConsumerGroupsTest > testDeleteWithMixOfSuccessAndError PASSED

kafka.admin.DeleteConsumerGroupsTest > 
testDeleteWithUnrecognizedNewConsumerOption STARTED

kafka.admin.DeleteConsumerGroupsTest > 
testDeleteWithUnrecognizedNewConsumerOption PASSED

kafka.admin.DeleteConsumerGroupsTest > testDeleteCmdAllGroups STARTED

kafka.api.PlaintextConsumerTest > testMultiConsumerStickyAssignment PASSED

kafka.api.PlaintextConsumerTest > testPauseStateNotPreservedByRebalance STARTED

kafka.admin.DeleteConsumerGroupsTest > testDeleteCmdAllGroups PASSED

kafka.admin.DeleteConsumerGroupsTest > testDeleteCmdEmptyGroup STARTED

kafka.api.PlaintextConsumerTest > testPauseStateNotPreservedByRebalance PASSED

kafka.api.PlaintextConsumerTest > 
testFetchHonoursFetchSizeIfLargeRecordNotFirst STARTED

kafka.admin.DeleteConsumerGroupsTest > testDeleteCmdEmptyGroup PASSED

kafka.admin.DeleteOffsetsConsumerGroupCommandIntegrationTest > 
testDeleteOffsetsOfStableConsumerGroupWithTopicOnly STARTED

kafka.admin.DeleteOffsetsConsumerGroupCommandIntegrationTest > 
testDeleteOffsetsOfStableConsumerGroupWithTopicOnly PASSED

kafka.admin.DeleteOffsetsConsumerGroupCommandIntegrationTest > 
testDeleteOffsetsOfEmptyConsumerGroupWithUnknownTopicOnly STARTED

kafka.api.PlaintextConsumerTest > 
testFetchHonoursFetchSizeIfLargeRecordNotF

[jira] [Created] (KAFKA-9814) Allow AdminClient.listTopics to list based on expression

[Viktor Somogyi-Vass (Jira)]

Viktor Somogyi-Vass created KAFKA-9814:
--

 Summary: Allow AdminClient.listTopics to list based on expression
 Key: KAFKA-9814
 URL: https://issues.apache.org/jira/browse/KAFKA-9814
 Project: Kafka
  Issue Type: Improvement
Affects Versions: 2.5.0
Reporter: Viktor Somogyi-Vass
Assignee: Viktor Somogyi-Vass


Currently listTopics would list all the topics in the cluster which often isn't 
desired due to the large number of topics and because the authorizer would 
check access for every topic which could result in various bloated access/audit 
logs that would make them hard to read and also in case of many topics it isn't 
really effective to return all of them due to the response's size.

To take an example let's have a user who issues a naive command such as:
{noformat}
kafka-topics --list --topic topic6 --bootstrap-server mycompany.com:9092 
--command-config client.properties
{noformat}
In this case the admin client would issue a metadata request for all the 
topics. When this request gets to the broker then it will try to check 
permissions to every topic in the cluster and then return those which are 
allowed for "describe" to the client. Then the command when it gets the answer 
will simply filter out {{topic6}} from the response.

Looking at this problem we can see that if we allow the broker to do the 
filtering then we can:
* do less permission checks
* return less data to the clients

Regarding the solution I see two path but both of them of course subject 
community discussion:
# modify the MetadataRequest and put some kind of regex in it 
# create a new list API that would contain this filter pattern
Perhaps the list API would make more sense as the MetadataRequest is currently 
used because it just fits but there is no real reason to extend it with this as 
the functionality won't be used elsewhere.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] KIP-586: Deprecate commit records without record metadata

[Randall Hauch]

Hi, Mario.

Thanks for creating this small but useful KIP! The only suggestion I have
is to include the JavaDoc for the method we want to deprecate so that we
can include the line:

* @deprecated Use {@link #commitRecord(SourceRecord, RecordMetadata)}


This will make it more clear that there are two methods with the same name
and we're deprecating only one of them.

Best regards,

Randall

On Wed, Apr 1, 2020 at 8:07 PM Mario Molina  wrote:

> Hi all,
>
> I'd like to use this thread to discuss KIP-586.
> You can see the detail at:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-586%3A+Deprecate+commit+records+without+record+metadata
>
> Best,
> Mario
>

[jira] [Created] (KAFKA-9815) Consumer may never re-join if inconsistent metadata is received once

[Rajini Sivaram (Jira)]

Rajini Sivaram created KAFKA-9815:
-

 Summary: Consumer may never re-join if inconsistent metadata is 
received once
 Key: KAFKA-9815
 URL: https://issues.apache.org/jira/browse/KAFKA-9815
 Project: Kafka
  Issue Type: Bug
  Components: consumer
Reporter: Rajini Sivaram
Assignee: Rajini Sivaram


KAFKA-9797 is the result of an incorrect rolling upgrade test where a new 
listener is added to brokers and set as the inter-broker listener within the 
same rolling upgrade. As a result, metadata is inconsistent across brokers 
until the rolling upgrade completes since interbroker communication is broken 
until all brokers have the new listener. The test fails due to consumer 
timeouts and sometimes this is because the upgrade takes longer than consumer 
timeout. But several logs show an issue with the consumer when one metadata 
response received during upgrade is different from the consumer's cached 
`assignmentSnapshot`, triggering rebalance.

In 
[https://github.com/apache/kafka/blob/7f640f13b4d486477035c3edb28466734f053beb/clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerCoordinator.java#L750,]
 we return true for `rejoinNeededOrPending()` if `assignmentSnapshot` is not 
the same as the current `metadataSnapshot`. We don't set `rejoinNeeded` in the 
instance, but we revoke partitions and send JoinGroup request. If the JoinGroup 
request fails and a subsequent metadata response contains the same snapshot 
value as the previously cached `assignmentSnapshot`, we never send `JoinGroup` 
again since snapshots match and `rejoinNeeded=false`. Partitions are not 
assigned to the consumer after this and the test fails because messages are not 
received.

Even though this particular system test isn't a valid upgrade scenario, we 
should fix the consumer, since temporary metadata differences between brokers 
can result in this scenario.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (KAFKA-9812) Integration tests hang and timeout the entire PR build on jenkins

[Matthias J. Sax (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-9812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax resolved KAFKA-9812.

Fix Version/s: 2.6.0
   Resolution: Fixed

> Integration tests hang and timeout the entire PR build on jenkins 
> --
>
> Key: KAFKA-9812
> URL: https://issues.apache.org/jira/browse/KAFKA-9812
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Konstantine Karantasis
>Assignee: John Roesler
>Priority: Major
>  Labels: flaky, flaky-build
> Fix For: 2.6.0
>
>
> Test {{org.apache.kafka.streams.integration.EosIntegrationTest > 
> shouldNotViolateEosIfOneTaskGetsFencedUsingIsolatedAppInstances[exactly_once_beta]}}
>  and possibly others have been spotted to hang indefinitely without entering 
> a {{FAILED}} state, which resulted in the whole jenkins to time out hours 
> later without a list of failures. 
> A suggested fix could possibly involve using a class wide {{Timeout}} rule 
> with reasonable timeouts for integration tests, as described here: 
> [https://github.com/junit-team/junit4/wiki/Timeout-for-tests]
> Snippet from the build log 
> [https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/1558/console] :
> {code:bash}
> org.apache.kafka.streams.integration.EosIntegrationTest > 
> shouldBeAbleToRunWithTwoSubtopologies[exactly_once_beta] STARTED 12:46:36 
> 12:46:36 org.apache.kafka.streams.integration.EosIntegrationTest > 
> shouldBeAbleToRunWithTwoSubtopologies[exactly_once_beta] PASSED 12:46:36 
> 12:46:36 org.apache.kafka.streams.integration.EosIntegrationTest > 
> shouldNotViolateEosIfOneTaskGetsFencedUsingIsolatedAppInstances[exactly_once_beta]
>  STARTED 15:12:14 Build timed out (after 270 minutes). Marking the build as 
> aborted. 15:12:15 Build was aborted 15:12:15 [FINDBUGS] Skipping publisher 
> since build result is ABORTED 15:12:15 Recording test results 15:12:15 
> Setting MAVEN_LATEST__HOME=/home/jenkins/tools/maven/latest/ 15:12:15 Setting 
> GRADLE_4_10_3_HOME=/home/jenkins/tools/gradle/4.10.3 15:12:15 15:12:15 
> org.apache.kafka.streams.integration.EosIntegrationTest > 
> shouldNotViolateEosIfOneTaskGetsFencedUsingIsolatedAppInstances[exactly_once_beta]
>  SKIPPED 15:12:16 15:12:16 > Task :streams:integrationTest FAILED 15:12:17 
> The message received from the daemon indicates that the daemon has 
> disappeared. 15:12:17 Build request sent: 
> Build{id=4f5e3086-ab55-48c5-a3a4-6f213565ac61, 
> currentDir=/home/jenkins/jenkins-slave/workspace/kafka-pr-jdk8-scala2.12} 
> 15:12:17 Attempting to read last messages from the daemon log... 15:12:17 
> Daemon pid: 27019 15:12:17 log file: 
> /home/jenkins/.gradle/daemon/5.6.2/daemon-27019.out.log 15:12:17 - Last 
> 20 lines from daemon log file - daemon-27019.out.log - 15:12:17 at 
> org.gradle.process.internal.DefaultExecHandle.execExceptionFor(DefaultExecHandle.java:237)
>  15:12:17 at 
> org.gradle.process.internal.DefaultExecHandle.setEndStateInfo(DefaultExecHandle.java:214)
>  15:12:17 at 
> org.gradle.process.internal.DefaultExecHandle.failed(DefaultExecHandle.java:364)
>  15:12:17 at 
> org.gradle.process.internal.ExecHandleRunner.run(ExecHandleRunner.java:87) 
> 15:12:17 at 
> org.gradle.internal.operations.CurrentBuildOperationPreservingRunnable.run(CurrentBuildOperationPreservingRunnable.java:42)
>  15:12:17 at 
> org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
>  15:12:17 at 
> org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
>  15:12:17 at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>  15:12:17 at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>  15:12:17 at 
> org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:56)
>  15:12:17 at java.lang.Thread.run(Thread.java:748) 15:12:17 Caused by: 
> java.lang.IllegalStateException: Shutdown in progress 15:12:17 at 
> java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:82) 
> 15:12:17 at java.lang.Runtime.removeShutdownHook(Runtime.java:239) 15:12:17 
> at 
> org.gradle.process.internal.shutdown.ShutdownHooks.removeShutdownHook(ShutdownHooks.java:33)
>  15:12:17 at 
> org.gradle.process.internal.DefaultExecHandle.setEndStateInfo(DefaultExecHandle.java:204)
>  15:12:17 at 
> org.gradle.process.internal.DefaultExecHandle.aborted(DefaultExecHandle.java:360)
>  15:12:17 at 
> org.gradle.process.internal.ExecHandleRunner.completed(ExecHandleRunner.java:108)
>  15:12:17 at 
> org.gradle.process.internal.ExecHandleRunner.run(ExecHandleRunner.java:84) 
> 15:12:17 ... 7 more 15:12:17 - End of the daemon log - 15:12:17 
> 15:12:17 15:12:17 FAILURE: Bui

Re: [DISCUSS] KIP-584: Versioning scheme for features

[Boyang Chen]

Hey Kowshik,

thanks for getting the KIP updated. The Zookeeper routing approach makes
sense and simplifies the changes.
Some follow-ups:

1. Do you mind updating the non-goal section as we are introducing a
--feature-force-downgrade to address downgrade concern?

2. For the flags `--feature` seems to be a redundant prefix, as the script
is already called `kafka-features.sh`. They could just be called
`--upgrade` and `--force-downgrade`.

3. I don't feel strong to require a confirmation for a normal feature
upgrade, unless there are other existing scripts doing so.

4. How could we know the existing feature versions when user are only
executing upgrades? Does the `kafka-features.sh` always send a
DescribeFeatureRequest to broker first?

5. I'm not 100% sure, but a script usually use the same flag once, so maybe
we should also do that for `--upgrade-feature`? Instead of flagging twice
for different features, a comma separated list of (feature:max_version)
will be expected, or something like that.

6. "The node data shall be readable via existing ZK tooling" Just trying to
clarify, we are not introducing ZK direct read tool in this KIP correct? As
for KIP-500 we are eventually going to deprecate all direct ZK access tools.

7. Could we have a centralized section called `Public Interfaces` to
summarize all the public API changes? This is a required section in a KIP.
And we should also write down the new error codes we will be introducing in
this KIP, and include both new and old error codes in the Response schema
comment if possible. For example, UpdateFeatureResponse could expect a
`NOT_CONTROLLER` error code.


Boyang

On Fri, Apr 3, 2020 at 3:15 AM Kowshik Prakasam 
wrote:

> Hi all,
>
> Any other feedback on this KIP before we start the vote?
>
>
> Cheers,
> Kowshik
>
> On Fri, Apr 3, 2020 at 1:27 AM Kowshik Prakasam 
> wrote:
>
> > Hey Jun,
> >
> > Thanks a lot for the great feedback! Please note that the design
> > has changed a little bit on the KIP, and we now propagate the finalized
> > features metadata only via ZK watches (instead of UpdateMetadataRequest
> > from the controller).
> >
> > Please find below my response to your questions/feedback, with the prefix
> > "(Kowshik):".
> >
> > > 100. UpdateFeaturesRequest/UpdateFeaturesResponse
> > > 100.1 Since this request waits for responses from brokers, should we
> add
> > a
> > > timeout in the request (like createTopicRequest)?
> >
> > (Kowshik): Great point! Done. I have added a timeout field. Note: we no
> > longer
> > wait for responses from brokers, since the design has been changed so
> that
> > the
> > features information is propagated via ZK. Nevertheless, it is right to
> > have a timeout
> > for the request.
> >
> > > 100.2 The response schema is a bit weird. Typically, the response just
> > > shows an error code and an error message, instead of echoing the
> request.
> >
> > (Kowshik): Great point! Yeah, I have modified it to just return an error
> > code and a message.
> > Previously it was not echoing the "request", rather it was returning the
> > latest set of
> > cluster-wide finalized features (after applying the updates). But you are
> > right,
> > the additional info is not required, so I have removed it from the
> > response schema.
> >
> > > 100.3 Should we add a separate request to list/describe the existing
> > > features?
> >
> > (Kowshik): This is already present in the KIP via the 'DescribeFeatures'
> > Admin API,
> > which, underneath covers uses the ApiVersionsRequest to list/describe the
> > existing features. Please read the 'Tooling support' section.
> >
> > > 100.4 We are mixing ADD_OR_UPDATE and DELETE in a single request. For
> > > DELETE, the version field doesn't make sense. So, I guess the broker
> just
> > > ignores this? An alternative way is to have a separate
> > DeleteFeaturesRequest
> >
> > (Kowshik): Great point! I have modified the KIP now to have 2 separate
> > controller APIs
> > serving these different purposes:
> > 1. updateFeatures
> > 2. deleteFeatures
> >
> > > 100.5 In UpdateFeaturesResponse, we have "The monotonically increasing
> > > version of the metadata for finalized features." I am wondering why the
> > > ordering is important?
> >
> > (Kowshik): In the latest KIP write-up, it is called epoch (instead of
> > version), and
> > it is just the ZK node version. Basically, this is the epoch for the
> > cluster-wide
> > finalized feature version metadata. This metadata is served to clients
> via
> > the
> > ApiVersionsResponse (for reads). We propagate updates from the
> '/features'
> > ZK node
> > to all brokers, via ZK watches setup by each broker on the '/features'
> > node.
> >
> > Now here is why the ordering is important:
> > ZK watches don't propagate at the same time. As a result, the
> > ApiVersionsResponse
> > is eventually consistent across brokers. This can introduce cases
> > where clients see an older lower epoch of the features metadata, after a
> > more recent
> > higher epoch was

Re: [DISCUSS] KIP-585: Conditional SMT

[Gunnar Morling]

Hi all,

Thanks a lot for this initiative, Tom!

To shed some light, the use case where this first came up, were issues
we saw with SMTs being applied to the different topics produced by the
Debezium change data capture connectors. There are different kinds of
topics (for change data, schema history, heartbeats etc.) and the
record structure to expect may vary between those. Hence we saw issues
with SMTs like ExtractField, which for instance only should be applied
to all change data topics but not the other ones.

I like the overall approach; for Debezium's purposes, the simple topic
matching and negation operators would be sufficient already. I agree
with Chris and would prefer one single condition attribute, which
contains a single condition or potentially a logical expression with
not, and, etc. I think it's less ambiguous, in particular when it
comes to ordering of the different conditions and determining their
precedence.

Would love to see this feature in one or another way in Connect.

Best,

--Gunnar



Am Do., 2. Apr. 2020 um 18:48 Uhr schrieb Tom Bentley :
>
> Hi Chris and Sönke,
>
> Using the numbering from Chris's email...
>
> 1. That's a good point, I'll see what is needed to make that work.
>
> 2. I'm happy enough to add support for "and" and "or" as part of this KIP
> if people can see a need for it.
>
> In a similar vein, I was wondering about whether it would be worthwhile
> having the equivalent of an "else" clause (what's in the KIP is basically
> an "if" statement). Without support for "else" I think people would often
> need two conditionals, with the condition of one being the negation of the
> condition of another.
>
> 3. I can see the attraction of an expression language. The pros include
> being terse and familiar to programmers and potentially very flexible if
> that's needed in the future. I had a play and implemented it using ANTLR
> and it's not difficult to write a grammar and implement the functions we've
> already discussed and get decent error messages when the expression is
> malformed. So on the one hand I quite like the idea. On the other hand it
> feels like overkill for the use cases that have actually been identified so
> far.
>
> @Sönke what do you make of the expression language idea?
>
> Kind regards,
>
> Tom
>
> On Wed, Apr 1, 2020 at 9:49 PM Christopher Egerton 
> wrote:
>
> > Hi Tom,
> >
> > This looks great and I'd love to see the out-of-the-box SMTs become even
> > more powerful with the improvements you've proposed! Got a few remarks and
> > would be interested in your thoughts:
> >
> > 1. Instead of the new "ConfigDef config(Map props)" method,
> > what would you think about adopting a similar approach as the framework
> > uses with connectors, by adding a "Config validate(Map
> > props)" method that can perform custom validation outside of what can be
> > performed by the ConfigDef's single-property-at-a-time validation? It may
> > be a little heavyweight for use with this particular SMT, but it'd provide
> > more flexibility for other SMT implementations and would mirror an API that
> > developers targeting the framework are likely already familiar with.
> > 2. The possibility for adding the logical operators "and" and "or" is
> > mentioned, but only as a potential future change and not one proposed by
> > this KIP. Why not include those operators sooner rather than later?
> > 3. The syntax for named conditions that are then referenced in logical
> > operators is... tricky. It took me a few attempts to grok the example
> > provided in the KIP after reading Sönke's question about the example for
> > negation. What about a more sophisticated but less verbose syntax that
> > supports a single configuration for the condition, even with logical
> > operators? I'm thinking something like
> > "transforms.conditionalExtract.condition: not(has-header:my-header)"
> > instead of the "transforms.conditionalExtract.condition: not:hasMyHeader"
> > and "transforms.conditionalExtract.condition.hasMyHeader:
> > has-header:my-header" properties. If support for a logical "and" is added,
> > this could then be expanded to something like
> > "transforms.conditionalExtract.condition: and(has-header(my-header),
> > not(topic-matches(my-prefix-.*)))". There would be additional complexity
> > here with the need to escape parentheses and commas that are intended to be
> > treated literally (as part of a header name, for example) instead of as
> > part of the syntax for the condition itself, but a little additional
> > complexity for edge cases like that may be warranted if it heavily reduces
> > complexity for the common cases. The rationale for the proposed
> > parentheses-based syntax here instead of what's mentioned in the KIP
> > (something like "and: , ") is to help with
> > readability; we probably wouldn't need that with the approach of naming
> > conditions via separate properties, but things may get a little nasty with
> > literal conditions included there, especially with

[jira] [Resolved] (KAFKA-9809) Shrink transaction timeout for Streams

[Boyang Chen (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-9809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Boyang Chen resolved KAFKA-9809.

Resolution: Fixed

> Shrink transaction timeout for Streams
> --
>
> Key: KAFKA-9809
> URL: https://issues.apache.org/jira/browse/KAFKA-9809
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Boyang Chen
>Assignee: Boyang Chen
>Priority: Major
>
> Per 447 documented, we need to shrink transaction timeout for Streams to 10 
> seconds.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Build failed in Jenkins: kafka-trunk-jdk8 #4397

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-9739: Fixes null key changing child node (#8400)


--
[...truncated 2.99 MB...]

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime STARTED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime PASSED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep PASSED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWindowStartTimestampWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWindowStartTimestampWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldReturnIsPersistent STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldReturnIsPersistent PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardClose 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardClose 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
PASSED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord PASSED

org.apache.kafka.streams.test.TestRecordTest > testToString STARTED

org.apache.kafka.streams.test.TestRecordTest > testToString PASSED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords STARTED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords PASSED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
STARTED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
PASSED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher STARTED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher PASSED

org.apache.kafka.streams.test.TestRecordTest > testFields STARTED

org.apache.kafka.streams.test.TestRecordTest > testFields PASSED

org.apache.kafka.streams.test.TestRecordTest > testProducerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testProducerRecord PASSED

org.apache.kafka.streams.test.TestRecordTest > testEqualsAndHashCode STARTED

org.apache.kafka.streams.test.TestRecordTest > testEqualsAndHashCode PASSED

> Task :streams:upgrade-system-tests-0100:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0100:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:classes UP-TO-DATE
> Task :streams:upgrade-system-tests-0100:checkstyleMain NO-SOURCE
> Task :streams:upgrade-system-tests-0100:compileTestJava
> Task :streams:upgrade-system-tests-0100:processTestResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:testClasses
> Task :streams:upgrade-system-tests-0100:checkstyleTest
> Task :streams:upgrade-system-tests-0100:spotbugsMain NO-SOURCE
> Task :streams:upgrade-system-tests-0100:test
> Task :streams:upgrade-system-tests-0101:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0101:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0101:classes UP-TO-DATE
> Task :streams:upgrade-system-tests-0101:checkstyleMain NO-SOURCE
> Task :streams:upgrade-system-tests-0101:compileTestJava
> Task :streams:upgrade-system-tests-0101:processTestResources NO-SOURCE
> Task :streams:upgrade-system-tests-0101:testClasses
> Task :streams:upgrade-system-tests-0101:checkstyleTest
> Task :streams:upgrade-system-tests-0101:spotbugsMain NO-SOURCE
> Task :streams:upgrade-system-tests-0101:test
> Task :streams:upgrade-system-tests-0102:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0102:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0102:classes UP-TO-DATE
> Task :streams:upgrade-system-tests-0102:checkstyleMain NO-SOURCE
> Task :streams:upgrade-system-tests-0102:compileTestJava
> Task :streams:upgrade-system-tests-0102:processTestResources NO-SOURCE
> Task :streams:up

Re: [DISCUSS] KIP-584: Versioning scheme for features

[Jun Rao]

Hi, Kowshik,

Thanks for the reply. A few more comments below.

100.6 For every new request, the admin needs to control who is allowed to
issue that request if security is enabled. So, we need to assign the new
request a ResourceType and possible AclOperations. See
https://cwiki.apache.org/confluence/display/KAFKA/KIP-455%3A+Create+an+Administrative+API+for+Replica+Reassignment
as
an example.

105. If we change delete to disable, it's better to do this consistently in
request protocol and admin api as well.

110. The minVersion/maxVersion for features use int64. Currently, our
release version schema is major.minor.bugfix (e.g. 2.5.0). It's possible
for new features to be included in minor releases too. Should we make the
feature versioning match the release versioning?

111. "During regular operations, the data in the ZK node can be mutated
only via a specific admin API served only by the controller." I am
wondering why can't the controller auto finalize a feature version after
all brokers are upgraded? For new users who download the latest version to
build a new cluster, it's inconvenient for them to have to manually enable
each feature.

112. DeleteFeaturesResponse: It seems the apiKey should be 49 instead of 48.

Jun


On Fri, Apr 3, 2020 at 1:27 AM Kowshik Prakasam 
wrote:

> Hey Jun,
>
> Thanks a lot for the great feedback! Please note that the design
> has changed a little bit on the KIP, and we now propagate the finalized
> features metadata only via ZK watches (instead of UpdateMetadataRequest
> from the controller).
>
> Please find below my response to your questions/feedback, with the prefix
> "(Kowshik):".
>
> > 100. UpdateFeaturesRequest/UpdateFeaturesResponse
> > 100.1 Since this request waits for responses from brokers, should we add
> a
> > timeout in the request (like createTopicRequest)?
>
> (Kowshik): Great point! Done. I have added a timeout field. Note: we no
> longer
> wait for responses from brokers, since the design has been changed so that
> the
> features information is propagated via ZK. Nevertheless, it is right to
> have a timeout
> for the request.
>
> > 100.2 The response schema is a bit weird. Typically, the response just
> > shows an error code and an error message, instead of echoing the request.
>
> (Kowshik): Great point! Yeah, I have modified it to just return an error
> code and a message.
> Previously it was not echoing the "request", rather it was returning the
> latest set of
> cluster-wide finalized features (after applying the updates). But you are
> right,
> the additional info is not required, so I have removed it from the response
> schema.
>
> > 100.3 Should we add a separate request to list/describe the existing
> > features?
>
> (Kowshik): This is already present in the KIP via the 'DescribeFeatures'
> Admin API,
> which, underneath covers uses the ApiVersionsRequest to list/describe the
> existing features. Please read the 'Tooling support' section.
>
> > 100.4 We are mixing ADD_OR_UPDATE and DELETE in a single request. For
> > DELETE, the version field doesn't make sense. So, I guess the broker just
> > ignores this? An alternative way is to have a separate
> DeleteFeaturesRequest
>
> (Kowshik): Great point! I have modified the KIP now to have 2 separate
> controller APIs
> serving these different purposes:
> 1. updateFeatures
> 2. deleteFeatures
>
> > 100.5 In UpdateFeaturesResponse, we have "The monotonically increasing
> > version of the metadata for finalized features." I am wondering why the
> > ordering is important?
>
> (Kowshik): In the latest KIP write-up, it is called epoch (instead of
> version), and
> it is just the ZK node version. Basically, this is the epoch for the
> cluster-wide
> finalized feature version metadata. This metadata is served to clients via
> the
> ApiVersionsResponse (for reads). We propagate updates from the '/features'
> ZK node
> to all brokers, via ZK watches setup by each broker on the '/features'
> node.
>
> Now here is why the ordering is important:
> ZK watches don't propagate at the same time. As a result, the
> ApiVersionsResponse
> is eventually consistent across brokers. This can introduce cases
> where clients see an older lower epoch of the features metadata, after a
> more recent
> higher epoch was returned at a previous point in time. We expect clients
> to always employ the rule that the latest received higher epoch of metadata
> always trumps an older smaller epoch. Those clients that are external to
> Kafka should strongly consider discovering the latest metadata once during
> startup from the brokers, and if required refresh the metadata periodically
> (to get the latest metadata).
>
> > 100.6 Could you specify the required ACL for this new request?
>
> (Kowshik): What is ACL, and how could I find out which one to specify?
> Please could you provide me some pointers? I'll be glad to update the
> KIP once I know the next steps.
>
> > 101. For the broker registration ZK node, should we bump up the version
> 

[jira] [Resolved] (KAFKA-9750) Flaky test kafka.server.ReplicaManagerTest.testFencedErrorCausedByBecomeLeader

[Jason Gustafson (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-9750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-9750.

Resolution: Fixed

> Flaky test kafka.server.ReplicaManagerTest.testFencedErrorCausedByBecomeLeader
> --
>
> Key: KAFKA-9750
> URL: https://issues.apache.org/jira/browse/KAFKA-9750
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Reporter: Bob Barrett
>Assignee: Chia-Ping Tsai
>Priority: Blocker
>  Labels: flaky-test
> Fix For: 2.5.0
>
>
> When running tests locally, I've seen that 1-2% of the time, 
> testFencedErrorCausedByBecomeLeader fails with
> {code:java}
> org.scalatest.exceptions.TestFailedException: the partition=test-topic-0 
> should be removed from pending 
> stateorg.scalatest.exceptions.TestFailedException: the partition=test-topic-0 
> should be removed from pending state
>  at 
> org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:530) at 
> org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:529) 
> at 
> org.scalatest.Assertions$.newAssertionFailedException(Assertions.scala:1389) 
> at org.scalatest.Assertions.fail(Assertions.scala:1091) at 
> org.scalatest.Assertions.fail$(Assertions.scala:1087) at 
> org.scalatest.Assertions$.fail(Assertions.scala:1389) at 
> kafka.server.ReplicaManagerTest.testFencedErrorCausedByBecomeLeader(ReplicaManagerTest.scala:248)
>  at jdk.internal.reflect.GeneratedMethodAccessor25.invoke(Unknown Source) at 
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  at java.base/java.lang.reflect.Method.invoke(Method.java:566) at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
>  at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>  at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
>  at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>  at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) 
> at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) 
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at 
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
>  at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
>  at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
>  at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at 
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at 
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at 
> org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at 
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at 
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at 
> org.junit.runners.ParentRunner.run(ParentRunner.java:413) at 
> org.junit.runner.JUnitCore.run(JUnitCore.java:137) at 
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
>  at 
> com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:40)
>  at 
> com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:230)
>  at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:58) {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] KIP-587 Suppress detailed responses for handled exceptions in security-sensitive environments

[Connor Penhale]

Hi Chris!

Thanks for your feedback! I'll number my responses to your questions / thoughts.

1. Apologies on that lack of clarity! I settled on "Detailed exception 
information has been suppressed. Please see logs." 
(https://github.com/apache/kafka/pull/8355/files#diff-64c265986e7bbe40cdd79f831e961907R34).
 Should I update the KIP to reflect what I've already thought about? It's my 
first one, not sure what the process should be for editing.

2. I was unaware of the REST extensions! I'll see if I can implement the same 
behavior as a REST extension. I agree that the KIP still has merit, regardless 
of the feasibility of the extension, but in regards to the 5th thought, this 
might make that decision easier.

3. I agree with your suggestion here. Absolutely ready to take the community 
feedback on what makes sense here.

4. I should note that while I emphasized uncaught exceptions, I mean all 
exceptions handled by the ExceptionMapper, including ConnectRestExceptions. An 
example of this is here: 
https://github.com/apache/kafka/pull/8355/files#diff-64c265986e7bbe40cdd79f831e961907R46

5. I didn't know how specific I should get if I had already taken a stab at 
implementing! I'm happy to edit this in whatever way we want to go about it.

Let me know if anyone has any other questions or feedback!


Thanks!
Connor

On 4/2/20, 3:58 PM, "Christopher Egerton"  wrote:

Hi Connor,

Great stuff! I generally like being able to see the stack trace of an
exception directly via the REST API but can definitely understand the
security concerns here. I've got a few questions/remarks about the KIP and
would be interested in your thoughts:

1. The KIP mentions a SUPRESSED_EXCEPTION_MESSAGE, but doesn't actually
outline what this message would actually be. It'd be great to see the
actual message in the KIP since people may have thoughts on what it should
be and want to comment on it as part of this discussion.

2. In the "Rejected Alternatives" section, an Nginx proxy is mentioned as
one possible way to filter out stack traces from the REST API. It seems
like a Connect REST extension (

https://kafka.apache.org/24/javadoc/index.html?org/apache/kafka/connect/rest/ConnectRestExtension.html)
would be a better alternative than an Nginx proxy; had you considered
utilizing one? I still think this KIP is worthwhile and a REST extension
shouldn't be necessary in order to lock down the REST API this way, but it
might be worth calling out as an alternative and perhaps even a workaround
in cases where users are stuck on a given version of Connect and can't
upgrade to 2.6 (or whichever version this KIP lands on) any time soon.

3. The "error.rest.response.message.detail.enabled" property is a bit of a
mouthful; it'd be great if we could come up with something more succinct.
What do you think about something like "rest.response.stack.traces"?

4. The KIP is targeted at stack traces for uncaught exceptions, but it's
also possible that stack traces get exposed in the REST API when querying
the status of a connector or one of its tasks. Was this intentional? If so,
it'd be great to call out why that kind of filtering is not required in the
"Rejected Alternatives" section, and if not, it's probably not too late to
consider modifying the KIP to cover those cases as well.

5. The KIP mentions creating a new, separate exception mapper class. This
seems like more of an implementation detail and something that can be
decided on during code review; unless it's critical to the functionality
that the KIP aims to accomplish, I'd suggest leaving that part out since it
shouldn't affect the impact on users of the Connect framework.

Thanks for the KIP, looking forward to seeing this happen!

Cheers,

Chris

On Thu, Apr 2, 2020 at 11:01 AM Connor Penhale 
wrote:

> Hello All!
>
> I’ve created the following KIP:
> 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-587:+Suppress+detailed+responses+for+handled+exceptions+in+security-sensitive+environments
>
> The PR that originated this discussion, is here:
> https://github.com/apache/kafka/pull/8355  It is based on 2.0, but I
> would be working on Kafka Connect in 2.6 to get this behavior changed to
> the community’s preference.
>
> Looking forward to working with everyone!
>
> Thanks!
> Connor
> ---
> Connor Penhale | Enterprise Architect, OpenLogic (https://openlogic.com/)
> Perforce (https://www.perforce.com/)
> Support: +1 866.399.6736
>
>
>
>
> This e-mail may contain information that is privileged or confidential. If
> you are not the intended recipient, please delete the e-mail and any
> attachments and notify us immediately.
>
>


CAUTION: This email originated from outside of the organization. Do not 
click on links or open attachme

Re: [DISCUSS] KIP-584: Versioning scheme for features

[Kowshik Prakasam]

Hey Boyang,

Thanks for the feedback! I've updated the KIP. Please find below my
response.

> 1. Do you mind updating the non-goal section as we are introducing a
> --feature-force-downgrade to address downgrade concern?

(Kowshik): This was already mentioned. Look for non-goal: 1-b.

> 2. For the flags `--feature` seems to be a redundant prefix, as the script
> is already called `kafka-features.sh`. They could just be called
> `--upgrade` and `--force-downgrade`.

(Kowshik): Great point! Done.

> 3. I don't feel strong to require a confirmation for a normal feature
> upgrade, unless there are other existing scripts doing so.

(Kowshik): Done. Removed now. We now ask for a confirmation only for
downgrades.

> 4. How could we know the existing feature versions when user are only
> executing upgrades? Does the `kafka-features.sh` always send a
> DescribeFeatureRequest to broker first?

(Kowshik): For deletes, yes it will make an ApiVersionsRequest call to show
the
versions of the features. Perhaps the ApiVersionsRequest can be sent
to just the controller to avoid questions on consistency, but that's
an implementation detail.

> 5. I'm not 100% sure, but a script usually use the same flag once, so
maybe
> we should also do that for `--upgrade-feature`? Instead of flagging twice
> for different features, a comma separated list of (feature:max_version)
> will be expected, or something like that.

(Kowshik): Done. I'm using a comma-separated list now.

> 6. "The node data shall be readable via existing ZK tooling" Just trying
to
> clarify, we are not introducing ZK direct read tool in this KIP correct?
As
> for KIP-500  we are
eventually going to deprecate all direct ZK access tools.

(Kowshik): Done. Yes, we are not intending to add such a tool. I was just
saying that
if we ever want to read it from ZK, then it's readable via ZK cli (in the
interim).
I have modified the text conveying the intent to support reads via
ApiVersionsRequest only (optionally this request can be directed at the
controller to
void questions on consistency, but that's an implementation detail).

> 7. Could we have a centralized section called `Public Interfaces` to
> summarize all the public API changes? This is a required section in a KIP.
> And we should also write down the new error codes we will be introducing
in
> this KIP, and include both new and old error codes in the Response schema
> comment if possible. For example, UpdateFeatureResponse could expect a
> `NOT_CONTROLLER` error code.

(Kowshik): Done. The error codes have been documented in the response
schemas now.
Added a new section titled "New or Changed Public Interfaces" summarizing
only the
changes made to the public interfaces.


Cheers,
Kowshik


On Fri, Apr 3, 2020 at 9:39 AM Boyang Chen 
wrote:

> Hey Kowshik,
>
> thanks for getting the KIP updated. The Zookeeper routing approach makes
> sense and simplifies the changes.
> Some follow-ups:
>
> 1. Do you mind updating the non-goal section as we are introducing a
> --feature-force-downgrade to address downgrade concern?
>
> 2. For the flags `--feature` seems to be a redundant prefix, as the script
> is already called `kafka-features.sh`. They could just be called
> `--upgrade` and `--force-downgrade`.
>
> 3. I don't feel strong to require a confirmation for a normal feature
> upgrade, unless there are other existing scripts doing so.
>
> 4. How could we know the existing feature versions when user are only
> executing upgrades? Does the `kafka-features.sh` always send a
> DescribeFeatureRequest to broker first?
>
> 5. I'm not 100% sure, but a script usually use the same flag once, so maybe
> we should also do that for `--upgrade-feature`? Instead of flagging twice
> for different features, a comma separated list of (feature:max_version)
> will be expected, or something like that.
>
> 6. "The node data shall be readable via existing ZK tooling" Just trying to
> clarify, we are not introducing ZK direct read tool in this KIP correct? As
> for KIP-500  we are
> eventually going to deprecate all direct ZK access tools.
>
> 7. Could we have a centralized section called `Public Interfaces` to
> summarize all the public API changes? This is a required section in a KIP.
> And we should also write down the new error codes we will be introducing in
> this KIP, and include both new and old error codes in the Response schema
> comment if possible. For example, UpdateFeatureResponse could expect a
> `NOT_CONTROLLER` error code.
>
>
> Boyang
>
> On Fri, Apr 3, 2020 at 3:15 AM Kowshik Prakasam 
> wrote:
>
> > Hi all,
> >
> > Any other feedback on this KIP before we start the vote?
> >
> >
> > Cheers,
> > Kowshik
> >
> > On Fri, Apr 3, 2020 at 1:27 AM Kowshik Prakasam 
> > wrote:
> >
> > > Hey Jun,
> > >
> > > Thanks a lot for the great feedback! Please note that the design
> > > has changed a little bit on the KIP, and we now propagate the f

Build failed in Jenkins: kafka-trunk-jdk11 #1319

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-9739: Fixes null key changing child node (#8400)


--
[...truncated 6.02 MB...]

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfInMemoryBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfInMemoryBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourcesThatMatchMultiplePattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourcesThatMatchMultiplePattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldPopulateGlobalStore[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldPopulateGlobalStore[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfPersistentBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfPersistentBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldAllowPrePopulatingStatesStoresWithCachingEnabled[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldAllowPrePopulatingStatesStoresWithCachingEnabled[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldRespectTaskIdling[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldRespectTaskIdling[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldApplyGlobalUpdatesCorrectlyInRecursiveTopologies[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldApplyGlobalUpdatesCorrectlyInRecursiveTopologies[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabl

[jira] [Created] (KAFKA-9816) Group Coordinator should be stricter about leader returned assignment

[Guozhang Wang (Jira)]

Guozhang Wang created KAFKA-9816:


 Summary: Group Coordinator should be stricter about leader 
returned assignment 
 Key: KAFKA-9816
 URL: https://issues.apache.org/jira/browse/KAFKA-9816
 Project: Kafka
  Issue Type: Improvement
  Components: core
Reporter: Guozhang Wang


At the moment, broker's group-coordinator does not require leader to provide 
assignment for all members but will simply default members without assignment 
with an empty byte buffer. However, since assignment bytes has get two versions 
now and hence we always have to try de-serialize the version first, an empty 
assignment could not be interpreted at the client at all.

Therefore, group coordinator should be more strict about the leader#assign 
function that each member should get non-empty assignment bytes even without 
assigned partitions.

Once we fixed the bug as discovered in 
https://issues.apache.org/jira/browse/KAFKA-9801, we should consider making 
this change since it would be no more compatibility concerns.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (KAFKA-9139) Dynamic broker config types aren't being discovered

[Brian Byrne (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-9139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brian Byrne resolved KAFKA-9139.

Resolution: Duplicate

> Dynamic broker config types aren't being discovered
> ---
>
> Key: KAFKA-9139
> URL: https://issues.apache.org/jira/browse/KAFKA-9139
> Project: Kafka
>  Issue Type: Bug
>Reporter: Brian Byrne
>Assignee: Brian Byrne
>Priority: Major
>
> The broker's dynamic config definition types aren't being properly 
> discovered, and therefore they're being considered "sensitive" when returned 
> to the client. This needs to be resolved. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] KIP-584: Versioning scheme for features

[Colin McCabe]

Hi Kowshik,

The discussion on ZooKeeper reads versus writes makes sense to me.  The 
important thing to keep in mind here is that in the bridge release, all brokers 
can read from ZooKeeper, but only the controller writes.

Why do we need both UpdateFeaturesRequest and DeleteFeaturesRequest?  It seems 
awkward to have "deleting" be a special case here when the general idea is that 
we have an RPC to change the supported feature flags.  Changing the feature 
level from 2 to 1 doesn't seem that different from changing it from 1 to not 
present.

It would be simpler to just say that a feature flag which doesn't appear in the 
znode is considered to be at version level 0.  This will also simplify the code 
a lot, I think, since you won't have to keep track of tricky distinctions 
between "disabled" and "enabled at version 0."  Then you would be able to just 
use an int in most places.

(By the way, I would propose the term "version level" for this number, since it 
avoids confusion with all the other meanings of the word "version" that we have 
in the code.)

Another thing to keep in mind is that if a request RPC is batch, the 
corresponding response RPC also needs to be batch.  In other words, you need 
multiple error codes, one for each feature flag whose level you are trying to 
change.  Unless the idea is that the whole change is a transaction that all 
either happens or doesn't?

Rather than FeatureUpdateType, I would just go with a boolean like "force."  
I'm not sure what other values we'd want to add to this later on, if it were an 
enum.  I think the boolean is clearer.

This ties in with my comment earlier, but for the result classes, we need 
methods other than just "all".  Batch operations aren't usable if you can't get 
the result per operation unless the semantics are transactional and it 
really is just everything succeeded or everything failed.

There are a bunch of Java interfaces described like FinalizedFeature, 
FeatureUpdate, UpdateFeaturesResult, and so on that should just be regular 
concrete Java classes.  In general we'd only use an interface if we wanted the 
caller to implement some kind of callback function.  We don't make classes that 
are just designed to hold data into interfaces, since that just imposes extra 
work on callers (they have to define their own concrete class for each 
interface just to use the API.)  There's also probably no reason to have these 
classes inherit from each other or have complex type relationships.  One more 
nitpick is that Kafka generally doesn't use "get" in the function names of 
accessors.

best,
Colin


On Fri, Apr 3, 2020, at 13:04, Kowshik Prakasam wrote:
> Hey Boyang,
> 
> Thanks for the feedback! I've updated the KIP. Please find below my
> response.
> 
> > 1. Do you mind updating the non-goal section as we are introducing a
> > --feature-force-downgrade to address downgrade concern?
> 
> (Kowshik): This was already mentioned. Look for non-goal: 1-b.
> 
> > 2. For the flags `--feature` seems to be a redundant prefix, as the script
> > is already called `kafka-features.sh`. They could just be called
> > `--upgrade` and `--force-downgrade`.
> 
> (Kowshik): Great point! Done.
> 
> > 3. I don't feel strong to require a confirmation for a normal feature
> > upgrade, unless there are other existing scripts doing so.
> 
> (Kowshik): Done. Removed now. We now ask for a confirmation only for
> downgrades.
> 
> > 4. How could we know the existing feature versions when user are only
> > executing upgrades? Does the `kafka-features.sh` always send a
> > DescribeFeatureRequest to broker first?
> 
> (Kowshik): For deletes, yes it will make an ApiVersionsRequest call to show
> the
> versions of the features. Perhaps the ApiVersionsRequest can be sent
> to just the controller to avoid questions on consistency, but that's
> an implementation detail.
> 
> > 5. I'm not 100% sure, but a script usually use the same flag once, so
> maybe
> > we should also do that for `--upgrade-feature`? Instead of flagging twice
> > for different features, a comma separated list of (feature:max_version)
> > will be expected, or something like that.
> 
> (Kowshik): Done. I'm using a comma-separated list now.
> 
> > 6. "The node data shall be readable via existing ZK tooling" Just trying
> to
> > clarify, we are not introducing ZK direct read tool in this KIP correct?
> As
> > for KIP-500  we are
> eventually going to deprecate all direct ZK access tools.
> 
> (Kowshik): Done. Yes, we are not intending to add such a tool. I was just
> saying that
> if we ever want to read it from ZK, then it's readable via ZK cli (in the
> interim).
> I have modified the text conveying the intent to support reads via
> ApiVersionsRequest only (optionally this request can be directed at the
> controller to
> void questions on consistency, but that's an implementation detail).
> 
> > 7. Could we have a centralized section called `Public Interfaces`

Re: [DISCUSS] KIP-587 Suppress detailed responses for handled exceptions in security-sensitive environments

[Colin McCabe]

Hi Connor,

If we are putting security-sensitive information into REST responses, that is a 
bug that needs to be fixed, not worked around with a configuration option.  Do 
you have an example of security-sensitive information appearing in the 
exception text?  Why do you feel that PCI-DSS requires this change?

By the way, the same concern applies to log messages.  We do not log sensitive 
information such as passwords to the log4j output.  If you know of that 
happening somewhere, please file a bug so it can be fixed.

best,
Colin


On Fri, Apr 3, 2020, at 12:56, Connor Penhale wrote:
> Hi Chris!
> 
> Thanks for your feedback! I'll number my responses to your questions / 
> thoughts.
> 
> 1. Apologies on that lack of clarity! I settled on "Detailed exception 
> information has been suppressed. Please see logs." 
> (https://github.com/apache/kafka/pull/8355/files#diff-64c265986e7bbe40cdd79f831e961907R34).
>  Should I update the KIP to reflect what I've already thought about? It's my 
> first one, not sure what the process should be for editing.
> 
> 2. I was unaware of the REST extensions! I'll see if I can implement 
> the same behavior as a REST extension. I agree that the KIP still has 
> merit, regardless of the feasibility of the extension, but in regards 
> to the 5th thought, this might make that decision easier.
> 
> 3. I agree with your suggestion here. Absolutely ready to take the 
> community feedback on what makes sense here.
> 
> 4. I should note that while I emphasized uncaught exceptions, I mean 
> all exceptions handled by the ExceptionMapper, including 
> ConnectRestExceptions. An example of this is here: 
> https://github.com/apache/kafka/pull/8355/files#diff-64c265986e7bbe40cdd79f831e961907R46
> 
> 5. I didn't know how specific I should get if I had already taken a 
> stab at implementing! I'm happy to edit this in whatever way we want to 
> go about it.
> 
> Let me know if anyone has any other questions or feedback!
> 
> 
> Thanks!
> Connor
> 
> On 4/2/20, 3:58 PM, "Christopher Egerton"  wrote:
> 
> Hi Connor,
> 
> Great stuff! I generally like being able to see the stack trace of an
> exception directly via the REST API but can definitely understand the
> security concerns here. I've got a few questions/remarks about the KIP and
> would be interested in your thoughts:
> 
> 1. The KIP mentions a SUPRESSED_EXCEPTION_MESSAGE, but doesn't actually
> outline what this message would actually be. It'd be great to see the
> actual message in the KIP since people may have thoughts on what it should
> be and want to comment on it as part of this discussion.
> 
> 2. In the "Rejected Alternatives" section, an Nginx proxy is 
> mentioned as
> one possible way to filter out stack traces from the REST API. It 
> seems
> like a Connect REST extension (
> 
> https://kafka.apache.org/24/javadoc/index.html?org/apache/kafka/connect/rest/ConnectRestExtension.html)
> would be a better alternative than an Nginx proxy; had you 
> considered
> utilizing one? I still think this KIP is worthwhile and a REST 
> extension
> shouldn't be necessary in order to lock down the REST API this way, 
> but it
> might be worth calling out as an alternative and perhaps even a 
> workaround
> in cases where users are stuck on a given version of Connect and 
> can't
> upgrade to 2.6 (or whichever version this KIP lands on) any time 
> soon.
> 
> 3. The "error.rest.response.message.detail.enabled" property is a bit of a
> mouthful; it'd be great if we could come up with something more succinct.
> What do you think about something like "rest.response.stack.traces"?
> 
> 4. The KIP is targeted at stack traces for uncaught exceptions, but it's
> also possible that stack traces get exposed in the REST API when querying
> the status of a connector or one of its tasks. Was this intentional? If 
> so,
> it'd be great to call out why that kind of filtering is not required in 
> the
> "Rejected Alternatives" section, and if not, it's probably not too late to
> consider modifying the KIP to cover those cases as well.
> 
> 5. The KIP mentions creating a new, separate exception mapper class. This
> seems like more of an implementation detail and something that can be
> decided on during code review; unless it's critical to the functionality
> that the KIP aims to accomplish, I'd suggest leaving that part out since 
> it
> shouldn't affect the impact on users of the Connect framework.
> 
> Thanks for the KIP, looking forward to seeing this happen!
> 
> Cheers,
> 
> Chris
> 
> On Thu, Apr 2, 2020 at 11:01 AM Connor Penhale 
> wrote:
> 
> > Hello All!
> >
> > I’ve created the following KIP:
> > 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-587:+Suppress+detailed+responses+for+handled+exceptions+in+security-sensitive+environments
> >
> > The PR that originat

Re: [DISCUSS] KIP-587 Suppress detailed responses for handled exceptions in security-sensitive environments

[Colin McCabe]

Also, if you do find a security issue, the process to follow is here: 
https://kafka.apache.org/project-security.html .

best,
Colin


On Fri, Apr 3, 2020, at 14:20, Colin McCabe wrote:
> Hi Connor,
> 
> If we are putting security-sensitive information into REST responses, 
> that is a bug that needs to be fixed, not worked around with a 
> configuration option.  Do you have an example of security-sensitive 
> information appearing in the exception text?  Why do you feel that 
> PCI-DSS requires this change?
> 
> By the way, the same concern applies to log messages.  We do not log 
> sensitive information such as passwords to the log4j output.  If you 
> know of that happening somewhere, please file a bug so it can be fixed.
> 
> best,
> Colin
> 
> 
> On Fri, Apr 3, 2020, at 12:56, Connor Penhale wrote:
> > Hi Chris!
> > 
> > Thanks for your feedback! I'll number my responses to your questions / 
> > thoughts.
> > 
> > 1. Apologies on that lack of clarity! I settled on "Detailed exception 
> > information has been suppressed. Please see logs." 
> > (https://github.com/apache/kafka/pull/8355/files#diff-64c265986e7bbe40cdd79f831e961907R34).
> >  Should I update the KIP to reflect what I've already thought about? It's 
> > my first one, not sure what the process should be for editing.
> > 
> > 2. I was unaware of the REST extensions! I'll see if I can implement 
> > the same behavior as a REST extension. I agree that the KIP still has 
> > merit, regardless of the feasibility of the extension, but in regards 
> > to the 5th thought, this might make that decision easier.
> > 
> > 3. I agree with your suggestion here. Absolutely ready to take the 
> > community feedback on what makes sense here.
> > 
> > 4. I should note that while I emphasized uncaught exceptions, I mean 
> > all exceptions handled by the ExceptionMapper, including 
> > ConnectRestExceptions. An example of this is here: 
> > https://github.com/apache/kafka/pull/8355/files#diff-64c265986e7bbe40cdd79f831e961907R46
> > 
> > 5. I didn't know how specific I should get if I had already taken a 
> > stab at implementing! I'm happy to edit this in whatever way we want to 
> > go about it.
> > 
> > Let me know if anyone has any other questions or feedback!
> > 
> > 
> > Thanks!
> > Connor
> > 
> > On 4/2/20, 3:58 PM, "Christopher Egerton"  wrote:
> > 
> > Hi Connor,
> > 
> > Great stuff! I generally like being able to see the stack trace of an
> > exception directly via the REST API but can definitely understand the
> > security concerns here. I've got a few questions/remarks about the KIP 
> > and
> > would be interested in your thoughts:
> > 
> > 1. The KIP mentions a SUPRESSED_EXCEPTION_MESSAGE, but doesn't actually
> > outline what this message would actually be. It'd be great to see the
> > actual message in the KIP since people may have thoughts on what it 
> > should
> > be and want to comment on it as part of this discussion.
> > 
> > 2. In the "Rejected Alternatives" section, an Nginx proxy is 
> > mentioned as
> > one possible way to filter out stack traces from the REST API. It 
> > seems
> > like a Connect REST extension (
> > 
> > https://kafka.apache.org/24/javadoc/index.html?org/apache/kafka/connect/rest/ConnectRestExtension.html)
> > would be a better alternative than an Nginx proxy; had you 
> > considered
> > utilizing one? I still think this KIP is worthwhile and a REST 
> > extension
> > shouldn't be necessary in order to lock down the REST API this way, 
> > but it
> > might be worth calling out as an alternative and perhaps even a 
> > workaround
> > in cases where users are stuck on a given version of Connect and 
> > can't
> > upgrade to 2.6 (or whichever version this KIP lands on) any time 
> > soon.
> > 
> > 3. The "error.rest.response.message.detail.enabled" property is a bit 
> > of a
> > mouthful; it'd be great if we could come up with something more 
> > succinct.
> > What do you think about something like "rest.response.stack.traces"?
> > 
> > 4. The KIP is targeted at stack traces for uncaught exceptions, but it's
> > also possible that stack traces get exposed in the REST API when 
> > querying
> > the status of a connector or one of its tasks. Was this intentional? If 
> > so,
> > it'd be great to call out why that kind of filtering is not required in 
> > the
> > "Rejected Alternatives" section, and if not, it's probably not too late 
> > to
> > consider modifying the KIP to cover those cases as well.
> > 
> > 5. The KIP mentions creating a new, separate exception mapper class. 
> > This
> > seems like more of an implementation detail and something that can be
> > decided on during code review; unless it's critical to the functionality
> > that the KIP aims to accomplish, I'd suggest leaving that part out 
> > since it
> > shouldn't affect the impact on users of the Connect framework.
> 

Re: [DISCUSS] KIP-586: Deprecate commit records without record metadata

[Mario Molina]

Hi,

I've updated the comment in the KIP (even though it was already included in
the PR).

Thanks!
Mario

On Fri, 3 Apr 2020 at 09:01, Randall Hauch  wrote:

> Hi, Mario.
>
> Thanks for creating this small but useful KIP! The only suggestion I have
> is to include the JavaDoc for the method we want to deprecate so that we
> can include the line:
>
> * @deprecated Use {@link #commitRecord(SourceRecord, RecordMetadata)}
>
>
> This will make it more clear that there are two methods with the same name
> and we're deprecating only one of them.
>
> Best regards,
>
> Randall
>
> On Wed, Apr 1, 2020 at 8:07 PM Mario Molina  wrote:
>
> > Hi all,
> >
> > I'd like to use this thread to discuss KIP-586.
> > You can see the detail at:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-586%3A+Deprecate+commit+records+without+record+metadata
> >
> > Best,
> > Mario
> >
>

Build failed in Jenkins: kafka-trunk-jdk8 #4398

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-9812: fix infinite loop in test code (#8411)

[manikumar] KAFKA-9775: Fix IllegalFormatConversionException in ToolsUtils


--
[...truncated 5.99 MB...]
org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForSmallerValue[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = false] PASSED

org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis STARTED

org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis PASSED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime STARTED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime PASSED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep PASSED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWindowStartTimestampWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWindowStartTimestampWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldReturnIsPersistent STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldReturnIsPersistent PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardClose 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardClose 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
PASSED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord PASSED

org.apache.kafka.streams.test.TestRecordTest > testToString STARTED

org.apache.kafka.streams.test.TestRecordTest > testToString PASSED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords STARTED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords PASSED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
STARTED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
PASSED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher STARTED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher PASSED

org.apache.kafka.streams.test.TestRecordTest > testFields STARTED

org.apache.kafka.streams.test.TestRecordTest > testFields PASSED

org.apache.kafka.streams.test.TestRecordTest > testProducerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testProducerRecord PASSED

org.apache.kafka.streams.test.TestRecordTest > testEqualsAndHashCode STARTED

org.apache.kafka.streams.test.TestRecordTest > testEqualsAndHashCode PASSED

> Task :streams:upgrade-system-tests-0100:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0100:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:classes UP-TO-DATE
> Task :streams:upgrade-system-tests-0100:checkstyleMain NO-SOURCE
> Task :streams:upgrade-system-tests-0100:compileTestJava
> Task :streams:upgrade-system-tests-0100:processTestResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:testClasses
> Task :streams:upgrade-system-tests-0100:checkstyleTest
> Task :streams:upgrade-system-tests-0100:spotbugsMain NO-SOURCE
> Task :streams:upgrade-system-tests-0100:test
> Task :streams:upgrade-system-tests-0101:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0101:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0101:classes UP-TO-DATE
> Task :s

[jira] [Created] (KAFKA-9817) Poor performance for electLeaders controller RPC

[Colin McCabe (Jira)]

Colin McCabe created KAFKA-9817:
---

 Summary: Poor performance for electLeaders controller RPC
 Key: KAFKA-9817
 URL: https://issues.apache.org/jira/browse/KAFKA-9817
 Project: Kafka
  Issue Type: Bug
Reporter: Colin McCabe


The electLeaders controller RPC has poor performance when the number of 
partitions is high.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Build failed in Jenkins: kafka-trunk-jdk11 #1320

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-9812: fix infinite loop in test code (#8411)

[manikumar] KAFKA-9775: Fix IllegalFormatConversionException in ToolsUtils

[github] KAFKA-9750; Fix race condition with log dir reassign completion (#8412)

[github] MINOR: clean up Streams assignment classes and tests (#8406)

[github] KAFKA-9810: Document Connect Root REST API on / (#8408)


--
[...truncated 3.00 MB...]

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfInMemoryBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfInMemoryBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourcesThatMatchMultiplePattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourcesThatMatchMultiplePattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldPopulateGlobalStore[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldPopulateGlobalStore[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfPersistentBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfPersistentBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldAllowPrePopulatingStatesStoresWithCachingEnabled[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldAllowPrePopulatingStatesStoresWithCachingEnabled[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldRespectTaskIdling[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldRespectTaskIdling[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldApplyGlobalUpdatesCorrectlyInRecursiveTopologies[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldApplyGlobalUpdatesCorrectlyInRecursiveTopologies[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] STARTED

org.apach

[jira] [Created] (KAFKA-9818) Flaky Test RecordCollectorTest.shouldNotThrowStreamsExceptionOnSubsequentCallIfASendFailsWithContinueExceptionHandler

[Sophie Blee-Goldman (Jira)]

Sophie Blee-Goldman created KAFKA-9818:
--

 Summary: Flaky Test 
RecordCollectorTest.shouldNotThrowStreamsExceptionOnSubsequentCallIfASendFailsWithContinueExceptionHandler
 Key: KAFKA-9818
 URL: https://issues.apache.org/jira/browse/KAFKA-9818
 Project: Kafka
  Issue Type: Bug
  Components: streams
Reporter: Sophie Blee-Goldman


h3. Error Message

java.lang.AssertionError
h3. Stacktrace

java.lang.AssertionError at org.junit.Assert.fail(Assert.java:87) at 
org.junit.Assert.assertTrue(Assert.java:42) at 
org.junit.Assert.assertTrue(Assert.java:53) at 
org.apache.kafka.streams.processor.internals.RecordCollectorTest.shouldNotThrowStreamsExceptionOnSubsequentCallIfASendFailsWithContinueExceptionHandler(RecordCollectorTest.java:521)
 at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method) at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.base/java.lang.reflect.Method.invoke(Method.java:566) at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
 at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
 at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
 at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
 at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) 
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) 
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at 
org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
 at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
 at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
 at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at 
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at 
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at 
org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at 
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at 
org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at 
org.junit.runners.ParentRunner.run(ParentRunner.java:413) at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110)
 at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
 at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
 at 
org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
 at 
org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
 at jdk.internal.reflect.GeneratedMethodAccessor20.invoke(Unknown Source) at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.base/java.lang.reflect.Method.invoke(Method.java:566) at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
 at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
 at 
org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
 at 
org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
 at com.sun.proxy.$Proxy5.processTestClass(Unknown Source) at 
org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:118)
 at jdk.internal.reflect.GeneratedMethodAccessor19.invoke(Unknown Source) at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.base/java.lang.reflect.Method.invoke(Method.java:566) at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
 at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
 at 
org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:182)
 at 
org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:164)
 at 
org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:412)
 at 
org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
 at 
org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExe

[jira] [Created] (KAFKA-9819) Flaky Test StoreChangelogReaderTest#shouldNotThrowOnUnknownRevokedPartition[0]

[Sophie Blee-Goldman (Jira)]

Sophie Blee-Goldman created KAFKA-9819:
--

 Summary: Flaky Test 
StoreChangelogReaderTest#shouldNotThrowOnUnknownRevokedPartition[0]
 Key: KAFKA-9819
 URL: https://issues.apache.org/jira/browse/KAFKA-9819
 Project: Kafka
  Issue Type: Bug
  Components: streams
Reporter: Sophie Blee-Goldman


h3. Error Message

java.lang.AssertionError: expected:<[test-reader Changelog partition unknown-0 
could not be found, it could be already cleaned up during the handlingof task 
corruption and never restore again]> but was:<[[AdminClient 
clientId=adminclient-91] Connection to node -1 (localhost/127.0.0.1:8080) could 
not be established. Broker may not be available., test-reader Changelog 
partition unknown-0 could not be found, it could be already cleaned up during 
the handlingof task corruption and never restore again]>
h3. Stacktrace

java.lang.AssertionError: expected:<[test-reader Changelog partition unknown-0 
could not be found, it could be already cleaned up during the handlingof task 
corruption and never restore again]> but was:<[[AdminClient 
clientId=adminclient-91] Connection to node -1 (localhost/127.0.0.1:8080) could 
not be established. Broker may not be available., test-reader Changelog 
partition unknown-0 could not be found, it could be already cleaned up during 
the handlingof task corruption and never restore again]>



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (KAFKA-9807) Race condition updating high watermark allows reads above LSO

[Jason Gustafson (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-9807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-9807.

Fix Version/s: 2.4.2
   2.5.0
   Resolution: Fixed

Resolving this. I will likely backport to older branches when I get a chance. I 
will also open separate jiras for some of the additional improvements suggested 
above.

> Race condition updating high watermark allows reads above LSO
> -
>
> Key: KAFKA-9807
> URL: https://issues.apache.org/jira/browse/KAFKA-9807
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.11.0.3, 1.0.2, 1.1.1, 2.0.1, 2.1.1, 2.2.2, 2.3.1, 2.4.1
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
>Priority: Major
> Fix For: 2.5.0, 2.4.2
>
>
> We had a transaction system test fail with the following error:
> {code}
> AssertionError: Detected 37 dups in concurrently consumed messages
> {code}
> After investigation, we found the duplicates were a result of the consumer 
> reading an aborted transaction, which should not be possible with the 
> read_committed isolation level.
> We tracked down the fetch request which returned the aborted data:
> {code}
> [2020-03-24 07:27:58,284] INFO Completed request:RequestHeader(apiKey=FETCH, 
> apiVersion=11, clientId=console-consumer, correlationId=283) -- 
> {replica_id=-1,max_wait_time=500,min_bytes=1,max_bytes=52428800,isolation_level=1,session_id=2043970605,session_epoch=87,topics=[{topic=output-topic,partitions=[{partition=1,current_leader_epoch=3,fetch_offset=48393,log_start_offset=-1,partition_max_bytes=1048576}]}],forgotten_topics_data=[],rack_id=},response:{throttle_time_ms=0,error_code=0,session_id=2043970605,responses=[{topic=output-topic,partition_responses=[{partition_header={partition=1,error_code=0,high_watermark=50646,last_stable_offset=50646,log_start_offset=0,aborted_transactions=[],preferred_read_replica=-1},record_set=FileRecords(size=31582,
>  file=/mnt/kafka/kafka-data-logs-1/output-topic-1/00045694.log, 
> start=37613, end=69195)}]}]} 
> {code}
> After correlating with the contents of the log segment 
> 00045694.log, we found that this fetch response included data 
> which was above the returned LSO which is 50646. In fact, the high watermark 
> matched the LSO in this case, so the data was above the high watermark as 
> well. 
> At the same time this request was received, we noted that the high watermark 
> was updated:
> {code}
> [2020-03-24 07:27:58,284] DEBUG [Partition output-topic-1 broker=3] High 
> watermark updated from (offset=50646 segment=[45694:68690]) to (offset=50683 
> segment=[45694:69195]) (kafka.cluster.Partition)
> {code}
> The position of the new high watermark matched the end position from the 
> fetch response, so that led us to believe there was a race condition with the 
> updating of this value. In the code, we have the following (abridged) logic 
> for fetching the LSO:
> {code}
> firstUnstableOffsetMetadata match {
>   case Some(offsetMetadata) if offsetMetadata.messageOffset < 
> highWatermark => offsetMetadata
>   case _ => fetchHighWatermarkMetadata
> }
> {code}
> If the first unstable offset is less than the high watermark, we should use 
> that; otherwise we use the high watermark. The problem is that the high 
> watermark referenced here could be updated between the range check and the 
> call to `fetchHighWatermarkMetadata`. If that happens, we would end up 
> reading data which is above the first unstable offset.
> The solution to fix this problem is to cache the high watermark value so that 
> it is used in both places. We may consider some additional improvements here 
> as well, such as fixing the inconsistency problem in the fetch response which 
> included data above the returned high watermark. We may also consider having 
> the client react more defensively by ignoring fetched data above the high 
> watermark. This would fix this problem for newer clients talking to older 
> brokers which might hit this problem.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Build failed in Jenkins: kafka-2.1-jdk8 #263

[Apache Jenkins Server]

See 


Changes:

[konstantine] KAFKA-9810: Document Connect Root REST API on / (#8408)


--
[...truncated 686.57 KB...]
  } else if (!path.equals(ZkUtils.ConsumersPath)) {
  ^
:126:
 object ZkUtils in package utils is deprecated: This is an internal class that 
is no longer used by Kafka and will be removed in a future release. Please use 
org.apache.kafka.clients.admin.AdminClient instead.
val unsecureZkUtils = ZkUtils(zkConnect, 6000, 6000, false) 
  ^
:140:
 object ZkUtils in package utils is deprecated: This is an internal class that 
is no longer used by Kafka and will be removed in a future release. Please use 
org.apache.kafka.clients.admin.AdminClient instead.
val unsecureZkUtils = ZkUtils(zkConnect, 6000, 6000, false)
  ^
:165:
 object ZkUtils in package utils is deprecated: This is an internal class that 
is no longer used by Kafka and will be removed in a future release. Please use 
org.apache.kafka.clients.admin.AdminClient instead.
for (path <- ZkUtils.SecureZkRootPaths) {
 ^
:181:
 object ZkUtils in package utils is deprecated: This is an internal class that 
is no longer used by Kafka and will be removed in a future release. Please use 
org.apache.kafka.clients.admin.AdminClient instead.
val unsecureZkUtils = ZkUtils(zkUrl, 6000, 6000, false)
  ^
:182:
 object ZkUtils in package utils is deprecated: This is an internal class that 
is no longer used by Kafka and will be removed in a future release. Please use 
org.apache.kafka.clients.admin.AdminClient instead.
val secureZkUtils = ZkUtils(zkUrl, 6000, 6000, true)
^
:195:
 class ZkUtils in package utils is deprecated: This is an internal class that 
is no longer used by Kafka and will be removed in a future release. Please use 
org.apache.kafka.clients.admin.AdminClient instead.
  private def testMigration(zkUrl: String, firstZk: ZkUtils, secondZk: ZkUtils) 
{
^
:195:
 class ZkUtils in package utils is deprecated: This is an internal class that 
is no longer used by Kafka and will be removed in a future release. Please use 
org.apache.kafka.clients.admin.AdminClient instead.
  private def testMigration(zkUrl: String, firstZk: ZkUtils, secondZk: ZkUtils) 
{
   ^
:197:
 object ZkUtils in package utils is deprecated: This is an internal class that 
is no longer used by Kafka and will be removed in a future release. Please use 
org.apache.kafka.clients.admin.AdminClient instead.
for (path <- ZkUtils.SecureZkRootPaths ++ ZkUtils.SensitiveZkRootPaths) {
 ^
:197:
 object ZkUtils in package utils is deprecated: This is an internal class that 
is no longer used by Kafka and will be removed in a future release. Please use 
org.apache.kafka.clients.admin.AdminClient instead.
for (path <- ZkUtils.SecureZkRootPaths ++ ZkUtils.SensitiveZkRootPaths) {
  ^
:210:
 object ZkUtils in package utils is deprecated: This is an internal class that 
is no longer used by Kafka and will be removed in a future release. Please use 
org.apache.kafka.clients.admin.AdminClient instead.
firstZk.createPersistentPath(ZkUtils.ConsumersPath)
 ^
:213:
 object ZkUtils in package utils is deprecated: This is an internal class that 
is no longer used by Kafka and will be removed in a

Build failed in Jenkins: kafka-trunk-jdk8 #4399

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-9750; Fix race condition with log dir reassign completion (#8412)

[github] MINOR: clean up Streams assignment classes and tests (#8406)

[github] KAFKA-9810: Document Connect Root REST API on / (#8408)

[github] KAFKA-9807; Protect LSO reads from concurrent high-watermark updates


--
[...truncated 2.99 MB...]
org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForSmallerValue[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = false] PASSED

org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis STARTED

org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis PASSED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime STARTED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime PASSED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep PASSED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWindowStartTimestampWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWindowStartTimestampWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldReturnIsPersistent STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldReturnIsPersistent PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardClose 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardClose 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
PASSED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord PASSED

org.apache.kafka.streams.test.TestRecordTest > testToString STARTED

org.apache.kafka.streams.test.TestRecordTest > testToString PASSED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords STARTED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords PASSED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
STARTED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
PASSED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher STARTED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher PASSED

org.apache.kafka.streams.test.TestRecordTest > testFields STARTED

org.apache.kafka.streams.test.TestRecordTest > testFields PASSED

org.apache.kafka.streams.test.TestRecordTest > testProducerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testProducerRecord PASSED

org.apache.kafka.streams.test.TestRecordTest > testEqualsAndHashCode STARTED

org.apache.kafka.streams.test.TestRecordTest > testEqualsAndHashCode PASSED

> Task :streams:upgrade-system-tests-0100:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0100:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:classes UP-TO-DATE
> Task :streams:upgrade-system-tests-0100:checkstyleMain NO-SOURCE
> Task :streams:upgrade-system-tests-0100:compileTestJava
> Task :streams:upgrade-system-tests-0100:processTestResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:testClasses
> Task :streams:upgrade-system-tests-0100:checkstyleTest
> Task :streams:upgrade-system-tests-0100:spotbugsMain NO-SOURCE
> Task :streams:upgrade-system-tests-0100:test
> Task :streams:upgrade-system-tests-0101:

[DISCUSS] KIP-590: Redirect Zookeeper Mutation Protocols to The Controller

[Boyang Chen]

Hey all,

I would like to start off the discussion for KIP-590, a follow-up
initiative after KIP-500:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-590%3A+Redirect+Zookeeper+Mutation+Protocols+to+The+Controller

This KIP proposes to migrate existing Zookeeper mutation paths, including
configuration, security and quota changes, to controller-only by always
routing these alterations to the controller.

Let me know your thoughts!

Best,
Boyang

[jira] [Created] (KAFKA-9820) validateMessagesAndAssignOffsetsCompressed allocates batch iterator which is not used

[Lucas Bradstreet (Jira)]

Lucas Bradstreet created KAFKA-9820:
---

 Summary: validateMessagesAndAssignOffsetsCompressed allocates 
batch iterator which is not used
 Key: KAFKA-9820
 URL: https://issues.apache.org/jira/browse/KAFKA-9820
 Project: Kafka
  Issue Type: Bug
  Components: core
Reporter: Lucas Bradstreet


KAFKA-8106 added a new skip key/value iterator that reduces allocations 
[https://github.com/apache/kafka/commit/3e9d1c1411c5268de382f9dfcc95bdf66d0063a0].

Unfortunately in LogValidator it creates that iterator but it never uses it, 
and this is quite expensive in terms of allocations.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Build failed in Jenkins: kafka-trunk-jdk11 #1321

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-9807; Protect LSO reads from concurrent high-watermark updates


--
[...truncated 3.00 MB...]

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfInMemoryBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfInMemoryBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourcesThatMatchMultiplePattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourcesThatMatchMultiplePattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldPopulateGlobalStore[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldPopulateGlobalStore[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfPersistentBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfPersistentBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldAllowPrePopulatingStatesStoresWithCachingEnabled[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldAllowPrePopulatingStatesStoresWithCachingEnabled[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldRespectTaskIdling[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldRespectTaskIdling[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldApplyGlobalUpdatesCorrectlyInRecursiveTopologies[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldApplyGlobalUpdatesCorrectlyInRecursiveTopologies[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetad

Re: [DISCUSS] KIP-584: Versioning scheme for features

[Kowshik Prakasam]

Hi Jun,

Thanks for the feedback and suggestions. Please find my response below.

> 100.6 For every new request, the admin needs to control who is allowed to
> issue that request if security is enabled. So, we need to assign the new
> request a ResourceType and possible AclOperations. See
>
https://cwiki.apache.org/confluence/display/KAFKA/KIP-455%3A+Create+an+Administrative+API+for+Replica+Reassignment
> as an example.

(Kowshik): I don't see any reference to the words ResourceType or
AclOperations
in the KIP. Please let me know how I can use the KIP that you linked to
know how to
setup the appropriate ResourceType and/or ClusterOperation?

> 105. If we change delete to disable, it's better to do this consistently
in
> request protocol and admin api as well.

(Kowshik): The API shouldn't be called 'disable' when it is deleting a
feature.
I've just changed the KIP to use 'delete'. I don't have a strong preference.

> 110. The minVersion/maxVersion for features use int64. Currently, our
> release version schema is major.minor.bugfix (e.g. 2.5.0). It's possible
> for new features to be included in minor releases too. Should we make the
> feature versioning match the release versioning?

(Kowshik): The release version can be mapped to a set of feature versions,
and this can be done, for example in the tool (or even external to the
tool).
Can you please clarify what I'm missing?

> 111. "During regular operations, the data in the ZK node can be mutated
> only via a specific admin API served only by the controller." I am
> wondering why can't the controller auto finalize a feature version after
> all brokers are upgraded? For new users who download the latest version to
> build a new cluster, it's inconvenient for them to have to manually enable
> each feature.

(Kowshik): I agree that there is a trade-off here, but it will help
to decide whether the automation can be thought through in the future
in a follow up KIP, or right now in this KIP. We may invest
in automation, but we have to decide whether we should do it
now or later.

For the inconvenience that you mentioned, do you think the problem that you
mentioned can be  overcome by asking for the cluster operator to run a
bootstrap script  when he/she knows that a specific AK release has been
almost completely deployed in a cluster for the first time? Idea is that the
bootstrap script will know how to map a specific AK release to finalized
feature versions, and run the `kafka-features.sh` tool appropriately against
the cluster.

Now, coming back to your automation proposal/question.
I do see the value of automated feature version finalization, but I also
see
that this will open up several questions and some risks, as explained below.
The answers to these depend on the definition of the automation we choose
to build, and how well does it fit into a kafka deployment.
Basically, it can be unsafe for the controller to finalize feature version
upgrades automatically, without learning about the intent of the cluster
operator.
1. We would sometimes want to lock feature versions only when we have
externally verified
the stability of the broker binary.
2. Sometimes only the cluster operator knows that a cluster upgrade is
complete,
and new brokers are highly unlikely to join the cluster.
3. Only the cluster operator knows that the intent is to deploy the same
version
of the new broker release across the entire cluster (i.e. the latest
downloaded version).
4. For downgrades, it appears the controller still needs some external input
(such as the proposed tool) to finalize a feature version downgrade.

If we have automation, that automation can end up failing in some of the
cases
above. Then, we need a way to declare that the cluster is "not ready" if the
controller cannot automatically finalize some basic required feature version
upgrades across the cluster. We need to make the cluster operator aware in
such a scenario (raise an alert or alike).

> 112. DeleteFeaturesResponse: It seems the apiKey should be 49 instead of
48.

(Kowshik): Done.


Cheers,
Kowshik

On Fri, Apr 3, 2020 at 11:24 AM Jun Rao  wrote:

> Hi, Kowshik,
>
> Thanks for the reply. A few more comments below.
>
> 100.6 For every new request, the admin needs to control who is allowed to
> issue that request if security is enabled. So, we need to assign the new
> request a ResourceType and possible AclOperations. See
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-455%3A+Create+an+Administrative+API+for+Replica+Reassignment
> as
> an example.
>
> 105. If we change delete to disable, it's better to do this consistently in
> request protocol and admin api as well.
>
> 110. The minVersion/maxVersion for features use int64. Currently, our
> release version schema is major.minor.bugfix (e.g. 2.5.0). It's possible
> for new features to be included in minor releases too. Should we make the
> feature versioning match the release versioning?
>
> 111. "During regular operations, the data in the ZK node ca

[jira] [Created] (KAFKA-9821) Partition may skip assignment with static members and incremental rebalances

[Guozhang Wang (Jira)]

Guozhang Wang created KAFKA-9821:


 Summary: Partition may skip assignment with static members and 
incremental rebalances
 Key: KAFKA-9821
 URL: https://issues.apache.org/jira/browse/KAFKA-9821
 Project: Kafka
  Issue Type: Bug
  Components: consumer, streams
Reporter: Guozhang Wang


When static membership (KIP-345) and incremental rebalancing (KIP-429) are 
turned on at the same time, that upon failure it is possible some partitions 
are not assigned to anyone. The event sequence is the following:

1. An assignment (task1) from rebalance is sent to an existing static member 
with owned list (partition1, partition2), hence upon receiving the assignment 
the static member is supposed to revoke partition2 and then re-join the group 
to trigger another rebalance.

2. The member crashed before re-join the group, lost all of its assigned 
partitions. However since this member is static with long session timeout, it 
was not kicked out of the group yet at the coordinator side.

3. The member resumes and then re-join with a known instance.id. The 
coordinator would not trigger a rebalance in this case and just give it the 
previous assignment (partition1), and since the member has forgot about its 
previous owned partitions it would just take partition1 and not re-join.

4. As a result the partition2 is not owned by this member any more but not 
re-assigned to anyone; until the next rebalance it would not be fetched by any 
member of the group.

The key here is that today we are relying on the member's local memory to 
calculate the added / revoked diff based on (owned, assigned). But if the 
member crashed and lost all of its owned partition, AND if it is a static 
member whose re-join would not trigger a new rebalance, this will break.

After thinking about that I think 1) on the consumer side, maybe we should 
augment the Assignment protocol with an error code indicating rebalance_needed 
which is going to be persisted on broker’s offset topic as well so that 
consumer member can learn about this without comparing the assignment with its 
owned partitions.

But 1) would affect the protocol and hence would only fix-forward future 
versions; so 2) on the streams side, maybe we should leverage the existing 
error field, to set REBALANCE_NEEDED for revocation as well (today it is only 
set for version probing and a couple edge cases) so that streams can, upon 
getting the error, force trigger a rebalance via the new consumer API. This 
means that we can fix-backward in 2.4/2.5 as well.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Jenkins build is back to normal : kafka-2.5-jdk8 #87

[Apache Jenkins Server]

See 

Jenkins build is back to normal : kafka-2.3-jdk8 #195

[Apache Jenkins Server]

See 

Re: 2.5.0 Producer.sendOffsetsToTransaction() and Backwards Compatibility

[Matthias J. Sax]

I guess you would need to catch the exception and retry?

It's a little unfortunate. Not sure if we could back-port the internal
producer config that we add in 2.6 for auto-downgrade to a 2.5 bug fix
release?


-Matthias


On 4/2/20 7:25 PM, Gary Russell wrote:
> Thanks Mattias
> 
>> Hence, why do you want/need to switch to the newer overload that only
> works for 2.5+ brokers?
> 
> So I can choose to use the producer per consumer thread Vs. the producer
> per group/topic/partition threading model for zombie fencing, based on the
> broker version.
> 
> I don't have the same luxury as kafka streams (i.e. don't use streams 2.6
> unless you have 2.5+ brokers).
> 
> I add new features with each minor release (and try to use the latest
> kafka-clients as they become available).
> 
> Users may want other new features, not related to EOS, and they might stay
> on old brokers.
> 
> Other users might want to take advantage of the improved performance of the
> new EOS so I need to support both APIs.
> 
> Many enterprises take forever to upgrade their brokers. I recently had a
> question of why my latest version won't work with an 0.9.x.x broker (sigh).
> 
> Spring versioning rules don't allow me to bump kafka-clients versions in a
> patch release so I am already supporting 4 active branches and I am trying
> to avoid supporting a fifth.
> 
> Thanks again.
> 
> 
> 
> 
> 
> 
> 
> 
> 
> On Thu, Apr 2, 2020 at 8:23 PM Matthias J. Sax  wrote:
> 
>> Gary,
>>
>> thanks for the question. We recently had a discussion about the exact
>> some topic:
>>
>> http://mail-archives.apache.org/mod_mbox/kafka-dev/202003.mbox/%3CCAJKanumaUg7bcRr%3DoZqQq9aWuO%3DfA5U1uvxAciB6RbYsvsEbYQ%40mail.gmail.com%3E
>>
>> Note that the "old" `sendOffsetsToTransaction(..., String groupId)` is
>> not deprecated. Hence, why do you want/need to switch to the newer
>> overload that only works for 2.5+ brokers? For many/most cases, the
>> "old" API that is compatible with older broker still does what you need
>> and there in not need to switch to the newer API.
>>
>>
>> -Matthias
>>
>>
>> On 4/2/20 1:54 PM, Gary Russell wrote:
>>> Thanks, Boyang,
>>>
>>> I maintain a framework (Spring for Apache Kafka) that sits on top of the
>>> clients, and I would like to be able to support all broker versions. I
>>> don't have control over what brokers my users are using.
>>>
>>> You guys have done a great job since 0.10.2.0 (I think) of supporting
>> older
>>> brokers from newer clients but this one's a blocker for me.
>>>
>>> My framework will enforce the proper semantics for EOS, depending on the
>>> broker version, but I need to know which model to use at runtime.
>>>
>>> As I said, I can have a property that the user can set to tell the
>>> framework that the broker is >= 2.5 but it would be cleaner if I could
>> stay
>>> away from that.
>>>
>>> Something like KafkaAdminClient.brokerApi() (or add the lowest API/broker
>>> version to describeCluster()), would be helpful.
>>>
>>> Worst case, I'll add a configuration option.
>>>
>>> Thanks.
>>>
>>> On Thu, Apr 2, 2020 at 4:45 PM Boyang Chen 
>>> wrote:
>>>
 Thanks for the question Gary. The reasoning for crash the new
 sendTxnOffsets API is because we don't want users to unconsciously
>> violate
 the EOS guarantee. In your case, using this API with 2.4.1 is not
>> supported
 anyway, so the upgrade path has to start from broker first to 2.5, and
>> then
 client binaries. Is there any further concern that blocks you from
>> getting
 the broker side upgrade first before using the new API?

 Boyang

 On Thu, Apr 2, 2020 at 1:37 PM Gary Russell 
>> wrote:

> Is there any way to determine the broker version in the kafka-clients?
>
> I need to determine whether I can use the new  sendOffsetsToTransaction
> with ConsumerGroupMetadata or use the old one.
>
> If I use the new API with a 2.4.1 broker, I get
>
> UpsupportedVersionException: Attempted to write a non-default
 generationId
> at version 2
>
> Alternatively, couldn't the client simply extract the groupId from the
> ConsumerGroupMetadata and use the old struct if the broker is too old?
>
> I'd rather not have a user property in my framework to tell us which
>> API
 to
> use.
>
> Thanks in advance.
>

>>>
>>
>>
> 



signature.asc
Description: OpenPGP digital signature

Build failed in Jenkins: kafka-2.4-jdk8 #182

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-9739:  Fixes null key changing child node (#8416)

[jason] KAFKA-9750; Fix race condition with log dir reassign completion (#8412)


--
[...truncated 5.58 MB...]
org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfTimestampIsDifferentForCompareValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfTimestampIsDifferentForCompareValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualWithNullForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualWithNullForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestampWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordWithExpectedRecordForCompareValueTimestamp 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordWithExpectedRecordForCompareValueTimestamp 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullExpectedRecordForCompareValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullExpectedRecordForCompareValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareKeyValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareKeyValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifie

Re: [DISCUSS] KIP-584: Versioning scheme for features

[Kowshik Prakasam]

Hi Colin,

Thanks for the feedback! I have updated the KIP based on your feedback.
Please find my response below.

> The discussion on ZooKeeper reads versus writes makes sense to me.  The
important thing to keep in mind here is that in the bridge release,
> all brokers can read from ZooKeeper, but only the controller writes.

(Kowshik): Great, thanks!

> Why do we need both UpdateFeaturesRequest and DeleteFeaturesRequest?  It
seems awkward to have "deleting" be a special case here when the
> general idea is that we have an RPC to change the supported feature
flags.  Changing the feature level from 2 to 1 doesn't seem that different
> from changing it from 1 to not present.

(Kowshik): Done, makes sense. I have updated the KIP to just use 1 API, and
that's the UpdateFeaturesRequest. For the deletion case, we can just ignore
the version number passed in the API (which is indicative of 'not present').

> It would be simpler to just say that a feature flag which doesn't appear
in the znode is considered to be at version level 0.  This will also
> simplify the code a lot, I think, since you won't have to keep track of
tricky distinctions between "disabled" and "enabled at version 0."
> Then you would be able to just use an int in most places.

(Kowshik): I'm not sure I understood why we want do it this way. If an
entry for some finalized feature is absent in '/features' node,
alternatively we can just treat this as a feature with a version that
was never finalized/enabled or it was deleted at some point. Then, we can
even allow for "enabled at version 0" as the {minVersion, maxVersion} range
can be any valid range, not necessarily minVersion > 0.

> (By the way, I would propose the term "version level" for this number,
since it avoids confusion with all the other meanings of the word
> "version" that we have in the code.)

(Kowshik): Good idea! I have updated the KIP to refer to "version level"
instead of version.

> Another thing to keep in mind is that if a request RPC is batch, the
corresponding response RPC also needs to be batch.  In other words, you
> need multiple error codes, one for each feature flag whose level you are
trying to change.  Unless the idea is that the whole change is a
> transaction that all either happens or doesn't?

(Kowshik): Yes, the whole change is a transaction. Either all provided
FeatureUpdate is carried out in ZK, or none happens. That's why we just
allow for a single error code field, as it is easier that way. This
transactional guarantee is mentioned under 'Proposed changes > New
controller API'

> Rather than FeatureUpdateType, I would just go with a boolean like
"force."  I'm not sure what other values we'd want to add to this later on,
> if it were an enum.  I think the boolean is clearer.

(Kowshik): Since we have decided to go just one API (i.e.
UpdateFeaturesRequest), it is better that FeatureUpdateType is an enum with
multiple values. A FeatureUpdateType is tied to a feature, and the possible
values are: ADD_OR_UPDATE, ADD_OR_UPDATE_ALLOW_DOWNGRADE, DELETE.

> This ties in with my comment earlier, but for the result classes, we need
methods other than just "all".  Batch operations aren't usable if
> you can't get the result per operation unless the semantics are
transactional and it really is just everything succeeded or everything
> failed.

(Kowshik): The semantics are transactional, as I explained above.

> There are a bunch of Java interfaces described like FinalizedFeature,
FeatureUpdate, UpdateFeaturesResult, and so on that should just be
> regular concrete Java classes.  In general we'd only use an interface if
we wanted the caller to implement some kind of callback function. We
> don't make classes that are just designed to hold data into interfaces,
since that just imposes extra work on callers (they have to define
> their own concrete class for each interface just to use the API.)
 There's also probably no reason to have these classes inherit from each
> other or have complex type relationships.  One more nitpick is that Kafka
generally doesn't use "get" in the function names of accessors.

(Kowshik): Done, I have changed the KIP. By 'interface', I just meant
interface from a pseudocode standpoint (i.e. it was just an abstraction
providing at least the specified behavior). Since that was a bit confusing,
I have now renamed it calling it a class. Also I have eliminated the type
relationships.


Cheers,
Kowshik

On Fri, Apr 3, 2020 at 5:54 PM Kowshik Prakasam 
wrote:

> Hi Jun,
>
> Thanks for the feedback and suggestions. Please find my response below.
>
> > 100.6 For every new request, the admin needs to control who is allowed to
> > issue that request if security is enabled. So, we need to assign the new
> > request a ResourceType and possible AclOperations. See
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-455%3A+Create+an+Administrative+API+for+Replica+Reassignment
> > as an example.
>
> (Kowshik): I don't see any reference to the words R

Build failed in Jenkins: kafka-trunk-jdk11 #1322

[Apache Jenkins Server]

See 


Changes:

[github] MINOR: Refactor StreamsProducer (#8380)


--
[...truncated 2.08 MB...]

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueWithSchemaFloat64 STARTED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueWithSchemaFloat64 PASSED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueWithSchemaString STARTED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueWithSchemaString PASSED

org.apache.kafka.connect.transforms.CastTest > testConfigEmpty STARTED

org.apache.kafka.connect.transforms.CastTest > testConfigEmpty PASSED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueSchemalessInt16 STARTED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueSchemalessInt16 PASSED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueSchemalessInt32 STARTED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueSchemalessInt32 PASSED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueSchemalessInt64 STARTED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueSchemalessInt64 PASSED

org.apache.kafka.connect.transforms.CastTest > castFieldsSchemaless STARTED

org.apache.kafka.connect.transforms.CastTest > castFieldsSchemaless PASSED

org.apache.kafka.connect.transforms.CastTest > testUnsupportedTargetType STARTED

org.apache.kafka.connect.transforms.CastTest > testUnsupportedTargetType PASSED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueWithSchemaInt16 STARTED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueWithSchemaInt16 PASSED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueWithSchemaInt32 STARTED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueWithSchemaInt32 PASSED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueWithSchemaInt64 STARTED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueWithSchemaInt64 PASSED

org.apache.kafka.connect.transforms.CastTest > 
castWholeBigDecimalRecordValueWithSchemaString STARTED

org.apache.kafka.connect.transforms.CastTest > 
castWholeBigDecimalRecordValueWithSchemaString PASSED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueWithSchemaBooleanFalse STARTED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueWithSchemaBooleanFalse PASSED

org.apache.kafka.connect.transforms.CastTest > testConfigInvalidMap STARTED

org.apache.kafka.connect.transforms.CastTest > testConfigInvalidMap PASSED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueWithSchemaBooleanTrue STARTED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueWithSchemaBooleanTrue PASSED

org.apache.kafka.connect.transforms.CastTest > castWholeRecordDefaultValue 
STARTED

org.apache.kafka.connect.transforms.CastTest > castWholeRecordDefaultValue 
PASSED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueWithSchemaInt8 STARTED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueWithSchemaInt8 PASSED

org.apache.kafka.connect.transforms.CastTest > castWholeRecordKeyWithSchema 
STARTED

org.apache.kafka.connect.transforms.CastTest > castWholeRecordKeyWithSchema 
PASSED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueSchemalessInt8 STARTED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueSchemalessInt8 PASSED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueSchemalessBooleanFalse STARTED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueSchemalessBooleanFalse PASSED

org.apache.kafka.connect.transforms.CastTest > testConfigInvalidSchemaType 
STARTED

org.apache.kafka.connect.transforms.CastTest > testConfigInvalidSchemaType 
PASSED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueSchemalessString STARTED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueSchemalessString PASSED

org.apache.kafka.connect.transforms.CastTest > castFieldsWithSchema STARTED

org.apache.kafka.connect.transforms.CastTest > castFieldsWithSchema PASSED

org.apache.kafka.connect.transforms.CastTest > castLogicalToString STARTED

org.apache.kafka.connect.transforms.CastTest > castLogicalToString PASSED

> Task :streams:examples:processResources NO-SOURCE
> Task :streams:examples:processTestResources NO-SOURCE
> Task :spotlessScala
> Task :spotlessScalaCheck
> Task :streams:streams-scala:processResources NO-SOURCE
> Task :streams:streams-scala:processTestResources
> Task :streams:test-utils:processTestResources
> Task :streams:upgrade-system-tests-0100:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0100:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:classes UP-TO-DATE
> Ta

Build failed in Jenkins: kafka-2.5-jdk8 #88

[Apache Jenkins Server]

See 


Changes:

[konstantine] KAFKA-9810: Document Connect Root REST API on / (#8408)

[jason] KAFKA-9807; Protect LSO reads from concurrent high-watermark updates


--
[...truncated 2.93 MB...]

org.apache.kafka.streams.MockProcessorContextTest > shouldCapturePunctuator 
PASSED

org.apache.kafka.streams.TestTopicsTest > testNonUsedOutputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonUsedOutputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testEmptyTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testEmptyTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testStartTimestamp STARTED

org.apache.kafka.streams.TestTopicsTest > testStartTimestamp PASSED

org.apache.kafka.streams.TestTopicsTest > testNegativeAdvance STARTED

org.apache.kafka.streams.TestTopicsTest > testNegativeAdvance PASSED

org.apache.kafka.streams.TestTopicsTest > shouldNotAllowToCreateWithNullDriver 
STARTED

org.apache.kafka.streams.TestTopicsTest > shouldNotAllowToCreateWithNullDriver 
PASSED

org.apache.kafka.streams.TestTopicsTest > testDuration STARTED

org.apache.kafka.streams.TestTopicsTest > testDuration PASSED

org.apache.kafka.streams.TestTopicsTest > testOutputToString STARTED

org.apache.kafka.streams.TestTopicsTest > testOutputToString PASSED

org.apache.kafka.streams.TestTopicsTest > testValue STARTED

org.apache.kafka.streams.TestTopicsTest > testValue PASSED

org.apache.kafka.streams.TestTopicsTest > testTimestampAutoAdvance STARTED

org.apache.kafka.streams.TestTopicsTest > testTimestampAutoAdvance PASSED

org.apache.kafka.streams.TestTopicsTest > testOutputWrongSerde STARTED

org.apache.kafka.streams.TestTopicsTest > testOutputWrongSerde PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputTopicWithNullTopicName STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputTopicWithNullTopicName PASSED

org.apache.kafka.streams.TestTopicsTest > testWrongSerde STARTED

org.apache.kafka.streams.TestTopicsTest > testWrongSerde PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMapWithNull STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMapWithNull PASSED

org.apache.kafka.streams.TestTopicsTest > testNonExistingOutputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonExistingOutputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testMultipleTopics STARTED

org.apache.kafka.streams.TestTopicsTest > testMultipleTopics PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValueList STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValueList PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputWithNullDriver STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputWithNullDriver PASSED

org.apache.kafka.streams.TestTopicsTest > testValueList STARTED

org.apache.kafka.streams.TestTopicsTest > testValueList PASSED

org.apache.kafka.streams.TestTopicsTest > testRecordList STARTED

org.apache.kafka.streams.TestTopicsTest > testRecordList PASSED

org.apache.kafka.streams.TestTopicsTest > testNonExistingInputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonExistingInputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMap STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMap PASSED

org.apache.kafka.streams.TestTopicsTest > testRecordsToList STARTED

org.apache.kafka.streams.TestTopicsTest > testRecordsToList PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValueListDuration STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValueListDuration PASSED

org.apache.kafka.streams.TestTopicsTest > testInputToString STARTED

org.apache.kafka.streams.TestTopicsTest > testInputToString PASSED

org.apache.kafka.streams.TestTopicsTest > testTimestamp STARTED

org.apache.kafka.streams.TestTopicsTest > testTimestamp PASSED

org.apache.kafka.streams.TestTopicsTest > testWithHeaders STARTED

org.apache.kafka.streams.TestTopicsTest > testWithHeaders PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValue STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValue PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateTopicWithNullTopicName STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateTopicWithNullTopicName PASSED

> Task :streams:upgrade-system-tests-0100:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0100:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:classes UP-TO-DATE
> Task :streams:upgrade-system-tests-0100:checkstyleMain NO-SOURCE
> Task :streams:upgrade-system-tests-0100:compileTestJava
> Task :streams:upgrade-system-tests-0100:processTestResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:testClasses
> Task :streams:upgrade-system-tests-0100:checkstyleTest
> Task :str

Build failed in Jenkins: kafka-trunk-jdk8 #4400

[Apache Jenkins Server]

See 


Changes:

[github] MINOR: Refactor StreamsProducer (#8380)


--
[...truncated 2.06 MB...]
org.apache.kafka.connect.transforms.CastTest > testConfigEmpty STARTED

org.apache.kafka.connect.transforms.CastTest > testConfigEmpty PASSED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueSchemalessInt16 STARTED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueSchemalessInt16 PASSED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueSchemalessInt32 STARTED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueSchemalessInt32 PASSED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueSchemalessInt64 STARTED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueSchemalessInt64 PASSED

org.apache.kafka.connect.transforms.CastTest > castFieldsSchemaless STARTED

org.apache.kafka.connect.transforms.CastTest > castFieldsSchemaless PASSED

org.apache.kafka.connect.transforms.CastTest > testUnsupportedTargetType STARTED

org.apache.kafka.connect.transforms.CastTest > testUnsupportedTargetType PASSED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueWithSchemaInt16 STARTED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueWithSchemaInt16 PASSED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueWithSchemaInt32 STARTED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueWithSchemaInt32 PASSED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueWithSchemaInt64 STARTED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueWithSchemaInt64 PASSED

org.apache.kafka.connect.transforms.CastTest > 
castWholeBigDecimalRecordValueWithSchemaString STARTED

org.apache.kafka.connect.transforms.CastTest > 
castWholeBigDecimalRecordValueWithSchemaString PASSED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueWithSchemaBooleanFalse STARTED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueWithSchemaBooleanFalse PASSED

org.apache.kafka.connect.transforms.CastTest > testConfigInvalidMap STARTED

org.apache.kafka.connect.transforms.CastTest > testConfigInvalidMap PASSED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueWithSchemaBooleanTrue STARTED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueWithSchemaBooleanTrue PASSED

org.apache.kafka.connect.transforms.CastTest > castWholeRecordDefaultValue 
STARTED

org.apache.kafka.connect.transforms.CastTest > castWholeRecordDefaultValue 
PASSED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueWithSchemaInt8 STARTED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueWithSchemaInt8 PASSED

org.apache.kafka.connect.transforms.CastTest > castWholeRecordKeyWithSchema 
STARTED

org.apache.kafka.connect.transforms.CastTest > castWholeRecordKeyWithSchema 
PASSED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueSchemalessInt8 STARTED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueSchemalessInt8 PASSED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueSchemalessBooleanFalse STARTED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueSchemalessBooleanFalse PASSED

org.apache.kafka.connect.transforms.CastTest > testConfigInvalidSchemaType 
STARTED

org.apache.kafka.connect.transforms.CastTest > testConfigInvalidSchemaType 
PASSED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueSchemalessString STARTED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueSchemalessString PASSED

org.apache.kafka.connect.transforms.CastTest > castFieldsWithSchema STARTED

org.apache.kafka.connect.transforms.CastTest > castFieldsWithSchema PASSED

org.apache.kafka.connect.transforms.CastTest > castLogicalToString STARTED

org.apache.kafka.connect.transforms.CastTest > castLogicalToString PASSED

> Task :streams:examples:processResources NO-SOURCE
> Task :streams:examples:processTestResources NO-SOURCE
> Task :spotlessScala
> Task :spotlessScalaCheck
> Task :streams:streams-scala:processResources NO-SOURCE
> Task :streams:streams-scala:processTestResources
> Task :streams:test-utils:processTestResources
> Task :streams:upgrade-system-tests-0100:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0100:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:classes UP-TO-DATE
> Task :streams:upgrade-system-tests-0100:checkstyleMain NO-SOURCE
> Task :streams:upgrade-system-tests-0100:compileTestJava
> Task :streams:upgrade-system-tests-0100:processTestResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:testClasses
> Task :streams:upgrade-system-tests-0100:checkstyleTest
> Task :streams:upgrade-system-tests-0100:spotbugsMain NO-SOURCE
> Task :

Re: [DISCUSS] KIP-579: new exception on min.insync.replicas > replication.factor

[James Cheng]

> On Apr 2, 2020, at 4:27 AM, Paolo Moriello  wrote:
> 
> Hi,
> 
> Thanks a lot for your feedback, I really appreciate your help on this.
> 
> Given what you suggested, I will take some time to update the kip with a
> proposal to make invalid configuration requests FAIL. This involves
> checking multiple code paths, as James was saying, to eg.: validate topic
> creation, topic-configuration changes, partition reassignment and broker
> configuration setup.
> 
> Regarding the latter, do you have any suggestion on what's the best thing
> to do? For instance, we know that we can specify min.insync.replicas at
> cluster level. At the same time, we can also specify
> default.replication.factor. If there is an inconsistency with this setup,
> do we want to make kafka fail at startup or allow the users to overwrite it
> at a later point? (I believe we should be strict here and fail at startup).
> A similar question applies to offsets.topic.replication.factor.
> 

Paolo,

I haven't had a chance to think of it deeply, but your idea of having kafka 
fail at startup makes sense to me. I'd like one of the committers to chime in 
about that idea, too.

Some broker default settings can also be set dynamically during runtime, so you 
will also have to catch/reject those.

-James

> Thanks,
> Paolo
> 
> On Wed, 1 Apr 2020 at 05:29, James Cheng  wrote:
> 
>> I agree that we should prevent the creation of such a topic configuration.
>> That would mean catching it at topic-creation time, as well as catching it
>> on any topic-configuration changes that might make min.isr > replication
>> factor.
>> 
>> Not sure how we would detect things if someone changed the broker-default
>> configuration. That could be tricky.
>> 
>> Btw, I was the person who filed the original JIRA and as Mickael guessed,
>> it was done by mistake.
>> 
>> -James
>> 
>>> On Mar 31, 2020, at 9:30 AM, Ismael Juma  wrote:
>>> 
>>> Hi Paolo,
>>> 
>>> Thanks for the KIP. Why would one want to set min.isr to be higher than
>>> replication factor even in that case? Mickael's suggestion seems better
>> to
>>> me.
>>> 
>>> Ismael
>>> 
>>> On Fri, Mar 13, 2020 at 10:28 AM Paolo Moriello <
>> paolomoriell...@gmail.com>
>>> wrote:
>>> 
 Hi Mickael,
 
 Thanks for your interest in this. The main motivation to NOT make topic
 creation fail when this mismatch happens is because at the moment it is
 possible to produce/consume on topics if acks is not set to all. I'm not
 sure we want to disable this behavior (as we would by failing at topic
 creation). That's why I decided to go for a softer approach, which at
>> least
 gives some more clarity to the users and avoids other issues mentioned
>> in
 the KIP.
 
 Let's see what others think!
 
 On Fri, 13 Mar 2020 at 17:16, Mickael Maison 
 wrote:
 
> Hi Paolo,
> 
> Thanks for looking at this issue. This can indeed be a source of
 confusion.
> 
> I'm wondering if we should prevent the creation of topics with
> min.insync.replicas > replication.factor?
> You listed that as a rejected alternative because it requires more
> changes. However, I can't think of any scenarios where a user would
> want to create such a topic. I'm guessing it's probably always by
> mistake.
> 
> Let's see what other people think but I think it's worth checking what
> needs to be done if we wanted to prevent topics with bogus configs
> 
> On Fri, Mar 13, 2020 at 3:28 PM Paolo Moriello
>  wrote:
>> 
>> Hi,
>> 
>> Following this Jira ticket (
> https://issues.apache.org/jira/browse/KAFKA-4680),
>> I've created a proposal (
>> 
> 
 
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-579%3A+new+exception+on+min.insync.replicas+%3E+replication.factor
> )
>> to add a new exception/error to be used on min.insync.replicas >
>> replication.factor.
>> 
>> The proposal aims to introduce a new exception specific for the
>> configuration mismatch above to be used when producers requires acks =
> all.
>> At the moment we are using NotEnoughReplicaException, which is a
> retriable
>> exception and is used to fail on insync replicas < min isr. Plan is to
> have
>> a new, non-retriable exception, to separate the two cases.
>> 
>> I've also submitted a PR for the change mentioned above:
>> https://github.com/apache/kafka/pull/8225
>> 
>> Please have a look and let me know what you think.
>> 
>> Thanks,
>> Paolo
> 
 
>> 
>>