[jira] [Resolved] (KAFKA-6601) Kafka manager does not provide consumer offset producer rate with kafka v2.10-0.10.2.0

[JIRA]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sönke Liebau resolved KAFKA-6601.
-
Resolution: Cannot Reproduce

Hi [~raj6329], 

I've just tested this with kafka_2.10_0.10.2.0 and there doesn't seem to be an 
issue with that metric. I can clearly see it with jconsole. Do you have an 
active consumer that is committing offsets to your cluster? The metric will 
after a server restart only show up once someone actually produces to that 
topic.

If that doesn't fix it, then I am afraid you should open an issue for this on 
the KafkaManager bugtracker, as I cannot see anything wrong on the Kafka side.

> Kafka manager does not provide consumer offset producer rate with kafka 
> v2.10-0.10.2.0
> --
>
> Key: KAFKA-6601
> URL: https://issues.apache.org/jira/browse/KAFKA-6601
> Project: Kafka
>  Issue Type: Bug
>Reporter: Rajendra Jangir
>Priority: Major
>
> I am using kafka-manager and kafka version 2.10-0.10.2.
> And I am not able to see producer rate for _consumer_offset topic._
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] KIP-253: Support in-order message delivery with partition expansion

[Jason Gustafson]

Hi Dong,

Great work on this proposal! Just a couple initial comments:

My understanding is that the consumer will block on a topic until the all
partitions have reached a certain partition epoch. What are the
implications if a partition is offline? If we observe an epoch change while
a partition is offline, it seems like we'd have to wait until the partition
is back online before we can begin consuming the new epoch. Otherwise we
will violate the ordering guarantees. Many use cases involve unordered
data, so this would be a kind of regression in behavior, wouldn't it? A
couple ideas:

1. Maybe we could have a topic configuration that controls whether or not
ordering on the topic needs to be strictly followed? If we don't care about
ordering, the consumer need not synchronize on epoch boundaries and we need
not care about offline partitions.
2. Waiting on all partitions allows for any key partitioning function. It's
good because it's general, but it is overly conservative when the
partitioning function has finer control over key movement. For example, if
the partitioner only allows for splits, then there is just one partition to
await before consuming a new epoch for any given partition. I am not sure
what it would look like, but I'm wondering if it would be possible to
leverage the custom partitioning logic on the consumer side as well to
avoid unneeded waiting.

I think piggybacking the epoch exchanges onto the consumer heartbeats is a
good idea. Just wanted to mention that consumers are not the only ones
using the heartbeat API. For example, Kafka Connect also uses the group
protocol to balance its load. Of course other use cases could leave these
fields empty, but it's a little odd to have the protocol tailored
specifically for one use case. To be honest, the group management protocol
is one of the messier Kafka APIs and I don't think anyone is satisfied with
the current approach. We need not redesign the whole thing in this KIP, but
it might be nice to consider some options so that we're sure we're either
heading in a better direction or at least not making things more confusing
than they already are. The challenge is that it's useful to have some
coordinator logic specific to the group type. I can imagine down the road
that other use cases may also have some custom metadata which they need to
piggyback on the heartbeat and they may also need the coordinator to do
some facilitation. Maybe the heartbeat protocol could be left generic and
we could have a separate module in the GroupCoordinator for custom consumer
logic? Not too sure the best way to go.

Thanks,
Jason


On Tue, Feb 27, 2018 at 11:49 PM, Stephane Maarek <
steph...@simplemachines.com.au> wrote:

> Sounds awesome !
> Are you planning to have auto scaling of partitions in a following KIP ?
> That would be the holy grail
>
> On 28 Feb. 2018 5:13 pm, "Dong Lin"  wrote:
>
> > Hey Jan,
> >
> > I am not sure if it is acceptable for producer to be stopped for a while,
> > particularly for online application which requires low latency. I am also
> > not sure how consumers can switch to a new topic. Does user application
> > needs to explicitly specify a different topic for producer/consumer to
> > subscribe to? It will be helpful for discussion if you can provide more
> > detail on the interface change for this solution.
> >
> > Thanks,
> > Dong
> >
> > On Mon, Feb 26, 2018 at 12:48 AM, Jan Filipiak  >
> > wrote:
> >
> > > Hi,
> > >
> > > just want to throw my though in. In general the functionality is very
> > > usefull, we should though not try to find the architecture to hard
> while
> > > implementing.
> > >
> > > The manual steps would be to
> > >
> > > create a new topic
> > > the mirrormake from the new old topic to the new topic
> > > wait for mirror making to catch up.
> > > then put the consumers onto the new topic
> > > (having mirrormaker spit out a mapping from old offsets to new
> > offsets:
> > > if topic is increased by factor X there is gonna be a clean
> > > mapping from 1 offset in the old topic to X offsets in the new topic,
> > > if there is no factor then there is no chance to generate a
> > > mapping that can be reasonable used for continuing)
> > > make consumers stop at appropriate points and continue consumption
> > > with offsets from the mapping.
> > > have the producers stop for a minimal time.
> > > wait for mirrormaker to finish
> > > let producer produce with the new metadata.
> > >
> > >
> > > Instead of implementing the approach suggest in the KIP which will
> leave
> > > log compacted topic completely crumbled and unusable.
> > > I would much rather try to build infrastructure to support the
> mentioned
> > > above operations more smoothly.
> > > Especially having producers stop and use another topic is difficult and
> > > it would be nice if one can trigger "invalid metadata" exceptions for
> > them
> > > and
> > > if one could give topics aliases so that their produces with the old
> > to

Re: [VOTE] 1.1.0 RC0

[Jason Gustafson]

Hey Damian,

I think we should consider https://issues.apache.org/jira/browse/KAFKA-6593
for the release. I have a patch available, but still working on validating
both the bug and the fix.

-Jason

On Wed, Feb 28, 2018 at 9:34 AM, Matthias J. Sax 
wrote:

> No. Both will be released.
>
> -Matthias
>
> On 2/28/18 6:32 AM, Marina Popova wrote:
> > Sorry, maybe a stupid question, but:
> >  I see that Kafka 1.0.1 RC2 is still not released, but now 1.1.0 RC0 is
> coming up...
> > Does it mean 1.0.1 will be abandoned and we should be looking forward to
> 1.1.0 instead?
> >
> > thanks!
> >
> > ​Sent with ProtonMail Secure Email.​
> >
> > ‐‐‐ Original Message ‐‐‐
> >
> > On February 26, 2018 6:28 PM, Vahid S Hashemian <
> vahidhashem...@us.ibm.com> wrote:
> >
> >> +1 (non-binding)
> >>
> >> Built the source and ran quickstart (including streams) successfully on
> >>
> >> Ubuntu (with both Java 8 and Java 9).
> >>
> >> I understand the Windows platform is not officially supported, but I ran
> >>
> >> the same on Windows 10, and except for Step 7 (Connect) everything else
> >>
> >> worked fine.
> >>
> >> There are a number of warning and errors (including
> >>
> >> java.lang.ClassNotFoundException). Here's the final error message:
> >>
> >>> bin\\windows\\connect-standalone.bat config\\connect-standalone.
> properties
> >>
> >> config\\connect-file-source.properties config\\connect-file-sink.
> properties
> >>
> >> ...
> >>
> >> \[2018-02-26 14:55:56,529\] ERROR Stopping after connector error
> >>
> >> (org.apache.kafka.connect.cli.ConnectStandalone)
> >>
> >> java.lang.NoClassDefFoundError:
> >>
> >> org/apache/kafka/connect/transforms/util/RegexValidator
> >>
> >> at
> >>
> >> org.apache.kafka.connect.runtime.SinkConnectorConfig.<
> clinit>(SinkConnectorConfig.java:46)
> >>
> >> at
> >>
> >>
> >> org.apache.kafka.connect.runtime.AbstractHerder.
> validateConnectorConfig(AbstractHerder.java:263)
> >>
> >> at
> >>
> >> org.apache.kafka.connect.runtime.standalone.StandaloneHerder.
> putConnectorConfig(StandaloneHerder.java:164)
> >>
> >> at
> >>
> >> org.apache.kafka.connect.cli.ConnectStandalone.main(
> ConnectStandalone.java:107)
> >>
> >> Caused by: java.lang.ClassNotFoundException:
> >>
> >> org.apache.kafka.connect.transforms.util.RegexValidator
> >>
> >> at
> >>
> >> java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(
> BuiltinClassLoader.java:582)
> >>
> >> at
> >>
> >> java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.
> loadClass(ClassLoaders.java:185)
> >>
> >> at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:496)
> >>
> >> ... 4 more
> >>
> >> Thanks for running the release.
> >>
> >> --Vahid
> >>
> >> From: Damian Guy damian@gmail.com
> >>
> >> To: dev@kafka.apache.org, us...@kafka.apache.org,
> >>
> >> kafka-clie...@googlegroups.com
> >>
> >> Date: 02/24/2018 08:16 AM
> >>
> >> Subject: \[VOTE\] 1.1.0 RC0
> >>
> >> Hello Kafka users, developers and client-developers,
> >>
> >> This is the first candidate for release of Apache Kafka 1.1.0.
> >>
> >> This is minor version release of Apache Kakfa. It Includes 29 new KIPs.
> >>
> >> Please see the release plan for more details:
> >>
> >> https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.
> apache.org_confluence_pages_viewpage.action-3FpageId-
> 3D71764913&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Q_
> itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc&m=K9Iz2hWA2pj4QGxW6fleW20K0M7oEe
> WCbqs5nbbUY0c&s=M1liORvtcIt7pZ8e5GnLr9a1i6SOUY4bvjHYOrY_zcE&e=
> >>
> >> A few highlights:
> >>
> >> -   Significant Controller improvements (much faster and session
> expiration
> >>
> >> edge cases fixed)
> >>
> >> -   Data balancing across log directories (JBOD)
> >> -   More efficient replication when the number of partitions is large
> >> -   Dynamic Broker Configs
> >> -   Delegation tokens (KIP-48)
> >> -   Kafka Streams API improvements (KIP-205 / 210 / 220 / 224 / 239)
> >>
> >> Release notes for the 1.1.0 release:
> >>
> >> https://urldefense.proofpoint.com/v2/url?u=http-3A__home.
> apache.org_-7Edamianguy_kafka-2D1.1.0-2Drc0_RELEASE-5FNOTES.
> html&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-
> kjJc7uSVcviKUc&m=K9Iz2hWA2pj4QGxW6fleW20K0M7oEe
> WCbqs5nbbUY0c&s=H6-O0mkXk2tT_7RlN4W9bJd_lpoOt5ranhTx28WdRnQ&e=
> >>
> >> \*\*\* Please download, test and vote by Wednesday, February 28th,
> 5pm PT
> >>
> >> Kafka's KEYS file containing PGP keys we use to sign the release:
> >>
> >> https://urldefense.proofpoint.com/v2/url?u=http-3A__kafka.
> apache.org_KEYS&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Q_
> itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc&m=K9Iz2hWA2pj4QGxW6fleW20K0M7oEe
> WCbqs5nbbUY0c&s=Eo5JrktOPUlA2-7W11222zSVYfR6oqzd9uiaUEod2D4&e=
> >>
> >> -   Release artifacts to be voted upon (source and binary):
> >>
> >> https://urldefense.proofpoint.com/v2/url?u=http-3A__home.
> apache.org_-7Edamianguy_kafka-2D1.1.0-2Drc0_&d=DwIBaQ&c=jf_
> iaSHvJObTbx-siA1ZOg&r=Q_itwloTQj3_xUKl7Nzswo6KE4Nj

Build failed in Jenkins: kafka-trunk-jdk7 #3219

[Apache Jenkins Server]

See 


Changes:

[me] MINOR: Extend release.py with a subcommand for staging docs into the

--
[...truncated 412.53 KB...]

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldRetainLatestEpochOnClearAllEarliest PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldPersistEpochsBetweenInstances STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldPersistEpochsBetweenInstances PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotClearAnythingIfOffsetToFirstOffset STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotClearAnythingIfOffsetToFirstOffset PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotLetOffsetsGoBackwardsEvenIfEpochsProgress STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotLetOffsetsGoBackwardsEvenIfEpochsProgress PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldGetFirstOffsetOfSubsequentEpochWhenOffsetRequestedForPreviousEpoch STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldGetFirstOffsetOfSubsequentEpochWhenOffsetRequestedForPreviousEpoch PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldUpdateOffsetBetweenEpochBoundariesOnClearEarliest2 STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldUpdateOffsetBetweenEpochBoundariesOnClearEarliest2 PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearEarliestOnEmptyCache 
STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearEarliestOnEmptyCache 
PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldPreserveResetOffsetOnClearEarliestIfOneExists STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldPreserveResetOffsetOnClearEarliestIfOneExists PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldUpdateOffsetBetweenEpochBoundariesOnClearEarliest STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldUpdateOffsetBetweenEpochBoundariesOnClearEarliest PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldReturnInvalidOffsetIfEpochIsRequestedWhichIsNotCurrentlyTracked STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldReturnInvalidOffsetIfEpochIsRequestedWhichIsNotCurrentlyTracked PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldFetchEndOffsetOfEmptyCache 
STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldFetchEndOffsetOfEmptyCache 
PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldRetainLatestEpochOnClearAllEarliestAndUpdateItsOffset STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldRetainLatestEpochOnClearAllEarliestAndUpdateItsOffset PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearAllEntries STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearAllEntries PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearLatestOnEmptyCache 
STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearLatestOnEmptyCache 
PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotResetEpochHistoryHeadIfUndefinedPassed STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotResetEpochHistoryHeadIfUndefinedPassed PASSED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldIncreaseLeaderEpochBetweenLeaderRestarts STARTED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldIncreaseLeaderEpochBetweenLeaderRestarts PASSED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldAddCurrentLeaderEpochToMessagesAsTheyAreWrittenToLeader STARTED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldAddCurrentLeaderEpochToMessagesAsTheyAreWrittenToLeader PASSED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldSendLeaderEpochRequestAndGetAResponse STARTED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldSendLeaderEpochRequestAndGetAResponse PASSED

kafka.server.epoch.OffsetsForLeaderEpochTest > shouldGetEpochsFromReplica 
STARTED

kafka.server.epoch.OffsetsForLeaderEpochTest > shouldGetEpochsFromReplica PASSED

kafka.server.epoch.OffsetsForLeaderEpochTest > 
shouldReturnUnknownTopicOrPartitionIfThrown STARTED

kafka.server.epoch.OffsetsForLeaderEpochTest > 
shouldReturnUnknownTopicOrPartitionIfThrown PASSED

kafka.server.epoch.OffsetsForLeaderEpochTest > 
shouldReturnNoLeaderForPartitionIfThrown STARTED

kafka.server.epoch.OffsetsForLeaderEpochTest > 
shouldReturnNoLeaderForPartitionIfThrown PASSED

kafka.server.epoch.EpochDrivenReplicationProtocolAcceptanceTest > 
shouldSurviveFastLeaderChange STARTED

kafka.server.epoch.EpochDrivenReplicationProtocolAcceptanceTest > 
shouldSurviveFastLeaderChange PASSED

kafka.server.epoch.EpochDrivenReplicationProtocolAcceptanceTest > 
offsetsShouldNotGoBackwards STARTED

kafka.server.epoch.EpochDrivenReplicationProtocolAcceptanceTest > 
offsetsShouldNotGoBackwards PASSED

kafka.server.epoch.EpochDrivenReplicationProtocolAcceptanceTest > 
shouldFollowLeaderEpochBasicWorkflow STARTED

kafka.server.epoch.EpochDrivenR

Re: [VOTE] 1.1.0 RC0

[Damian Guy]

Hi Jason,

Ok - thanks. Let me know how you get on.

Cheers,
Damian

On Wed, 28 Feb 2018 at 19:23 Jason Gustafson  wrote:

> Hey Damian,
>
> I think we should consider
> https://issues.apache.org/jira/browse/KAFKA-6593
> for the release. I have a patch available, but still working on validating
> both the bug and the fix.
>
> -Jason
>
> On Wed, Feb 28, 2018 at 9:34 AM, Matthias J. Sax 
> wrote:
>
> > No. Both will be released.
> >
> > -Matthias
> >
> > On 2/28/18 6:32 AM, Marina Popova wrote:
> > > Sorry, maybe a stupid question, but:
> > >  I see that Kafka 1.0.1 RC2 is still not released, but now 1.1.0 RC0 is
> > coming up...
> > > Does it mean 1.0.1 will be abandoned and we should be looking forward
> to
> > 1.1.0 instead?
> > >
> > > thanks!
> > >
> > > ​Sent with ProtonMail Secure Email.​
> > >
> > > ‐‐‐ Original Message ‐‐‐
> > >
> > > On February 26, 2018 6:28 PM, Vahid S Hashemian <
> > vahidhashem...@us.ibm.com> wrote:
> > >
> > >> +1 (non-binding)
> > >>
> > >> Built the source and ran quickstart (including streams) successfully
> on
> > >>
> > >> Ubuntu (with both Java 8 and Java 9).
> > >>
> > >> I understand the Windows platform is not officially supported, but I
> ran
> > >>
> > >> the same on Windows 10, and except for Step 7 (Connect) everything
> else
> > >>
> > >> worked fine.
> > >>
> > >> There are a number of warning and errors (including
> > >>
> > >> java.lang.ClassNotFoundException). Here's the final error message:
> > >>
> > >>> bin\\windows\\connect-standalone.bat config\\connect-standalone.
> > properties
> > >>
> > >> config\\connect-file-source.properties config\\connect-file-sink.
> > properties
> > >>
> > >> ...
> > >>
> > >> \[2018-02-26 14:55:56,529\] ERROR Stopping after connector error
> > >>
> > >> (org.apache.kafka.connect.cli.ConnectStandalone)
> > >>
> > >> java.lang.NoClassDefFoundError:
> > >>
> > >> org/apache/kafka/connect/transforms/util/RegexValidator
> > >>
> > >> at
> > >>
> > >> org.apache.kafka.connect.runtime.SinkConnectorConfig.<
> > clinit>(SinkConnectorConfig.java:46)
> > >>
> > >> at
> > >>
> > >>
> > >> org.apache.kafka.connect.runtime.AbstractHerder.
> > validateConnectorConfig(AbstractHerder.java:263)
> > >>
> > >> at
> > >>
> > >> org.apache.kafka.connect.runtime.standalone.StandaloneHerder.
> > putConnectorConfig(StandaloneHerder.java:164)
> > >>
> > >> at
> > >>
> > >> org.apache.kafka.connect.cli.ConnectStandalone.main(
> > ConnectStandalone.java:107)
> > >>
> > >> Caused by: java.lang.ClassNotFoundException:
> > >>
> > >> org.apache.kafka.connect.transforms.util.RegexValidator
> > >>
> > >> at
> > >>
> > >> java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(
> > BuiltinClassLoader.java:582)
> > >>
> > >> at
> > >>
> > >> java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.
> > loadClass(ClassLoaders.java:185)
> > >>
> > >> at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:496)
> > >>
> > >> ... 4 more
> > >>
> > >> Thanks for running the release.
> > >>
> > >> --Vahid
> > >>
> > >> From: Damian Guy damian@gmail.com
> > >>
> > >> To: dev@kafka.apache.org, us...@kafka.apache.org,
> > >>
> > >> kafka-clie...@googlegroups.com
> > >>
> > >> Date: 02/24/2018 08:16 AM
> > >>
> > >> Subject: \[VOTE\] 1.1.0 RC0
> > >>
> > >> Hello Kafka users, developers and client-developers,
> > >>
> > >> This is the first candidate for release of Apache Kafka 1.1.0.
> > >>
> > >> This is minor version release of Apache Kakfa. It Includes 29 new
> KIPs.
> > >>
> > >> Please see the release plan for more details:
> > >>
> > >> https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.
> > apache.org_confluence_pages_viewpage.action-3FpageId-
> > 3D71764913&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Q_
> >
> itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc&m=K9Iz2hWA2pj4QGxW6fleW20K0M7oEe
> > WCbqs5nbbUY0c&s=M1liORvtcIt7pZ8e5GnLr9a1i6SOUY4bvjHYOrY_zcE&e=
> > >>
> > >> A few highlights:
> > >>
> > >> -   Significant Controller improvements (much faster and session
> > expiration
> > >>
> > >> edge cases fixed)
> > >>
> > >> -   Data balancing across log directories (JBOD)
> > >> -   More efficient replication when the number of partitions is large
> > >> -   Dynamic Broker Configs
> > >> -   Delegation tokens (KIP-48)
> > >> -   Kafka Streams API improvements (KIP-205 / 210 / 220 / 224 / 239)
> > >>
> > >> Release notes for the 1.1.0 release:
> > >>
> > >> https://urldefense.proofpoint.com/v2/url?u=http-3A__home.
> > apache.org_-7Edamianguy_kafka-2D1.1.0-2Drc0_RELEASE-5FNOTES.
> > html&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-
> > kjJc7uSVcviKUc&m=K9Iz2hWA2pj4QGxW6fleW20K0M7oEe
> > WCbqs5nbbUY0c&s=H6-O0mkXk2tT_7RlN4W9bJd_lpoOt5ranhTx28WdRnQ&e=
> > >>
> > >> \*\*\* Please download, test and vote by Wednesday, February 28th,
> > 5pm PT
> > >>
> > >> Kafka's KEYS file containing PGP keys we use to sign the release:
> > >>
> > >> https://urldefense.proofpoint.com/v2/url?u=http-3A__kafka.
> > apac

Re: [DISCUSS] KIP-211: Revise Expiration Semantics of Consumer Group Offsets

[Jason Gustafson]

Hey Vahid,

Thanks for the response. Replies below:


> 1. I think my suggestion in the KIP was more towards ignoring the client
> provided values and use a large enough broker config value instead. It
> seems the question comes down to whether we still want to honor the
> `retention_time` field in the old requests. With the new request (as per
> this KIP) the client would not be able to overwrite the broker retention
> config. Your suggestion provides kind of a back door for the overwrite.
> Also, since different offset commits associated with a group can
> potentially use different `retention_time` values, it's probably
> reasonable to use the maximum of all those values (including the broker
> config) as the group offset retention.


Mainly I wanted to ensure that we would be holding offsets at least as long
as what was requested by older clients. If we hold it for longer, that's
probably fine, but there may be application behavior which would break if
offsets are expired earlier than expected.

2. If I'm not mistake you are referring to potential changes in
> `GROUP_METADATA_VALUE_SCHEMA`. I saw this as an internal implementation
> matter and frankly, have not fully thought about it, but I agree that it
> needs to be updated to include either the timestamp the group becomes
> `Empty` or maybe the expiration timestamp of the group. And perhaps, we
> would not need to store per partition offset expiration timestamp anymore.
> Is there a particular reason for your suggestion of storing the timestamp
> the group becomes `Empty`, vs the expiration timestamp of the group?


Although it is not exposed to clients, we still have to manage
compatibility of the schema across versions, so I think we should include
it in the KIP. The reason I was thinking of using the time that the group
became Empty is that the configured timeout might change. I think my
expectation as a user would be that a timeout change would also apply to
existing groups, but I'm not sure if there are any reasons not to so.

3. To limit the scope of the KIP I would prefer to handle this matter
> separately if it doesn't have to be addressed as part of this change. It
> probably needs be addressed at some point and I'll mention it in the KIP
> so we have it documented. Do you think my suggestion of manually removing
> topic offsets from group (as an interim solution) is worth additional
> discussion / implementation?


I think manual removal of offsets for this case is a bit of a tough sell
for usability. Did you imagine it happening automatically in the consumer
through an API?

I'm finding it increasingly frustrating that the generic group coordinator
is limited in its decision making since it cannot see the subscription
metadata. It is the same problem in Dong's KIP. I think I would suggest
that, at a minimum, we leave the door open to enforcing offset expiration
either 1) when the group becomes empty, and 2) when the corresponding
partition is removed from the subscription. Perhaps that means we need to
keep the individual offset expiration timestamp after all. Actually we
would probably need it anyway to handle "simple" consumer groups which are
always Empty.

One additional note: I have seen recently a case where the offset cache
caused an OOM on the broker. I looked into it and found that most of the
cache was used for storing console consumer offsets. I know you had a patch
before which turned off auto-commit when the groupId was generated by
ConsoleConsumer. Maybe we could lump that change into this KIP?

Thanks,
Jason




On Fri, Feb 23, 2018 at 4:08 PM, Vahid S Hashemian <
vahidhashem...@us.ibm.com> wrote:

> Hi Jason,
>
> Thanks a lot for reviewing the KIP.
>
> 1. I think my suggestion in the KIP was more towards ignoring the client
> provided values and use a large enough broker config value instead. It
> seems the question comes down to whether we still want to honor the
> `retention_time` field in the old requests. With the new request (as per
> this KIP) the client would not be able to overwrite the broker retention
> config. Your suggestion provides kind of a back door for the overwrite.
> Also, since different offset commits associated with a group can
> potentially use different `retention_time` values, it's probably
> reasonable to use the maximum of all those values (including the broker
> config) as the group offset retention.
>
> 2. If I'm not mistake you are referring to potential changes in
> `GROUP_METADATA_VALUE_SCHEMA`. I saw this as an internal implementation
> matter and frankly, have not fully thought about it, but I agree that it
> needs to be updated to include either the timestamp the group becomes
> `Empty` or maybe the expiration timestamp of the group. And perhaps, we
> would not need to store per partition offset expiration timestamp anymore.
> Is there a particular reason for your suggestion of storing the timestamp
> the group becomes `Empty`, vs the expiration timestamp of the group?
>
> 3. To limit the 

Build failed in Jenkins: kafka-trunk-jdk8 #2444

[Apache Jenkins Server]

See 


Changes:

[me] MINOR: Extend release.py with a subcommand for staging docs into the

--
[...truncated 415.50 KB...]
kafka.message.ByteBufferMessageSetTest > testWrittenEqualsRead PASSED

kafka.message.ByteBufferMessageSetTest > testWriteTo STARTED

kafka.message.ByteBufferMessageSetTest > testWriteTo PASSED

kafka.message.ByteBufferMessageSetTest > testEquals STARTED

kafka.message.ByteBufferMessageSetTest > testEquals PASSED

kafka.message.ByteBufferMessageSetTest > testSizeInBytes STARTED

kafka.message.ByteBufferMessageSetTest > testSizeInBytes PASSED

kafka.message.ByteBufferMessageSetTest > testIterator STARTED

kafka.message.ByteBufferMessageSetTest > testIterator PASSED

kafka.metrics.KafkaTimerTest > testKafkaTimer STARTED

kafka.metrics.KafkaTimerTest > testKafkaTimer PASSED

kafka.metrics.MetricsTest > testMetricsReporterAfterDeletingTopic STARTED

kafka.metrics.MetricsTest > testMetricsReporterAfterDeletingTopic PASSED

kafka.metrics.MetricsTest > testSessionExpireListenerMetrics STARTED

kafka.metrics.MetricsTest > testSessionExpireListenerMetrics PASSED

kafka.metrics.MetricsTest > 
testBrokerTopicMetricsUnregisteredAfterDeletingTopic STARTED

kafka.metrics.MetricsTest > 
testBrokerTopicMetricsUnregisteredAfterDeletingTopic PASSED

kafka.metrics.MetricsTest > testClusterIdMetric STARTED

kafka.metrics.MetricsTest > testClusterIdMetric PASSED

kafka.metrics.MetricsTest > testControllerMetrics STARTED

kafka.metrics.MetricsTest > testControllerMetrics PASSED

kafka.metrics.MetricsTest > testWindowsStyleTagNames STARTED

kafka.metrics.MetricsTest > testWindowsStyleTagNames PASSED

kafka.metrics.MetricsTest > testMetricsLeak STARTED

kafka.metrics.MetricsTest > testMetricsLeak PASSED

kafka.metrics.MetricsTest > testBrokerTopicMetricsBytesInOut STARTED

kafka.metrics.MetricsTest > testBrokerTopicMetricsBytesInOut PASSED

kafka.javaapi.consumer.ZookeeperConsumerConnectorTest > testBasic STARTED

kafka.javaapi.consumer.ZookeeperConsumerConnectorTest > testBasic PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testSizeInBytesWithCompression 
STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > testSizeInBytesWithCompression 
PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > 
testIteratorIsConsistentWithCompression STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > 
testIteratorIsConsistentWithCompression PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testIteratorIsConsistent 
STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > testIteratorIsConsistent PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testEqualsWithCompression 
STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > testEqualsWithCompression 
PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testWrittenEqualsRead STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > testWrittenEqualsRead PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testEquals STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > testEquals PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testSizeInBytes STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > testSizeInBytes PASSED

kafka.security.token.delegation.DelegationTokenManagerTest > 
testPeriodicTokenExpiry STARTED

kafka.security.token.delegation.DelegationTokenManagerTest > 
testPeriodicTokenExpiry PASSED

kafka.security.token.delegation.DelegationTokenManagerTest > 
testTokenRequestsWithDelegationTokenDisabled STARTED

kafka.security.token.delegation.DelegationTokenManagerTest > 
testTokenRequestsWithDelegationTokenDisabled PASSED

kafka.security.token.delegation.DelegationTokenManagerTest > testDescribeToken 
STARTED

kafka.security.token.delegation.DelegationTokenManagerTest > testDescribeToken 
PASSED

kafka.security.token.delegation.DelegationTokenManagerTest > testCreateToken 
STARTED

kafka.security.token.delegation.DelegationTokenManagerTest > testCreateToken 
PASSED

kafka.security.token.delegation.DelegationTokenManagerTest > testExpireToken 
STARTED

kafka.security.token.delegation.DelegationTokenManagerTest > testExpireToken 
PASSED

kafka.security.token.delegation.DelegationTokenManagerTest > testRenewToken 
STARTED

kafka.security.token.delegation.DelegationTokenManagerTest > testRenewToken 
PASSED

kafka.security.auth.ResourceTypeTest > testJavaConversions STARTED

kafka.security.auth.ResourceTypeTest > testJavaConversions PASSED

kafka.security.auth.ResourceTypeTest > testFromString STARTED

kafka.security.auth.ResourceTypeTest > testFromString PASSED

kafka.security.auth.OperationTest > testJavaConversions STARTED

kafka.security.auth.OperationTest > testJavaConversions PASSED

kafka.security.auth.PermissionTypeTest > testJavaConversions STARTED

kafka.security.auth.PermissionTypeTest > testJavaConversions PASSED

kafka.security.auth.PermissionTypeTest > testFromString STARTE

Re: [DISCUSS]: KIP-230: Name Windowing Joins

[Guozhang Wang]

Hi Matthias,

I've also made a pass over the KIP, aside from the-other-Matthias's
comment, I'm wondering if you have scenarios that want to distinguish the
two internal topics of the join?

Currently we use "-this" and "-other" suffix for the topics. So for example:

stream1.join(stream2, ...)  // stream1 will be materialized with "-this",
and stream2 with "-other"

While:

stream2.join(stream1, ...)  // stream2 will be materialized with "-this",
and stream1 with "-other"


If we think it is reasonable to require users be aware that the above join
situations are not exactly the same, then the current naming is fine; if we
want them to be mutually reused (I'm not sure if this is a common case?)
then we probably need to consider something new?



Guozhang





On Tue, Feb 13, 2018 at 7:02 PM, Matthias Margush <
matthias.marg...@gmail.com> wrote:

> Thanks for the reminder! I need to do some wordsmithing based on the
> feedback I’ve gotten. I’ll do that soon (hopefully).
> On Tue, Feb 13, 2018 at 1:45 PM Matthias J. Sax 
> wrote:
>
> > Is there any updates for this KIP?
> >
> > -Matthias
> >
> > On 12/28/17 12:27 PM, Matthias J. Sax wrote:
> > > Thanks for updating the KIP.
> > >
> > > The code-diff is a  little hard to read. It's better to so something
> > > similar as in this KIP:
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 245%3A+Use+Properties+instead+of+StreamsConfig+in+KafkaStreams+constructor
> > >
> > > (Just as an example. Maybe take a look as other KIPs, too.)
> > >
> > > Some side remarks:
> > >  - please update the link to the DISCUSS thread
> > >  - there are some typos: Kstream -> KStream; Topology Builder exception
> > > -> TopologyBuilderException
> > >
> > >
> > > You propose to add `otherValueSerde(final String joinName)` -- I guess
> > > the method name is a c&p error and method name must be updated?
> > >
> > > Changes to internal classes like `KStreamImpl` are not required in the
> > > KIP as those as implementation details. The KIP should focus on public
> > > changes.
> > >
> > >
> > > -Matthias
> > >
> > >
> > > On 12/26/17 11:19 AM, Matthias Margush wrote:
> > >> Greetings.
> > >>
> > >> Thanks for the comments and suggestions. I updated the KIP with these
> > >> proposals for the questions posed by Matt & Matthias:
> > >>
> > >> *Can you please c&p the corresponding content instead of just
> > >> putting links? A KIP should be a self-contained Wiki page. Also, if we
> > add
> > >> a optional config parameter, how would we specify it? **Please list
> all
> > >> changes to want to apply to `Joined` class.*
> > >>
> > >> I added more details around the proposed changes directly to the KIP.
> > >>
> > >> *I will point out that your KIP doesn't outline what would happen if
> > >> you picked a name that resulted in a non unique topic name? What would
> > be
> > >> the error handling behavior there?*
> > >>
> > >> Looking at the current behavior of methods that allow the user to
> > specify
> > >> names for internal resources (e.g. `reduce`, `aggregate`), I added a
> > >> proposal that the code generate a similar exception if a name conflict
> > is
> > >> detected in the topology:
> > >>
> > >> org.apache.kafka.streams.errors.TopologyBuilderException: "Invalid
> > topology
> > >> building: Topic reduction-same-name-repartition has already been
> > registered
> > >> by another source."
> > >>
> > >> *What is the impact on KStream-KTable join?*
> > >>
> > >> Proposed that kstream-ktable joins similarly make use of the provided
> > >> joinName when generating internal repartition topics.
> > >>
> > >> On Mon, Dec 4, 2017 at 2:57 PM Matthias J. Sax  >
> > >> wrote:
> > >>
> > >>> Matthias,
> > >>>
> > >>> thanks for the KIP.
> > >>>
> > >>> Can you please c&p the corresponding content instead of just putting
> > >>> links? A KIP should be a self-contained Wiki page.
> > >>>
> > >>> Also, if we add a optional config parameter, how would we specify it?
> > >>> Please list all changes to want to apply to `Joined` class.
> > >>>
> > >>> Furthermore, `Joined` is also used for KStream-KTable join but the
> KIP
> > >>> only talks about windowed joins (ie, KStream-KTream join). What the
> > >>> impact on KStream-KTable join?
> > >>>
> > >>>
> > >>> -Matthias
> > >>>
> > >>> On 11/29/17 6:09 AM, Matt Farmer wrote:
> >  Hi Matthias,
> > 
> >  I certainly have found the auto-generated names unwieldy while doing
> >  cluster administration.
> > 
> >  I will point out that your KIP doesn't outline what would happen if
> > you
> >  picked a name that resulted in a non unique topic name? What would
> be
> > the
> >  error handling behavior there?
> > 
> >  On Wed, Nov 29, 2017 at 9:03 AM Matthias Margush <
> > >>> matthias.marg...@gmail.com>
> >  wrote:
> > 
> > > Hi everyone,
> > >
> > > I created this KIP to allow windowing joins to be named. If named,
> > then
> > >>> the
> > > associated internal topic names would be de

Re: Are there plans to migrate some/all of the command line tools to use the new AdminClient?

[Viktor Somogyi]

Hi Sönke,

There are a couple. The umbrella jira for these tasks is KAFKA-3268
 . I personally raised
KIP-248

to
refactor the kafka-configs.sh scripts.
I'd be happy to get some feedback on that (see the discussion thread for
it) but if you're interested in refactoring a command, then I'd be happy to
coordinate with you on that.

Viktor

On Thu, Feb 22, 2018 at 4:29 PM, Sönke Liebau <
soenke.lie...@opencore.com.invalid> wrote:

> I've dug around jira and the list of KIPs for a bit now, but could not
> really find anything specific on plans to move the command line tools over
> to the new AdminClient. Did I miss something or is that not currently
> planned?
>
> Most of the current command line tools require access to Zookeeper, which
> becomes a bit of an issue once you enable zookeeper acls, as you need to
> kinit with a broker keytab to be allowed write access which is somewhat of
> a security concern. Also, if you want to firewall Zookeeper of from the
> rest of the world any management command would need to be run from a
> cluster machine.
> None of this is an actual issue, it just required some additional effort
> for cluster administration, however in a larger corporate environment I
> can't imagine this would go down well with security audit guys and related
> persons.
>
> Using the AdminClient on the other hand allows to give specific users the
> right to create topics/acls etc.which is checked by the brokers and
> requires no access to Zookeeper by anybody except the brokers.
>
> Maybe we could add a --use-adminclient parameter to the command line tools
> sort of similar to the --new-consumer parameter to keep the old
> functionality while enabling us to slowly move things over to the
> AdminClient implementation?
>
> Best regards,
> Sönke
>

Re: Hello

[Ted Yu]

Impressive work.

Under 3.4.1, New Consumer Configs, there are several entries which are not
translated.

Is there plan to translate them ?

Cheers

On Tue, Feb 27, 2018 at 1:48 AM, 程威  wrote:

> Hello,we are a Open source Chinese document organization ,now we are
> spending time on translating kafka document into Chinese. This is 1.0.0
> version link https://github.com/apachecn/kafka-doc-zh/tree/1.0.0。
> We want to further cooperation with you, let more Chinese developers more
> convenient to use the document.
>
>

Re: Hello

[Guozhang Wang]

Thanks for sharing this Wei !

Just to let you know that there is a major web docs revamping discussion
coming along which aims to re-write the docs in non-html scripting:


https://issues.apache.org/jira/browse/KAFKA-2967

https://github.com/apache/kafka/pull/4536#discussion_r171336905

Guozhang


On Wed, Feb 28, 2018 at 1:51 PM, Ted Yu  wrote:

> Impressive work.
>
> Under 3.4.1, New Consumer Configs, there are several entries which are not
> translated.
>
> Is there plan to translate them ?
>
> Cheers
>
> On Tue, Feb 27, 2018 at 1:48 AM, 程威  wrote:
>
> > Hello,we are a Open source Chinese document organization ,now we are
> > spending time on translating kafka document into Chinese. This is 1.0.0
> > version link https://github.com/apachecn/kafka-doc-zh/tree/1.0.0。
> > We want to further cooperation with you, let more Chinese developers more
> > convenient to use the document.
> >
> >
>



-- 
-- Guozhang

Re: [DISCUSS] KIP-253: Support in-order message delivery with partition expansion

[Jan Filipiak]

Hi Dong,

I tried to focus on what the steps are one can currently perform to 
expand or shrink a keyed topic while maintaining a top notch semantics.
I can understand that there might be confusion about "stopping the 
consumer". It is exactly the same as proposed in the KIP. there needs to be
a time the producers agree on the new partitioning. The extra semantics 
I want to put in there is that we have a possibility to wait until all 
the existing data
is copied over into the new partitioning scheme. When I say stopping I 
think more of having a memory barrier that ensures the ordering. I am 
still aming for latencies  on the scale of leader failovers.


Consumers have to explicitly adapt the new partitioning scheme in the 
above scenario. The reason is that in these cases where you are 
dependent on a particular partitioning scheme, you also have other 
topics that have co-partition enforcements or the kind -frequently. 
Therefore all your other input topics might need to grow accordingly.



What I was suggesting was to streamline all these operations as best as 
possible to have "real" partition grow and shrinkage going on. Migrating 
the producers to a new partitioning scheme can be much more streamlined 
with proper broker support for this. Migrating consumer is a step that 
might be made completly unnecessary if - for example streams - takes the 
gcd as partitioning scheme instead of enforcing 1 to 1. Connect 
consumers and other consumers should be fine anyways.


I hope this makes more clear where I was aiming at. The rest needs to be 
figured out. The only danger i see is that when we are introducing this 
feature as supposed in the KIP, it wont help any people depending on  
log compaction.


The other thing I wanted to mention is that I believe the current 
suggestion (without copying data over) can be implemented in pure 
userland with a custom partitioner and a small feedbackloop from 
ProduceResponse => Partitionier in coorporation with a change management 
system.


Best Jan







On 28.02.2018 07:13, Dong Lin wrote:

Hey Jan,

I am not sure if it is acceptable for producer to be stopped for a while,
particularly for online application which requires low latency. I am also
not sure how consumers can switch to a new topic. Does user application
needs to explicitly specify a different topic for producer/consumer to
subscribe to? It will be helpful for discussion if you can provide more
detail on the interface change for this solution.

Thanks,
Dong

On Mon, Feb 26, 2018 at 12:48 AM, Jan Filipiak 
wrote:


Hi,

just want to throw my though in. In general the functionality is very
usefull, we should though not try to find the architecture to hard while
implementing.

The manual steps would be to

create a new topic
the mirrormake from the new old topic to the new topic
wait for mirror making to catch up.
then put the consumers onto the new topic
 (having mirrormaker spit out a mapping from old offsets to new offsets:
 if topic is increased by factor X there is gonna be a clean
mapping from 1 offset in the old topic to X offsets in the new topic,
 if there is no factor then there is no chance to generate a
mapping that can be reasonable used for continuing)
 make consumers stop at appropriate points and continue consumption
with offsets from the mapping.
have the producers stop for a minimal time.
wait for mirrormaker to finish
let producer produce with the new metadata.


Instead of implementing the approach suggest in the KIP which will leave
log compacted topic completely crumbled and unusable.
I would much rather try to build infrastructure to support the mentioned
above operations more smoothly.
Especially having producers stop and use another topic is difficult and
it would be nice if one can trigger "invalid metadata" exceptions for them
and
if one could give topics aliases so that their produces with the old topic
will arrive in the new topic.

The downsides are obvious I guess ( having the same data twice for the
transition period, but kafka tends to scale well with datasize). So its a
nicer fit into the architecture.

I further want to argument that the functionality by the KIP can
completely be implementing in "userland" with a custom partitioner that
handles the transition as needed. I would appreciate if someone could point
out what a custom partitioner couldn't handle in this case?

With the above approach, shrinking a topic becomes the same steps. Without
loosing keys in the discontinued partitions.

Would love to hear what everyone thinks.

Best Jan


















On 11.02.2018 00:35, Dong Lin wrote:


Hi all,

I have created KIP-253: Support in-order message delivery with partition
expansion. See
https://cwiki.apache.org/confluence/display/KAFKA/KIP-253%
3A+Support+in-order+message+delivery+with+partition+expansion
.

This KIP provides a way to allow messages of the same key from the same
producer to be consumed in the same order they are produced ev