Re: Documentation build system

[Sönke Liebau]

Hi,

I'm in favor of moving away from HTML as well, while I have not had
direct exposure to the pain points that Ewen mentions it is in general
not pleasant to write docs in I think and bears a lot of risk of
parsing failures etc.

I can't vote for using rst, not having used it, but I am happy to vote
against markdown, as my own experience is that you hit its limitations
fairly quickly and start looking for ways around these limitatios,
which would likely make us end up in a similarly hacky place as where
we are today, just on top of a different basis.

Best regards,
Sönke

On Wed, Feb 7, 2018 at 2:55 AM, Guozhang Wang  wrote:
> Ewen,
>
> Thanks for re-picking this up again. I'm big +1 as I was two years ago :P
> One thing that may still worth pointing out is that migrating from html
> means that for any edits it would require one more compilation step to
> review / compare the diffs if it is not only wording but also formatting /
> displaying purposed. Personally I think it is worthy.
>
> As for rst v.s. markdown v.s. anything else, I'm also inclining to rst but
> admittedly because I am familiar with rst than markdown as well. You have
> listed quite a list of pros of rst on the ticket and I do not have more to
> add.
>
>
> Guozhang
>
>
> On Tue, Feb 6, 2018 at 4:09 PM, Ewen Cheslack-Postava 
> wrote:
>
>> Hi all,
>>
>> I just wrote a note in https://issues.apache.org/jira/browse/KAFKA-2967
>> with a proposal for changing how docs are written. I want to move on this
>> soon if possible and normally would just leave the discussion to the JIRA,
>> but as I think this is something everyone has an opinion on and affects
>> everyone contributing to the project, I figured I'd send this quick note to
>> increase the likelihood people see it and have a chance to weigh in.
>>
>> Thanks,
>> -Ewen
>>
>
>
>
> --
> -- Guozhang



-- 
Sönke Liebau
Partner
Tel. +49 179 7940878
OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany

Jenkins build is back to normal : kafka-trunk-jdk7 #3161

[Apache Jenkins Server]

See 

[jira] [Resolved] (KAFKA-6524) kafka mirror can't producer internal topic

[Ahmed Madkour (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmed Madkour resolved KAFKA-6524.
--
Resolution: Information Provided

> kafka mirror can't producer internal topic 
> ---
>
> Key: KAFKA-6524
> URL: https://issues.apache.org/jira/browse/KAFKA-6524
> Project: Kafka
>  Issue Type: Bug
>  Components: producer 
>Affects Versions: 1.0.0
>Reporter: Ahmed Madkour
>Priority: Minor
>
> We are using kafka-mirror-maker.sh to consume data from a 3 brokers kafka 
> cluster and producer the data to another single broker kafka cluster
> We want to include internal topics so we added the following in the consumer 
> configuration
> exclude.internal.topics=false
> We keep receiving the following errors:
> {code:java}
> org.apache.kafka.common.errors.InvalidTopicException: The request attempted 
> to perform an operation on an invalid topic.
>  ERROR Error when sending message to topic __consumer_offsets with key: 43 
> bytes, value: 28 bytes with error: 
> (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
> {code}
> It seems that the producer can't access the internal topic __consumer_offsets.
> Any way to fix that?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : kafka-trunk-jdk8 #2395

[Apache Jenkins Server]

See 

[jira] [Resolved] (KAFKA-6532) Delegation token internals should not impact public interfaces

[Rajini Sivaram (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajini Sivaram resolved KAFKA-6532.
---
   Resolution: Fixed
Fix Version/s: 1.1.0

> Delegation token internals should not impact public interfaces
> --
>
> Key: KAFKA-6532
> URL: https://issues.apache.org/jira/browse/KAFKA-6532
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Reporter: Rajini Sivaram
>Assignee: Rajini Sivaram
>Priority: Major
> Fix For: 1.1.0
>
>
> We need to make sure that code related to the internal delegation tokens 
> implementation doesn't have any impact on public interfaces, including 
> customizable callback handlers from KIP-86.
>  # KafkaPrincipal has a public _tokenAuthenticated()_ method. Principal 
> builders are configurable and we now expect custom principal builders to set 
> this value. Since we allow the same endpoint to be used for basic SCRAM and 
> delegation tokens, the configured principal builder needs a way of detecting 
> token authentication. Default principal builder does this using internal 
> SCRAM implementation code. It will be better if configurable principal 
> builders didn't have to set this flag at all.
>  # It will be better to replace 
> _o.a.k.c.security.scram.DelegationTokenAuthenticationCallback_ with a more 
> generic _ScramExtensionsCallback_. This will allow us to add more extensions 
> in future and it will also enable custom Scram extensions.
>  # _ScramCredentialCallback_ was extended to add _tokenOwner_ and mechanism. 
> Mechanism is determined during SASL handshake and shouldn't be configurable 
> in a callback handler. _ScramCredentialCallback_ is being made a public 
> interface in KIP-86 with configurable callback handlers. Since delegation 
> token implementation is internal and not extensible, _tokenOwner_ should be 
> in a delegation-token-specific callback.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] KIP-248 Create New ConfigCommand That Uses The New AdminClient

[Viktor Somogyi]

Hi Rajini,

I think it makes sense absolutely and even we could do it for AlterQuotas
as we will have the same problem there.
Updated my KIP

to
reflect these changes:
- proposed protocol changes
- created a AdminOperation type to represent the Add/Set/Delete triplet.
(Put in the org.apache.kafka.clients.admin package)

Please let me know if I missed something that you thought otherwise.

Regards,
Viktor


On Tue, Feb 6, 2018 at 1:31 PM, Rajini Sivaram 
wrote:

> Hi Viktor,
>
> While implementing KAFKA-6494, I realised that there is a mismatch between
> the --alter command of ConfigCommand and AlterConfigs request.
> ConfigCommand uses --add-config and --delete-config to make incremental
> updates. --add-config reads all the configs from ZooKeeper and adds the
> delta provided on top of that. AlterConfigs request currently sets the
> whole properties object, so you need to know the full set of properties of
> an entity to use AlterConfigs request through the AdminClient. We don't
> allow sensitive configs to be read using AdminClient, so we can't read and
> add configs as we do with ZooKeeper. So we need a protocol change to make
> this work. I didn't want to make this change after KIP freeze, so perhaps
> we could include this in your KIP? We could perhaps add a mode
> (SET/ADD/DELETE) for AlterConfigs request where SET matches the existing
> behaviour for backward compatibility and ConfigCommand uses ADD/DELETE.
>
> Thoughts?
>
> Regards,
>
> Rajini
>
> On Fri, Jan 19, 2018 at 12:57 PM, Viktor Somogyi 
> wrote:
>
> > Hi Rajini,
> >
> > Ok, I think I got you. I wasn't calculating with the fact that the parent
> > might not be set, therefore it could be a default user as well or even
> the
> > default client if nothing else is set (supposing we're talking about the
> >  example). So if I'm correct, the quota will be applied in
> > the order of the above points. In this case your argument is absolutely
> > valid. I'll modify the QuotaSource.
> >
> > About your last point: yes, I was hesitating a lot. I thought the
> interface
> > would be simpler but after removing the helpers it's not that scary
> > afterall :).
> > I'll start the vote.
> >
> > Viktor
> >
> >
> > On Thu, Jan 18, 2018 at 7:59 PM, Rajini Sivaram  >
> > wrote:
> >
> > > Hi Viktor,
> > >
> > > Thanks for the updates.
> > >
> > > *QuotaSource* currently has *Self/Default/Parent*. Not sure if that is
> > > sufficient.
> > > For the entity , quota could be used from any of these
> > > configs:
> > >
> > >1. /config/users//clients/
> > >2. /config/users//clients/
> > >3. /config/users/
> > >4. /config/users//clients/
> > >5. /config/users//clients/
> > >6. /config/users/
> > >7. /config/clients/
> > >8. /config/clients/
> > >
> > > So perhaps we should have a *QuotaSource* entry for each of these
> eight?
> > >
> > > A couple of minor points:
> > >
> > >- *Help Message* still uses --config.properties
> > >- The other AdminClient APIs don't use aliases for various
> > collections.
> > >So not sure if we need the aliases here. I think you can leave it
> > as-is
> > > and
> > >see what others think.
> > >
> > > Yes, please do start the voting thread to make it in time for the KIP
> > > freeze.
> > >
> > > Thank you,
> > >
> > > Rajini
> > >
> > >
> > > On Thu, Jan 18, 2018 at 6:15 PM, Viktor Somogyi <
> viktorsomo...@gmail.com
> > >
> > > wrote:
> > >
> > > > Rajini, I have updated the KIP as agreed. Could you please have a
> > second
> > > > look at it?
> > > > I have also added a section about SCRAM:
> > > > "To enable describing and altering SCRAM credentials we will use the
> > > > DescribeConfigs and AlterConfigs protocols. There are no changes in
> the
> > > > protocol's structure but we will allow the USER resource type to be
> > > passed
> > > > in the protocol. When this happens, the server will know that SCRAM
> > > configs
> > > > are asked and will send them in the response.  In case of
> AlterConfigs
> > > if a
> > > > USER resource type is passed it will validate if there are only SCRAM
> > > > credentials are changed. If not, then will fail with
> > > > InvalidRequestException
> > > > ."
> > > >
> > > > If you don't have any comments, we might start voting as we're close
> to
> > > KIP
> > > > freeze.
> > > >
> > > > On Thu, Jan 18, 2018 at 12:12 PM, Viktor Somogyi <
> > > viktorsomo...@gmail.com>
> > > > wrote:
> > > >
> > > > > 3. Ok, I'll remove this from the KIP for now and perhaps add a
> future
> > > > > considerations section with the idea.
> > > > >
> > > > > 9. Ok, I can do that.
> > > > >
> > > > > On Thu, Jan 18, 2018 at 11:18 AM, Rajini Sivaram <
> > > > rajinisiva...@gmail.com>
> > > > > wrote:
> > > > >
> > > > >> Hi Viktor,
> > > > >>
> > > > >> 3. Agree that it would be better to use something like
> > > ConfigEntityList
> > > > >> rather than ListQuotas. But

Problem in use of Kafka connect to import and export data.

[Himanshu Rawani]

Hello Team,

I am replicating the same steps mentioned in your quickstart. I have
written some message to test.txt file through the below commands mentioned.

> echo foo> test.txt
> echo bar>> test.txt
But when I am running below command to instantiate connectors
".\bin\windows\connect-standalone.bat
.\config\connect-standalone.properties
.\config\connect-file-source.properties
.\config\connect-file-sink.properties"
It is throwing me following exceptions:

Please help me out whats wrong with it.

Trace:
[2018-02-07 17:08:35,822] WARN could not get type for name
org.osgi.framework.BundleListener from any class loader
(org.reflections.Reflections)
org.reflections.ReflectionsException: could not get type for name
org.osgi.framework.BundleListener
at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:390)
at
org.reflections.Reflections.expandSuperTypes(Reflections.java:381)
at org.reflections.Reflections.(Reflections.java:126)
at
org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:258)
at
org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:201)
at
org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:162)
at
org.apache.kafka.connect.runtime.isolation.Plugins.(Plugins.java:47)
at
org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:75)
Caused by: java.lang.ClassNotFoundException:
org.osgi.framework.BundleListener
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:388)
... 7 more
[2018-02-07 17:08:35,830] WARN could not get type for name
org.apache.kafka.common.utils.MockTime from any class loader
(org.reflections.Reflections)
org.reflections.ReflectionsException: could not get type for name
org.apache.kafka.common.utils.MockTime
at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:390)
at
org.reflections.Reflections.expandSuperTypes(Reflections.java:381)
at org.reflections.Reflections.(Reflections.java:126)
at
org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:258)
at
org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:201)
at
org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:162)
at
org.apache.kafka.connect.runtime.isolation.Plugins.(Plugins.java:47)
at
org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:75)
Caused by: java.lang.ClassNotFoundException:
org.apache.kafka.common.utils.MockTime
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:388)
... 7 more
[2018-02-07 17:08:35,838] WARN could not get type for name
com.google.gson.JsonDeserializer from any class loader
(org.reflections.Reflections)
org.reflections.ReflectionsException: could not get type for name
com.google.gson.JsonDeserializer
at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:390)
at
org.reflections.Reflections.expandSuperTypes(Reflections.java:381)
at org.reflections.Reflections.(Reflections.java:126)
at
org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:258)
at
org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:201)
at
org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:162)
at
org.apache.kafka.connect.runtime.isolation.Plugins.(Plugins.java:47)
at
org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:75)
Caused by: java.lang.ClassNotFoundException:
com.google.gson.JsonDeserializer
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:388)
... 7 more
[2018-02-07 17:08:35,847] WARN could not get type for name
org.scalatest.junit.JUnitSuite from any class loader
(org.re

Re: [DISCUSS] KIP-248 Create New ConfigCommand That Uses The New AdminClient

[Rajini Sivaram]

Hi Viktor,

Thanks for the updates. Looks good, just a few minor comments:

   1. AdminOperation - could be AlterOperation since it is only applied to
   'Alter'?
   2. Don't think we need `Unknown` type to process old requests. We can
   use `Set` as the default for alter requests with version 0.
   3. There is a typo in
   
https://cwiki.apache.org/confluence/display/KAFKA/KIP-248+-+Create+New+ConfigCommand+That+Uses+The+New+AdminClient#KIP-248-CreateNewConfigCommandThatUsesTheNewAdminClient-AdminClientAPIs
:
   AdminOperation enum has a constructor QuotaType.


On Wed, Feb 7, 2018 at 4:53 PM, Viktor Somogyi 
wrote:

> Hi Rajini,
>
> I think it makes sense absolutely and even we could do it for AlterQuotas
> as we will have the same problem there.
> Updated my KIP
>  ConfigCommand+That+Uses+The+New+AdminClient>
> to
> reflect these changes:
> - proposed protocol changes
> - created a AdminOperation type to represent the Add/Set/Delete triplet.
> (Put in the org.apache.kafka.clients.admin package)
>
> Please let me know if I missed something that you thought otherwise.
>
> Regards,
> Viktor
>
>
> On Tue, Feb 6, 2018 at 1:31 PM, Rajini Sivaram 
> wrote:
>
> > Hi Viktor,
> >
> > While implementing KAFKA-6494, I realised that there is a mismatch
> between
> > the --alter command of ConfigCommand and AlterConfigs request.
> > ConfigCommand uses --add-config and --delete-config to make incremental
> > updates. --add-config reads all the configs from ZooKeeper and adds the
> > delta provided on top of that. AlterConfigs request currently sets the
> > whole properties object, so you need to know the full set of properties
> of
> > an entity to use AlterConfigs request through the AdminClient. We don't
> > allow sensitive configs to be read using AdminClient, so we can't read
> and
> > add configs as we do with ZooKeeper. So we need a protocol change to make
> > this work. I didn't want to make this change after KIP freeze, so perhaps
> > we could include this in your KIP? We could perhaps add a mode
> > (SET/ADD/DELETE) for AlterConfigs request where SET matches the existing
> > behaviour for backward compatibility and ConfigCommand uses ADD/DELETE.
> >
> > Thoughts?
> >
> > Regards,
> >
> > Rajini
> >
> > On Fri, Jan 19, 2018 at 12:57 PM, Viktor Somogyi <
> viktorsomo...@gmail.com>
> > wrote:
> >
> > > Hi Rajini,
> > >
> > > Ok, I think I got you. I wasn't calculating with the fact that the
> parent
> > > might not be set, therefore it could be a default user as well or even
> > the
> > > default client if nothing else is set (supposing we're talking about
> the
> > >  example). So if I'm correct, the quota will be applied
> in
> > > the order of the above points. In this case your argument is absolutely
> > > valid. I'll modify the QuotaSource.
> > >
> > > About your last point: yes, I was hesitating a lot. I thought the
> > interface
> > > would be simpler but after removing the helpers it's not that scary
> > > afterall :).
> > > I'll start the vote.
> > >
> > > Viktor
> > >
> > >
> > > On Thu, Jan 18, 2018 at 7:59 PM, Rajini Sivaram <
> rajinisiva...@gmail.com
> > >
> > > wrote:
> > >
> > > > Hi Viktor,
> > > >
> > > > Thanks for the updates.
> > > >
> > > > *QuotaSource* currently has *Self/Default/Parent*. Not sure if that
> is
> > > > sufficient.
> > > > For the entity , quota could be used from any of
> these
> > > > configs:
> > > >
> > > >1. /config/users//clients/
> > > >2. /config/users//clients/
> > > >3. /config/users/
> > > >4. /config/users//clients/
> > > >5. /config/users//clients/
> > > >6. /config/users/
> > > >7. /config/clients/
> > > >8. /config/clients/
> > > >
> > > > So perhaps we should have a *QuotaSource* entry for each of these
> > eight?
> > > >
> > > > A couple of minor points:
> > > >
> > > >- *Help Message* still uses --config.properties
> > > >- The other AdminClient APIs don't use aliases for various
> > > collections.
> > > >So not sure if we need the aliases here. I think you can leave it
> > > as-is
> > > > and
> > > >see what others think.
> > > >
> > > > Yes, please do start the voting thread to make it in time for the KIP
> > > > freeze.
> > > >
> > > > Thank you,
> > > >
> > > > Rajini
> > > >
> > > >
> > > > On Thu, Jan 18, 2018 at 6:15 PM, Viktor Somogyi <
> > viktorsomo...@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > Rajini, I have updated the KIP as agreed. Could you please have a
> > > second
> > > > > look at it?
> > > > > I have also added a section about SCRAM:
> > > > > "To enable describing and altering SCRAM credentials we will use
> the
> > > > > DescribeConfigs and AlterConfigs protocols. There are no changes in
> > the
> > > > > protocol's structure but we will allow the USER resource type to be
> > > > passed
> > > > > in the protocol. When this happens, the server will know that SCRAM
> > > > configs
> > > > > are asked and wil

Re: [VOTE] 1.0.1 RC0

[Vahid S Hashemian]

Hi Ewen,

+1

Building from source and running the quickstart were successful on Ubuntu 
and Windows 10.

Thanks for running the release.
--Vahid



From:   Ewen Cheslack-Postava 
To: dev@kafka.apache.org, us...@kafka.apache.org, 
kafka-clie...@googlegroups.com
Date:   02/05/2018 07:49 PM
Subject:[VOTE] 1.0.1 RC0



Hello Kafka users, developers and client-developers,

Sorry for a bit of delay, but I've now prepared the first candidate for
release of Apache Kafka 1.0.1.

This is a bugfix release for the 1.0 branch that was first released with
1.0.0 about 3 months ago. We've fixed 46 significant issues since that
release. Most of these are non-critical, but in aggregate these fixes will
have significant impact. A few of the more significant fixes include:

* KAFKA-6277: Make loadClass thread-safe for class loaders of Connect
plugins
* KAFKA-6185: Selector memory leak with high likelihood of OOM in case of
down conversion
* KAFKA-6269: KTable state restore fails after rebalance
* KAFKA-6190: GlobalKTable never finishes restoring when consuming
transactional messages

Release notes for the 1.0.1 release:
https://urldefense.proofpoint.com/v2/url?u=http-3A__home.apache.org_-7Eewencp_kafka-2D1.0.1-2Drc0_RELEASE-5FNOTES.html&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc&m=Z45uiGWoLkCQ5hYes5SiOy1n_pA3ih4Cvmr5W32xx98&s=l1iKa9gDVsN8n73JUsdMj2b_8vCXjo6ZlhPjlHnwLa4&e=


*** Please download, test and vote by Thursday, Feb 8, 12pm PT ***

Kafka's KEYS file containing PGP keys we use to sign the release:
https://urldefense.proofpoint.com/v2/url?u=http-3A__kafka.apache.org_KEYS&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc&m=Z45uiGWoLkCQ5hYes5SiOy1n_pA3ih4Cvmr5W32xx98&s=FMJWV-i3KbNT9eWV7mxnb9vLofAG8UOyqf13nC60HT0&e=


* Release artifacts to be voted upon (source and binary):
https://urldefense.proofpoint.com/v2/url?u=http-3A__home.apache.org_-7Eewencp_kafka-2D1.0.1-2Drc0_&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc&m=Z45uiGWoLkCQ5hYes5SiOy1n_pA3ih4Cvmr5W32xx98&s=wfEb6h21ejMltBiWDsND5C_iAR1asfxwSVKbbmNwDRQ&e=


* Maven artifacts to be voted upon:
https://urldefense.proofpoint.com/v2/url?u=https-3A__repository.apache.org_content_groups_staging_&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc&m=Z45uiGWoLkCQ5hYes5SiOy1n_pA3ih4Cvmr5W32xx98&s=YVQzF4zQchi3ru3UYkgkhgC2LnRRf_NFl1iJId4Iw2Q&e=


* Javadoc:
https://urldefense.proofpoint.com/v2/url?u=http-3A__home.apache.org_-7Eewencp_kafka-2D1.0.1-2Drc0_javadoc_&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc&m=Z45uiGWoLkCQ5hYes5SiOy1n_pA3ih4Cvmr5W32xx98&s=Y7hXIhHxDGb-M7d6kLZaargoYcLW6kH3agSdqO1SuwQ&e=


* Tag to be voted upon (off 1.0 branch) is the 1.0.1 tag:
https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_kafka_tree_1.0.1-2Drc0&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc&m=Z45uiGWoLkCQ5hYes5SiOy1n_pA3ih4Cvmr5W32xx98&s=L729TlgNpT-y8WQzeZTsNATg1zFfAsCpXBhXfbu6UXk&e=



* Documentation:
https://urldefense.proofpoint.com/v2/url?u=http-3A__kafka.apache.org_10_documentation.html&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc&m=Z45uiGWoLkCQ5hYes5SiOy1n_pA3ih4Cvmr5W32xx98&s=DYynoi4X5K3p9DwzxkGYp8vprFK4qvPPQtO1IvQEbME&e=


* Protocol:
https://urldefense.proofpoint.com/v2/url?u=http-3A__kafka.apache.org_10_protocol.html&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc&m=Z45uiGWoLkCQ5hYes5SiOy1n_pA3ih4Cvmr5W32xx98&s=_BLA3u9JgZKeJ0Kwij9_2J3lnxt8rCCXmptRh4OUPic&e=



Please test and verify the release artifacts and submit a vote for this 
RC,
or report any issues so we can fix them and get a new RC out ASAP! 
Although
this release vote requires PMC votes to pass, testing, votes, and bug
reports are valuable and appreciated from everyone.

Thanks,
Ewen

Bay Area Apache Kafka Meetup - Morning of Feb 20

[Vahid S Hashemian]

Kafka users and developers,

The next *Bay Area Apache Kafka Meetup* is on the *morning of Feb 20* and 
is hosted by *Index Developer Conference* at Moscone West in San Francisco
.
Meetup Info: https://www.meetup.com/KafkaBayArea/events/247433783/
Registration Link: https://ibm.co/2n742Jn (required)

Promo code for free meetup registration: CD1KAFKA
Promo code for free meetup registration + free full Index pass (Feb 
20-22): IND18FULL (expires Feb 12, 11:59 PST)

Detailed instructions and agenda can be found at the meetup link above.

Hope to see you there.
--Vahid

Build failed in Jenkins: kafka-1.1-jdk7 #19

[Apache Jenkins Server]

See 


Changes:

[matthias] KAFKA-6367: StateRestoreListener use actual last restored offset for

--
[...truncated 411.82 KB...]

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewConsumerConfigWithAutoOffsetResetAndMatchingFromBeginning 
STARTED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewConsumerConfigWithAutoOffsetResetAndMatchingFromBeginning 
PASSED

kafka.tools.ConsoleConsumerTest > shouldStopWhenOutputCheckErrorFails STARTED

kafka.tools.ConsoleConsumerTest > shouldStopWhenOutputCheckErrorFails PASSED

kafka.tools.ConsoleConsumerTest > 
shouldExitOnInvalidConfigWithAutoOffsetResetAndConflictingFromBeginningNewConsumer
 STARTED

kafka.tools.ConsoleConsumerTest > 
shouldExitOnInvalidConfigWithAutoOffsetResetAndConflictingFromBeginningNewConsumer
 PASSED

kafka.tools.ConsoleConsumerTest > 
shouldSetAutoResetToSmallestWhenFromBeginningConfigured STARTED

kafka.tools.ConsoleConsumerTest > 
shouldSetAutoResetToSmallestWhenFromBeginningConfigured PASSED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewConsumerConfigWithAutoOffsetResetEarliest STARTED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewConsumerConfigWithAutoOffsetResetEarliest PASSED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewSimpleConsumerValidConfigWithStringOffset STARTED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewSimpleConsumerValidConfigWithStringOffset PASSED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidOldConsumerConfigWithAutoOffsetResetLargest STARTED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidOldConsumerConfigWithAutoOffsetResetLargest PASSED

kafka.tools.ConsoleConsumerTest > shouldParseConfigsFromFile STARTED

kafka.tools.ConsoleConsumerTest > shouldParseConfigsFromFile PASSED

kafka.tools.ConsoleConsumerTest > groupIdsProvidedInDifferentPlacesMustMatch 
STARTED

kafka.tools.ConsoleConsumerTest > groupIdsProvidedInDifferentPlacesMustMatch 
PASSED

kafka.tools.ConsoleConsumerTest > 
shouldExitOnInvalidConfigWithAutoOffsetResetAndConflictingFromBeginningOldConsumer
 STARTED

kafka.tools.ConsoleConsumerTest > 
shouldExitOnInvalidConfigWithAutoOffsetResetAndConflictingFromBeginningOldConsumer
 PASSED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewSimpleConsumerValidConfigWithNumericOffset STARTED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewSimpleConsumerValidConfigWithNumericOffset PASSED

kafka.tools.ConsoleConsumerTest > testDefaultConsumer STARTED

kafka.tools.ConsoleConsumerTest > testDefaultConsumer PASSED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewConsumerConfigWithAutoOffsetResetLatest STARTED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewConsumerConfigWithAutoOffsetResetLatest PASSED

kafka.tools.ConsoleConsumerTest > shouldParseValidOldConsumerValidConfig STARTED

kafka.tools.ConsoleConsumerTest > shouldParseValidOldConsumerValidConfig PASSED

kafka.message.MessageCompressionTest > testCompressSize STARTED

kafka.message.MessageCompressionTest > testCompressSize PASSED

kafka.message.MessageCompressionTest > testSimpleCompressDecompress STARTED

kafka.message.MessageCompressionTest > testSimpleCompressDecompress PASSED

kafka.message.MessageTest > testChecksum STARTED

kafka.message.MessageTest > testChecksum PASSED

kafka.message.MessageTest > testInvalidTimestamp STARTED

kafka.message.MessageTest > testInvalidTimestamp PASSED

kafka.message.MessageTest > testIsHashable STARTED

kafka.message.MessageTest > testIsHashable PASSED

kafka.message.MessageTest > testInvalidTimestampAndMagicValueCombination STARTED

kafka.message.MessageTest > testInvalidTimestampAndMagicValueCombination PASSED

kafka.message.MessageTest > testExceptionMapping STARTED

kafka.message.MessageTest > testExceptionMapping PASSED

kafka.message.MessageTest > testFieldValues STARTED

kafka.message.MessageTest > testFieldValues PASSED

kafka.message.MessageTest > testInvalidMagicByte STARTED

kafka.message.MessageTest > testInvalidMagicByte PASSED

kafka.message.MessageTest > testEquality STARTED

kafka.message.MessageTest > testEquality PASSED

kafka.message.ByteBufferMessageSetTest > testMessageWithProvidedOffsetSeq 
STARTED

kafka.message.ByteBufferMessageSetTest > testMessageWithProvidedOffsetSeq PASSED

kafka.message.ByteBufferMessageSetTest > testValidBytes STARTED

kafka.message.ByteBufferMessageSetTest > testValidBytes PASSED

kafka.message.ByteBufferMessageSetTest > testValidBytesWithCompression STARTED

kafka.message.ByteBufferMessageSetTest > testValidBytesWithCompression PASSED

kafka.message.ByteBufferMessageSetTest > 
testWriteToChannelThatConsumesPartially STARTED

kafka.message.ByteBufferMessageSetTest > 
testWriteToChannelThatConsumesPartially PASSED

kafka.message.ByteBufferMessageSetTest > testIteratorIsConsistent STARTED

kafka.message.ByteBufferMessageSetTest > testIteratorIsConsistent PASSED

kafka.message.ByteBu

[jira] [Resolved] (KAFKA-4641) Improve test coverage of StreamsThread

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang resolved KAFKA-4641.
--
   Resolution: Fixed
 Assignee: Guozhang Wang
Fix Version/s: 1.2.0

> Improve test coverage of StreamsThread
> --
>
> Key: KAFKA-4641
> URL: https://issues.apache.org/jira/browse/KAFKA-4641
> Project: Kafka
>  Issue Type: Sub-task
>  Components: streams
>Affects Versions: 0.11.0.0
>Reporter: Damian Guy
>Assignee: Guozhang Wang
>Priority: Minor
>  Labels: newbie
> Fix For: 1.2.0
>
>
> Some methods in {{StreamThread}} have little or no coverage.
> In particular:
> {{maybeUpdateStandbyTasks}} has little to no coverage
> Committing of StandbyTasks in {{commitAll}}
> {{maybePunctuate}}
> {{commitOne}} - no tests for exceptions
> {{unAssignChangeLogPartitions} - no tests for exceptions
> {{addStreamsTask}} - no tests for exceptions
> {{runLoop}}
> Please see coverage report attached to parent



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] KIP-251: Allow timestamp manipulation in Processor API

[Guozhang Wang]

I intend to suggest we piggyback this change, just for connivence.


Guozhang

On Tue, Feb 6, 2018 at 7:25 PM, Matthias J. Sax 
wrote:

> I am open to remove the index API and also open to piggyback this change
> on this KIP... Not sure if this was the intention of your comment? Or
> should be have a separate KIP for this?
>
> Not sure, what other think about removing the index API?
>
>
> -Matthias
>
> On 2/5/18 4:13 PM, Guozhang Wang wrote:
> > This is not included in this KIP, but I'm wondering if we should still
> > support forwarding to a child with index.
> >
> > Because 1) if users are using DSL, they would probably never use this API
> > since the ordering of the child is hard to reason from a DSL anyways.
> >2) if users are using PAPI, they would just use the other
> > overload with child name.
> >
> > In fact the index currently is only used at KStream.branch internally,
> and
> > maybe we can just keep a map from predicate to child name inside Filter
> > operator, so that we do not use it at all. Just as a hindsight, I feel
> this
> > overloading API is not useful to users at all.
> >
> >
> > Guozhang
> >
> >
> > On Thu, Feb 1, 2018 at 5:05 PM, Matthias J. Sax 
> > wrote:
> >
> >> Thanks.
> >>
> >> I updated the KIP accordingly and started work on the PR to see if this
> >> `To` interface work nicely.
> >>
> >> -Matthias
> >>
> >> On 2/1/18 4:00 PM, Ted Yu wrote:
> >>> Yeah.
> >>> Cleaner in this formation.
> >>>
> >>> On Thu, Feb 1, 2018 at 3:59 PM, Bill Bejeck  wrote:
> >>>
>  `To` works for me.
> 
>  Thanks,
>  Bill
> 
>  On Thu, Feb 1, 2018 at 6:47 PM, Matthias J. Sax <
> matth...@confluent.io>
>  wrote:
> 
> > @Paolo:
> >
> > The timestamp will be used to set the message/record metadata
> timestamp
> > on `Producer.send(new ProducerRecord(...,timestamp,...))`.
> >
> > @Bill,Ted:
> >
> > Might be a good idea. I was thinking about the name, and came up with
>  `To`:
> >
> >> context.forward(key, value, To.child("processorX").
> withTimestamp(5));
> >> context.forward(key, value, To.child(1).withTimestamp(10));
> >
> > Without specifying the downstream child processor:
> >
> >> context.forward(key, value, To.all().withTimestamp(5));
> >
> > WDYT?
> >
> >
> > -Matthias
> >
> > On 2/1/18 8:45 AM, Ted Yu wrote:
> >> I like Bill's idea (pending a better name for the Forwarded).
> >>
> >> Cheers
> >>
> >> On Thu, Feb 1, 2018 at 7:47 AM, Bill Bejeck 
> >> wrote:
> >>
> >>> Hi Matthias,
> >>>
> >>> Thanks for the KIP!
> >>>
> >>> Could we consider taking an approach similar to what was done in
>  KIP-182
> >>> with regards to overloading?
> >>>
> >>> Meaning we could add a "Forwarded" object (horrible name I know)
> with
> >>> methods withTimestamp, withChildName, and withChildIndex. To handle
>  the
> >>> case when both a child-name and child-index is provided we could
> >> throw
> > an
> >>> exception.
> >>>
> >>> Then we could reduce the overloaded {{forward}} methods from 6 to
> 2.
> >>>
> >>> Thanks,
> >>> Bill
> >>>
> >>>
> >>> On Thu, Feb 1, 2018 at 3:49 AM, Paolo Patierno  >
> > wrote:
> >>>
>  Hi Matthias,
> 
>  just a question : what will be the timestamp "type" in the new
>  message
> > on
>  the wire ?
> 
>  Thanks,
>  Paolo.
>  
>  From: Matthias J. Sax 
>  Sent: Wednesday, January 31, 2018 2:06 AM
>  To: dev@kafka.apache.org
>  Subject: [DISCUSS] KIP-251: Allow timestamp manipulation in
> >> Processor
> > API
> 
>  Hi,
> 
>  I want to propose a new KIP for Kafka Streams that allows
> timestamp
>  manipulation at Processor API level.
> 
>  https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>  251%3A+Allow+timestamp+manipulation+in+Processor+API
> 
>  Looking forward to your feedback.
> 
> 
>  -Matthias
> 
> 
> >>>
> >>
> >
> >
> 
> >>>
> >>
> >>
> >
> >
>
>


-- 
-- Guozhang

Re: [DISCUSS] KIP-228 Negative record timestamp support

[Guozhang Wang]

TL.DR: I'd agree with Colin here.


I admit that define -1 == unknown is not a good idea, and if we design the
system from scratch we should not do so.

But Colin's point is valid, that it has been treated like this in past
versions, and I have seen producer client users doing so indeed. So
changing it could potentially break lots of users.

Thinking about this from the user base respective:

1. I'm a happy user as of today and will never use a negative timestamp. I
do not care about this KIP though I do not want to change "-1 ==
unknown" semantics since I'm using it.
2. I'm a happy user that I do not use -1 as well. I will not use negative
timestamps so I do not care about this KIP at all.
3. I'm not happy because I cannot use negative timestamps, I would want
this KIP, while I'm also OK to keep -1 as is knowing it is only a special
case that I will need to work around; think about the most simple hack: one
line change to use 0 or -2 when I want to mean -1.
4. I'm not happy because I cannot use negative timestamps, I would love
this KIP, and I do care about -1 because to me it could be very common or I
would not want to special handle it.

If our expectation that population wise, 2 > 1 >> 3 >> 4, then Colin's
suggestion makes sense to me.

On the other side, I tend to not introduce new configs if their targeted
audience is too small, this is what we learned from core broker configs
that more configs would tend to make users more confused over time.
Although we can argue that for client configs, we have made them on three
levels (HIGH, MEDIUM, LOW) and expect those LOW configs to be of interest
only for a small group of users and hence can be less confusing to normal
users as they'd probably not even pay attention to them, to me this -1 use
case is still too small even for a new LOW level config.





Guozhang



On Tue, Jan 16, 2018 at 1:36 PM, Colin McCabe  wrote:

> I think we should just keep -1 as a special value, and allow negative
> timestamps.  It just means one missing millisecond in 1969, right?
>
> There is just a lot of code everywhere checking for -1, and changing it
> now would be really likely to be buggy.
>
> We probably also want to limit the range of timestamps as well, so that we
> can calculate timestamp1 - timestamp2 without overflowing a 64-bit long.
> If we cap timestamps to be in the range (-2**62,  2**62), we can still
> represent dates about 146 million years into the future or past.
>
> best,
> Colin
>
>
> On Tue, Jan 16, 2018, at 13:03, Konstantin Chukhlomin wrote:
> > Hi all,
> >
> > I tried to summarize below all approaches we were discussing.
> >
> > In case there are some difficulties in email formatting, see GoogleDocs:
> > https://docs.google.com/document/d/1RjlcebpigOj9DyLCedyRxki9nZcFd
> kBchy-k7BiThMc/edit?usp=sharing
> >  kBchy-k7BiThMc/edit?usp=sharing>
> >
> > Personally, I find options A01 and A02/B03 very convenient.
> >
> > Please, let me know what you think and feel free to add more options.
> >
> > Thanks,
> > Konstantin
> >
> > –
> >
> > Preconditions:
> > we would need to remote negative timestamp check from producer anyway
> > no existing data should have -1 timestamp unless it is NO_TIMESTAMP
> > older client are not embedding a timestamp: default to -1 on the broker
> side...
> > APPEND_TIME would still work for older producers
> >
> > Can we still use −1 as NO_TIMESTAMP value?
> > Yes: A01 Keep using −1ms as a NO_TIMESTAMP value
> > No, A02 We want to change the semantics of −1ms as an actual timestamp.
> > For example, use Long.MIN_VALUE as "NO_TIMESTAMP".
> > How do we indicate new timestamp format?
> > A02 In the message itself. That way one topic may contain messages of
> > different types.
> > How?
> > B01 Create new timestamp type and use one of the reserved bits
> > B02 Use one reserved bit in the message as a flag
> > B03 Introduce new message format version
> > By the topic property: `allow.negative.timestamp`.
> > That way messages with CreateTime could have negative value.
> > "false" by default for existing topics.
> > Should `allow.negative.timestamp` be enabled by default for new topics?
> > Yes. C01: We want it to be future standard
> > No. C02: We want it to be client specific case by case decision.
> > Do we allow changing that property for existing topics?
> > Yes
> > Yes, but only if existing topic have no records with negative timestamps
> > already
> > No: old behavior and new behavior are not mixed
> > Can old producers write to topics with that flag enabled?
> > Yes, as long as we can differentiate old and new messages: B01-03
> > No, broker will write old "NO_TIMESTAMP" as a −1ms
> > No, broker will throw an error.
> > By the broker config: `allow.negative.timestamp`.
> > What to do with old topics?
> > D01: Convert all messages in all topics to the new format (B01-03)
> > during the update phase.
> > D02: Keep old messages, but w

Build failed in Jenkins: kafka-1.1-jdk7 #20

[Apache Jenkins Server]

See 


Changes:

[matthias] MINOR: update upgrade notes for Streams API; message format 0.10

[matthias] MINOR: Add attributes `processedKeys` and `processedValues` to

--
[...truncated 412.05 KB...]

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldIncrementEpochAndUpdateMetadataOnHandleInitPidWhenExistingCompleteTransaction
 STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldIncrementEpochAndUpdateMetadataOnHandleInitPidWhenExistingCompleteTransaction
 PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldGenerateNewProducerIdIfEpochsExhausted STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldGenerateNewProducerIdIfEpochsExhausted PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithNotCoordinatorOnInitPidWhenNotCoordinator STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithNotCoordinatorOnInitPidWhenNotCoordinator PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithConcurrentTransactionOnAddPartitionsWhenStateIsPrepareAbort 
STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithConcurrentTransactionOnAddPartitionsWhenStateIsPrepareAbort 
PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldInitPidWithEpochZeroForNewTransactionalId STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldInitPidWithEpochZeroForNewTransactionalId PASSED

kafka.coordinator.transaction.TransactionStateManagerTest > 
testAppendTransactionToLogWhileProducerFenced STARTED

kafka.coordinator.transaction.TransactionStateManagerTest > 
testAppendTransactionToLogWhileProducerFenced PASSED

kafka.coordinator.transaction.TransactionStateManagerTest > 
testCompleteTransitionWhenAppendSucceeded STARTED

kafka.coordinator.transaction.TransactionStateManagerTest > 
testCompleteTransitionWhenAppendSucceeded PASSED

kafka.coordinator.transaction.TransactionStateManagerTest > 
testAppendFailToCoordinatorNotAvailableError STARTED

kafka.coordinator.transaction.TransactionStateManagerTest > 
testAppendFailToCoordinatorNotAvailableError PASSED

kafka.coordinator.transaction.TransactionStateManagerTest > 
testAppendFailToUnknownError STARTED

kafka.coordinator.transaction.TransactionStateManagerTest > 
testAppendFailToUnknownError PASSED

kafka.coordinator.transaction.TransactionStateManagerTest > 
shouldReturnNotCooridnatorErrorIfTransactionIdPartitionNotOwned STARTED

kafka.coordinator.transaction.TransactionStateManagerTest > 
shouldReturnNotCooridnatorErrorIfTransactionIdPartitionNotOwned PASSED

kafka.coordinator.transaction.TransactionStateManagerTest > 
testValidateTransactionTimeout STARTED

kafka.coordinator.transaction.TransactionStateManagerTest > 
testValidateTransactionTimeout PASSED

kafka.coordinator.transaction.TransactionStateManagerTest > 
shouldWriteTxnMarkersForTransactionInPreparedCommitState STARTED

kafka.coordinator.transaction.TransactionStateManagerTest > 
shouldWriteTxnMarkersForTransactionInPreparedCommitState PASSED

kafka.coordinator.transaction.TransactionStateManagerTest > 
shouldOnlyConsiderTransactionsInTheOngoingStateToAbort STARTED

kafka.coordinator.transaction.TransactionStateManagerTest > 
shouldOnlyConsiderTransactionsInTheOngoingStateToAbort PASSED

kafka.coordinator.transaction.TransactionStateManagerTest > 
shouldRemoveCompleteAbortExpiredTransactionalIds STARTED

kafka.coordinator.transaction.TransactionStateManagerTest > 
shouldRemoveCompleteAbortExpiredTransactionalIds PASSED

kafka.coordinator.transaction.TransactionStateManagerTest > 
testAppendTransactionToLogWhilePendingStateChanged STARTED

kafka.coordinator.transaction.TransactionStateManagerTest > 
testAppendTransactionToLogWhilePendingStateChanged PASSED

kafka.coordinator.transaction.TransactionStateManagerTest > 
testLoadAndRemoveTransactionsForPartition STARTED

kafka.coordinator.transaction.TransactionStateManagerTest > 
testLoadAndRemoveTransactionsForPartition PASSED

kafka.coordinator.transaction.TransactionStateManagerTest > 
testAppendFailToNotCoordinatorError STARTED

kafka.coordinator.transaction.TransactionStateManagerTest > 
testAppendFailToNotCoordinatorError PASSED

kafka.coordinator.transaction.TransactionStateManagerTest > 
shouldNotRemovePrepareCommitTransactionalIds STARTED

kafka.coordinator.transaction.TransactionStateManagerTest > 
shouldNotRemovePrepareCommitTransactionalIds PASSED

kafka.coordinator.transaction.TransactionStateManagerTest > 
testAppendFailToCoordinatorLoadingError STARTED

kafka.coordinator.transaction.TransactionStateManagerTest > 
testAppendFailToCoordinatorLoadingError PASSED

kafka.coordinator.transaction.TransactionStateManagerTest > 
shouldNotRemoveOngoingTransactionalIds STARTED

kafka.coordinator.transaction.TransactionStat

[jira] [Created] (KAFKA-6540) Consumer lag metric is not updated when a partition is paused

[Jason Gustafson (JIRA)]

Jason Gustafson created KAFKA-6540:
--

 Summary: Consumer lag metric is not updated when a partition is 
paused
 Key: KAFKA-6540
 URL: https://issues.apache.org/jira/browse/KAFKA-6540
 Project: Kafka
  Issue Type: Bug
Reporter: Jason Gustafson


When a partition is paused, we no longer include it in fetches, which means we 
do not get updates to the high watermark. Since lag is computed based on the 
high watermark we've received in the most recent fetch, this means that the 
reported lag of a paused partitions will be stuck at whatever value it had when 
the partition was paused. A possible workaround is to continue fetching the 
partition, but set the max requested bytes for that partition to 0.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] KIP-232: Detect outdated metadata using leaderEpoch and partitionEpoch

[Jun Rao]

Hi, Dong,

Thanks for the reply. The general idea that you had for adding partitions
is similar to what we had in mind. It would be useful to make this more
general, allowing adding an arbitrary number of partitions (instead of just
doubling) and potentially removing partitions as well. The following is the
high level idea from the discussion with Colin, Jason and Ismael.

* To change the number of partitions from X to Y in a topic, the controller
marks all existing X partitions as read-only and creates Y new partitions.
The new partitions are writable and are tagged with a higher repartition
epoch (RE).

* The controller propagates the new metadata to every broker. Once the
leader of a partition is marked as read-only, it rejects the produce
requests on this partition. The producer will then refresh the metadata and
start publishing to the new writable partitions.

* The consumers will then be consuming messages in RE order. The consumer
coordinator will only assign partitions in the same RE to consumers. Only
after all messages in an RE are consumed, will partitions in a higher RE be
assigned to consumers.

As Colin mentioned, if we do the above, we could potentially (1) use a
globally unique partition id, or (2) use a globally unique topic id to
distinguish recreated partitions due to topic deletion.

So, perhaps we can sketch out the re-partitioning KIP a bit more and see if
there is any overlap with KIP-232. Would you be interested in doing that?
If not, we can do that next week.

Jun


On Tue, Feb 6, 2018 at 11:30 AM, Dong Lin  wrote:

> Hey Jun,
>
> Interestingly I am also planning to sketch a KIP to allow partition
> expansion for keyed topics after this KIP. Since you are already doing
> that, I guess I will just share my high level idea here in case it is
> helpful.
>
> The motivation for the KIP is that we currently lose order guarantee for
> messages with the same key if we expand partitions of keyed topic.
>
> The solution can probably be built upon the following ideas:
>
> - Partition number of the keyed topic should always be doubled (or
> multiplied by power of 2). Given that we select a partition based on
> hash(key) % partitionNum, this should help us ensure that, a message
> assigned to an existing partition will not be mapped to another existing
> partition after partition expansion.
>
> - Producer includes in the ProduceRequest some information that helps
> ensure that messages produced ti a partition will monotonically increase in
> the partitionNum of the topic. In other words, if broker receives a
> ProduceRequest and notices that the producer does not know the partition
> number has increased, broker should reject this request. That "information"
> maybe leaderEpoch, max partitionEpoch of the partitions of the topic, or
> simply partitionNum of the topic. The benefit of this property is that we
> can keep the new logic for in-order message consumption entirely in how
> consumer leader determines the partition -> consumer mapping.
>
> - When consumer leader determines partition -> consumer mapping, leader
> first reads the start position for each partition using OffsetFetchRequest.
> If start position are all non-zero, then assignment can be done in its
> current manner. The assumption is that, a message in the new partition
> should only be consumed after all messages with the same key produced
> before it has been consumed. Since some messages in the new partition has
> been consumed, we should not worry about consuming messages out-of-order.
> This benefit of this approach is that we can avoid unnecessary overhead in
> the common case.
>
> - If the consumer leader finds that the start position for some partition
> is 0. Say the current partition number is 18 and the partition index is 12,
> then consumer leader should ensure that messages produced to partition 12 -
> 18/2 = 3 before the first message of partition 12 is consumed, before it
> assigned partition 12 to any consumer in the consumer group. Since we have
> a "information" that is monotonically increasing per partition, consumer
> can read the value of this information from the first message in partition
> 12, get the offset corresponding to this value in partition 3, assign
> partition except for partition 12 (and probably other new partitions) to
> the existing consumers, waiting for the committed offset to go beyond this
> offset for partition 3, and trigger rebalance again so that partition 3 can
> be reassigned to some consumer.
>
>
> Thanks,
> Dong
>
>
> On Tue, Feb 6, 2018 at 10:10 AM, Jun Rao  wrote:
>
> > Hi, Dong,
> >
> > Thanks for the KIP. It looks good overall. We are working on a separate
> KIP
> > for adding partitions while preserving the ordering guarantees. That may
> > require another flavor of partition epoch. It's not very clear whether
> that
> > partition epoch can be merged with the partition epoch in this KIP. So,
> > perhaps you can wait on this a bit until we post the other KIP in the
> next
> 

Jenkins build is back to normal : kafka-1.1-jdk7 #21

[Apache Jenkins Server]

See 

Build failed in Jenkins: kafka-trunk-jdk9 #378

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-6519; Reduce log level for normal replica fetch errors (#4501)

--
[...truncated 1.47 MB...]
kafka.tools.ConsoleConsumerTest > 
shouldParseValidOldConsumerConfigWithAutoOffsetResetSmallest PASSED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewConsumerConfigWithNoOffsetReset STARTED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewConsumerConfigWithNoOffsetReset PASSED

kafka.tools.ConsoleConsumerTest > shouldLimitReadsToMaxMessageLimit STARTED

kafka.tools.ConsoleConsumerTest > shouldLimitReadsToMaxMessageLimit PASSED

kafka.tools.ConsoleConsumerTest > shouldParseValidNewConsumerValidConfig STARTED

kafka.tools.ConsoleConsumerTest > shouldParseValidNewConsumerValidConfig PASSED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewConsumerConfigWithAutoOffsetResetAndMatchingFromBeginning 
STARTED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewConsumerConfigWithAutoOffsetResetAndMatchingFromBeginning 
PASSED

kafka.tools.ConsoleConsumerTest > shouldStopWhenOutputCheckErrorFails STARTED

kafka.tools.ConsoleConsumerTest > shouldStopWhenOutputCheckErrorFails PASSED

kafka.tools.ConsoleConsumerTest > 
shouldExitOnInvalidConfigWithAutoOffsetResetAndConflictingFromBeginningNewConsumer
 STARTED

kafka.tools.ConsoleConsumerTest > 
shouldExitOnInvalidConfigWithAutoOffsetResetAndConflictingFromBeginningNewConsumer
 PASSED

kafka.tools.ConsoleConsumerTest > 
shouldSetAutoResetToSmallestWhenFromBeginningConfigured STARTED

kafka.tools.ConsoleConsumerTest > 
shouldSetAutoResetToSmallestWhenFromBeginningConfigured PASSED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewConsumerConfigWithAutoOffsetResetEarliest STARTED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewConsumerConfigWithAutoOffsetResetEarliest PASSED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewSimpleConsumerValidConfigWithStringOffset STARTED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewSimpleConsumerValidConfigWithStringOffset PASSED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidOldConsumerConfigWithAutoOffsetResetLargest STARTED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidOldConsumerConfigWithAutoOffsetResetLargest PASSED

kafka.tools.ConsoleConsumerTest > shouldParseConfigsFromFile STARTED

kafka.tools.ConsoleConsumerTest > shouldParseConfigsFromFile PASSED

kafka.tools.ConsoleConsumerTest > groupIdsProvidedInDifferentPlacesMustMatch 
STARTED

kafka.tools.ConsoleConsumerTest > groupIdsProvidedInDifferentPlacesMustMatch 
PASSED

kafka.tools.ConsoleConsumerTest > 
shouldExitOnInvalidConfigWithAutoOffsetResetAndConflictingFromBeginningOldConsumer
 STARTED

kafka.tools.ConsoleConsumerTest > 
shouldExitOnInvalidConfigWithAutoOffsetResetAndConflictingFromBeginningOldConsumer
 PASSED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewSimpleConsumerValidConfigWithNumericOffset STARTED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewSimpleConsumerValidConfigWithNumericOffset PASSED

kafka.tools.ConsoleConsumerTest > testDefaultConsumer STARTED

kafka.tools.ConsoleConsumerTest > testDefaultConsumer PASSED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewConsumerConfigWithAutoOffsetResetLatest STARTED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewConsumerConfigWithAutoOffsetResetLatest PASSED

kafka.tools.ConsoleConsumerTest > shouldParseValidOldConsumerValidConfig STARTED

kafka.tools.ConsoleConsumerTest > shouldParseValidOldConsumerValidConfig PASSED

kafka.message.MessageCompressionTest > testCompressSize STARTED

kafka.message.MessageCompressionTest > testCompressSize PASSED

kafka.message.MessageCompressionTest > testSimpleCompressDecompress STARTED

kafka.message.MessageCompressionTest > testSimpleCompressDecompress PASSED

kafka.message.MessageTest > testChecksum STARTED

kafka.message.MessageTest > testChecksum PASSED

kafka.message.MessageTest > testInvalidTimestamp STARTED

kafka.message.MessageTest > testInvalidTimestamp PASSED

kafka.message.MessageTest > testIsHashable STARTED

kafka.message.MessageTest > testIsHashable PASSED

kafka.message.MessageTest > testInvalidTimestampAndMagicValueCombination STARTED

kafka.message.MessageTest > testInvalidTimestampAndMagicValueCombination PASSED

kafka.message.MessageTest > testExceptionMapping STARTED

kafka.message.MessageTest > testExceptionMapping PASSED

kafka.message.MessageTest > testFieldValues STARTED

kafka.message.MessageTest > testFieldValues PASSED

kafka.message.MessageTest > testInvalidMagicByte STARTED

kafka.message.MessageTest > testInvalidMagicByte PASSED

kafka.message.MessageTest > testEquality STARTED

kafka.message.MessageTest > testEquality PASSED

kafka.message.ByteBufferMessageSetTest > testMessageWithProvidedOffsetSeq 
STARTED

kafka.message.ByteBufferMessageSetTest > testMessageWithProvidedOffsetSeq PASSED

kafka.message.ByteBufferMessage

Build failed in Jenkins: kafka-1.1-jdk7 #22

[Apache Jenkins Server]

See 


Changes:

[jason] KAFKA-6519; Reduce log level for normal replica fetch errors (#4501)

--
[...truncated 411.36 KB...]

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldIncrementEpochAndUpdateMetadataOnHandleInitPidWhenExistingCompleteTransaction
 STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldIncrementEpochAndUpdateMetadataOnHandleInitPidWhenExistingCompleteTransaction
 PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldGenerateNewProducerIdIfEpochsExhausted STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldGenerateNewProducerIdIfEpochsExhausted PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithNotCoordinatorOnInitPidWhenNotCoordinator STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithNotCoordinatorOnInitPidWhenNotCoordinator PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithConcurrentTransactionOnAddPartitionsWhenStateIsPrepareAbort 
STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithConcurrentTransactionOnAddPartitionsWhenStateIsPrepareAbort 
PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldInitPidWithEpochZeroForNewTransactionalId STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldInitPidWithEpochZeroForNewTransactionalId PASSED

kafka.coordinator.transaction.TransactionStateManagerTest > 
testAppendTransactionToLogWhileProducerFenced STARTED

kafka.coordinator.transaction.TransactionStateManagerTest > 
testAppendTransactionToLogWhileProducerFenced PASSED

kafka.coordinator.transaction.TransactionStateManagerTest > 
testCompleteTransitionWhenAppendSucceeded STARTED

kafka.coordinator.transaction.TransactionStateManagerTest > 
testCompleteTransitionWhenAppendSucceeded PASSED

kafka.coordinator.transaction.TransactionStateManagerTest > 
testAppendFailToCoordinatorNotAvailableError STARTED

kafka.coordinator.transaction.TransactionStateManagerTest > 
testAppendFailToCoordinatorNotAvailableError PASSED

kafka.coordinator.transaction.TransactionStateManagerTest > 
testAppendFailToUnknownError STARTED

kafka.coordinator.transaction.TransactionStateManagerTest > 
testAppendFailToUnknownError PASSED

kafka.coordinator.transaction.TransactionStateManagerTest > 
shouldReturnNotCooridnatorErrorIfTransactionIdPartitionNotOwned STARTED

kafka.coordinator.transaction.TransactionStateManagerTest > 
shouldReturnNotCooridnatorErrorIfTransactionIdPartitionNotOwned PASSED

kafka.coordinator.transaction.TransactionStateManagerTest > 
testValidateTransactionTimeout STARTED

kafka.coordinator.transaction.TransactionStateManagerTest > 
testValidateTransactionTimeout PASSED

kafka.coordinator.transaction.TransactionStateManagerTest > 
shouldWriteTxnMarkersForTransactionInPreparedCommitState STARTED

kafka.coordinator.transaction.TransactionStateManagerTest > 
shouldWriteTxnMarkersForTransactionInPreparedCommitState PASSED

kafka.coordinator.transaction.TransactionStateManagerTest > 
shouldOnlyConsiderTransactionsInTheOngoingStateToAbort STARTED

kafka.coordinator.transaction.TransactionStateManagerTest > 
shouldOnlyConsiderTransactionsInTheOngoingStateToAbort PASSED

kafka.coordinator.transaction.TransactionStateManagerTest > 
shouldRemoveCompleteAbortExpiredTransactionalIds STARTED

kafka.coordinator.transaction.TransactionStateManagerTest > 
shouldRemoveCompleteAbortExpiredTransactionalIds PASSED

kafka.coordinator.transaction.TransactionStateManagerTest > 
testAppendTransactionToLogWhilePendingStateChanged STARTED

kafka.coordinator.transaction.TransactionStateManagerTest > 
testAppendTransactionToLogWhilePendingStateChanged PASSED

kafka.coordinator.transaction.TransactionStateManagerTest > 
testLoadAndRemoveTransactionsForPartition STARTED

kafka.coordinator.transaction.TransactionStateManagerTest > 
testLoadAndRemoveTransactionsForPartition PASSED

kafka.coordinator.transaction.TransactionStateManagerTest > 
testAppendFailToNotCoordinatorError STARTED

kafka.coordinator.transaction.TransactionStateManagerTest > 
testAppendFailToNotCoordinatorError PASSED

kafka.coordinator.transaction.TransactionStateManagerTest > 
shouldNotRemovePrepareCommitTransactionalIds STARTED

kafka.coordinator.transaction.TransactionStateManagerTest > 
shouldNotRemovePrepareCommitTransactionalIds PASSED

kafka.coordinator.transaction.TransactionStateManagerTest > 
testAppendFailToCoordinatorLoadingError STARTED

kafka.coordinator.transaction.TransactionStateManagerTest > 
testAppendFailToCoordinatorLoadingError PASSED

kafka.coordinator.transaction.TransactionStateManagerTest > 
shouldNotRemoveOngoingTransactionalIds STARTED

kafka.coordinator.transaction.TransactionStateManagerTest > 
shouldNotRemoveOngoingTransactionalIds PASSED

kafka.coord

Build failed in Jenkins: kafka-trunk-jdk8 #2398

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-6519; Reduce log level for normal replica fetch errors (#4501)

[wangguoz] MINOR: adding system tests for how streams functions with broker 
faiures

[wangguoz] MINOR: Update TupleForwarder comment (#4414)

[wangguoz] KAFKA-6405: Fix incorrect comment in MetadataUpdater (#4361)

--
[...truncated 414.17 KB...]

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldPersistEpochsBetweenInstances PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotClearAnythingIfOffsetToFirstOffset STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotClearAnythingIfOffsetToFirstOffset PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotLetOffsetsGoBackwardsEvenIfEpochsProgress STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotLetOffsetsGoBackwardsEvenIfEpochsProgress PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldGetFirstOffsetOfSubsequentEpochWhenOffsetRequestedForPreviousEpoch STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldGetFirstOffsetOfSubsequentEpochWhenOffsetRequestedForPreviousEpoch PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldUpdateOffsetBetweenEpochBoundariesOnClearEarliest2 STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldUpdateOffsetBetweenEpochBoundariesOnClearEarliest2 PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearEarliestOnEmptyCache 
STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearEarliestOnEmptyCache 
PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldPreserveResetOffsetOnClearEarliestIfOneExists STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldPreserveResetOffsetOnClearEarliestIfOneExists PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldUpdateOffsetBetweenEpochBoundariesOnClearEarliest STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldUpdateOffsetBetweenEpochBoundariesOnClearEarliest PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldReturnInvalidOffsetIfEpochIsRequestedWhichIsNotCurrentlyTracked STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldReturnInvalidOffsetIfEpochIsRequestedWhichIsNotCurrentlyTracked PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldFetchEndOffsetOfEmptyCache 
STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldFetchEndOffsetOfEmptyCache 
PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldRetainLatestEpochOnClearAllEarliestAndUpdateItsOffset STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldRetainLatestEpochOnClearAllEarliestAndUpdateItsOffset PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearAllEntries STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearAllEntries PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearLatestOnEmptyCache 
STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearLatestOnEmptyCache 
PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotResetEpochHistoryHeadIfUndefinedPassed STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotResetEpochHistoryHeadIfUndefinedPassed PASSED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldIncreaseLeaderEpochBetweenLeaderRestarts STARTED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldIncreaseLeaderEpochBetweenLeaderRestarts PASSED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldAddCurrentLeaderEpochToMessagesAsTheyAreWrittenToLeader STARTED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldAddCurrentLeaderEpochToMessagesAsTheyAreWrittenToLeader PASSED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldSendLeaderEpochRequestAndGetAResponse STARTED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldSendLeaderEpochRequestAndGetAResponse PASSED

kafka.server.epoch.OffsetsForLeaderEpochTest > shouldGetEpochsFromReplica 
STARTED

kafka.server.epoch.OffsetsForLeaderEpochTest > shouldGetEpochsFromReplica PASSED

kafka.server.epoch.OffsetsForLeaderEpochTest > 
shouldReturnUnknownTopicOrPartitionIfThrown STARTED

kafka.server.epoch.OffsetsForLeaderEpochTest > 
shouldReturnUnknownTopicOrPartitionIfThrown PASSED

kafka.server.epoch.OffsetsForLeaderEpochTest > 
shouldReturnNoLeaderForPartitionIfThrown STARTED

kafka.server.epoch.OffsetsForLeaderEpochTest > 
shouldReturnNoLeaderForPartitionIfThrown PASSED

kafka.server.epoch.EpochDrivenReplicationProtocolAcceptanceTest > 
shouldSurviveFastLeaderChange STARTED

kafka.server.epoch.EpochDrivenReplicationProtocolAcceptanceTest > 
shouldSurviveFastLeaderChange PASSED

kafka.server.epoch.EpochDrivenReplicationProtocolAcceptanceTest > 
offsetsShouldNotGoBackwards STARTED

kafka.server.epoch.EpochDrivenReplicationProtocolAcceptanceTest > 
offsetsShouldNotGoBackwards PASSED

kafka.server.epoch.EpochDrivenReplicationProtocolAcceptanceTest > 
shouldFollowLeaderEpochBasicWorkflow STARTED

Ability to raise a KIP

[Andy Bryant]

Hi there

I'd like the ability to raise a KIP.

I intend to raise on to suggest adding KTable.selectKey method which
differs in behaviour from KStream.selectKey in that it knows when the
selected key changes for a given record so can react appropriately.

Thanks
Andy Bryant

Build failed in Jenkins: kafka-trunk-jdk9 #379

[Apache Jenkins Server]

See 


Changes:

[wangguoz] MINOR: adding system tests for how streams functions with broker 
faiures

[wangguoz] MINOR: Update TupleForwarder comment (#4414)

[wangguoz] KAFKA-6405: Fix incorrect comment in MetadataUpdater (#4361)

--
[...truncated 1.47 MB...]

kafka.api.SaslClientsWithInvalidCredentialsTest > 
testProducerWithAuthenticationFailure STARTED

kafka.api.SaslClientsWithInvalidCredentialsTest > 
testProducerWithAuthenticationFailure PASSED

kafka.api.SaslClientsWithInvalidCredentialsTest > 
testConsumerGroupServiceWithAuthenticationFailure STARTED

kafka.api.SaslClientsWithInvalidCredentialsTest > 
testConsumerGroupServiceWithAuthenticationFailure PASSED

kafka.api.SaslClientsWithInvalidCredentialsTest > 
testManualAssignmentConsumerWithAutoCommitDisabledWithAuthenticationFailure 
STARTED

kafka.api.SaslClientsWithInvalidCredentialsTest > 
testManualAssignmentConsumerWithAutoCommitDisabledWithAuthenticationFailure 
PASSED

kafka.api.SaslClientsWithInvalidCredentialsTest > 
testKafkaAdminClientWithAuthenticationFailure STARTED

kafka.api.SaslClientsWithInvalidCredentialsTest > 
testKafkaAdminClientWithAuthenticationFailure PASSED

kafka.api.SaslClientsWithInvalidCredentialsTest > 
testConsumerWithAuthenticationFailure STARTED

kafka.api.SaslClientsWithInvalidCredentialsTest > 
testConsumerWithAuthenticationFailure PASSED

kafka.api.DelegationTokenEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaSubscribe STARTED

kafka.api.DelegationTokenEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaSubscribe PASSED

kafka.api.DelegationTokenEndToEndAuthorizationTest > 
testProduceConsumeViaAssign STARTED

kafka.api.DelegationTokenEndToEndAuthorizationTest > 
testProduceConsumeViaAssign PASSED

kafka.api.DelegationTokenEndToEndAuthorizationTest > 
testNoConsumeWithDescribeAclViaAssign STARTED

kafka.api.DelegationTokenEndToEndAuthorizationTest > 
testNoConsumeWithDescribeAclViaAssign PASSED

kafka.api.DelegationTokenEndToEndAuthorizationTest > 
testNoConsumeWithDescribeAclViaSubscribe STARTED

kafka.api.DelegationTokenEndToEndAuthorizationTest > 
testNoConsumeWithDescribeAclViaSubscribe PASSED

kafka.api.DelegationTokenEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaAssign STARTED

kafka.api.DelegationTokenEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaAssign PASSED

kafka.api.DelegationTokenEndToEndAuthorizationTest > testNoGroupAcl STARTED

kafka.api.DelegationTokenEndToEndAuthorizationTest > testNoGroupAcl PASSED

kafka.api.DelegationTokenEndToEndAuthorizationTest > 
testNoProduceWithDescribeAcl STARTED

kafka.api.DelegationTokenEndToEndAuthorizationTest > 
testNoProduceWithDescribeAcl PASSED

kafka.api.DelegationTokenEndToEndAuthorizationTest > 
testProduceConsumeViaSubscribe STARTED

kafka.api.DelegationTokenEndToEndAuthorizationTest > 
testProduceConsumeViaSubscribe PASSED

kafka.api.DelegationTokenEndToEndAuthorizationTest > 
testNoProduceWithoutDescribeAcl STARTED

kafka.api.DelegationTokenEndToEndAuthorizationTest > 
testNoProduceWithoutDescribeAcl PASSED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testTwoConsumersWithDifferentSaslCredentials STARTED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testTwoConsumersWithDifferentSaslCredentials PASSED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaSubscribe STARTED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaSubscribe PASSED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > testProduceConsumeViaAssign 
STARTED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > testProduceConsumeViaAssign 
PASSED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testNoConsumeWithDescribeAclViaAssign STARTED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testNoConsumeWithDescribeAclViaAssign PASSED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testNoConsumeWithDescribeAclViaSubscribe STARTED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testNoConsumeWithDescribeAclViaSubscribe PASSED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaAssign STARTED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaAssign PASSED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > testNoGroupAcl STARTED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > testNoGroupAcl PASSED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > testNoProduceWithDescribeAcl 
STARTED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > testNoProduceWithDescribeAcl 
PASSED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testProduceConsumeViaSubscribe STARTED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testProduceConsumeViaSubscribe PASSED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testNoProduceWithoutDescribeAcl START

Eclipse or Intellij workspace setup

[Ramesh Kolli]

Hi Team,

I am newbie to Kafka dev forum, I would like to be part of kafka
development. I tried to setup Eclipse and Intellij workspace setup by
follow below link(
https://cwiki.apache.org/confluence/display/KAFKA/Developer+Setup). But, I
am not to setup compile successfully, after importing project, i am getting
lot of error.

Can someone please help me setup the workspace?


Regards,
Ramesh

Re: Eclipse or Intellij workspace setup

[Ray Chiang]

Here's what I did recently with OS X/IntelliJ.  I hadn't quite fleshed 
out all the instructions to put up an updated Wiki version yet:


- Use Homebrew to install gradle/scala@2.11 (use --with-docs option)/sbt
- git clone Kafka
- Run gradle/gradlew commands as documented in 
https://cwiki.apache.org/confluence/display/KAFKA/Developer+Setup

- Upgrade to latest IntelliJ
- Import Kafka into IntelliJ
  - Point to local Gradle command
- Set up pointer to Scala SDK (OSX: File->Other Settings->Default 
Project Structure)

  - Change Compiler Classpath jars to Homebrew directory
    - scala-compiler.jar
    - scala-library.jar
    - scala-reflect.jar
  - Change Standard Library jars to Homebrew directory
    - Classes
  - scala-library.jar
  - scala-library--sources.jar (TBD)
  - scala-reflect.jar
  - scala-reflect--sources.jar (TBD)
    - Javadoc
  - scala-library--javadoc.jar
  - scala-reflect--javadoc.jar
- Install IntelliJ Scala plugin

-Ray

On 2/7/18 11:06 PM, Ramesh Kolli wrote:

Hi Team,

I am newbie to Kafka dev forum, I would like to be part of kafka
development. I tried to setup Eclipse and Intellij workspace setup by
follow below link(
https://cwiki.apache.org/confluence/display/KAFKA/Developer+Setup). But, I
am not to setup compile successfully, after importing project, i am getting
lot of error.

Can someone please help me setup the workspace?


Regards,
Ramesh

[jira] [Created] (KAFKA-6541) StackOverflow exceptions in thread 'kafka-coordinator-heartbeat-thread

[Anh Le (JIRA)]

Anh Le created KAFKA-6541:
-

 Summary: StackOverflow exceptions in thread 
'kafka-coordinator-heartbeat-thread
 Key: KAFKA-6541
 URL: https://issues.apache.org/jira/browse/KAFKA-6541
 Project: Kafka
  Issue Type: Bug
  Components: clients
Affects Versions: 1.0.0
 Environment: Linux
Reporter: Anh Le


There's something wrong with our client library when sending heart beats. This 
bug seems to be identical to this one: 
[http://mail-archives.apache.org/mod_mbox/kafka-users/201712.mbox/%3CCALte62w6=pJObC+i36BkoqbOLTKsQ=nrddv6dm8abfwb5ps...@mail.gmail.com%3E]

 

Here's the log:

 

{{2018-02-08 13:55:01,102 ERROR 
org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread 
Uncaught exception in thread 'kafka-coordinator-heartbeat-thread | 
default-group':}}
{{java.lang.StackOverflowError: null}}
{{ at java.lang.StringBuilder.append(StringBuilder.java:136)}}
{{ at 
org.slf4j.helpers.MessageFormatter.safeObjectAppend(MessageFormatter.java:302)}}
{{ at 
org.slf4j.helpers.MessageFormatter.deeplyAppendParameter(MessageFormatter.java:271)}}
{{ at 
org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:233)}}
{{ at 
org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:173)}}
{{ at 
ch.qos.logback.classic.spi.LoggingEvent.getFormattedMessage(LoggingEvent.java:293)}}
{{ at 
ch.qos.logback.classic.spi.LoggingEvent.prepareForDeferredProcessing(LoggingEvent.java:206)}}
{{ at 
ch.qos.logback.core.OutputStreamAppender.subAppend(OutputStreamAppender.java:223)}}
{{ at 
ch.qos.logback.core.OutputStreamAppender.append(OutputStreamAppender.java:102)}}
{{ at 
ch.qos.logback.core.UnsynchronizedAppenderBase.doAppend(UnsynchronizedAppenderBase.java:84)}}
{{ at 
ch.qos.logback.core.spi.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:51)}}
{{ at ch.qos.logback.classic.Logger.appendLoopOnAppenders(Logger.java:270)}}
{{ at ch.qos.logback.classic.Logger.callAppenders(Logger.java:257)}}
{{ at 
ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:421)}}
{{ at ch.qos.logback.classic.Logger.filterAndLog_1(Logger.java:398)}}
{{ at ch.qos.logback.classic.Logger.info(Logger.java:583)}}
{{ at 
org.apache.kafka.common.utils.LogContext$KafkaLogger.info(LogContext.java:341)}}
{{ at 
org.apache.kafka.clients.consumer.internals.AbstractCoordinator.coordinatorDead(AbstractCoordinator.java:649)}}
{{ at 
org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onFailure(AbstractCoordinator.java:797)}}
{{ at 
org.apache.kafka.clients.consumer.internals.RequestFuture$1.onFailure(RequestFuture.java:209)}}
{{ at 
org.apache.kafka.clients.consumer.internals.RequestFuture.fireFailure(RequestFuture.java:177)}}
{{ at 
org.apache.kafka.clients.consumer.internals.RequestFuture.raise(RequestFuture.java:147)}}
{{ at 
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler.fireCompletion(ConsumerNetworkClient.java:496)}}
{{ at 
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.firePendingCompletedRequests(ConsumerNetworkClient.java:353)}}
{{ at 
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.failUnsentRequests(ConsumerNetworkClient.java:416)}}
{{ at 
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.disconnect(ConsumerNetworkClient.java:388)}}
{{ at 
org.apache.kafka.clients.consumer.internals.AbstractCoordinator.coordinatorDead(AbstractCoordinator.java:653)}}
{{ at 
org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onFailure(AbstractCoordinator.java:797)}}
{{ at 
org.apache.kafka.clients.consumer.internals.RequestFuture$1.onFailure(RequestFuture.java:209)}}
{{ at 
org.apache.kafka.clients.consumer.internals.RequestFuture.fireFailure(RequestFuture.java:177)}}
{{ at 
org.apache.kafka.clients.consumer.internals.RequestFuture.raise(RequestFuture.java:147)}}
{{ at 
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler.fireCompletion(ConsumerNetworkClient.java:496)}}
{{ at 
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.firePendingCompletedRequests(ConsumerNetworkClient.java:353)}}
{{ at 
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.failUnsentRequests(ConsumerNetworkClient.java:416)}}
{{ at 
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.disconnect(ConsumerNetworkClient.java:388)}}
{{ at 
org.apache.kafka.clients.consumer.internals.AbstractCoordinator.coordinatorDead(AbstractCoordinator.java:653)}}
{{ at 
org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onFailure(AbstractCoordinator.java:797)}}
{{ at 
org.apache.kafka.clients.consumer.internals.RequestFuture$1.onFailure(RequestFuture.java:209)}}
{{ at 
org.apache.kafka.clients.consumer.internals.Reques

Re: [VOTE] KIP-232: Detect outdated metadata using leaderEpoch and partitionEpoch

[Dong Lin]

Hey Jun,

Sure, I will come up with a KIP this week. I think there is a way to allow
partition expansion to arbitrary number without introducing new concepts
such as read-only partition or repartition epoch.

Thanks,
Dong

On Wed, Feb 7, 2018 at 5:28 PM, Jun Rao  wrote:

> Hi, Dong,
>
> Thanks for the reply. The general idea that you had for adding partitions
> is similar to what we had in mind. It would be useful to make this more
> general, allowing adding an arbitrary number of partitions (instead of just
> doubling) and potentially removing partitions as well. The following is the
> high level idea from the discussion with Colin, Jason and Ismael.
>
> * To change the number of partitions from X to Y in a topic, the controller
> marks all existing X partitions as read-only and creates Y new partitions.
> The new partitions are writable and are tagged with a higher repartition
> epoch (RE).
>
> * The controller propagates the new metadata to every broker. Once the
> leader of a partition is marked as read-only, it rejects the produce
> requests on this partition. The producer will then refresh the metadata and
> start publishing to the new writable partitions.
>
> * The consumers will then be consuming messages in RE order. The consumer
> coordinator will only assign partitions in the same RE to consumers. Only
> after all messages in an RE are consumed, will partitions in a higher RE be
> assigned to consumers.
>
> As Colin mentioned, if we do the above, we could potentially (1) use a
> globally unique partition id, or (2) use a globally unique topic id to
> distinguish recreated partitions due to topic deletion.
>
> So, perhaps we can sketch out the re-partitioning KIP a bit more and see if
> there is any overlap with KIP-232. Would you be interested in doing that?
> If not, we can do that next week.
>
> Jun
>
>
> On Tue, Feb 6, 2018 at 11:30 AM, Dong Lin  wrote:
>
> > Hey Jun,
> >
> > Interestingly I am also planning to sketch a KIP to allow partition
> > expansion for keyed topics after this KIP. Since you are already doing
> > that, I guess I will just share my high level idea here in case it is
> > helpful.
> >
> > The motivation for the KIP is that we currently lose order guarantee for
> > messages with the same key if we expand partitions of keyed topic.
> >
> > The solution can probably be built upon the following ideas:
> >
> > - Partition number of the keyed topic should always be doubled (or
> > multiplied by power of 2). Given that we select a partition based on
> > hash(key) % partitionNum, this should help us ensure that, a message
> > assigned to an existing partition will not be mapped to another existing
> > partition after partition expansion.
> >
> > - Producer includes in the ProduceRequest some information that helps
> > ensure that messages produced ti a partition will monotonically increase
> in
> > the partitionNum of the topic. In other words, if broker receives a
> > ProduceRequest and notices that the producer does not know the partition
> > number has increased, broker should reject this request. That
> "information"
> > maybe leaderEpoch, max partitionEpoch of the partitions of the topic, or
> > simply partitionNum of the topic. The benefit of this property is that we
> > can keep the new logic for in-order message consumption entirely in how
> > consumer leader determines the partition -> consumer mapping.
> >
> > - When consumer leader determines partition -> consumer mapping, leader
> > first reads the start position for each partition using
> OffsetFetchRequest.
> > If start position are all non-zero, then assignment can be done in its
> > current manner. The assumption is that, a message in the new partition
> > should only be consumed after all messages with the same key produced
> > before it has been consumed. Since some messages in the new partition has
> > been consumed, we should not worry about consuming messages out-of-order.
> > This benefit of this approach is that we can avoid unnecessary overhead
> in
> > the common case.
> >
> > - If the consumer leader finds that the start position for some partition
> > is 0. Say the current partition number is 18 and the partition index is
> 12,
> > then consumer leader should ensure that messages produced to partition
> 12 -
> > 18/2 = 3 before the first message of partition 12 is consumed, before it
> > assigned partition 12 to any consumer in the consumer group. Since we
> have
> > a "information" that is monotonically increasing per partition, consumer
> > can read the value of this information from the first message in
> partition
> > 12, get the offset corresponding to this value in partition 3, assign
> > partition except for partition 12 (and probably other new partitions) to
> > the existing consumers, waiting for the committed offset to go beyond
> this
> > offset for partition 3, and trigger rebalance again so that partition 3
> can
> > be reassigned to some consumer.
> >
> >
> > Thanks,
> > Dong
> >
> >
> > O

Re: [DISCUSS] KIP-250 Add Support for Quorum-based Producer Acknowledgment

[Dong Lin]

Hey Colin,

Yeah I think you have a good point that controller is currently a
scalability bottleneck in Kafka and it will be great if we can re-design it
in distributed way using some consensus protocol. I have not gone through
the available options here but I imagine this will require a lot of
discussion and thinking. Before we have a solid proposal for controller,
maybe we should assume it will still be only entity that determines the
global partition assignment etc. Then it is probably OK for us to still
rely on controller for the purpose of this KIP.

Thanks,
Dong

On Tue, Feb 6, 2018 at 10:58 PM, Colin McCabe  wrote:

> Hi Dong & Guozhang,
>
> Thanks... this is a really interesting discussion!
>
> My first thought is, why should the controller be involved in choosing the
> leader of the quorum?  That seems like an internal quorum issue.  All of
> the state that is needed to select the proper leader is on the nodes
> themselves.  None of it is on the controller.
>
> We know why the current system requires the controller to be in the loop--
> it's because we don't have a consensus protocol that allows the nodes to
> decide for themselves, so an external system needs to step in.  But with a
> consensus protocol, the leader should be chosen by the consensus protocol
> itself.  This is the approach taken by other systems such as Apache Kudu
> and Apache ZooKeeper. Kudu uses Raft. ZK uses ZAB.  Probably, we should
> choose the quorum leader by using something like the Raft protocol.  [
> https://raft.github.io/ ]
>
> The only thing that the controller needs to do is to tell clients where to
> find the nodes in the quorum for each partition.  Once the client has found
> the partition, the controller is out of the loop.  The client can talk to
> any node in the quorum and learn from that node what the current leader is,
> by following the consensus protocol.  The client will cache this
> information for efficiency, but it's just soft state.  There's no reason to
> store it in ZooKeeper or persist it across system restarts.  A new leader
> will be elected if the system is restarted anyway.  The only thing that
> needs to be persisted in ZooKeeper is the mapping from partitions to sets
> of brokers.
>
> Involving the controller in leadership changes is a scalability
> bottleneck.  If the quorum needs to ask the controller for permission every
> time it changes its leader, this doesn't scale well as the number of
> partitions increases.  It also decreases our availability-- if the
> controller is unavailable, the leadership change must be delayed.  As far
> as I can tell, this strategy hurts availability without even increasing
> consistency.
>
> best,
> Colin
>
>
> On Sat, Feb 3, 2018, at 10:03, Dong Lin wrote:
> > Hey Guozhang,
> >
> > I don't have very detailed design. But I have some high level idea that
> can
> > probably work. Here is how it looks like:
> >
> > - When controller needs to do leadership movement
> >- Controller sends a request asking the brokers involved in the
> > migration to use acks=[isr_set] and wait for response
> >- Controller updates znodes, sends LeaderAndIsrRequest to migrate
> > leadership and wait for response.
> >- Controller sends a request asking brokers involves in the migration
> to
> > use acks=[min_isr_num]. This improves tail latency when there is no
> > leadership movement.
> >
> > Personally I find this approach to be relatively straightforward. It
> wraps
> > around the existing logic of leadership movement and we can still reply
> on
> > the existing logic to avoid data loss in case of network partition. Does
> > this sound reasonable?
> >
> > Thanks,
> > Dong
> >
> > On Fri, Feb 2, 2018 at 11:26 PM, Guozhang Wang 
> wrote:
> >
> > > Hi Dong,
> > >
> > > Could you elaborate a bit more how controller could affect leaders to
> > > switch between all and quorum?
> > >
> > >
> > > Guozhang
> > >
> > >
> > > On Fri, Feb 2, 2018 at 10:12 PM, Dong Lin  wrote:
> > >
> > >> Hey Guazhang,
> > >>
> > >> Got it. Thanks for the detailed explanation. I guess my point is that
> we
> > >> can probably achieve the best of both worlds, i.e. maintain the
> existing
> > >> behavior of ack="all" while improving the tail latency.
> > >>
> > >> Thanks,
> > >> Dong
> > >>
> > >>
> > >>
> > >> On Fri, Feb 2, 2018 at 8:43 PM, Guozhang Wang 
> wrote:
> > >>
> > >>> Hi Dong,
> > >>>
> > >>> Yes, in terms of fault tolerance "quorum" does not do better than
> "all",
> > >>> as I said, with {min.isr} to X+1 Kafka is able to tolerate X
> failures only.
> > >>> So if A and B are partitioned off at the same time, then there are
> two
> > >>> concurrent failures and we do not guarantee all acked messages will
> be
> > >>> retained.
> > >>>
> > >>> The goal of my approach is to maintain the behavior of ack="all",
> which
> > >>> happen to do better than what Kafka is actually guaranteed: when
> both A and
> > >>> B are partitioned off, produced records will not be acked since "all"
> > >>> r