Re: [VOTE] KIP-266: Add TimeoutException for KafkaConsumer#position

[khaireddine Rezgui]

+1

2018-05-07 20:35 GMT+01:00 Bill Bejeck :

> +1
>
> Thanks,
> Bill
>
> On Fri, May 4, 2018 at 7:21 PM, Richard Yu 
> wrote:
>
> > Hi all, I would like to bump this thread since discussion in the KIP
> > appears to be reaching its conclusion.
> >
> >
> >
> > On Thu, Mar 15, 2018 at 3:30 PM, Richard Yu 
> > wrote:
> >
> > > Hi all,
> > >
> > > Since there does not seem to be too much discussion in KIP-266, I will
> be
> > > starting a voting thread.
> > > Here is the link to KIP-266 for reference:
> > >
> > > https://cwiki.apache.org/confluence/pages/viewpage.
> > action?pageId=75974886
> > >
> > > Recently, I have made some updates to the KIP. To reiterate, I have
> > > included KafkaConsumer's commitSync,
> > > poll, and committed in the KIP. (we will be adding to a
> TimeoutException
> > > to them as well, in a similar manner
> > > to what we will be doing for position())
> > >
> > > Thanks,
> > > Richard Yu
> > >
> > >
> >
>



-- 
Ingénieur en informatique

Re: [DISCUSS] KIP-294 - Enable TLS hostname verification by default

[Rajini Sivaram]

Anyone have any concerns on this change? If there are no other comments
till tomorrow, I will start vote since it is a straightforward KIP.

On Fri, May 4, 2018 at 11:10 PM, Rajini Sivaram 
wrote:

> Hi all,
>
> I have submitted KIP-294 to make the default value of the config `
> ssl.endpoint.identification.algorithm` secure:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 294+-+Enable+TLS+hostname+verification+by+default
>
> Feedback is welcome.
>
> Regards,
>
> Rajini
>
>
>

[jira] [Created] (KAFKA-6883) KafkaShortnamer should allow to convert Kerberos principal name to upper case user name

[Attila Sasvari (JIRA)]

Attila Sasvari created KAFKA-6883:
-

 Summary: KafkaShortnamer should allow to convert Kerberos 
principal name to upper case user name
 Key: KAFKA-6883
 URL: https://issues.apache.org/jira/browse/KAFKA-6883
 Project: Kafka
  Issue Type: Improvement
Reporter: Attila Sasvari


KAFKA-5764 implemented support to convert Kerberos principal name to lower case 
Linux user name via auth_to_local rules. 

As a follow-up, KafkaShortnamer could be further extended to allow converting 
principal names to uppercase by appending /U to the rule.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] KIP-255: OAuth Authentication via SASL/OAUTHBEARER

[Ron Dagostino]

HI everyone.  Can we get 2 more binding votes on this KIP (and non-binding
votes, too)?

Ron

On Fri, May 4, 2018 at 11:53 AM, Rajini Sivaram 
wrote:

> Hi Ron,
>
> +1 (binding)
>
> Thanks for the KIP!
>
> Regards,
>
> Rajini
>
> On Fri, May 4, 2018 at 4:55 AM, Ron Dagostino  wrote:
>
> > Hi everyone.  I would like to start the vote for KIP-255:
> > https://cwiki.apache.org/confluence/pages/viewpage.
> action?pageId=75968876
> >
> > This KIP proposes to add the following functionality related to
> > SASL/OAUTHBEARER:
> >
> > 1) Allow clients (both brokers when SASL/OAUTHBEARER is the inter-broker
> > protocol as well as non-broker clients) to flexibly retrieve an access
> > token from an OAuth 2 authorization server based on the declaration of a
> > custom login CallbackHandler implementation and have that access token
> > transparently and automatically transmitted to a broker for
> authentication.
> >
> > 2) Allow brokers to flexibly validate provided access tokens when a
> client
> > establishes a connection based on the declaration of a custom SASL Server
> > CallbackHandler implementation.
> >
> > 3) Provide implementations of the above retrieval and validation features
> > based on an unsecured JSON Web Token that function out-of-the-box with
> > minimal configuration required (i.e. implementations of the two types of
> > callback handlers mentioned above will be used by default with no need to
> > explicitly declare them).
> >
> > 4) Allow clients (both brokers when SASL/OAUTHBEARER is the inter-broker
> > protocol as well as non-broker clients) to transparently retrieve a new
> > access token in the background before the existing access token expires
> in
> > case the client has to open new connections.
> >
> > Thanks,
> >
> > Ron
> >
>

Re: [DISCUSS] KIP-292: Add transformValues() method to KTable

[Damian Guy]

Initially i thought materializing a store would be overkill, but from a
consistency point of view it makes sense to add an overload that takes a
`Materialized` and only create the store if that is supplied.

On Sun, 6 May 2018 at 17:52 Matthias J. Sax  wrote:

> Andy,
>
> thanks for the KIP. I don't have any further comments.
>
> My 2cents about Guozhang's questions: as I like consistent behavior, I
> think transfromValues() should behave the same way as filter() and
> mapValues().
>
>
> -Matthias
>
> On 5/2/18 2:24 PM, Guozhang Wang wrote:
> > Hello Andy,
> >
> > Thanks for the KIP. The motivation and the general proposal looks good to
> > me. I think in KTable it is indeed valuable to add the functions that
> does
> > not change key, such as mapValues, transformValues, and filter.
> >
> > There are a few meta comments I have about the semantics of the newly
> added
> > functions:
> >
> > 1) For the resulted KTable, how should its "queryableStoreName()" be
> > returning?
> >
> > 2) More specifically, how do we decide if the resulted KTable is to be
> > materialized or not? E.g. if there is no store names provided then it is
> > likely that the resulted KTable is not materialized, or at least not
> > logically materialized and not be queryable. What if there is at least
> one
> > state store provided? Will any of them be provided as the materialized
> > store, or should we still add a Materialized parameter for this purpose?
> >
> > 3) For its internal implementations, how should the key/value serde,
> > sendOldValues flag etc be inherited from its parent processor node?
> >
> >
> > Guozhang
> >
> >
> > On Wed, May 2, 2018 at 12:43 PM, Andy Coates  wrote:
> >
> >> Hi everyone,
> >>
> >> I would like to start a discussion for KIP 292. I would appreciate it if
> >> you could review and provide feedback.
> >>
> >> KIP: KIP-292: Add transformValues() method to KTable
> >>  >> 292%3A+Add+transformValues%28%29+method+to+KTable>
> >> Jira: KAFKA-6849 
> >>
> >>PR: #4959 
> >>
> >>
> >>
> >> Thanks,
> >>
> >> Andy
> >>
> >
> >
> >
>
>

Re: [VOTE] KIP-282: Add the listener name to the authentication context

[Mickael Maison]

Bumping as it still needs a few votes =)

On Wed, Apr 25, 2018 at 8:24 PM, Mickael Maison
 wrote:
> I've updated the KIP to simply expose the String value instead of the
> full ListenerName object.
>
> On Wed, Apr 25, 2018 at 8:10 PM, Mickael Maison
>  wrote:
>> Thanks for the heads up Ismael. In that case, I also think it makes
>> sense to just use the String representation.
>>
>> I'll update the KIP to reflect that change
>>
>> On Wed, Apr 25, 2018 at 7:25 PM, Rajini Sivaram  
>> wrote:
>>> Sorry, I hadn't noticed that it was using `ListenerName`. I think we should
>>> return just a String.
>>>
>>> On Wed, Apr 25, 2018 at 7:02 PM, Ismael Juma  wrote:
>>>
 ListenerName is in an internal package, so we would have to move it before
 exposing it in a public API.

 Ismael

 On Wed, 25 Apr 2018, 01:52 Mickael Maison, 
 wrote:

 > Hi,
 >
 > There has been no objections in the DISCUSS thread so I'd like to
 > start a vote on KIP-282:
 > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
 > 282%3A+Add+the+listener+name+to+the+authentication+context
 >
 > Thanks
 >

Re: [VOTE] KIP-277 - Fine Grained ACL for CreateTopics API

[Edoardo Comar]

Hi,
bumping the thread as the current vote count for this KIP is
2 binding +1
5 non-binding +1

so still missing a binding vote please

thanks,
Edo


On 30 April 2018 at 12:49, Manikumar  wrote:

> +1 (non-binding)
>
> Thanks
>
> On Thu, Apr 26, 2018 at 9:59 PM, Colin McCabe  wrote:
>
> > +1 (non-binding)
> >
> > best,
> > Colin
> >
> >
> > On Wed, Apr 25, 2018, at 02:45, Edoardo Comar wrote:
> > > Hi,
> > >
> > > The discuss thread on KIP-277 (
> > > https://www.mail-archive.com/dev@kafka.apache.org/msg86540.html )
> > > seems to have been fruitful and concerns have been addressed, please
> > allow
> > > me start a vote on it:
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 277+-+Fine+Grained+ACL+for+CreateTopics+API
> > >
> > > I will update the small PR to the latest KIP semantics if the vote
> > passes
> > > (as I hope :-).
> > >
> > > cheers
> > > Edo
> > > --
> > >
> > > Edoardo Comar
> > >
> > > IBM Message Hub
> > >
> > > IBM UK Ltd, Hursley Park, SO21 2JN
> > > Unless stated otherwise above:
> > > IBM United Kingdom Limited - Registered in England and Wales with
> number
> > > 741598.
> > > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6
> > 3AU
> >
>



-- 
"When the people fear their government, there is tyranny; when the
government fears the people, there is liberty." [Thomas Jefferson]

Kafka system tests contribution

[Andriy Sorokhtey]

Hello Kafka team

I’d like to contribute to the Kafka system tests.

I’ve tried to execute system tests locally and I have some issues. Can
anyone give me a hand to figure out what’s wrong?

So, I see that multiple system tests are failing when I try to run it with
the docker or with vagrant.
Maybe there is some way to debug it using PyCharm. For example, put some
breakpoint and start debugging, when the test goes to the breakpoint I’d
like to go to instances and check what’s going on there.
I’ll be thankful for any advice.

 Here is an example of one test failure:

> [INFO:2018-05-03 06:37:19,861]: Triggering test 1 of 37...
> [INFO:2018-05-03 06:37:19,870]: RunnerClient: Loading test {'directory':
> '/opt/kafka-dev/tests/kafkatest/tests/streams', 'file_name':
> 'streams_broker_compatibility_test.py', 'method_name':
> 'test_compatible_brokers_eos_disabled', 'cls_name':
> 'StreamsBrokerCompatibility', 'injected_args': {'broker_version':
> '0.10.1.1'}}
> [INFO:2018-05-03 06:37:19,874]: RunnerClient:
> kafkatest.tests.streams.streams_broker_compatibility_test.StreamsBrokerCompatibility.test_compatible_brokers_eos_disabled.broker_version=
> 0.10.1.1: Setting up...
> [INFO:2018-05-03 06:37:22,484]: RunnerClient:
> kafkatest.tests.streams.streams_broker_compatibility_test.StreamsBrokerCompatibility.test_compatible_brokers_eos_disabled.broker_version=
> 0.10.1.1: Running...
> [INFO:2018-05-03 06:38:34,129]: RunnerClient:
> kafkatest.tests.streams.streams_broker_compatibility_test.StreamsBrokerCompatibility.test_compatible_brokers_eos_disabled.broker_version=
> 0.10.1.1: FAIL: Never saw message indicating StreamsTest finished startup
> on ducker@ducker05
> Traceback (most recent call last):
> File
> "/usr/local/lib/python2.7/dist-packages/ducktape/tests/runner_client.py",
> line 132, in run
> data = self.run_test()
> File
> "/usr/local/lib/python2.7/dist-packages/ducktape/tests/runner_client.py",
> line 185, in run_test
> return self.test_context.function(self.test)
> File "/usr/local/lib/python2.7/dist-packages/ducktape/mark/_mark.py", line
> 324, in wrapper
> return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
> File
> "/opt/kafka-dev/tests/kafkatest/tests/streams/streams_broker_compatibility_test.py",
> line 84, in test_compatible_brokers_eos_disabled
> processor.start()
> File
> "/usr/local/lib/python2.7/dist-packages/ducktape/services/service.py", line
> 234, in start
> self.start_node(node)
> File "/opt/kafka-dev/tests/kafkatest/services/streams.py", line 138, in
> start_node
> monitor.wait_until('StreamsTest instance started', timeout_sec=60,
> err_msg="Never saw message indicating StreamsTest finished startup on " +
> str(node.account))
> File
> "/usr/local/lib/python2.7/dist-packages/ducktape/cluster/remoteaccount.py",
> line 668, in wait_until
> allow_fail=True) == 0, **kwargs)
> File "/usr/local/lib/python2.7/dist-packages/ducktape/utils/util.py", line
> 36, in wait_until
> raise TimeoutError(err_msg)
> TimeoutError: Never saw message indicating StreamsTest finished startup on
> ducker@ducker05


If I figure out what's wrong I can try to fix other tests.

-- 

*Sincerely*
*Andriy Sorokhtey*
+380681890146

Re: please subscribe me to dev list

[Matthias J. Sax]

It's self-service: https://kafka.apache.org/contact

-Matthias


On 5/7/18 5:13 PM, Ravi Chinoy wrote:



signature.asc
Description: OpenPGP digital signature

[jira] [Created] (KAFKA-6884) ConsumerGroupCommand should use new AdminClient

[Jason Gustafson (JIRA)]

Jason Gustafson created KAFKA-6884:
--

 Summary: ConsumerGroupCommand should use new AdminClient
 Key: KAFKA-6884
 URL: https://issues.apache.org/jira/browse/KAFKA-6884
 Project: Kafka
  Issue Type: Improvement
Reporter: Jason Gustafson


Now that we have KIP-222, we should update consumer-groups.sh to use the new 
AdminClient.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: kafka-trunk-jdk7 #3407

[Apache Jenkins Server]

See 


Changes:

[wangguoz] MINOR: Make Serdes less confusing in Scala (#4963)

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on H26 (ubuntu xenial) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/trunk^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/trunk^{commit} # timeout=10
Checking out Revision b88d70b53290af715034a1f772a271f7e44505fd 
(refs/remotes/origin/trunk)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f b88d70b53290af715034a1f772a271f7e44505fd
Commit message: "MINOR: Make Serdes less confusing in Scala (#4963)"
 > git rev-list --no-walk 2b5a59406622affa1e333e073546f075f59b4ac9 # timeout=10
Setting GRADLE_3_5_HOME=/home/jenkins/tools/gradle/3.5
[kafka-trunk-jdk7] $ /bin/bash -xe /tmp/jenkins9214342773236464869.sh
+ rm -rf 
+ /home/jenkins/tools/gradle/3.5/bin/gradle
To honour the JVM settings for this build a new JVM will be forked. Please 
consider using the daemon: 
https://docs.gradle.org/3.5/userguide/gradle_daemon.html.
Daemon will be stopped at the end of the build stopping after processing

FAILURE: Build failed with an exception.

* Where:
Build file ' 
line: 25

* What went wrong:
A problem occurred evaluating root project 'kafka-trunk-jdk7'.
> unable to create new native thread

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output.

BUILD FAILED

Total time: 6.015 secs
Build step 'Execute shell' marked build as failure
[FINDBUGS] Collecting findbugs analysis files...
Setting GRADLE_3_5_HOME=/home/jenkins/tools/gradle/3.5
[FINDBUGS] Searching for all files in 
 that match the pattern 
**/build/reports/findbugs/*.xml
[FINDBUGS] Parsing 1 file in 

[FINDBUGS] Successfully parsed file 

 with 0 unique warnings and 0 duplicates.
Setting GRADLE_3_5_HOME=/home/jenkins/tools/gradle/3.5
Setting GRADLE_3_5_HOME=/home/jenkins/tools/gradle/3.5
 Using GitBlamer to create author and commit information for all 
warnings.
 GIT_COMMIT=b88d70b53290af715034a1f772a271f7e44505fd, 
workspace=
[FINDBUGS] Computing warning deltas based on reference build #3406
Recording test results
Setting GRADLE_3_5_HOME=/home/jenkins/tools/gradle/3.5
ERROR: Step ?Publish JUnit test result report? failed: Test reports were found 
but none of them are new. Did leafNodes run? 
For example, 

 is 3 days 21 hr old

Setting GRADLE_3_5_HOME=/home/jenkins/tools/gradle/3.5
Not sending mail to unregistered user wangg...@gmail.com

Build failed in Jenkins: kafka-trunk-jdk10 #84

[Apache Jenkins Server]

See 


Changes:

[wangguoz] MINOR: Make Serdes less confusing in Scala (#4963)

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on H26 (ubuntu xenial) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/trunk^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/trunk^{commit} # timeout=10
Checking out Revision b88d70b53290af715034a1f772a271f7e44505fd 
(refs/remotes/origin/trunk)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f b88d70b53290af715034a1f772a271f7e44505fd
Commit message: "MINOR: Make Serdes less confusing in Scala (#4963)"
 > git rev-list --no-walk 2b5a59406622affa1e333e073546f075f59b4ac9 # timeout=10
Setting GRADLE_4_4_HOME=/home/jenkins/tools/gradle/4.4
[kafka-trunk-jdk10] $ /bin/bash -xe /tmp/jenkins248596064039456076.sh
+ rm -rf 
+ /home/jenkins/tools/gradle/4.4/bin/gradle
[0.119s][warning][os,thread] Failed to start thread - pthread_create failed 
(EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached.
[0.120s][warning][os,thread] Failed to start thread - pthread_create failed 
(EAGAIN) for attributes: stacksize: 1024k, guardsize: 4k, detached.
Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.NoClassDefFoundError: Could not initialize 
class sun.launcher.LauncherHelper$ResourceBundleHolder
at 
java.base/sun.launcher.LauncherHelper.getLocalizedMessage(LauncherHelper.java:351)
at java.base/sun.launcher.LauncherHelper.abort(LauncherHelper.java:514)
at 
java.base/sun.launcher.LauncherHelper.loadMainClass(LauncherHelper.java:671)
at 
java.base/sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:547)
Build step 'Execute shell' marked build as failure
Recording test results
Setting GRADLE_4_4_HOME=/home/jenkins/tools/gradle/4.4
ERROR: Step ?Publish JUnit test result report? failed: Test reports were found 
but none of them are new. Did leafNodes run? 
For example, 

 is 12 days old

Setting GRADLE_4_4_HOME=/home/jenkins/tools/gradle/4.4
Not sending mail to unregistered user wangg...@gmail.com
Not sending mail to unregistered user git...@alasdairhodge.co.uk

[jira] [Created] (KAFKA-6885) DescribeConfigs synonyms are are identical to parent entry for BROKER resources

[Magnus Edenhill (JIRA)]

Magnus Edenhill created KAFKA-6885:
--

 Summary: DescribeConfigs synonyms are are identical to parent 
entry for BROKER resources
 Key: KAFKA-6885
 URL: https://issues.apache.org/jira/browse/KAFKA-6885
 Project: Kafka
  Issue Type: Bug
  Components: admin
Affects Versions: 1.1.0
Reporter: Magnus Edenhill


The DescribeConfigs protocol response for BROKER resources returns synonyms for 
various configuration entries, such as "log.dir".

The list of synonyms returned are identical to their parent configuration 
entry, rather than the actual synonyms.

For example, for the broker "log.dir" config entry it returns one synonym, also 
named "log.dir" rather than "log.dirs" or whatever the synonym is supposed to 
be.

 

This does not seem to happen for TOPIC resources.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-6886) Externalize Secrets for Kafka Connect Configurations

[Robert Yokota (JIRA)]

Robert Yokota created KAFKA-6886:


 Summary: Externalize Secrets for Kafka Connect Configurations
 Key: KAFKA-6886
 URL: https://issues.apache.org/jira/browse/KAFKA-6886
 Project: Kafka
  Issue Type: New Feature
  Components: KafkaConnect
Reporter: Robert Yokota
Assignee: Robert Yokota
 Fix For: 2.0.0


Kafka Connect's connector configurations have plaintext passwords, and Connect 
stores these in cleartext either on the filesystem (for standalone mode) or in 
internal topics (for distributed mode). 

Connect should not store or transmit cleartext passwords in connector 
configurations. Secrets in stored connector configurations should be allowed to 
be replaced with references to values stored in external secret management 
systems. Connect should provide an extension point for adding customized 
integrations, as well as provide a file-based extension as an example. Second, 
a Connect runtime should be allowed to be configured to use one or more of 
these extensions, and allow connector configurations to use placeholders that 
will be resolved by the runtime before passing the complete connector 
configurations to connectors. This will allow existing connectors to not see 
any difference in the configurations that Connect provides to them at startup. 
And third, Connect's API should be changed to allow a connector to obtain the 
latest connector configuration at any time.





--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[DISCUSS] KIP-297: Externalizing Secrets for Connect Configurations

[Robert Yokota]

Hi,

I would like to start a discussion for KIP-297 to externalize secrets from
Kafka Connect configurations.  Any feedback is appreciated.
<
https://cwiki.apache.org/confluence/display/KAFKA/KIP-297%3A+Externalizing+Secrets+for+Connect+Configurations
>

JIRA: 

Thanks in advance,
Robert

Jenkins build is back to normal : kafka-trunk-jdk8 #2626

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-295: Add Streams Configuration Allowing for Optional Topology Optimization

[Bill Bejeck]

Thanks for the comments Guozhang and Ted.

Guozhang:

  1) I'll update the KIP in the "Compatibility, Deprecation and Migration
Plan" with the expected impact of turning on optimization. But at this
point, I have not identified a migration plan that doesn't involve having
to stop all instances and restart.

  2) Setting the type to String was just so we could have the default of
null, indicating run no optimizations. As for partially enabling
optimizations, I'm not sure I had that in mind, at least at this point.
 To me having the topology optimized should be an "all or nothing"
proposition.  For now, I'll change the type to boolean (with a default
value of false) to better reflect the intent of the configuration.

Ted, thanks again for the comments.

The intent of the new configuration, as I mentioned above, is whether to
turn optimization on or off in aggregate.  The main reason for having the
configuration is for backward compatibility.  As optimization may result in
a slightly different topology from the original one, we need to allow users
to turn it off until they are ready for migrating to the new topology.

I don't think having to select each optimization is a feasible solution.  I
say this for few reasons:

   1. Maintenance will be an issue.  While the initial target is only a
   small number of optimizations, but as the number grows, having to maintain
   the number of options in the code will difficult at best.
   2. Users probably don't want to reason about which combination of
   optimizations to have.  Thinking about which optimizations to turn on or
   off raises the complexity of deploying an application.
   3. When using a relational database or other another framework that may
   optimize queries, as far as I know, it's not a choice of which
   optimizations to have, they are performed automatically.

Does this make sense?


On Mon, May 7, 2018 at 7:49 PM, Ted Yu  wrote:

> There are 4 subtasks for KAFKA-6034
>
> If each optimization can be switched on/off, there should be 4 enums for
> the switch.
>
> On Mon, May 7, 2018 at 4:39 PM, Guozhang Wang  wrote:
>
> > Hello Bill,
> >
> > Thanks for the KIP. My comments are the following:
> >
> > 1) We should state clearly what are the expected impact in
> "Compatibility,
> > Deprecation, and Migration Plan" when optimization is turned on.
> >
> > 2) I'm wondering why we define this config as a String typed, rather than
> > boolean typed? Or are you expecting to extend it to allow partially
> > enabling part of the optimizations in the future?
> >
> >
> > Hi Ted,
> >
> > The cover JIRA for topology optimization can be found here:
> > https://issues.apache.org/jira/browse/KAFKA-6034
> >
> >
> > Guozhang
> >
> >
> > On Mon, May 7, 2018 at 11:04 AM, Ted Yu  wrote:
> >
> > > Which JIRA is for the Topology Optimization itself ?
> > >
> > > Thanks
> > >
> > > On Mon, May 7, 2018 at 10:26 AM, Bill Bejeck 
> wrote:
> > >
> > > > All,
> > > > I'd like to start a discussion about adding a configuration parameter
> > > > allowing for the forthcoming topology optimization to be optional via
> > > > configuration.
> > > >
> > > >
> > > > The KIP can be found here:
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 295%3A+Add+Streams+
> > > > Configuration+Allowing+for+Optional+Topology+Optimization
> > > >
> > > > Thanks,
> > > > Bill
> > > >
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>

CFP: Kafka Summit San Francisco, 2018

[Jun Rao]

Hi, Everyone,

Just want to let you know that Kafka Summit San Francisco, 2018 is open for
submissions. The deadline for submission is Jun. 16. The conference itself
will be held on Oct. 16 - Oct. 17.

More details can be found at
https://kafka-summit.org/kafka-summit-san-francisco-2018/speakers/

Thanks,

Jun

Re: [DISCUSS] KIP-295: Add Streams Configuration Allowing for Optional Topology Optimization

[Ted Yu]

Bill:That makes sense.
Using boolean should suffice.
 Original message From: Bill Bejeck  Date: 
5/8/18  2:48 PM  (GMT-08:00) To: dev@kafka.apache.org Subject: Re: [DISCUSS] 
KIP-295: Add Streams Configuration Allowing for Optional Topology Optimization 
Thanks for the comments Guozhang and Ted.

Guozhang:

  1) I'll update the KIP in the "Compatibility, Deprecation and Migration
Plan" with the expected impact of turning on optimization. But at this
point, I have not identified a migration plan that doesn't involve having
to stop all instances and restart.

  2) Setting the type to String was just so we could have the default of
null, indicating run no optimizations. As for partially enabling
optimizations, I'm not sure I had that in mind, at least at this point.
 To me having the topology optimized should be an "all or nothing"
proposition.  For now, I'll change the type to boolean (with a default
value of false) to better reflect the intent of the configuration.

Ted, thanks again for the comments.

The intent of the new configuration, as I mentioned above, is whether to
turn optimization on or off in aggregate.  The main reason for having the
configuration is for backward compatibility.  As optimization may result in
a slightly different topology from the original one, we need to allow users
to turn it off until they are ready for migrating to the new topology.

I don't think having to select each optimization is a feasible solution.  I
say this for few reasons:

   1. Maintenance will be an issue.  While the initial target is only a
   small number of optimizations, but as the number grows, having to maintain
   the number of options in the code will difficult at best.
   2. Users probably don't want to reason about which combination of
   optimizations to have.  Thinking about which optimizations to turn on or
   off raises the complexity of deploying an application.
   3. When using a relational database or other another framework that may
   optimize queries, as far as I know, it's not a choice of which
   optimizations to have, they are performed automatically.

Does this make sense?


On Mon, May 7, 2018 at 7:49 PM, Ted Yu  wrote:

> There are 4 subtasks for KAFKA-6034
>
> If each optimization can be switched on/off, there should be 4 enums for
> the switch.
>
> On Mon, May 7, 2018 at 4:39 PM, Guozhang Wang  wrote:
>
> > Hello Bill,
> >
> > Thanks for the KIP. My comments are the following:
> >
> > 1) We should state clearly what are the expected impact in
> "Compatibility,
> > Deprecation, and Migration Plan" when optimization is turned on.
> >
> > 2) I'm wondering why we define this config as a String typed, rather than
> > boolean typed? Or are you expecting to extend it to allow partially
> > enabling part of the optimizations in the future?
> >
> >
> > Hi Ted,
> >
> > The cover JIRA for topology optimization can be found here:
> > https://issues.apache.org/jira/browse/KAFKA-6034
> >
> >
> > Guozhang
> >
> >
> > On Mon, May 7, 2018 at 11:04 AM, Ted Yu  wrote:
> >
> > > Which JIRA is for the Topology Optimization itself ?
> > >
> > > Thanks
> > >
> > > On Mon, May 7, 2018 at 10:26 AM, Bill Bejeck 
> wrote:
> > >
> > > > All,
> > > > I'd like to start a discussion about adding a configuration parameter
> > > > allowing for the forthcoming topology optimization to be optional via
> > > > configuration.
> > > >
> > > >
> > > > The KIP can be found here:
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 295%3A+Add+Streams+
> > > > Configuration+Allowing+for+Optional+Topology+Optimization
> > > >
> > > > Thanks,
> > > > Bill
> > > >
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>

Re: [DISCUSS] KIP-295: Add Streams Configuration Allowing for Optional Topology Optimization

[Matthias J. Sax]

One thought I want to bring up about switching optimization on/off:

While for the initial release, a boolean flag seems to be sufficient, I
could imagine that we apply different and potentially
upgrade-incompatible optimizations in future releases. Thus, to me it
would make sense to use a String type, to indicate what optimizations
are possible based on the release. For example, in next release we
accept `null` for disabled and "2.0". If there are any incompatible
changes, people can stay with "2.0" optimizations level when upgrading
to "2.1" while new applications can use "2.1" optimization level. Old
applications would need to do an offline upgrade to get "2.1" optimizations.

I agree with Bill, that switching individual optimizations on/off is too
fine grained and hard to maintain. However, for compatibility, it might
make sense, to have certain "levels of optimizations" (based on the
release) that allow users to stay with on an older level for upgrade
purpose. Using the release numbers to encode those "levels" is easy to
understand for users and should minimize the mental effort to get the
config right.

Let me know what you think about this.


-Matthias

On 5/8/18 3:08 PM, Ted Yu wrote:
> Bill:That makes sense.
> Using boolean should suffice.
>  Original message From: Bill Bejeck  Date: 
> 5/8/18  2:48 PM  (GMT-08:00) To: dev@kafka.apache.org Subject: Re: [DISCUSS] 
> KIP-295: Add Streams Configuration Allowing for Optional Topology 
> Optimization 
> Thanks for the comments Guozhang and Ted.
> 
> Guozhang:
> 
>   1) I'll update the KIP in the "Compatibility, Deprecation and Migration
> Plan" with the expected impact of turning on optimization. But at this
> point, I have not identified a migration plan that doesn't involve having
> to stop all instances and restart.
> 
>   2) Setting the type to String was just so we could have the default of
> null, indicating run no optimizations. As for partially enabling
> optimizations, I'm not sure I had that in mind, at least at this point.
>  To me having the topology optimized should be an "all or nothing"
> proposition.  For now, I'll change the type to boolean (with a default
> value of false) to better reflect the intent of the configuration.
> 
> Ted, thanks again for the comments.
> 
> The intent of the new configuration, as I mentioned above, is whether to
> turn optimization on or off in aggregate.  The main reason for having the
> configuration is for backward compatibility.  As optimization may result in
> a slightly different topology from the original one, we need to allow users
> to turn it off until they are ready for migrating to the new topology.
> 
> I don't think having to select each optimization is a feasible solution.  I
> say this for few reasons:
> 
>    1. Maintenance will be an issue.  While the initial target is only a
>    small number of optimizations, but as the number grows, having to maintain
>    the number of options in the code will difficult at best.
>    2. Users probably don't want to reason about which combination of
>    optimizations to have.  Thinking about which optimizations to turn on or
>    off raises the complexity of deploying an application.
>    3. When using a relational database or other another framework that may
>    optimize queries, as far as I know, it's not a choice of which
>    optimizations to have, they are performed automatically.
> 
> Does this make sense?
> 
> 
> On Mon, May 7, 2018 at 7:49 PM, Ted Yu  wrote:
> 
>> There are 4 subtasks for KAFKA-6034
>>
>> If each optimization can be switched on/off, there should be 4 enums for
>> the switch.
>>
>> On Mon, May 7, 2018 at 4:39 PM, Guozhang Wang  wrote:
>>
>>> Hello Bill,
>>>
>>> Thanks for the KIP. My comments are the following:
>>>
>>> 1) We should state clearly what are the expected impact in
>> "Compatibility,
>>> Deprecation, and Migration Plan" when optimization is turned on.
>>>
>>> 2) I'm wondering why we define this config as a String typed, rather than
>>> boolean typed? Or are you expecting to extend it to allow partially
>>> enabling part of the optimizations in the future?
>>>
>>>
>>> Hi Ted,
>>>
>>> The cover JIRA for topology optimization can be found here:
>>> https://issues.apache.org/jira/browse/KAFKA-6034
>>>
>>>
>>> Guozhang
>>>
>>>
>>> On Mon, May 7, 2018 at 11:04 AM, Ted Yu  wrote:
>>>
 Which JIRA is for the Topology Optimization itself ?

 Thanks

 On Mon, May 7, 2018 at 10:26 AM, Bill Bejeck 
>> wrote:

> All,
> I'd like to start a discussion about adding a configuration parameter
> allowing for the forthcoming topology optimization to be optional via
> configuration.
>
>
> The KIP can be found here:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
 295%3A+Add+Streams+
> Configuration+Allowing+for+Optional+Topology+Optimization
>
> Thanks,
> Bill
>

>>>
>>>
>>>
>>> --
>>> -- Guozhang
>>>
>>



signature.asc
Descripti

[jira] [Resolved] (KAFKA-6292) KafkaConsumer ran into Unknown error fetching data for topic-partition caused by integer overflow in FileLogInputStream

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-6292.

Resolution: Fixed

> KafkaConsumer ran into Unknown error fetching data for topic-partition caused 
> by integer overflow in FileLogInputStream 
> 
>
> Key: KAFKA-6292
> URL: https://issues.apache.org/jira/browse/KAFKA-6292
> Project: Kafka
>  Issue Type: Bug
>  Components: log
>Affects Versions: 0.11.0.0, 0.11.0.1, 1.0.0, 0.11.0.2
> Environment: OS:Red Hat Enterprise Linux Server release 7.3 (Maipo)
> Kafka: kafka_2.12-0.11.0.0
> JDK: jdk1.8.0_121
>Reporter: Terence Yi
>Priority: Major
>  Labels: easyfix, reliability
> Fix For: 2.0.0
>
>
> Steps to reproduce:
> * Broker config to reproduce this bug:
> {code:java}
>   # The maximum size of a log segment file. When this size is reached a new 
> log segment will be created.
> #2G
> log.segment.bytes=2147483647
> {code}
> * Setups:
> producer sends messages constantly. 
> consumer polling
> topic has 1 partitions and replication factor 1.
> min.insync.replicas=1
> producer has "acks=all"
> consumer has default "enable.auto.commit=false"
> consumer manually commitSync offsets after handling messages.
> kafka in standalone
> * Observe log in consumer side(for me running 12 hours)
> {code:java}
> 2017-12-18 07:11:01.013 WARN sep105v1 
> [app-consumer-subscription-pool-4-thread-20] 
> org.apache.kafka.clients.consumer.internals.Fetcher {} Unknown error fetching 
> data for topic-partition DDI.DISPATCHER.P_TVIN.W_SL.P_appx.P_ul.P_pos-0
> {code}
> * Observe server.log in Kafka/logs
> {code:java}
> [2017-12-14 04:52:21,144] ERROR [Replica Manager on Broker 3]: Error 
> processing fetch operation on partition 
> DDI.DISPATCHER.P_TVIN.W_SL.P_appx.P_ul.P_pos-0, offset 4043314339 
> (kafka.server.ReplicaManager)
> org.apache.kafka.common.KafkaException: java.io.EOFException: Failed to read 
> `log header` from file channel `sun.nio.ch.FileChannelImpl@5604ea91`. 
> Expected to read 17 bytes, but reached end of file after reading 0 bytes. 
> Started read from position 2147483643.
> at 
> org.apache.kafka.common.record.RecordBatchIterator.makeNext(RecordBatchIterator.java:40)
> at 
> org.apache.kafka.common.record.RecordBatchIterator.makeNext(RecordBatchIterator.java:24)
> at 
> org.apache.kafka.common.utils.AbstractIterator.maybeComputeNext(AbstractIterator.java:79)
> at 
> org.apache.kafka.common.utils.AbstractIterator.hasNext(AbstractIterator.java:45)
> at 
> org.apache.kafka.common.record.FileRecords.searchForOffsetWithSize(FileRecords.java:279)
> at kafka.log.LogSegment.translateOffset(LogSegment.scala:176)
> at kafka.log.LogSegment.read(LogSegment.scala:228)
> at kafka.log.Log.read(Log.scala:938)
> at kafka.server.ReplicaManager.read$1(ReplicaManager.scala:719)
> at 
> kafka.server.ReplicaManager.$anonfun$readFromLocalLog$6(ReplicaManager.scala:780)
> at 
> scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:59)
> at 
> scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:52)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
> at 
> kafka.server.ReplicaManager.readFromLocalLog(ReplicaManager.scala:779)
> at kafka.server.ReplicaManager.fetchMessages(ReplicaManager.scala:617)
> at kafka.server.KafkaApis.handleFetchRequest(KafkaApis.scala:615)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:98)
> at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:66)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.EOFException: Failed to read `log header` from file 
> channel `sun.nio.ch.FileChannelImpl@5604ea91`. Expected to read 17 bytes, but 
> reached end of file after reading 0 bytes. Started read from position 
> 2147483643.
> at org.apache.kafka.common.utils.Utils.readFullyOrFail(Utils.java:751)
> at 
> org.apache.kafka.common.record.FileLogInputStream.nextBatch(FileLogInputStream.java:66)
> at 
> org.apache.kafka.common.record.FileLogInputStream.nextBatch(FileLogInputStream.java:40)
> at 
> org.apache.kafka.common.record.RecordBatchIterator.makeNext(RecordBatchIterator.java:35)
> ... 18 more
> {code}
> * Impact:
> # After EOF exception occurs, the consumer will failed to consume the remain 
> message
> # After the segments log files which cause the EOF exception has been deleted 
> by the log Cleaner thread. Consumer recovered to consumer message.
> # Have no impact from the view of producer
> * Analysis:
> # Kafka log file list:
> {

Re: [DISCUSS] KIP-295: Add Streams Configuration Allowing for Optional Topology Optimization

[Guozhang Wang]

Thanks Matthias.

I was also thinking about whether in the future we'd want to enable
optimizations at different levels that may or may not impact compatibility.
That's why I asked if we have thought about "allowing part of the
optimizations in the future".

With that in mind, I'd change my preference and take string typed config.
Even if we ended up with no finer grained optimizations in the future we
can still have the string typed parameter with only two allowed values,
like what we did for EOS. But I think in 2.0 allowing any not-null string
values as enabled is still a bit odd, so how about we make two string
values, like `none` (default value) and `all`?


Guozhang


On Tue, May 8, 2018 at 3:35 PM, Matthias J. Sax 
wrote:

> One thought I want to bring up about switching optimization on/off:
>
> While for the initial release, a boolean flag seems to be sufficient, I
> could imagine that we apply different and potentially
> upgrade-incompatible optimizations in future releases. Thus, to me it
> would make sense to use a String type, to indicate what optimizations
> are possible based on the release. For example, in next release we
> accept `null` for disabled and "2.0". If there are any incompatible
> changes, people can stay with "2.0" optimizations level when upgrading
> to "2.1" while new applications can use "2.1" optimization level. Old
> applications would need to do an offline upgrade to get "2.1"
> optimizations.
>
> I agree with Bill, that switching individual optimizations on/off is too
> fine grained and hard to maintain. However, for compatibility, it might
> make sense, to have certain "levels of optimizations" (based on the
> release) that allow users to stay with on an older level for upgrade
> purpose. Using the release numbers to encode those "levels" is easy to
> understand for users and should minimize the mental effort to get the
> config right.
>
> Let me know what you think about this.
>
>
> -Matthias
>
> On 5/8/18 3:08 PM, Ted Yu wrote:
> > Bill:That makes sense.
> > Using boolean should suffice.
> >  Original message From: Bill Bejeck 
> Date: 5/8/18  2:48 PM  (GMT-08:00) To: dev@kafka.apache.org Subject: Re:
> [DISCUSS] KIP-295: Add Streams Configuration Allowing for Optional Topology
> Optimization
> > Thanks for the comments Guozhang and Ted.
> >
> > Guozhang:
> >
> >   1) I'll update the KIP in the "Compatibility, Deprecation and Migration
> > Plan" with the expected impact of turning on optimization. But at this
> > point, I have not identified a migration plan that doesn't involve having
> > to stop all instances and restart.
> >
> >   2) Setting the type to String was just so we could have the default of
> > null, indicating run no optimizations. As for partially enabling
> > optimizations, I'm not sure I had that in mind, at least at this point.
> >  To me having the topology optimized should be an "all or nothing"
> > proposition.  For now, I'll change the type to boolean (with a default
> > value of false) to better reflect the intent of the configuration.
> >
> > Ted, thanks again for the comments.
> >
> > The intent of the new configuration, as I mentioned above, is whether to
> > turn optimization on or off in aggregate.  The main reason for having the
> > configuration is for backward compatibility.  As optimization may result
> in
> > a slightly different topology from the original one, we need to allow
> users
> > to turn it off until they are ready for migrating to the new topology.
> >
> > I don't think having to select each optimization is a feasible
> solution.  I
> > say this for few reasons:
> >
> >1. Maintenance will be an issue.  While the initial target is only a
> >small number of optimizations, but as the number grows, having to
> maintain
> >the number of options in the code will difficult at best.
> >2. Users probably don't want to reason about which combination of
> >optimizations to have.  Thinking about which optimizations to turn on
> or
> >off raises the complexity of deploying an application.
> >3. When using a relational database or other another framework that
> may
> >optimize queries, as far as I know, it's not a choice of which
> >optimizations to have, they are performed automatically.
> >
> > Does this make sense?
> >
> >
> > On Mon, May 7, 2018 at 7:49 PM, Ted Yu  wrote:
> >
> >> There are 4 subtasks for KAFKA-6034
> >>
> >> If each optimization can be switched on/off, there should be 4 enums for
> >> the switch.
> >>
> >> On Mon, May 7, 2018 at 4:39 PM, Guozhang Wang 
> wrote:
> >>
> >>> Hello Bill,
> >>>
> >>> Thanks for the KIP. My comments are the following:
> >>>
> >>> 1) We should state clearly what are the expected impact in
> >> "Compatibility,
> >>> Deprecation, and Migration Plan" when optimization is turned on.
> >>>
> >>> 2) I'm wondering why we define this config as a String typed, rather
> than
> >>> boolean typed? Or are you expecting to extend it to allow partially
> >

Re: [DISCUSS] KIP-292: Add transformValues() method to KTable

[Guozhang Wang]

To follow on Matthias and Damian's comments here:

If we are going to add the overload functions as

```
 KTable transformValues(final ValueTransformerSupplier valueTransformerSupplier,
   final String... stateStoreNames,
   final Materialized materialized);

 KTable transformValues(final ValueTransformerWithKeySupplier valueTransformerSupplier,
   final String... stateStoreNames,
 final Materialized materialized);
```

Then are we going to still only allow the valueTransofmer.init() /
process() to be able to access N stores, with N stores specified with the
stateStoreNames, but not the one specified in materialized.name()?
Personally I think it should be the case as the materialized store should
be managed by the Streams library itself, but we should probably help users
to understand if they have some stores used for the same purpose (storing
the value that are going to be sent to the downstream changelog stream of
KTable), they should save that store and not creating by themselves as it
will be auto created by the Streams library.


Guozhang




On Tue, May 8, 2018 at 7:45 AM, Damian Guy  wrote:

> Initially i thought materializing a store would be overkill, but from a
> consistency point of view it makes sense to add an overload that takes a
> `Materialized` and only create the store if that is supplied.
>
> On Sun, 6 May 2018 at 17:52 Matthias J. Sax  wrote:
>
> > Andy,
> >
> > thanks for the KIP. I don't have any further comments.
> >
> > My 2cents about Guozhang's questions: as I like consistent behavior, I
> > think transfromValues() should behave the same way as filter() and
> > mapValues().
> >
> >
> > -Matthias
> >
> > On 5/2/18 2:24 PM, Guozhang Wang wrote:
> > > Hello Andy,
> > >
> > > Thanks for the KIP. The motivation and the general proposal looks good
> to
> > > me. I think in KTable it is indeed valuable to add the functions that
> > does
> > > not change key, such as mapValues, transformValues, and filter.
> > >
> > > There are a few meta comments I have about the semantics of the newly
> > added
> > > functions:
> > >
> > > 1) For the resulted KTable, how should its "queryableStoreName()" be
> > > returning?
> > >
> > > 2) More specifically, how do we decide if the resulted KTable is to be
> > > materialized or not? E.g. if there is no store names provided then it
> is
> > > likely that the resulted KTable is not materialized, or at least not
> > > logically materialized and not be queryable. What if there is at least
> > one
> > > state store provided? Will any of them be provided as the materialized
> > > store, or should we still add a Materialized parameter for this
> purpose?
> > >
> > > 3) For its internal implementations, how should the key/value serde,
> > > sendOldValues flag etc be inherited from its parent processor node?
> > >
> > >
> > > Guozhang
> > >
> > >
> > > On Wed, May 2, 2018 at 12:43 PM, Andy Coates 
> wrote:
> > >
> > >> Hi everyone,
> > >>
> > >> I would like to start a discussion for KIP 292. I would appreciate it
> if
> > >> you could review and provide feedback.
> > >>
> > >> KIP: KIP-292: Add transformValues() method to KTable
> > >>  > >> 292%3A+Add+transformValues%28%29+method+to+KTable>
> > >> Jira: KAFKA-6849 
> > >>
> > >>PR: #4959 
> > >>
> > >>
> > >>
> > >> Thanks,
> > >>
> > >> Andy
> > >>
> > >
> > >
> > >
> >
> >
>



-- 
-- Guozhang

Build failed in Jenkins: kafka-trunk-jdk8 #2627

[Apache Jenkins Server]

See 


Changes:

[ismael] KAFKA-6879; Invoke session init callbacks outside lock to avoid

[mjsax] MINOR: add equals()/hashCode() for Produced/Consumed (#4979)

--
[...truncated 419.73 KB...]

kafka.admin.ResetConsumerGroupOffsetTest > testResetOffsetsShiftPlus STARTED

kafka.admin.ResetConsumerGroupOffsetTest > testResetOffsetsShiftPlus PASSED

kafka.admin.ResetConsumerGroupOffsetTest > testResetOffsetsToLatest STARTED

kafka.admin.ResetConsumerGroupOffsetTest > testResetOffsetsToLatest PASSED

kafka.admin.ResetConsumerGroupOffsetTest > 
testResetOffsetsNewConsumerExistingTopic STARTED

kafka.admin.ResetConsumerGroupOffsetTest > 
testResetOffsetsNewConsumerExistingTopic PASSED

kafka.admin.ResetConsumerGroupOffsetTest > 
testResetOffsetsShiftByLowerThanEarliest STARTED

kafka.admin.ResetConsumerGroupOffsetTest > 
testResetOffsetsShiftByLowerThanEarliest PASSED

kafka.admin.ResetConsumerGroupOffsetTest > testResetOffsetsByDuration STARTED

kafka.admin.ResetConsumerGroupOffsetTest > testResetOffsetsByDuration PASSED

kafka.admin.ResetConsumerGroupOffsetTest > testResetOffsetsToLocalDateTime 
STARTED

kafka.admin.ResetConsumerGroupOffsetTest > testResetOffsetsToLocalDateTime 
PASSED

kafka.admin.ResetConsumerGroupOffsetTest > 
testResetOffsetsToEarliestOnTopicsAndPartitions STARTED

kafka.admin.ResetConsumerGroupOffsetTest > 
testResetOffsetsToEarliestOnTopicsAndPartitions PASSED

kafka.admin.ResetConsumerGroupOffsetTest > testResetOffsetsToEarliestOnTopics 
STARTED

kafka.admin.ResetConsumerGroupOffsetTest > testResetOffsetsToEarliestOnTopics 
PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > shouldFailIfBlankArg STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > shouldFailIfBlankArg PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowVerifyWithoutReassignmentOption STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowVerifyWithoutReassignmentOption PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowGenerateWithoutBrokersOption STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowGenerateWithoutBrokersOption PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowTopicsOptionWithVerify STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowTopicsOptionWithVerify PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowGenerateWithThrottleOption STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowGenerateWithThrottleOption PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > shouldFailIfNoArgs STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > shouldFailIfNoArgs PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowExecuteWithoutReassignmentOption STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowExecuteWithoutReassignmentOption PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowBrokersListWithVerifyOption STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowBrokersListWithVerifyOption PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldCorrectlyParseValidMinimumExecuteOptions STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldCorrectlyParseValidMinimumExecuteOptions PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowGenerateWithReassignmentOption STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowGenerateWithReassignmentOption PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldCorrectlyParseValidMinimumGenerateOptions STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldCorrectlyParseValidMinimumGenerateOptions PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowGenerateWithoutBrokersAndTopicsOptions STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowGenerateWithoutBrokersAndTopicsOptions PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowThrottleWithVerifyOption STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowThrottleWithVerifyOption PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > shouldUseDefaultsIfEnabled 
STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > shouldUseDefaultsIfEnabled 
PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldAllowThrottleOptionOnExecute STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldAllowThrottleOptionOnExecute PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowExecuteWithBrokers STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowExecuteWithBrokers PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowExecuteWithTopicsOption STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowExecuteWithTopicsOption PASSED

kafka.admin.ReassignPartit

Jenkins build is back to normal : kafka-trunk-jdk10 #85

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-295: Add Streams Configuration Allowing for Optional Topology Optimization

[Matthias J. Sax]

Sounds good to me.

On 5/8/18 5:47 PM, Guozhang Wang wrote:
> Thanks Matthias.
> 
> I was also thinking about whether in the future we'd want to enable
> optimizations at different levels that may or may not impact compatibility.
> That's why I asked if we have thought about "allowing part of the
> optimizations in the future".
> 
> With that in mind, I'd change my preference and take string typed config.
> Even if we ended up with no finer grained optimizations in the future we
> can still have the string typed parameter with only two allowed values,
> like what we did for EOS. But I think in 2.0 allowing any not-null string
> values as enabled is still a bit odd, so how about we make two string
> values, like `none` (default value) and `all`?
> 
> 
> Guozhang
> 
> 
> On Tue, May 8, 2018 at 3:35 PM, Matthias J. Sax 
> wrote:
> 
>> One thought I want to bring up about switching optimization on/off:
>>
>> While for the initial release, a boolean flag seems to be sufficient, I
>> could imagine that we apply different and potentially
>> upgrade-incompatible optimizations in future releases. Thus, to me it
>> would make sense to use a String type, to indicate what optimizations
>> are possible based on the release. For example, in next release we
>> accept `null` for disabled and "2.0". If there are any incompatible
>> changes, people can stay with "2.0" optimizations level when upgrading
>> to "2.1" while new applications can use "2.1" optimization level. Old
>> applications would need to do an offline upgrade to get "2.1"
>> optimizations.
>>
>> I agree with Bill, that switching individual optimizations on/off is too
>> fine grained and hard to maintain. However, for compatibility, it might
>> make sense, to have certain "levels of optimizations" (based on the
>> release) that allow users to stay with on an older level for upgrade
>> purpose. Using the release numbers to encode those "levels" is easy to
>> understand for users and should minimize the mental effort to get the
>> config right.
>>
>> Let me know what you think about this.
>>
>>
>> -Matthias
>>
>> On 5/8/18 3:08 PM, Ted Yu wrote:
>>> Bill:That makes sense.
>>> Using boolean should suffice.
>>>  Original message From: Bill Bejeck 
>> Date: 5/8/18  2:48 PM  (GMT-08:00) To: dev@kafka.apache.org Subject: Re:
>> [DISCUSS] KIP-295: Add Streams Configuration Allowing for Optional Topology
>> Optimization
>>> Thanks for the comments Guozhang and Ted.
>>>
>>> Guozhang:
>>>
>>>   1) I'll update the KIP in the "Compatibility, Deprecation and Migration
>>> Plan" with the expected impact of turning on optimization. But at this
>>> point, I have not identified a migration plan that doesn't involve having
>>> to stop all instances and restart.
>>>
>>>   2) Setting the type to String was just so we could have the default of
>>> null, indicating run no optimizations. As for partially enabling
>>> optimizations, I'm not sure I had that in mind, at least at this point.
>>>  To me having the topology optimized should be an "all or nothing"
>>> proposition.  For now, I'll change the type to boolean (with a default
>>> value of false) to better reflect the intent of the configuration.
>>>
>>> Ted, thanks again for the comments.
>>>
>>> The intent of the new configuration, as I mentioned above, is whether to
>>> turn optimization on or off in aggregate.  The main reason for having the
>>> configuration is for backward compatibility.  As optimization may result
>> in
>>> a slightly different topology from the original one, we need to allow
>> users
>>> to turn it off until they are ready for migrating to the new topology.
>>>
>>> I don't think having to select each optimization is a feasible
>> solution.  I
>>> say this for few reasons:
>>>
>>>1. Maintenance will be an issue.  While the initial target is only a
>>>small number of optimizations, but as the number grows, having to
>> maintain
>>>the number of options in the code will difficult at best.
>>>2. Users probably don't want to reason about which combination of
>>>optimizations to have.  Thinking about which optimizations to turn on
>> or
>>>off raises the complexity of deploying an application.
>>>3. When using a relational database or other another framework that
>> may
>>>optimize queries, as far as I know, it's not a choice of which
>>>optimizations to have, they are performed automatically.
>>>
>>> Does this make sense?
>>>
>>>
>>> On Mon, May 7, 2018 at 7:49 PM, Ted Yu  wrote:
>>>
 There are 4 subtasks for KAFKA-6034

 If each optimization can be switched on/off, there should be 4 enums for
 the switch.

 On Mon, May 7, 2018 at 4:39 PM, Guozhang Wang 
>> wrote:

> Hello Bill,
>
> Thanks for the KIP. My comments are the following:
>
> 1) We should state clearly what are the expected impact in
 "Compatibility,
> Deprecation, and Migration Plan" when optimization is turned on.
>
> 2) I'm wonderi

Re: [DISCUSS] KIP-295: Add Streams Configuration Allowing for Optional Topology Optimization

[Bill Bejeck]

Thanks, Guozhang and Matthias for the comments.

I was thinking of changing the config type back to a String and enforcing
the values to be "true" or "false", but "none" or "all" is just as good.

Since those values seem to work, I'll update the KIP accordingly.

Thanks,
Bill

On Tue, May 8, 2018 at 9:38 PM, Matthias J. Sax 
wrote:

> Sounds good to me.
>
> On 5/8/18 5:47 PM, Guozhang Wang wrote:
> > Thanks Matthias.
> >
> > I was also thinking about whether in the future we'd want to enable
> > optimizations at different levels that may or may not impact
> compatibility.
> > That's why I asked if we have thought about "allowing part of the
> > optimizations in the future".
> >
> > With that in mind, I'd change my preference and take string typed config.
> > Even if we ended up with no finer grained optimizations in the future we
> > can still have the string typed parameter with only two allowed values,
> > like what we did for EOS. But I think in 2.0 allowing any not-null string
> > values as enabled is still a bit odd, so how about we make two string
> > values, like `none` (default value) and `all`?
> >
> >
> > Guozhang
> >
> >
> > On Tue, May 8, 2018 at 3:35 PM, Matthias J. Sax 
> > wrote:
> >
> >> One thought I want to bring up about switching optimization on/off:
> >>
> >> While for the initial release, a boolean flag seems to be sufficient, I
> >> could imagine that we apply different and potentially
> >> upgrade-incompatible optimizations in future releases. Thus, to me it
> >> would make sense to use a String type, to indicate what optimizations
> >> are possible based on the release. For example, in next release we
> >> accept `null` for disabled and "2.0". If there are any incompatible
> >> changes, people can stay with "2.0" optimizations level when upgrading
> >> to "2.1" while new applications can use "2.1" optimization level. Old
> >> applications would need to do an offline upgrade to get "2.1"
> >> optimizations.
> >>
> >> I agree with Bill, that switching individual optimizations on/off is too
> >> fine grained and hard to maintain. However, for compatibility, it might
> >> make sense, to have certain "levels of optimizations" (based on the
> >> release) that allow users to stay with on an older level for upgrade
> >> purpose. Using the release numbers to encode those "levels" is easy to
> >> understand for users and should minimize the mental effort to get the
> >> config right.
> >>
> >> Let me know what you think about this.
> >>
> >>
> >> -Matthias
> >>
> >> On 5/8/18 3:08 PM, Ted Yu wrote:
> >>> Bill:That makes sense.
> >>> Using boolean should suffice.
> >>>  Original message From: Bill Bejeck  >
> >> Date: 5/8/18  2:48 PM  (GMT-08:00) To: dev@kafka.apache.org Subject:
> Re:
> >> [DISCUSS] KIP-295: Add Streams Configuration Allowing for Optional
> Topology
> >> Optimization
> >>> Thanks for the comments Guozhang and Ted.
> >>>
> >>> Guozhang:
> >>>
> >>>   1) I'll update the KIP in the "Compatibility, Deprecation and
> Migration
> >>> Plan" with the expected impact of turning on optimization. But at this
> >>> point, I have not identified a migration plan that doesn't involve
> having
> >>> to stop all instances and restart.
> >>>
> >>>   2) Setting the type to String was just so we could have the default
> of
> >>> null, indicating run no optimizations. As for partially enabling
> >>> optimizations, I'm not sure I had that in mind, at least at this point.
> >>>  To me having the topology optimized should be an "all or nothing"
> >>> proposition.  For now, I'll change the type to boolean (with a default
> >>> value of false) to better reflect the intent of the configuration.
> >>>
> >>> Ted, thanks again for the comments.
> >>>
> >>> The intent of the new configuration, as I mentioned above, is whether
> to
> >>> turn optimization on or off in aggregate.  The main reason for having
> the
> >>> configuration is for backward compatibility.  As optimization may
> result
> >> in
> >>> a slightly different topology from the original one, we need to allow
> >> users
> >>> to turn it off until they are ready for migrating to the new topology.
> >>>
> >>> I don't think having to select each optimization is a feasible
> >> solution.  I
> >>> say this for few reasons:
> >>>
> >>>1. Maintenance will be an issue.  While the initial target is only a
> >>>small number of optimizations, but as the number grows, having to
> >> maintain
> >>>the number of options in the code will difficult at best.
> >>>2. Users probably don't want to reason about which combination of
> >>>optimizations to have.  Thinking about which optimizations to turn
> on
> >> or
> >>>off raises the complexity of deploying an application.
> >>>3. When using a relational database or other another framework that
> >> may
> >>>optimize queries, as far as I know, it's not a choice of which
> >>>optimizations to have, they are performed automatically.
> >>>
> >>> Does this make se

Jenkins build is back to normal : kafka-trunk-jdk7 #3408

[Apache Jenkins Server]

See 

Is Connector#initialize(ConnectorContext, List>) a TODO method?

[Chia-Ping Tsai]

hi Kafka,

I'm reading the connector flow. And I noticed the methods 
"initialize(ConnectorContext, List>)" and "reconfigure" are 
not used in production code. Hence, are they still in TODO state? Or they are 
designed to use rather than to override?

Best Regards,
Chia-Ping

Re: [DISCUSS] KIP-297: Externalizing Secrets for Connect Configurations

[Magesh Nandakumar]

Hi Robert,

Thanks for the KIP. I think, this will be a great addition to the
framework. I think, will be great if the KIP can elaborate a little bit
more on how implementations would look like with an example.
Also, would be good to provide a reference implementation as well.

The other questions I had were

1.  How would the framework get the delayMs for void scheduleConfigReload(
long delayMs);
2. Would the start methods in SourceTask and SinkTask get the configs with
all the indirect references resolved. If so, trying to understand
the intent of the config() in SourceTaskContext and the SinkTaskContext
3. What if the provider itself needs some kind of secrets to be configured
to connect to it? I assume that's out of scope for this proposal but wanted
to clarify it.

Thanks
Magesh

On Tue, May 8, 2018 at 1:52 PM, Robert Yokota  wrote:

> Hi,
>
> I would like to start a discussion for KIP-297 to externalize secrets from
> Kafka Connect configurations.  Any feedback is appreciated.
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 297%3A+Externalizing+Secrets+for+Connect+Configurations
> >
>
> JIRA: 
>
> Thanks in advance,
> Robert
>

Re: [DISCUSS] KIP-285: Connect Rest Extension Plugin

[Magesh Nandakumar]

Hi All,

I have updated the KIP to reflect changes based on the PR
https://github.com/apache/kafka/pull/4931. Its mostly has minor changes to
the interfaces and includes details on packages for the interfaces and the
classes. Let me know your thoughts.

Thanks
Magesh

On Fri, Apr 27, 2018 at 12:03 PM, Randall Hauch  wrote:

> Great work, Magesh. I like the overall approach a lot, so I left some
> pretty nuanced comments about specific details.
>
> Best regards,
>
> Randall
>
> On Wed, Apr 25, 2018 at 3:03 PM, Magesh Nandakumar 
> wrote:
>
> > Thanks Randall for your thoughts. I have created a replica of the
> required
> > entities in the draft implementation. If you can take a look at the PR
> and
> > let me know your thoughts, I will update the KIP to reflect the same
> >
> > https://github.com/apache/kafka/pull/4931
> >
> > On Tue, Apr 24, 2018 at 11:44 AM, Randall Hauch 
> wrote:
> >
> > > Magesh, I think our last emails cross in mid-stream.
> > >
> > > We definitely want to put the new public interfaces/classes in the API
> > > module, and implementation in the runtime module. Yes, this will affect
> > the
> > > design, since for example we don't want to expose runtime types to the
> > API,
> > > and we want to prevent breaking changes. We don't really want to move
> the
> > > REST entities if we don't have to, since that may break projects that
> are
> > > extending the runtime module -- even though the runtime module is not a
> > > public API we still want to _try_ to change things.
> > >
> > > Do you want to try to create a prototype to see what kind of impact and
> > > choices we'll have to make?
> > >
> > > Best regards,
> > >
> > > Randall
> > >
> > > On Tue, Apr 24, 2018 at 12:48 PM, Randall Hauch 
> > wrote:
> > >
> > > > Thanks for updating the KIP, Magesh. You've resolved all of my
> > concerns,
> > > > though I have one more: we should specify the package names for all
> new
> > > > interfaces/classes.
> > > >
> > > > I'm looking forward to more feedback from others.
> > > >
> > > > Best regards,
> > > >
> > > > Randall
> > > >
> > > > On Fri, Apr 20, 2018 at 12:17 AM, Magesh Nandakumar <
> > > mage...@confluent.io>
> > > > wrote:
> > > >
> > > >> Hi All,
> > > >>
> > > >> I have updated the KIP with following changes
> > > >>
> > > >>1. Expanded the Motivation section
> > > >>2. Included details about the interface in the public interface
> > > section
> > > >>3. Modified the config name to rest.extension.classes
> > > >>4. Modified the ConnectRestExtension to include Configurable
> > instead
> > > of
> > > >>ResourceConfig
> > > >>5. Modified the "Rest Extension Integration with Connect" in
> > > "Proposed
> > > >>Approach" to include a new Custom implementation for Configurable
> > > >>6. Provided examples for the Java Service provider mechanism
> > > >>7. Included a reference implementation in scope
> > > >>
> > > >> Kindly let me know your thoughts on the updates.
> > > >>
> > > >> Thanks
> > > >> Magesh
> > > >>
> > > >> On Thu, Apr 19, 2018 at 10:39 AM, Magesh Nandakumar <
> > > mage...@confluent.io
> > > >> >
> > > >> wrote:
> > > >>
> > > >> > Hi Randall,
> > > >> >
> > > >> > Thanks for your feedback. I also would like to go with
> > > >> > rest.extension.classes`.
> > > >> >
> > > >> > For exposing Configurable, my original intention was just to
> expose
> > > that
> > > >> > to the extension because that's all one needs to register JAX RS
> > > >> resources.
> > > >> > The fact that we use Jersey shouldn't even be exposed in the
> > > interface.
> > > >> > Hence it doesn't affect the public API by any means.
> > > >> >
> > > >> > I will update the KIP and let everyone know.
> > > >> >
> > > >> > Thanks
> > > >> > Magesh
> > > >> >
> > > >> > On Thu, Apr 19, 2018 at 9:51 AM, Randall Hauch 
> > > >> wrote:
> > > >> >
> > > >> >> On Thu, Apr 12, 2018 at 10:59 AM, Magesh Nandakumar <
> > > >> mage...@confluent.io
> > > >> >> >
> > > >> >> wrote:
> > > >> >>
> > > >> >> > Hi Randall,
> > > >> >> >
> > > >> >> > Thanks a lot for your feedback.
> > > >> >> >
> > > >> >> > I will update the KIP to reflect your comments in (1), (2), (7)
> > and
> > > >> (8).
> > > >> >> >
> > > >> >>
> > > >> >> Looking forward to these.
> > > >> >>
> > > >> >>
> > > >> >> >
> > > >> >> > For comment # (3) , while it would be great to automatically
> > > >> configure
> > > >> >> the
> > > >> >> > Rest Extensions, I would prefer that to be specified
> explicitly.
> > > Lets
> > > >> >> > assume a connector archive includes a implementation for the
> > > >> >> RestExtension
> > > >> >> > to do authentication using some header. We don't want this to
> be
> > > >> >> > automatically included. Having said that I think that the
> config
> > > key
> > > >> >> name
> > > >> >> > should probably be changed to something like "rest.extension"
> or
> > > >> >> > "rest.extension.class".
> > > >> >> >
> > > >> >>
> > > >> >> That's a good point. I do like `rest.extension.class` (or
>

Jenkins build is back to normal : kafka-trunk-jdk8 #2628

[Apache Jenkins Server]

See 

[jira] [Resolved] (KAFKA-6390) Update ZooKeeper to 3.4.11, Gradle and other minor updates

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-6390.

Resolution: Fixed

> Update ZooKeeper to 3.4.11, Gradle and other minor updates
> --
>
> Key: KAFKA-6390
> URL: https://issues.apache.org/jira/browse/KAFKA-6390
> Project: Kafka
>  Issue Type: Bug
>Reporter: Ismael Juma
>Assignee: Ismael Juma
>Priority: Major
> Fix For: 2.0.0
>
>
> https://issues.apache.org/jira/browse/ZOOKEEPER-2614 is a helpful fix.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-6887) Error in I/O with host (java.io.EOFException) raised in producer

[Mohammad Tanvir Tasnim (JIRA)]

Mohammad Tanvir Tasnim created KAFKA-6887:
-

 Summary: Error in I/O with host (java.io.EOFException) raised in 
producer
 Key: KAFKA-6887
 URL: https://issues.apache.org/jira/browse/KAFKA-6887
 Project: Kafka
  Issue Type: Bug
  Components: clients
Affects Versions: 0.8.2.1, 0.9.0.0
Reporter: Mohammad Tanvir Tasnim


In a situation with a Kafka broker in 0.9 and producers still in 0.8.2.x, 
producers seems to raise the following after a variable amount of time since 
start :

{noformat}
2016-01-29 14:33:13,066 WARN [] o.a.k.c.n.Selector: Error in I/O with 
172.22.2.170
java.io.EOFException: null
at 
org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:62) 
~[org.apache.kafka.kafka-clients-0.8.2.0.jar:na]
at org.apache.kafka.common.network.Selector.poll(Selector.java:248) 
~[org.apache.kafka.kafka-clients-0.8.2.0.jar:na]
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:192) 
[org.apache.kafka.kafka-clients-0.8.2.0.jar:na]
at 
org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:191) 
[org.apache.kafka.kafka-clients-0.8.2.0.jar:na]
at 
org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:122) 
[org.apache.kafka.kafka-clients-0.8.2.0.jar:na]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_66-internal]
{noformat}

This can be reproduced successfully by doing the following :

 * Start a 0.8.2 producer connected to the 0.9 broker
 * Wait 15 minutes, exactly
 * See the error in the producer logs.

Oddly, this also shows up in an active producer but after 10 minutes of 
activity.

Kafka's server.properties :

{noformat}
broker.id=1
listeners=PLAINTEXT://:9092
port=9092
num.network.threads=2
num.io.threads=2
socket.send.buffer.bytes=1048576
socket.receive.buffer.bytes=1048576
socket.request.max.bytes=104857600
log.dirs=/mnt/data/kafka
num.partitions=4
auto.create.topics.enable=false
delete.topic.enable=true
num.recovery.threads.per.data.dir=1
log.retention.hours=48
log.retention.bytes=524288000
log.segment.bytes=52428800
log.retention.check.interval.ms=6
log.roll.hours=24
log.cleanup.policy=delete
log.cleaner.enable=true
zookeeper.connect=127.0.0.1:2181
zookeeper.connection.timeout.ms=100
{noformat}

Producer's configuration :

{noformat}
compression.type = none
metric.reporters = []
metadata.max.age.ms = 30
metadata.fetch.timeout.ms = 6
acks = all
batch.size = 16384
reconnect.backoff.ms = 10
bootstrap.servers = [127.0.0.1:9092]
receive.buffer.bytes = 32768
retry.backoff.ms = 500
buffer.memory = 33554432
timeout.ms = 3
key.serializer = class 
org.apache.kafka.common.serialization.StringSerializer
retries = 3
max.request.size = 500
block.on.buffer.full = true
value.serializer = class 
org.apache.kafka.common.serialization.StringSerializer
metrics.sample.window.ms = 3
send.buffer.bytes = 131072
max.in.flight.requests.per.connection = 5
metrics.num.samples = 2
linger.ms = 0
client.id = 
{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: kafka-trunk-jdk8 #2629

[Apache Jenkins Server]

See 


Changes:

[jqin] KAFKA-6877; Remove completedFetch upon a failed parse if it contains no

--
[...truncated 419.82 KB...]
kafka.admin.ResetConsumerGroupOffsetTest > testResetOffsetsToSpecificOffset 
PASSED

kafka.admin.ResetConsumerGroupOffsetTest > testResetOffsetsShiftPlus STARTED

kafka.admin.ResetConsumerGroupOffsetTest > testResetOffsetsShiftPlus PASSED

kafka.admin.ResetConsumerGroupOffsetTest > testResetOffsetsToLatest STARTED

kafka.admin.ResetConsumerGroupOffsetTest > testResetOffsetsToLatest PASSED

kafka.admin.ResetConsumerGroupOffsetTest > 
testResetOffsetsNewConsumerExistingTopic STARTED

kafka.admin.ResetConsumerGroupOffsetTest > 
testResetOffsetsNewConsumerExistingTopic PASSED

kafka.admin.ResetConsumerGroupOffsetTest > 
testResetOffsetsShiftByLowerThanEarliest STARTED

kafka.admin.ResetConsumerGroupOffsetTest > 
testResetOffsetsShiftByLowerThanEarliest PASSED

kafka.admin.ResetConsumerGroupOffsetTest > testResetOffsetsByDuration STARTED

kafka.admin.ResetConsumerGroupOffsetTest > testResetOffsetsByDuration PASSED

kafka.admin.ResetConsumerGroupOffsetTest > testResetOffsetsToLocalDateTime 
STARTED

kafka.admin.ResetConsumerGroupOffsetTest > testResetOffsetsToLocalDateTime 
PASSED

kafka.admin.ResetConsumerGroupOffsetTest > 
testResetOffsetsToEarliestOnTopicsAndPartitions STARTED

kafka.admin.ResetConsumerGroupOffsetTest > 
testResetOffsetsToEarliestOnTopicsAndPartitions PASSED

kafka.admin.ResetConsumerGroupOffsetTest > testResetOffsetsToEarliestOnTopics 
STARTED

kafka.admin.ResetConsumerGroupOffsetTest > testResetOffsetsToEarliestOnTopics 
PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > shouldFailIfBlankArg STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > shouldFailIfBlankArg PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowVerifyWithoutReassignmentOption STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowVerifyWithoutReassignmentOption PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowGenerateWithoutBrokersOption STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowGenerateWithoutBrokersOption PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowTopicsOptionWithVerify STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowTopicsOptionWithVerify PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowGenerateWithThrottleOption STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowGenerateWithThrottleOption PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > shouldFailIfNoArgs STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > shouldFailIfNoArgs PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowExecuteWithoutReassignmentOption STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowExecuteWithoutReassignmentOption PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowBrokersListWithVerifyOption STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowBrokersListWithVerifyOption PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldCorrectlyParseValidMinimumExecuteOptions STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldCorrectlyParseValidMinimumExecuteOptions PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowGenerateWithReassignmentOption STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowGenerateWithReassignmentOption PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldCorrectlyParseValidMinimumGenerateOptions STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldCorrectlyParseValidMinimumGenerateOptions PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowGenerateWithoutBrokersAndTopicsOptions STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowGenerateWithoutBrokersAndTopicsOptions PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowThrottleWithVerifyOption STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowThrottleWithVerifyOption PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > shouldUseDefaultsIfEnabled 
STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > shouldUseDefaultsIfEnabled 
PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldAllowThrottleOptionOnExecute STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldAllowThrottleOptionOnExecute PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowExecuteWithBrokers STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowExecuteWithBrokers PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowExecuteWithTopicsOption STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowExecuteWithTopicsOption PASSED

kafka.

[jira] [Resolved] (KAFKA-6879) Controller deadlock following session expiration

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-6879.

Resolution: Fixed

> Controller deadlock following session expiration
> 
>
> Key: KAFKA-6879
> URL: https://issues.apache.org/jira/browse/KAFKA-6879
> Project: Kafka
>  Issue Type: Bug
>  Components: controller
>Affects Versions: 1.1.0
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
>Priority: Critical
> Fix For: 2.0.0, 1.1.1
>
>
> We have observed an apparent deadlock situation which occurs following a 
> session expiration. The suspected deadlock occurs between the zookeeper 
> "initializationLock" and the latch inside the Expire event which we use to 
> ensure all events have been handled.
> In the logs, we see the "Session expired" message following acquisition of 
> the initialization lock: 
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/zookeeper/ZooKeeperClient.scala#L358
> But we never see any logs indicating that the new session is being 
> initialized. In fact, the controller logs are basically empty from that point 
> on. The problem we suspect is that completion of the 
> {{beforeInitializingSession}} callback requires that all events have finished 
> processing in order to count down the latch: 
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/controller/KafkaController.scala#L1525.
> But an event which was dequeued just prior to the acquisition of the write 
> lock may be unable to complete because it is awaiting acquisition of the 
> initialization lock: 
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/zookeeper/ZooKeeperClient.scala#L137.
> The impact is that the broker continues in a zombie state. It continues 
> fetching and is periodically added to ISRs, but it never receives any further 
> requests from the controller since it is not registered.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)