[jira] [Created] (KAFKA-7236) Add --critical-partitions option to describe topics command

[Kevin Lu (JIRA)]

Kevin Lu created KAFKA-7236:
---

 Summary: Add --critical-partitions option to describe topics 
command
 Key: KAFKA-7236
 URL: https://issues.apache.org/jira/browse/KAFKA-7236
 Project: Kafka
  Issue Type: Improvement
  Components: tools
Reporter: Kevin Lu
Assignee: Kevin Lu


A topic partition can be in one of four states (assuming replication factor of 
3):

 

(ISR = in sync replica)

 

3/3 ISRs: OK

2/3 ISRs: WARNING (under-replicated partition)

1/3 ISRs: CRITICAL (under-replicated partition)

0/3 ISRs: FATAL (offline/unavailable partition)

 

TopicCommand already has the --under-replicated-partitions and 
--unavailable-partitions flags, but it would be beneficial to include an 
additional --critical-partitions option that specifically lists out partitions 
in CRITICAL state (only one remaining ISR left).

 

With this new option, Kafka users can use this option to identify the exact 
topic partitions that are critical and need immediate repartitioning. Kafka 
users can also set up critical alerts to trigger when the output of this 
command contains partitions.

 

A couple cases where identifying this CRITICAL state is useful in alerting:
 * Users that have a large amount of topics in a single cluster, making it 
incredibly hard to manually repartition all topics that have under-replicated 
partitions, so they only take action when it hits CRITICAL state
 * Users with a high replication-factor that can tolerate some broker failures 
and only take action when it hits CRITICAL state



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[DISCUSS] KIP-351: Add --critical-partitions option to describe topics command

[Kevin Lu]

Hi friends!

This thread is to discuss KIP-351

!

I am proposing to add a --critical-partitions option to the describe topics
command that will only list out topic partitions that have 1 ISR left (RF >
1) as they would be in a critical state and need immediate repartitioning.

I wonder if the name "critical" is appropriate?

Thoughts?

Thanks!

Regards,
Kevin

Re: [DISCUSS] KIP-351: Add --critical-partitions option to describe topics command

[Mickael Maison]

What about also adding a --under-minisr-partitions option?

That would match the
"kafka.server:type=ReplicaManager,name=UnderMinIsrPartitionCount"
broker metric and it's usually pretty relevant when investigating
issues

On Thu, Aug 2, 2018 at 8:54 AM, Kevin Lu  wrote:
> Hi friends!
>
> This thread is to discuss KIP-351
> 
> !
>
> I am proposing to add a --critical-partitions option to the describe topics
> command that will only list out topic partitions that have 1 ISR left (RF >
> 1) as they would be in a critical state and need immediate repartitioning.
>
> I wonder if the name "critical" is appropriate?
>
> Thoughts?
>
> Thanks!
>
> Regards,
> Kevin

Re: [DISCUSSION] KIP-336: Consolidate ExtendedSerializer/Serializer and ExtendedDeserializer/Deserializer

[Viktor Somogyi]

Hi Chia-Ping,

Sorry for the delay on this. One thought though: looking at current
implementations on github they seemed a bit elaborate, which makes me think
that people don't usually want to use it as a lambda. But since in your KIP
you added it, what was your use case there?

"Q: Which implementation is suitable for serialize()/deserialize()? maybe
just throw exception?"

For now I think l it'll be useful to not throw exceptions but I have to
check if there is a use case for this. Do you have any use cases btw? :)

Viktor

On Tue, Jul 31, 2018 at 12:24 PM Chia-Ping Tsai  wrote:

> Bumping up!
>
> On 2018/07/09 08:50:07, Viktor Somogyi  wrote:
> > Hi folks,
> >
> > I've published KIP-336 which is about consolidating the
> > Serializer/Deserializer interfaces.
> >
> > Basically the story here is when ExtendedSerializer and
> > ExtendedDeserializer were added we still supported Java 7 and therefore
> had
> > to use compatible constructs which now seem unnecessary since we dropped
> > support for Java 7. Now in this KIP I propose a way to deprecate those
> > patterns:
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=87298242
> >
> > I'd be happy to receive some feedback about the KIP I published.
> >
> > Cheers,
> > Viktor
> >
>

Re: [DISCUSS] KIP-334 Include partitions in exceptions raised during consumer record deserialization/validation

[Stanislav Kozlovski]

Hi group,

I've updated the KIP and PR with the discussed interface changes.
I am also starting a voting thread

Best,
Stanislav

On Thu, Aug 2, 2018 at 1:27 AM Jason Gustafson  wrote:

> Hey Stanislav,
>
> Just to make sure I understood you right - you propose not exposing any new
> > exception types but rather the interface itself only?
>
>
> Yes, exactly. Our exception hierarchy is a bit of a mess to be honest.
> Interfaces are more flexible and here it simplifies the error handling.
>
> Regardless, I believe this is best left out for another KIP as I feel it
> > would warrant a bigger discussion
>
>
> Ok, that's fair. I thought I'd suggest it here just to see if there was any
> interest in the community. At least with this KIP, users have a viable way
> to skip past bad data if they wish.
>
> -Jason
>
> On Tue, Jul 31, 2018 at 2:42 AM, Stanislav Kozlovski <
> stanis...@confluent.io
> > wrote:
>
> > Hey Jason,
> >
> > Just to make sure I understood you right - you propose not exposing any
> new
> > exception types but rather the interface itself only? So a sample code
> > dealing with this would be something like:
> >
> > try {
> > // ...
> > } catch (KafkaException e) {
> > if (e instanceof UnconsumableRecordException) {
> >   // handle retry
> > }
> > }
> >
> > If that is the case, I like it better.
> >
> >
> > In regards to automatic handling of unconsumable messages - I like that
> > idea too. To me, a callback seems like the more straightforward
> approach. A
> > config such as `seek.past.unconsumable.record` limits the behavior too
> > much
> > in my opinion, I believe giving them the option to implement a (or use
> the
> > default) callback is better in that way.
> > Regardless, I believe this is best left out for another KIP as I feel it
> > would warrant a bigger discussion
> >
> > Best,
> > Stanislav
> >
> > On Mon, Jul 30, 2018 at 9:34 PM Jason Gustafson 
> > wrote:
> >
> > > Hey Stanislav,
> > >
> > > Thanks for the KIP. I think the goal is to allow users to seek past a
> > > records which cannot be parsed for whatever reason. However, it's a
> > little
> > > annoying that you need to catch two separate types to handle this. I'm
> > > wondering if it makes sense to expose an interface like
> > > `UnconsumableRecordException` or something like that. The consumer
> could
> > > then have separate internal exception types which extend from
> > > InvalidRecordException and SerializationException respectively and
> > > implement `UnconsumableRecordException`. That would simplify the
> handling
> > > and users could check the cause if they cared which case it was.
> > >
> > > Another question for consideration. I'd imagine some users would find
> it
> > > helpful to seek past failed messages automatically. If there is a
> corrupt
> > > record, for example, there's almost nothing you can do except seek past
> > it
> > > anyway. I'm wondering if there should be a config for this or if users
> > > should be able to install a callback of some sorts to handle failed
> > > records. Not sure if this is that big of a problem for users, but
> > > interested to hear others thoughts.
> > >
> > > Thanks,
> > > Jason
> > >
> > > On Fri, Jul 20, 2018 at 6:32 PM, Stanislav Kozlovski <
> > > stanis...@confluent.io
> > > > wrote:
> > >
> > > > Hi Ted,
> > > >
> > > > I do plan to start one. When is the appropriate time? My reasoning
> was
> > > that
> > > > people would like to view the changes first
> > > >
> > > > On Fri, Jul 20, 2018, 6:21 PM Ted Yu  wrote:
> > > >
> > > > > Hi, Stanislav:
> > > > > Do you plan to start VOTE thread ?
> > > > >
> > > > > Cheers
> > > > >
> > > > > On Fri, Jul 20, 2018 at 6:11 PM Stanislav Kozlovski <
> > > > > stanis...@confluent.io>
> > > > > wrote:
> > > > >
> > > > > > Hey group,
> > > > > >
> > > > > > I added a Pull Request for this KIP - here it is
> > > > > > https://github.com/apache/kafka/pull/5410
> > > > > > Please take a look.
> > > > > >
> > > > > > Best,
> > > > > > Stanislav
> > > > > >
> > > > > > On Thu, Jul 5, 2018 at 11:06 AM Ismael Juma 
> > > wrote:
> > > > > >
> > > > > > > Yes, the Scala consumers have been removed in 2.0.0, which
> > > simplifies
> > > > > > some
> > > > > > > of this. The following commit was an initial step in unifying
> the
> > > > > > exception
> > > > > > > handling:
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > https://github.com/apache/kafka/commit/
> > 96bcfdfc7c9aac075635b2034e65e4
> > > > 12a725672e
> > > > > > >
> > > > > > > But more can be done as you mentioned.
> > > > > > >
> > > > > > > Ismael
> > > > > > >
> > > > > > > On 5 Jul 2018 9:36 am, "Stanislav Kozlovski" <
> > > stanis...@confluent.io
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > Hey Ismael,
> > > > > > >
> > > > > > > It is only slightly related - my PR would attach two new
> > attributes
> > > > and
> > > > > > > also touch upon deserialization exceptions.
> > > > > > >
> > > > > > > But this PR did provide me with

[jira] [Created] (KAFKA-7237) Add explicit fatal marker to fatal error messages

[Rajini Sivaram (JIRA)]

Rajini Sivaram created KAFKA-7237:
-

 Summary: Add explicit fatal marker to fatal error messages
 Key: KAFKA-7237
 URL: https://issues.apache.org/jira/browse/KAFKA-7237
 Project: Kafka
  Issue Type: Bug
  Components: core
Reporter: Rajini Sivaram


We currently use FATAL markers for fatal error messages in core. But only 
logback supports markers out-of-the-box and log4j simply logs these as ERROR 
messages, ignoring the marker. We should perhaps include a String marker within 
these messages to make it easier to identify fatal messages in logs.

>From https://www.slf4j.org/faq.html:

 

While markers are part of the SLF4J API, only logback supports markers off the 
shelf. For example, if you add the {{%marker}}conversion word to its pattern, 
logback's {{PatternLayout}} will add marker data to its output. Marker data can 
be used to [filter messages|http://logback.qos.ch/manual/filters.html] or even 
[trigger|http://logback.qos.ch/manual/appenders.html#OnMarkerEvaluator] an 
outgoing email [at the end of an individual 
transaction|http://logback.qos.ch/recipes/emailPerTransaction.html].

In combination with logging frameworks such as log4j and java.util.logging 
which do not support markers, marker data will be silently ignored.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[VOTE] KIP-334 Include partitions in exceptions raised during consumer record deserialization/validation

[Stanislav Kozlovski]

Hey everybody,

I'd like to start a vote thread for KIP-334 Include partitions in
exceptions raised during consumer record deserialization/validation


-- 
Best,
Stanislav

Re: [Vote] KIP-321: Update TopologyDescription to better represent Source and Sink Nodes

[Damian Guy]

You have 3 binding votes, so i'll defer to the others.

On Thu, 2 Aug 2018 at 04:41 Nishanth Pradeep  wrote:

> The only issue I see with this is that Sink#topic would also need to be
> Optional as was pointed out already. Since Sink#topic is a preexisting
> method, changing its return type would break backwards compatibility.
>
> On the other hand, it might be worth it to rip that bandaid now.
>
> Best,
> Nishanth Pradeep
>
> On Wed, Aug 1, 2018 at 11:56 AM Guozhang Wang  wrote:
>
>> For source node, only one of `Set topicsSet` and `TopicPattern
>> topicPattern()` will be specified by the user. Similarly for sink node,
>> only one of `String` and `TopicNameExtractor` will be specified by the
>> user. Although I've not seen Nishanth's updated PR, I think when it is not
>> specified today we will return null in that case.
>>
>> If we want to improve on this situation with Optional, we'd need to do it
>> on all of these functions. Also note that for `Source#toString()` and
>> `Sink#toString()` we should only include the specified field in the
>> resulted representation.
>>
>>
>> Guozhang
>>
>> On Wed, Aug 1, 2018 at 5:08 AM, Damian Guy  wrote:
>>
>> > Ewen - no as I don't believe they are never null. Whereas the
>> > topicNameExtractor method returns null if it is the default extractor or
>> > the extractor. So i think this would be better to be optional as it is
>> > optionally returning a TopicNameExtractor
>> >
>> > On Tue, 31 Jul 2018 at 23:01 Ewen Cheslack-Postava 
>> > wrote:
>> >
>> > > Generally +1 (binding)
>> > >
>> > > It would be helpful to just provide the full, updated interfaces in
>> the
>> > > KIP and mark things as new with comments if needed. I had to go back
>> and
>> > > read the discussion thread to make sure I was understanding the intent
>> > > correctly.
>> > >
>> > > Damian -- if we make that Optional, shouldn't the methods on Source
>> also
>> > > be Optional types?
>> > >
>> > > -Ewen
>> > >
>> > > On Mon, Jul 30, 2018 at 11:13 PM Damian Guy 
>> > wrote:
>> > >
>> > >> Hi Nishanth,
>> > >>
>> > >> I have one nit on the KIP. I think the topicNameExtractor method
>> should
>> > >> return Optional rather than null.
>> > >> Sorry I'm late here.
>> > >>
>> > >> Thanks,
>> > >> Damian
>> > >>
>> > >> On Tue, 31 Jul 2018 at 01:14 Nishanth Pradeep > >
>> > >> wrote:
>> > >>
>> > >> > We need one more binding vote.
>> > >> >
>> > >> > Binding Votes:
>> > >> >
>> > >> >- Matthias J. Sax
>> > >> >- Guozhang Wong
>> > >> >
>> > >> > Community Votes:
>> > >> >
>> > >> >- Bill Bejeck
>> > >> >- Ted Yu
>> > >> >
>> > >> > Best,
>> > >> > Nishanth Pradeep
>> > >> >
>> > >> > On Fri, Jul 27, 2018 at 10:02 AM Bill Bejeck 
>> > wrote:
>> > >> >
>> > >> > > Thanks for the KIP!
>> > >> > >
>> > >> > > +1
>> > >> > >
>> > >> > > -Bill
>> > >> > >
>> > >> > > On Thu, Jul 26, 2018 at 2:39 AM Guozhang Wang <
>> wangg...@gmail.com>
>> > >> > wrote:
>> > >> > >
>> > >> > > > +1
>> > >> > > >
>> > >> > > > On Wed, Jul 25, 2018 at 11:13 PM, Matthias J. Sax <
>> > >> > matth...@confluent.io
>> > >> > > >
>> > >> > > > wrote:
>> > >> > > >
>> > >> > > > > +1 (binding)
>> > >> > > > >
>> > >> > > > > -Matthias
>> > >> > > > >
>> > >> > > > > On 7/25/18 7:47 PM, Ted Yu wrote:
>> > >> > > > > > +1
>> > >> > > > > >
>> > >> > > > > > On Wed, Jul 25, 2018 at 7:24 PM Nishanth Pradeep <
>> > >> > > > nishanth...@gmail.com>
>> > >> > > > > > wrote:
>> > >> > > > > >
>> > >> > > > > >> Hello,
>> > >> > > > > >>
>> > >> > > > > >> I'm calling a vote for KIP-321:
>> > >> > > > > >>
>> > >> > > > > >>
>> > >> > > > > >>
>> > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> > 321%3A+Update+
>> > >> > > > > TopologyDescription+to+better+represent+Source+and+Sink+Nodes
>> > >> > > > > >>
>> > >> > > > > >> Best,
>> > >> > > > > >> Nishanth Pradeep
>> > >> > > > > >>
>> > >> > > > > >
>> > >> > > > >
>> > >> > > > >
>> > >> > > >
>> > >> > > >
>> > >> > > > --
>> > >> > > > -- Guozhang
>> > >> > > >
>> > >> > >
>> > >> >
>> > >>
>> > >
>> >
>>
>>
>>
>> --
>> -- Guozhang
>>
>

Build failed in Jenkins: kafka-trunk-jdk8 #2854

[Apache Jenkins Server]

See 

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on H20 (ubuntu xenial) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from 
https://github.com/apache/kafka.git
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:888)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1155)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186)
at hudson.scm.SCM.checkout(SCM.java:504)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
at hudson.model.Run.execute(Run.java:1794)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
Caused by: hudson.plugins.git.GitException: Command "git fetch --tags 
--progress https://github.com/apache/kafka.git 
+refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: error: Could not read 24d0d3351a43f52f6f688143672d7c57c5f19595
error: Could not read 1e98b2aa5027eb60ee1a112a7f8ec45b9c47
error: Could not read 689bac3c6fcae927ccee89c8d4fd54c2fae83be1
error: Could not read 3af2cecb9d84369021549946db95dbef9bfe68fa
remote: Counting objects: 2657, done.
remote: Compressing objects:  16% (1/6)   remote: Compressing objects:  
33% (2/6)   remote: Compressing objects:  50% (3/6)   remote: 
Compressing objects:  66% (4/6)   remote: Compressing objects:  83% 
(5/6)   remote: Compressing objects: 100% (6/6)   remote: 
Compressing objects: 100% (6/6), done.
Receiving objects:   0% (1/2657)   Receiving objects:   1% (27/2657)   
Receiving objects:   2% (54/2657)   Receiving objects:   3% (80/2657)   
Receiving objects:   4% (107/2657)   Receiving objects:   5% (133/2657)   
Receiving objects:   6% (160/2657)   Receiving objects:   7% (186/2657)   
Receiving objects:   8% (213/2657)   Receiving objects:   9% (240/2657)   
Receiving objects:  10% (266/2657)   Receiving objects:  11% (293/2657)   
Receiving objects:  12% (319/2657)   Receiving objects:  13% (346/2657)   
Receiving objects:  14% (372/2657)   Receiving objects:  15% (399/2657)   
Receiving objects:  16% (426/2657)   Receiving objects:  17% (452/2657)   
Receiving objects:  18% (479/2657)   Receiving objects:  19% (505/2657)   
Receiving objects:  20% (532/2657)   Receiving objects:  21% (558/2657)   
Receiving objects:  22% (585/2657)   Receiving objects:  23% (612/2657)   
Receiving objects:  24% (638/2657)   Receiving objects:  25% (665/2657)   
Receiving objects:  26% (691/2657)   Receiving objects:  27% (718/2657)   
Receiving objects:  28% (744/2657)   Receiving objects:  29% (771/2657)   
Receiving objects:  30% (798/2657)   Receiving objects:  31% (824/2657)   
Receiving objects:  32% (851/2657)   Receiving objects:  33% (877/2657)   
Receiving objects:  34% (904/2657)   Receiving objects:  35% (930/2657)   
Receiving objects:  36% (957/2657)   Receiving objects:  37% (984/2657)   
Receiving objects:  38% (1010/2657)   Receiving objects:  39% (1037/2657)   
Receiving objects:  40% (1063/2657)   Receiving objects:  41% (1090/2657)   
Receiving objects:  42% (1116/2657)   Receiving objects:  43% (1143/2657)   
Receiving objects:  44% (1170/2657)   Receiving objects:  45% (1196/2657)   
Receiving objects:  46% (1223/2657)   Receiving objects:  47% (1249/2657)   
Receiving objects:  48% (1276/2657)   Receiving objects:  49% (1302/2657)   
Receiving objects:  50% (1329/2657)   Receiving objects:  51% (1356/2657)   
Receiving objects:  52% (1382/2657)   Receiving objects:  53% (1409/2657)   
Receiving objects:  54% (1435/2657)   Receiving objects:  55% (1462/2657)   
Receiving objects:  56% (1488/2657)   Receiving objects:  57% (1515/2657)   
Receiving objects:  58% (1542/2657)   Receiving objects:  59% (1568/2657)   
Receiving objects:  60% (1595/2657)   Receiving objects:  61% (1621/2657)   
Receiving objects:  62% (1648/2657)   Receiving objects:  63% (1674/2657)   
Receiving objects:  64% (1701/2657)   Receiving obj

[jira] [Created] (KAFKA-7238) Producer can't update Metadata when two brokers break down and restart the first one

[little brother ma (JIRA)]

little brother ma created KAFKA-7238:


 Summary: Producer  can't update Metadata  when two brokers break 
down and restart the first one  
 Key: KAFKA-7238
 URL: https://issues.apache.org/jira/browse/KAFKA-7238
 Project: Kafka
  Issue Type: Bug
  Components: clients
Affects Versions: 0.11.0.3, 0.11.0.1
 Environment: os: 14.04.1-Ubuntu
java: 1.8.0_151-b12
Reporter: little brother ma


There is a kafka cluster with two brokers( broker ids are 70 and 67),I create a 
topic named topic2 with 2 partitions and 1 replica, and partition 0  is on the 
broker 70,and partition 1  is on the broker 67.

 While sending datas， I stop broker 70 first, and wait 5 ms， stop broker 67 . 
And then ,I restart  broker 70,  the  producer client can't update metadata  
and don't  switch to connet broker 70，otherwise， it always try to connect 
broker 67.

The log before stop the broker:

send
The message size: 12, key: null, partition:0, offset:29
[2018-08-02 19:59:10,180] DEBUG Sending metadata request (type=MetadataRequest, 
topics=topic2) to node 10.93.132.67:9092 (id: 67 rack: null) 
(org.apache.kafka.clients.NetworkClient)
[2018-08-02 19:59:10,184] DEBUG Updated cluster metadata version 11 to 
Cluster(id = 3fL1MXQtRo6Ujmxa9dS3eA, nodes = [10.93.132.70:9092 (id: 70 rack: 
null), 10.93.132.67:9092 (id: 67 rack: null)], partitions = [Partition(topic = 
topic2, partition = 1, leader = 70, replicas = [70], isr = [70]), 
Partition(topic = topic2, partition = 0, leader = 67, replicas = [67], isr = 
[67])]) (org.apache.kafka.clients.Metadata)
send
The message size: 12, key: null, partition:1, offset:69
send
The message size: 12, key: null, partition:0, offset:30
send
The message size: 12, key: null, partition:1, offset:70
send
The message size: 12, key: null, partition:0, offset:31
send
The message size: 12, key: null, partition:1, offset:71
send
The message size: 12, key: null, partition:0, offset:32

 

 

The log after stop the broker 70  :

[2018-08-02 20:02:50,049] DEBUG Node 70 disconnected. 
(org.apache.kafka.clients.NetworkClient)
[2018-08-02 20:02:50,049] WARN Connection to node 70 could not be established. 
Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2018-08-02 20:02:50,049] DEBUG Give up sending metadata request since no node 
is available (org.apache.kafka.clients.NetworkClient)
[2018-08-02 20:02:50,100] DEBUG Give up sending metadata request since no node 
is available (org.apache.kafka.clients.NetworkClient)
[2018-08-02 20:02:50,151] DEBUG Initialize connection to node 10.93.132.70:9092 
(id: 70 rack: null) for sending metadata request 
(org.apache.kafka.clients.NetworkClient)
[2018-08-02 20:02:50,151] DEBUG Initiating connection to node 10.93.132.70:9092 
(id: 70 rack: null) (org.apache.kafka.clients.NetworkClient)
send
The message size: 12, key: null, partition:0, offset:95
[2018-08-02 20:02:51,156] DEBUG Connection with /10.93.132.70 disconnected 
(org.apache.kafka.common.network.Selector)
java.net.ConnectException: Connection refused: no further information
 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
 at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
 at 
org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:50)
 at 
org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:95)
 at 
org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:361)
 at org.apache.kafka.common.network.Selector.poll(Selector.java:326)
 at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:433)
 at 
org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:954)
 at java.lang.Thread.run(Thread.java:745)
[2018-08-02 20:02:51,157] DEBUG Node 70 disconnected. 
(org.apache.kafka.clients.NetworkClient)
[2018-08-02 20:02:51,157] WARN Connection to node 70 could not be established. 
Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2018-08-02 20:02:51,157] DEBUG Give up sending metadata request since no node 
is available (org.apache.kafka.clients.NetworkClient)
[2018-08-02 20:02:51,208] DEBUG Initialize connection to node 10.93.132.70:9092 
(id: 70 rack: null) for sending metadata request 
(org.apache.kafka.clients.NetworkClient)
[2018-08-02 20:02:51,208] DEBUG Initiating connection to node 10.93.132.70:9092 
(id: 70 rack: null) (org.apache.kafka.clients.NetworkClient)
[2018-08-02 20:02:52,216] DEBUG Connection with /10.93.132.70 disconnected 
(org.apache.kafka.common.network.Selector)
java.net.ConnectException: Connection refused: no further information
 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
 at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImp

Re: [VOTE] KIP-334 Include partitions in exceptions raised during consumer record deserialization/validation

[Ted Yu]

+1
 Original message From: Stanislav Kozlovski 
 Date: 8/2/18  2:41 AM  (GMT-08:00) To: 
dev@kafka.apache.org Subject: [VOTE] KIP-334 Include partitions in exceptions 
raised during consumer record deserialization/validation 
Hey everybody,

I'd like to start a vote thread for KIP-334 Include partitions in
exceptions raised during consumer record deserialization/validation


-- 
Best,
Stanislav

Build failed in Jenkins: kafka-trunk-jdk10 #355

[Apache Jenkins Server]

See 


Changes:

[rajinisivaram] MINOR: Implement toString() in config validator classes (#5401)

--
[...truncated 1.98 MB...]
org.apache.kafka.streams.StreamsConfigTest > shouldAcceptExactlyOnce PASSED

org.apache.kafka.streams.StreamsConfigTest > 
testGetGlobalConsumerConfigsWithGlobalConsumerOverridenPrefix STARTED

org.apache.kafka.streams.StreamsConfigTest > 
testGetGlobalConsumerConfigsWithGlobalConsumerOverridenPrefix PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSetInternalLeaveGroupOnCloseConfigToFalseInConsumer STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSetInternalLeaveGroupOnCloseConfigToFalseInConsumer PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedPropertiesThatAreNotPartOfConsumerConfig STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedPropertiesThatAreNotPartOfConsumerConfig PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldForwardCustomConfigsWithNoPrefixToAllClients STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldForwardCustomConfigsWithNoPrefixToAllClients PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldResetToDefaultIfRestoreConsumerAutoCommitIsOverridden STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldResetToDefaultIfRestoreConsumerAutoCommitIsOverridden PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSpecifyCorrectValueSerdeClassOnError STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSpecifyCorrectValueSerdeClassOnError PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldResetToDefaultIfGlobalConsumerAutoCommitIsOverridden STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldResetToDefaultIfGlobalConsumerAutoCommitIsOverridden PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldResetToDefaultIfConsumerIsolationLevelIsOverriddenIfEosEnabled STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldResetToDefaultIfConsumerIsolationLevelIsOverriddenIfEosEnabled PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldBeSupportNonPrefixedGlobalConsumerConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldBeSupportNonPrefixedGlobalConsumerConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportNonPrefixedAdminConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportNonPrefixedAdminConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldNotOverrideUserConfigCommitIntervalMsIfExactlyOnceEnabled STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldNotOverrideUserConfigCommitIntervalMsIfExactlyOnceEnabled PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportMultipleBootstrapServers STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportMultipleBootstrapServers PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSpecifyNoOptimizationWhenNotExplicitlyAddedToConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSpecifyNoOptimizationWhenNotExplicitlyAddedToConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldThrowConfigExceptionWhenOptimizationConfigNotValueInRange STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldThrowConfigExceptionWhenOptimizationConfigNotValueInRange PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldThrowStreamsExceptionIfKeySerdeConfigFails STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldThrowStreamsExceptionIfKeySerdeConfigFails PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportNonPrefixedProducerConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportNonPrefixedProducerConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > testGetRestoreConsumerConfigs 
STARTED

org.apache.kafka.streams.StreamsConfigTest > testGetRestoreConsumerConfigs 
PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldThrowExceptionIfBootstrapServersIsNotSet STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldThrowExceptionIfBootstrapServersIsNotSet PASSED

org.apache.kafka.streams.StreamsConfigTest > 
consumerConfigMustUseAdminClientConfigForRetries STARTED

org.apache.kafka.streams.StreamsConfigTest > 
consumerConfigMustUseAdminClientConfigForRetries PASSED

org.apache.kafka.streams.StreamsConfigTest > 
testGetMainConsumerConfigsWithMainConsumerOverridenPrefix STARTED

org.apache.kafka.streams.StreamsConfigTest > 
testGetMainConsumerConfigsWithMainConsumerOverridenPrefix PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldBeSupportNonPrefixedRestoreConsumerConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldBeSupportNonPrefixedRestoreConsumerConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedRestoreConsumerConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedRestoreConsumerConfigs PA

Re: [Vote] KIP-321: Update TopologyDescription to better represent Source and Sink Nodes

[Guozhang Wang]

I think leaving the current return value to be null-able is okay, as long
as it is well documented in java doc.


Guozhang

On Thu, Aug 2, 2018 at 3:13 AM, Damian Guy  wrote:

> You have 3 binding votes, so i'll defer to the others.
>
> On Thu, 2 Aug 2018 at 04:41 Nishanth Pradeep 
> wrote:
>
> > The only issue I see with this is that Sink#topic would also need to be
> > Optional as was pointed out already. Since Sink#topic is a preexisting
> > method, changing its return type would break backwards compatibility.
> >
> > On the other hand, it might be worth it to rip that bandaid now.
> >
> > Best,
> > Nishanth Pradeep
> >
> > On Wed, Aug 1, 2018 at 11:56 AM Guozhang Wang 
> wrote:
> >
> >> For source node, only one of `Set topicsSet` and `TopicPattern
> >> topicPattern()` will be specified by the user. Similarly for sink node,
> >> only one of `String` and `TopicNameExtractor` will be specified by the
> >> user. Although I've not seen Nishanth's updated PR, I think when it is
> not
> >> specified today we will return null in that case.
> >>
> >> If we want to improve on this situation with Optional, we'd need to do
> it
> >> on all of these functions. Also note that for `Source#toString()` and
> >> `Sink#toString()` we should only include the specified field in the
> >> resulted representation.
> >>
> >>
> >> Guozhang
> >>
> >> On Wed, Aug 1, 2018 at 5:08 AM, Damian Guy 
> wrote:
> >>
> >> > Ewen - no as I don't believe they are never null. Whereas the
> >> > topicNameExtractor method returns null if it is the default extractor
> or
> >> > the extractor. So i think this would be better to be optional as it is
> >> > optionally returning a TopicNameExtractor
> >> >
> >> > On Tue, 31 Jul 2018 at 23:01 Ewen Cheslack-Postava  >
> >> > wrote:
> >> >
> >> > > Generally +1 (binding)
> >> > >
> >> > > It would be helpful to just provide the full, updated interfaces in
> >> the
> >> > > KIP and mark things as new with comments if needed. I had to go back
> >> and
> >> > > read the discussion thread to make sure I was understanding the
> intent
> >> > > correctly.
> >> > >
> >> > > Damian -- if we make that Optional, shouldn't the methods on Source
> >> also
> >> > > be Optional types?
> >> > >
> >> > > -Ewen
> >> > >
> >> > > On Mon, Jul 30, 2018 at 11:13 PM Damian Guy 
> >> > wrote:
> >> > >
> >> > >> Hi Nishanth,
> >> > >>
> >> > >> I have one nit on the KIP. I think the topicNameExtractor method
> >> should
> >> > >> return Optional rather than null.
> >> > >> Sorry I'm late here.
> >> > >>
> >> > >> Thanks,
> >> > >> Damian
> >> > >>
> >> > >> On Tue, 31 Jul 2018 at 01:14 Nishanth Pradeep <
> nishanth...@gmail.com
> >> >
> >> > >> wrote:
> >> > >>
> >> > >> > We need one more binding vote.
> >> > >> >
> >> > >> > Binding Votes:
> >> > >> >
> >> > >> >- Matthias J. Sax
> >> > >> >- Guozhang Wong
> >> > >> >
> >> > >> > Community Votes:
> >> > >> >
> >> > >> >- Bill Bejeck
> >> > >> >- Ted Yu
> >> > >> >
> >> > >> > Best,
> >> > >> > Nishanth Pradeep
> >> > >> >
> >> > >> > On Fri, Jul 27, 2018 at 10:02 AM Bill Bejeck 
> >> > wrote:
> >> > >> >
> >> > >> > > Thanks for the KIP!
> >> > >> > >
> >> > >> > > +1
> >> > >> > >
> >> > >> > > -Bill
> >> > >> > >
> >> > >> > > On Thu, Jul 26, 2018 at 2:39 AM Guozhang Wang <
> >> wangg...@gmail.com>
> >> > >> > wrote:
> >> > >> > >
> >> > >> > > > +1
> >> > >> > > >
> >> > >> > > > On Wed, Jul 25, 2018 at 11:13 PM, Matthias J. Sax <
> >> > >> > matth...@confluent.io
> >> > >> > > >
> >> > >> > > > wrote:
> >> > >> > > >
> >> > >> > > > > +1 (binding)
> >> > >> > > > >
> >> > >> > > > > -Matthias
> >> > >> > > > >
> >> > >> > > > > On 7/25/18 7:47 PM, Ted Yu wrote:
> >> > >> > > > > > +1
> >> > >> > > > > >
> >> > >> > > > > > On Wed, Jul 25, 2018 at 7:24 PM Nishanth Pradeep <
> >> > >> > > > nishanth...@gmail.com>
> >> > >> > > > > > wrote:
> >> > >> > > > > >
> >> > >> > > > > >> Hello,
> >> > >> > > > > >>
> >> > >> > > > > >> I'm calling a vote for KIP-321:
> >> > >> > > > > >>
> >> > >> > > > > >>
> >> > >> > > > > >>
> >> > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >> > 321%3A+Update+
> >> > >> > > > > TopologyDescription+to+better+represent+Source+and+Sink+
> Nodes
> >> > >> > > > > >>
> >> > >> > > > > >> Best,
> >> > >> > > > > >> Nishanth Pradeep
> >> > >> > > > > >>
> >> > >> > > > > >
> >> > >> > > > >
> >> > >> > > > >
> >> > >> > > >
> >> > >> > > >
> >> > >> > > > --
> >> > >> > > > -- Guozhang
> >> > >> > > >
> >> > >> > >
> >> > >> >
> >> > >>
> >> > >
> >> >
> >>
> >>
> >>
> >> --
> >> -- Guozhang
> >>
> >
>



-- 
-- Guozhang

Jenkins build is back to normal : kafka-trunk-jdk8 #2855

[Apache Jenkins Server]

See 

Re: [VOTE] KIP-334 Include partitions in exceptions raised during consumer record deserialization/validation

[Jason Gustafson]

Hey Stanislav,

I should have noticed it earlier from your example, but I just realized
that interfaces don't mix well with exceptions. There is no way to catch
the interface type, which means you have to depend on instanceof checks,
which is not very conventional. At the moment, we raise
SerializationException if there is a problem parsing the error, and we
raise KafkaException if the record fails its CRC check. Since
SerializationException extends KafkaExeption, it seems like we can handle
both cases in a compatible way by handling both cases with a single type
that extends SerializationException. Maybe something like
RecordDeserializationException?

Thanks,
Jason

On Thu, Aug 2, 2018 at 5:45 AM, Ted Yu  wrote:

> +1
>  Original message From: Stanislav Kozlovski <
> stanis...@confluent.io> Date: 8/2/18  2:41 AM  (GMT-08:00) To:
> dev@kafka.apache.org Subject: [VOTE] KIP-334 Include partitions in
> exceptions raised during consumer record deserialization/validation
> Hey everybody,
>
> I'd like to start a vote thread for KIP-334 Include partitions in
> exceptions raised during consumer record deserialization/validation
>  >
>
> --
> Best,
> Stanislav
>

Re: [DISCUSS] KIP-351: Add --critical-partitions option to describe topics command

[Kevin Lu]

Hi Mickael,

Thanks for the suggestion!

Correct me if I am mistaken, but if a producer attempts to send to a
partition that is under min ISR (and ack=all or -1) then the send will fail
with a NotEnoughReplicas or NotEnoughReplicasAfterAppend exception? At this
point, client-side has already suffered failure but the server-side is
still fine for now?

If the above is true, then this would be a FATAL case for producers.

Would it be valuable to include the CRITICAL case where a topic partition
has exactly min ISR so that Kafka operators can take action so it does not
become FATAL? This could be in the same option or a new one.

Thanks!

Regards,
Kevin

On Thu, Aug 2, 2018 at 2:27 AM Mickael Maison 
wrote:

> What about also adding a --under-minisr-partitions option?
>
> That would match the
> "kafka.server:type=ReplicaManager,name=UnderMinIsrPartitionCount"
> broker metric and it's usually pretty relevant when investigating
> issues
>
> On Thu, Aug 2, 2018 at 8:54 AM, Kevin Lu  wrote:
> > Hi friends!
> >
> > This thread is to discuss KIP-351
> > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-351%3A+Add+--critical-partitions+option+to+describe+topics+command
> >
> > !
> >
> > I am proposing to add a --critical-partitions option to the describe
> topics
> > command that will only list out topic partitions that have 1 ISR left
> (RF >
> > 1) as they would be in a critical state and need immediate
> repartitioning.
> >
> > I wonder if the name "critical" is appropriate?
> >
> > Thoughts?
> >
> > Thanks!
> >
> > Regards,
> > Kevin
>

Re: [DISCUSS] KIP-346 - Limit blast radius of log compaction failure

[Colin McCabe]

On Wed, Aug 1, 2018, at 11:35, James Cheng wrote:
> I’m a little confused about something. Is this KIP focused on log 
> cleaner exceptions in general, or focused on log cleaner exceptions due 
> to disk failures?
> 
> Will max.uncleanable.partitions apply to all exceptions (including log 
> cleaner logic errors) or will it apply to only disk I/o exceptions?

There is no difference between "log cleaner exceptions in general" and "log 
cleaner exceptions due to disk failures."

For example, if the data on disk is corrupted we might read a 4-byte size as -1 
instead of 100.  Then we would get a BufferUnderFlowException later on.  This 
is a subclass of RuntimeException rather than IOException, of course, but it 
does result from a disk problem.  Or we might get exceptions while validating 
checksums, which may or may not be IOE (I haven't looked).

Of course, the log cleaner itself may have a bug, which results in it throwing 
an exception even if the disk does not have a problem.  We clearly want to fix 
these bugs.  But there's no way for the program itself to know that it has a 
bug and act differently.  If an exception occurs, we must assume there is a 
disk problem.

> 
> I can understand taking the disk offline if there have been “N” I/O 
> exceptions. Disk errors are user fixable (by replacing the affected 
> disk). It turns an invisible (soft?) failure into a visible hard 
> failure. And the I/O exceptions are possibly already causing problems, 
> so it makes sense to limit their impact.
> 
> But I’m not sure if it makes sense to take a disk offline after”N” logic 
> errors in the log cleaner. If a log cleaner logic error happens, it’s 
> rarely user fixable. And it will likely several partitions at once, so 
> you’re likely to bump up against the max.uncleanable.partitions limit 
> more quickly. If a disk was taken due to logic errors, I’m not sure what 
> the user would do.

I don't agree that log cleaner bugs "will likely [affect] several partitions at 
once."  Most of the ones I've looked at only affect one or two partitions.  In 
particular the ones that resulted from over-eagerness to use 32-bit math on 
64-bit values.

If the log cleaner is so buggy that it's useless (the scenario you're 
describing), and you want to put off an upgrade, then you can set 
max.uncleanable.partitions to the maximum value to ignore failures.

best,
Colin


> 
> -James
> 
> Sent from my iPhone
> 
> > On Aug 1, 2018, at 9:11 AM, Stanislav Kozlovski  
> > wrote:
> > 
> > Yes, good catch. Thank you, James!
> > 
> > Best,
> > Stanislav
> > 
> >> On Wed, Aug 1, 2018 at 5:05 PM James Cheng  wrote:
> >> 
> >> Can you update the KIP to say what the default is for
> >> max.uncleanable.partitions?
> >> 
> >> -James
> >> 
> >> Sent from my iPhone
> >> 
> >>> On Jul 31, 2018, at 9:56 AM, Stanislav Kozlovski 
> >> wrote:
> >>> 
> >>> Hey group,
> >>> 
> >>> I am planning on starting a voting thread tomorrow. Please do reply if
> >> you
> >>> feel there is anything left to discuss.
> >>> 
> >>> Best,
> >>> Stanislav
> >>> 
> >>> On Fri, Jul 27, 2018 at 11:05 PM Stanislav Kozlovski <
> >> stanis...@confluent.io>
> >>> wrote:
> >>> 
>  Hey, Ray
>  
>  Thanks for pointing that out, it's fixed now
>  
>  Best,
>  Stanislav
>  
> > On Fri, Jul 27, 2018 at 9:43 PM Ray Chiang  wrote:
> > 
> > Thanks.  Can you fix the link in the "KIPs under discussion" table on
> > the main KIP landing page
> > <
> > 
> >> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals#
> >>> ?
> > 
> > I tried, but the Wiki won't let me.
> > 
> > -Ray
> > 
> >> On 7/26/18 2:01 PM, Stanislav Kozlovski wrote:
> >> Hey guys,
> >> 
> >> @Colin - good point. I added some sentences mentioning recent
> > improvements
> >> in the introductory section.
> >> 
> >> *Disk Failure* - I tend to agree with what Colin said - once a disk
> > fails,
> >> you don't want to work with it again. As such, I've changed my mind
> >> and
> >> believe that we should mark the LogDir (assume its a disk) as offline
> >> on
> >> the first `IOException` encountered. This is the LogCleaner's current
> >> behavior. We shouldn't change that.
> >> 
> >> *Respawning Threads* - I believe we should never re-spawn a thread.
> >> The
> >> correct approach in my mind is to either have it stay dead or never
> >> let
> > it
> >> die in the first place.
> >> 
> >> *Uncleanable-partition-names metric* - Colin is right, this metric is
> >> unneeded. Users can monitor the `uncleanable-partitions-count` metric
> > and
> >> inspect logs.
> >> 
> >> 
> >> Hey Ray,
> >> 
> >>> 2) I'm 100% with James in agreement with setting up the LogCleaner to
> >>> skip over problematic partitions instead of dying.
> >> I think we can do this for every exception that isn't `IOException`.
> > This
> >> will

Re: [Vote] KIP-321: Update TopologyDescription to better represent Source and Sink Nodes

[Matthias J. Sax]

I agree with Guozhang.

Breaking compatibility is not acceptable.

If we want the change to use `Optional`, we should deprecate the current
method and explain that it return type will change in next major release
3.0.0 and create a ticket for this change that we can tackle when time
comes.



-Matthias

On 8/2/18 9:10 AM, Guozhang Wang wrote:
> I think leaving the current return value to be null-able is okay, as long
> as it is well documented in java doc.
> 
> 
> Guozhang
> 
> On Thu, Aug 2, 2018 at 3:13 AM, Damian Guy  wrote:
> 
>> You have 3 binding votes, so i'll defer to the others.
>>
>> On Thu, 2 Aug 2018 at 04:41 Nishanth Pradeep 
>> wrote:
>>
>>> The only issue I see with this is that Sink#topic would also need to be
>>> Optional as was pointed out already. Since Sink#topic is a preexisting
>>> method, changing its return type would break backwards compatibility.
>>>
>>> On the other hand, it might be worth it to rip that bandaid now.
>>>
>>> Best,
>>> Nishanth Pradeep
>>>
>>> On Wed, Aug 1, 2018 at 11:56 AM Guozhang Wang 
>> wrote:
>>>
 For source node, only one of `Set topicsSet` and `TopicPattern
 topicPattern()` will be specified by the user. Similarly for sink node,
 only one of `String` and `TopicNameExtractor` will be specified by the
 user. Although I've not seen Nishanth's updated PR, I think when it is
>> not
 specified today we will return null in that case.

 If we want to improve on this situation with Optional, we'd need to do
>> it
 on all of these functions. Also note that for `Source#toString()` and
 `Sink#toString()` we should only include the specified field in the
 resulted representation.


 Guozhang

 On Wed, Aug 1, 2018 at 5:08 AM, Damian Guy 
>> wrote:

> Ewen - no as I don't believe they are never null. Whereas the
> topicNameExtractor method returns null if it is the default extractor
>> or
> the extractor. So i think this would be better to be optional as it is
> optionally returning a TopicNameExtractor
>
> On Tue, 31 Jul 2018 at 23:01 Ewen Cheslack-Postava >>
> wrote:
>
>> Generally +1 (binding)
>>
>> It would be helpful to just provide the full, updated interfaces in
 the
>> KIP and mark things as new with comments if needed. I had to go back
 and
>> read the discussion thread to make sure I was understanding the
>> intent
>> correctly.
>>
>> Damian -- if we make that Optional, shouldn't the methods on Source
 also
>> be Optional types?
>>
>> -Ewen
>>
>> On Mon, Jul 30, 2018 at 11:13 PM Damian Guy 
> wrote:
>>
>>> Hi Nishanth,
>>>
>>> I have one nit on the KIP. I think the topicNameExtractor method
 should
>>> return Optional rather than null.
>>> Sorry I'm late here.
>>>
>>> Thanks,
>>> Damian
>>>
>>> On Tue, 31 Jul 2018 at 01:14 Nishanth Pradeep <
>> nishanth...@gmail.com
>
>>> wrote:
>>>
 We need one more binding vote.

 Binding Votes:

- Matthias J. Sax
- Guozhang Wong

 Community Votes:

- Bill Bejeck
- Ted Yu

 Best,
 Nishanth Pradeep

 On Fri, Jul 27, 2018 at 10:02 AM Bill Bejeck 
> wrote:

> Thanks for the KIP!
>
> +1
>
> -Bill
>
> On Thu, Jul 26, 2018 at 2:39 AM Guozhang Wang <
 wangg...@gmail.com>
 wrote:
>
>> +1
>>
>> On Wed, Jul 25, 2018 at 11:13 PM, Matthias J. Sax <
 matth...@confluent.io
>>
>> wrote:
>>
>>> +1 (binding)
>>>
>>> -Matthias
>>>
>>> On 7/25/18 7:47 PM, Ted Yu wrote:
 +1

 On Wed, Jul 25, 2018 at 7:24 PM Nishanth Pradeep <
>> nishanth...@gmail.com>
 wrote:

> Hello,
>
> I'm calling a vote for KIP-321:
>
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 321%3A+Update+
>>> TopologyDescription+to+better+represent+Source+and+Sink+
>> Nodes
>
> Best,
> Nishanth Pradeep
>

>>>
>>>
>>
>>
>> --
>> -- Guozhang
>>
>

>>>
>>
>



 --
 -- Guozhang

>>>
>>
> 
> 
> 



signature.asc
Description: OpenPGP digital signature

Build failed in Jenkins: kafka-trunk-jdk10 #356

[Apache Jenkins Server]

See 

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on H31 (ubuntu xenial) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from 
https://github.com/apache/kafka.git
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:888)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1155)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186)
at hudson.scm.SCM.checkout(SCM.java:504)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
at hudson.model.Run.execute(Run.java:1794)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
Caused by: hudson.plugins.git.GitException: Command "git fetch --tags 
--progress https://github.com/apache/kafka.git 
+refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: error: Could not read 24d0d3351a43f52f6f688143672d7c57c5f19595
error: Could not read 1e98b2aa5027eb60ee1a112a7f8ec45b9c47
error: Could not read 689bac3c6fcae927ccee89c8d4fd54c2fae83be1
remote: Counting objects: 4146, done.
remote: Compressing objects:   4% (1/24)   remote: Compressing objects: 
  8% (2/24)   remote: Compressing objects:  12% (3/24)   
remote: Compressing objects:  16% (4/24)   remote: Compressing objects: 
 20% (5/24)   remote: Compressing objects:  25% (6/24)   
remote: Compressing objects:  29% (7/24)   remote: Compressing objects: 
 33% (8/24)   remote: Compressing objects:  37% (9/24)   
remote: Compressing objects:  41% (10/24)   remote: Compressing 
objects:  45% (11/24)   remote: Compressing objects:  50% (12/24)   
remote: Compressing objects:  54% (13/24)   remote: Compressing 
objects:  58% (14/24)   remote: Compressing objects:  62% (15/24)   
remote: Compressing objects:  66% (16/24)   remote: Compressing 
objects:  70% (17/24)   remote: Compressing objects:  75% (18/24)   
remote: Compressing objects:  79% (19/24)   remote: Compressing 
objects:  83% (20/24)   remote: Compressing objects:  87% (21/24)   
remote: Compressing objects:  91% (22/24)   remote: Compressing 
objects:  95% (23/24)   remote: Compressing objects: 100% (24/24)   
remote: Compressing objects: 100% (24/24), done.
Receiving objects:   0% (1/4146)   Receiving objects:   1% (42/4146)   
Receiving objects:   2% (83/4146)   Receiving objects:   3% (125/4146)   
Receiving objects:   4% (166/4146)   Receiving objects:   5% (208/4146)   
Receiving objects:   6% (249/4146)   Receiving objects:   7% (291/4146)   
Receiving objects:   8% (332/4146)   Receiving objects:   9% (374/4146)   
Receiving objects:  10% (415/4146)   Receiving objects:  11% (457/4146)   
Receiving objects:  12% (498/4146)   Receiving objects:  13% (539/4146)   
Receiving objects:  14% (581/4146)   Receiving objects:  15% (622/4146)   
Receiving objects:  16% (664/4146)   Receiving objects:  17% (705/4146)   
Receiving objects:  18% (747/4146)   Receiving objects:  19% (788/4146)   
Receiving objects:  20% (830/4146)   Receiving objects:  21% (871/4146)   
Receiving objects:  22% (913/4146)   Receiving objects:  23% (954/4146)   
Receiving objects:  24% (996/4146)   Receiving objects:  25% (1037/4146)   
Receiving objects:  26% (1078/4146)   Receiving objects:  27% (1120/4146)   
Receiving objects:  28% (1161/4146)   Receiving objects:  29% (1203/4146)   
Receiving objects:  30% (1244/4146)   Receiving objects:  31% (1286/4146)   
Receiving objects:  32% (1327/4146)   Receiving objects:  33% (1369/4146)   
Receiving objects:  34% (1410/4146)   Receiving objects:  35% (1452/4146)   
Receiving objects:  36% (1493/4146)   Receiving objects:  37% (1535/4146)   
Receiving objects:  38% (1576/4146)   Receiving objects:  39% (1617/4146)   
Receiving objects:  40% (1659/4146)   Receiving objects:  41% (1700/41

Re: [DISCUSS] KIP-289: Improve the default group id behavior in KafkaConsumer

[Colin McCabe]

Thanks, Jason.  I don't have a very strong opinion on this.  But like you said, 
if we skip bumping the RPC versions, this would be a smaller change, which 
might be good.

best,
Colin


On Wed, Aug 1, 2018, at 17:43, Jason Gustafson wrote:
> Hey Vahid,
> 
> I talked with Colin offline. I think specifically he felt the version bump
> on the broker was overkill since the broker still has to support the empty
> group id for older versions. I had thought that eventually we would be able
> to remove those old versions, but it's true that this may not happen until
> indefinitely far in the future. I think the main improvement here is
> changing the default group.id to null instead of "". I could go either way
> on whether bumping the protocol is useful. I do think it is helpful though
> to signal clearly that it its use is deprecated and discouraged, especially
> in light of the ACL problem. I guess we could just deprecate the use on the
> client. What do you think?
> 
> Thanks,
> Jason
> 
> On Wed, Aug 1, 2018 at 3:19 PM, Vahid S Hashemian  > wrote:
> 
> > Thanks Jason for responding to Colin's concerns.
> >
> > If there are no other comment / feedback / objection I'll start a vote
> > soon.
> >
> > Thanks.
> > --Vahid
> >
> >
> >
> > From:   Jason Gustafson 
> > To: dev 
> > Date:   07/27/2018 10:38 AM
> > Subject:Re: [DISCUSS] KIP-289: Improve the default group id
> > behavior in KafkaConsumer
> >
> >
> >
> > Hey Colin,
> >
> > The problem is both that the empty group id is the default value and that
> > it is actually accepted by the broker for offset commits. Combine that
> > with
> > the fact that auto commit is enabled by default and you users get
> > surprising behavior. If you look at a random Kafka cluster, you'll
> > probably
> > find a bunch of inadvertent offset commits for the empty group id. I was
> > hoping we could distinguish between users who are using the empty group id
> > as an accident of the default configuration and those who use it
> > intentionally. By default, there will be no group id and the consumer will
> > not commit offsets. If a user has actually intentionally used the empty
> > group id, however, it will continue to work. I actually think there are
> > probably very few people doing this (maybe even no one), but I thought we
> > might err on the side of compatibility.
> >
> > The big incompatible change here is having brokers reject using
> > assign(...)
> > > with empty / null group.id.
> >
> >
> > This is not correct. In the proposal, the broker will only reject the
> > empty
> > group id for the new version of OffsetCommit. Older clients, which cannot
> > be changed, will continue to work because the old versions of the
> > OffsetCommit API still accept the empty group id. The null group id is
> > different from the empty group id: it is not allowed in any version of the
> > API. It is basically a way to indicate that the consumer has no dependence
> > on the coordinator at all, which we actually have a surprising number of
> > use cases for. Furthermore, if a user has an actual need for the empty
> > group id, it will still be allowed. We are just deprecating it.
> >
> > -Jason
> >
> > On Fri, Jul 27, 2018 at 9:56 AM, Colin McCabe  wrote:
> >
> > > Sorry if this is a silly question, but what's the rationale for
> > switching
> > > to using null for the default group id, rather than the empty string?
> > > Continuing to use the empty string seems like less churn.  And after
> > all,
> > > we're not using the empty string group name for anything else.
> > >
> > > The big incompatible change here is having brokers reject using
> > > assign(...) with empty / null group.id.  If I understand correctly, the
> > > KIP proposes that this change be made on the brokers on the next
> > > incompatible Kafka release.  But that has nothing to do with client
> > > versions.  Why not just have a broker config which controls this?  Maybe
> > "
> > > allow.assign.empty.group.id", or something like that.  At first, the
> > > default will be true, and then eventually we can flip it over to false.
> > >
> > > It seems like the main rationale for tying this behavior to the Kafka
> > > client version is to force people to stop using the empty group id so
> > that
> > > they can upgrade their clients.  But it's also possible that people will
> > > stop upgrading their Kafka clients instead.  That would be pretty
> > negative
> > > since  they'd miss out on any efficiency and feature improvements in the
> > > new clients and eventually have to do more protocol downgrading, etc.
> > >
> > > best,
> > > Colin
> > >
> > >
> > > On Thu, Jul 26, 2018, at 11:50, Vahid S Hashemian wrote:
> > > > Hi Jason,
> > > >
> > > > That makes sense.
> > > > I have updated the KIP based on the recent feedback.
> > > >
> > > > Thanks!
> > > > --Vahid
> > > >
> > > >
> > > >
> > > >
> > > > From:   Jason Gustafson 
> > > > To: dev 
> > > > Date:   07/25/2018 02:23 PM
> > > > Subject:Re: [DISCUS

Re: [DISCUSS] KIP-332: Update AclCommand to use AdminClient API

[Colin McCabe]

+1 for starting the vote

cheers,
Colin


On Wed, Aug 1, 2018, at 08:46, Manikumar wrote:
> Hi all,
> 
> If there are no concerns, I will start the voting process soon.
> 
> Thanks
> 
> On Tue, Jul 31, 2018 at 9:08 AM Manikumar  wrote:
> 
> > Hi Colin,
> >
> > Yes,  "--authorizer-properties" option is not required with
> > "--bootstrap-server" option. Updated the KIP.
> >
> >
> > Thanks,
> >
> > On Tue, Jul 31, 2018 at 1:30 AM Ted Yu  wrote:
> >
> >> Look good to me.
> >>
> >> On Mon, Jul 23, 2018 at 7:30 AM Manikumar 
> >> wrote:
> >>
> >> > Hi all,
> >> >
> >> > I have created a KIP to use AdminClient API in AclCommand
> >> (kafka-acls.sh)
> >> >
> >> > *
> >> >
> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-332%3A+Update+AclCommand+to+use+AdminClient+API*
> >> > <
> >> >
> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-332%3A+Update+AclCommand+to+use+AdminClient+API
> >> > >
> >> >
> >> > Please take a look.
> >> >
> >> > Thanks,
> >> >
> >>
> >

Re: [DISCUSS] KIP-346 - Limit blast radius of log compaction failure

[Ray Chiang]

I see this as a fix for the LogCleaner.  Uncaught exceptions kill the 
CleanerThread and this is viewed as undesired behavior.  Some other ways 
to think of this fix:


1) If you have occasional corruption in some log segments, then with 
each broker restart, the LogCleaner will lose its state, re-find all the 
corrupted log segments, and will skip them in future runs.  In these 
cases, you will see a non-zero value for uncleanable-partitions-count 
and look in the broker logs to see if this is fixable in some way or it 
will decrement once the log segment is no longer retained.


2) If you have increasing disk corruption, then this is a way to 
potentially catch increasing corruption.  It's not a perfect approach, 
but as we've discussed before, hard drive failures tend to cascade.  
This is a useful side effect of LogCleaning.


3) If you have a non-zero uncleanable-partitions-count, you can look in 
the logs, compare the replicated partitions across brokers, use 
DumpLogSegments to possibly find/fix/delete the corrupted record(s).  
Just from the cases I've seen, this type of corruption is fixable 
roughly 30% of the time.


-Ray


On 8/1/18 11:35 AM, James Cheng wrote:

I’m a little confused about something. Is this KIP focused on log cleaner 
exceptions in general, or focused on log cleaner exceptions due to disk 
failures?

Will max.uncleanable.partitions apply to all exceptions (including log cleaner 
logic errors) or will it apply to only disk I/o exceptions?

I can understand taking the disk offline if there have been “N” I/O exceptions. 
Disk errors are user fixable (by replacing the affected disk). It turns an 
invisible (soft?) failure into a visible hard failure. And the I/O exceptions 
are possibly already causing problems, so it makes sense to limit their impact.

But I’m not sure if it makes sense to take a disk offline after”N” logic errors 
in the log cleaner. If a log cleaner logic error happens, it’s rarely user 
fixable. And it will likely several partitions at once, so you’re likely to 
bump up against the max.uncleanable.partitions limit more quickly. If a disk 
was taken due to logic errors, I’m not sure what the user would do.

-James

Sent from my iPhone


On Aug 1, 2018, at 9:11 AM, Stanislav Kozlovski  wrote:

Yes, good catch. Thank you, James!

Best,
Stanislav


On Wed, Aug 1, 2018 at 5:05 PM James Cheng  wrote:

Can you update the KIP to say what the default is for
max.uncleanable.partitions?

-James

Sent from my iPhone


On Jul 31, 2018, at 9:56 AM, Stanislav Kozlovski 

wrote:

Hey group,

I am planning on starting a voting thread tomorrow. Please do reply if

you

feel there is anything left to discuss.

Best,
Stanislav

On Fri, Jul 27, 2018 at 11:05 PM Stanislav Kozlovski <

stanis...@confluent.io>

wrote:


Hey, Ray

Thanks for pointing that out, it's fixed now

Best,
Stanislav


On Fri, Jul 27, 2018 at 9:43 PM Ray Chiang  wrote:

Thanks.  Can you fix the link in the "KIPs under discussion" table on
the main KIP landing page
<


https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals#

?

I tried, but the Wiki won't let me.

-Ray


On 7/26/18 2:01 PM, Stanislav Kozlovski wrote:
Hey guys,

@Colin - good point. I added some sentences mentioning recent

improvements

in the introductory section.

*Disk Failure* - I tend to agree with what Colin said - once a disk

fails,

you don't want to work with it again. As such, I've changed my mind

and

believe that we should mark the LogDir (assume its a disk) as offline

on

the first `IOException` encountered. This is the LogCleaner's current
behavior. We shouldn't change that.

*Respawning Threads* - I believe we should never re-spawn a thread.

The

correct approach in my mind is to either have it stay dead or never

let

it

die in the first place.

*Uncleanable-partition-names metric* - Colin is right, this metric is
unneeded. Users can monitor the `uncleanable-partitions-count` metric

and

inspect logs.


Hey Ray,


2) I'm 100% with James in agreement with setting up the LogCleaner to
skip over problematic partitions instead of dying.

I think we can do this for every exception that isn't `IOException`.

This

will future-proof us against bugs in the system and potential other

errors.

Protecting yourself against unexpected failures is always a good thing

in

my mind, but I also think that protecting yourself against bugs in the
software is sort of clunky. What does everybody think about this?


4) The only improvement I can think of is that if such an
error occurs, then have the option (configuration setting?) to

create a

.skip file (or something similar).

This is a good suggestion. Have others also seen corruption be

generally

tied to the same segment?

On Wed, Jul 25, 2018 at 11:55 AM Dhruvil Shah 

wrote:

For the cleaner thread specifically, I do not think respawning will

help at

all because we are more than likely to run into the same issue again

which

would end up crashing the cleaner

Re: [DISCUSS] KIP-346 - Limit blast radius of log compaction failure

[Ray Chiang]

One more thing occurred to me.  Should the configuration property be 
named "max.uncleanable.partitions.per.disk" instead?


-Ray


On 8/1/18 9:11 AM, Stanislav Kozlovski wrote:

Yes, good catch. Thank you, James!

Best,
Stanislav

On Wed, Aug 1, 2018 at 5:05 PM James Cheng  wrote:


Can you update the KIP to say what the default is for
max.uncleanable.partitions?

-James

Sent from my iPhone


On Jul 31, 2018, at 9:56 AM, Stanislav Kozlovski 

wrote:

Hey group,

I am planning on starting a voting thread tomorrow. Please do reply if

you

feel there is anything left to discuss.

Best,
Stanislav

On Fri, Jul 27, 2018 at 11:05 PM Stanislav Kozlovski <

stanis...@confluent.io>

wrote:


Hey, Ray

Thanks for pointing that out, it's fixed now

Best,
Stanislav


On Fri, Jul 27, 2018 at 9:43 PM Ray Chiang  wrote:

Thanks.  Can you fix the link in the "KIPs under discussion" table on
the main KIP landing page
<


https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals#

?

I tried, but the Wiki won't let me.

-Ray


On 7/26/18 2:01 PM, Stanislav Kozlovski wrote:
Hey guys,

@Colin - good point. I added some sentences mentioning recent

improvements

in the introductory section.

*Disk Failure* - I tend to agree with what Colin said - once a disk

fails,

you don't want to work with it again. As such, I've changed my mind

and

believe that we should mark the LogDir (assume its a disk) as offline

on

the first `IOException` encountered. This is the LogCleaner's current
behavior. We shouldn't change that.

*Respawning Threads* - I believe we should never re-spawn a thread.

The

correct approach in my mind is to either have it stay dead or never

let

it

die in the first place.

*Uncleanable-partition-names metric* - Colin is right, this metric is
unneeded. Users can monitor the `uncleanable-partitions-count` metric

and

inspect logs.


Hey Ray,


2) I'm 100% with James in agreement with setting up the LogCleaner to
skip over problematic partitions instead of dying.

I think we can do this for every exception that isn't `IOException`.

This

will future-proof us against bugs in the system and potential other

errors.

Protecting yourself against unexpected failures is always a good thing

in

my mind, but I also think that protecting yourself against bugs in the
software is sort of clunky. What does everybody think about this?


4) The only improvement I can think of is that if such an
error occurs, then have the option (configuration setting?) to

create a

.skip file (or something similar).

This is a good suggestion. Have others also seen corruption be

generally

tied to the same segment?

On Wed, Jul 25, 2018 at 11:55 AM Dhruvil Shah 

wrote:

For the cleaner thread specifically, I do not think respawning will

help at

all because we are more than likely to run into the same issue again

which

would end up crashing the cleaner. Retrying makes sense for transient
errors or when you believe some part of the system could have healed
itself, both of which I think are not true for the log cleaner.

On Wed, Jul 25, 2018 at 11:08 AM Ron Dagostino 

wrote:

<<
in

an

infinite loop which consumes resources and fires off continuous log
messages.
Hi Colin.  In case it could be relevant, one way to mitigate this

effect

is

to implement a backoff mechanism (if a second respawn is to occur

then

wait

for 1 minute before doing it; then if a third respawn is to occur

wait

for

2 minutes before doing it; then 4 minutes, 8 minutes, etc. up to

some

max

wait time).

I have no opinion on whether respawn is appropriate or not in this

context,

but a mitigation like the increasing backoff described above may be
relevant in weighing the pros and cons.

Ron

On Wed, Jul 25, 2018 at 1:26 PM Colin McCabe 

wrote:

On Mon, Jul 23, 2018, at 23:20, James Cheng wrote:
Hi Stanislav! Thanks for this KIP!

I agree that it would be good if the LogCleaner were more tolerant

of

errors. Currently, as you said, once it dies, it stays dead.

Things are better now than they used to be. We have the metric
   kafka.log:type=LogCleanerManager,name=time-since-last-run-ms
which we can use to tell us if the threads are dead. And as of

1.1.0,

we

have KIP-226, which allows you to restart the log cleaner thread,
without requiring a broker restart.


https://cwiki.apache.org/confluence/display/KAFKA/KIP-226+-+Dynamic+Broker+Configuration

<

https://cwiki.apache.org/confluence/display/KAFKA/KIP-226+-+Dynamic+Broker+Configuration

I've only read about this, I haven't personally tried it.

Thanks for pointing this out, James!  Stanislav, we should probably

add a

sentence or two mentioning the KIP-226 changes somewhere in the

KIP.

Maybe

in the intro section?

I think it's clear that requiring the users to manually restart the

log

cleaner is not a very good solution.  But it's good to know that

it's a

possibility on some older releases.


Some comments:
* I like the idea of having the log cleaner continue to c

Build failed in Jenkins: kafka-2.0-jdk8 #98

[Apache Jenkins Server]

See 

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on H34 (ubuntu xenial) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from 
https://github.com/apache/kafka.git
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:888)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1155)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186)
at hudson.scm.SCM.checkout(SCM.java:504)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
at hudson.model.Run.execute(Run.java:1794)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
Caused by: hudson.plugins.git.GitException: Command "git fetch --tags 
--progress https://github.com/apache/kafka.git 
+refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: error: Could not read 08c465028d057ac23cdfe6d57641fe40240359dd
error: missing object referenced by 'refs/tags/1.1.1-rc0'
error: Could not read 0f3affc0f40751dc8fd064b36b6e859728f63e37
error: Could not read 95fbb2e03f4fe79737c71632e0ef2dfdcfb85a69
error: Could not read 737bf43bb4e78d2d7a0ee53c27527b479972ebf8
error: Could not read 1ed1daefbc2d72e9b501b94d8c99e874b89f1137
remote: Counting objects: 6771, done.
remote: Compressing objects:   3% (1/27)   remote: Compressing objects: 
  7% (2/27)   remote: Compressing objects:  11% (3/27)   
remote: Compressing objects:  14% (4/27)   remote: Compressing objects: 
 18% (5/27)   remote: Compressing objects:  22% (6/27)   
remote: Compressing objects:  25% (7/27)   remote: Compressing objects: 
 29% (8/27)   remote: Compressing objects:  33% (9/27)   
remote: Compressing objects:  37% (10/27)   remote: Compressing 
objects:  40% (11/27)   remote: Compressing objects:  44% (12/27)   
remote: Compressing objects:  48% (13/27)   remote: Compressing 
objects:  51% (14/27)   remote: Compressing objects:  55% (15/27)   
remote: Compressing objects:  59% (16/27)   remote: Compressing 
objects:  62% (17/27)   remote: Compressing objects:  66% (18/27)   
remote: Compressing objects:  70% (19/27)   remote: Compressing 
objects:  74% (20/27)   remote: Compressing objects:  77% (21/27)   
remote: Compressing objects:  81% (22/27)   remote: Compressing 
objects:  85% (23/27)   remote: Compressing objects:  88% (24/27)   
remote: Compressing objects:  92% (25/27)   remote: Compressing 
objects:  96% (26/27)   remote: Compressing objects: 100% (27/27)   
remote: Compressing objects: 100% (27/27), done.
Receiving objects:   0% (1/6771)   Receiving objects:   1% (68/6771)   
Receiving objects:   2% (136/6771)   Receiving objects:   3% (204/6771)   
Receiving objects:   4% (271/6771)   Receiving objects:   5% (339/6771)   
Receiving objects:   6% (407/6771)   Receiving objects:   7% (474/6771)   
Receiving objects:   8% (542/6771)   Receiving objects:   9% (610/6771)   
Receiving objects:  10% (678/6771)   Receiving objects:  11% (745/6771)   
Receiving objects:  12% (813/6771)   Receiving objects:  13% (881/6771)   
Receiving objects:  14% (948/6771)   Receiving objects:  15% (1016/6771)   
Receiving objects:  16% (1084/6771)   Receiving objects:  17% (1152/6771)   
Receiving objects:  18% (1219/6771)   Receiving objects:  19% (1287/6771)   
Receiving objects:  20% (1355/6771)   Receiving objects:  21% (1422/6771)   
Receiving objects:  22% (1490/6771)   Receiving objects:  23% (1558/6771)   
Receiving objects:  24% (1626/6771)   Receiving objects:  25% (1693/6771)   
Receiving objects:  26% (1761/6771)   Receiving objects:  27% (1829/6771)   
Receiving objects:  28% (1896/6771)   Receiving objects:  29% (1964/6771)   
Receiving objects:  30% (2032/6771)   Receiving objects:  31% (2100/6771)   
Receiving objects:  32% (2167/67

KIP-352: Distinguish URPs caused by reassignment

[Jason Gustafson]

Hey All,

Another day, another KIP. This one is hopefully straightforward:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-352%3A+Distinguish+URPs+caused+by+reassignment.
Have a look and let me know what you think!

Thanks,
Jason

Re: [DISCUSS] KIP-289: Improve the default group id behavior in KafkaConsumer

[Vahid S Hashemian]

Hi Jason, Colin,

Thanks for your feedback.

If we are going to keep supporting the empty group id like before then it 
makes sense to do what you suggested.

Though, IMHO if using the empty group id for committing offsets is a bad 
practice we should stop supporting it at some point; and it would be 
better to do it sooner before potentially more users adopt that.
I understand that changing the default would reduce the chance of someone 
using it, but it doesn't stop them. Note that we did not receive any 
objections to dropping the support even right-away (a good sign).
We could keep supporting the old API version in case someone must use this 
group id like before. But at least we won't expose new users to this bad 
practice.

I, too, do now have a strong opinion about this. So if the current KIP 
should not move forward as is, I'm okay with just changing the default and 
deprecating the use on the client, as you suggested.

Thanks.
--Vahid



From:   Colin McCabe 
To: dev@kafka.apache.org
Date:   08/02/2018 11:23 AM
Subject:Re: [DISCUSS] KIP-289: Improve the default group id 
behavior in KafkaConsumer



Thanks, Jason.  I don't have a very strong opinion on this.  But like you 
said, if we skip bumping the RPC versions, this would be a smaller change, 
which might be good.

best,
Colin


On Wed, Aug 1, 2018, at 17:43, Jason Gustafson wrote:
> Hey Vahid,
> 
> I talked with Colin offline. I think specifically he felt the version 
bump
> on the broker was overkill since the broker still has to support the 
empty
> group id for older versions. I had thought that eventually we would be 
able
> to remove those old versions, but it's true that this may not happen 
until
> indefinitely far in the future. I think the main improvement here is
> changing the default group.id to null instead of "". I could go either 
way
> on whether bumping the protocol is useful. I do think it is helpful 
though
> to signal clearly that it its use is deprecated and discouraged, 
especially
> in light of the ACL problem. I guess we could just deprecate the use on 
the
> client. What do you think?
> 
> Thanks,
> Jason
> 
> On Wed, Aug 1, 2018 at 3:19 PM, Vahid S Hashemian 
 > wrote:
> 
> > Thanks Jason for responding to Colin's concerns.
> >
> > If there are no other comment / feedback / objection I'll start a vote
> > soon.
> >
> > Thanks.
> > --Vahid
> >
> >
> >
> > From:   Jason Gustafson 
> > To: dev 
> > Date:   07/27/2018 10:38 AM
> > Subject:Re: [DISCUSS] KIP-289: Improve the default group id
> > behavior in KafkaConsumer
> >
> >
> >
> > Hey Colin,
> >
> > The problem is both that the empty group id is the default value and 
that
> > it is actually accepted by the broker for offset commits. Combine that
> > with
> > the fact that auto commit is enabled by default and you users get
> > surprising behavior. If you look at a random Kafka cluster, you'll
> > probably
> > find a bunch of inadvertent offset commits for the empty group id. I 
was
> > hoping we could distinguish between users who are using the empty 
group id
> > as an accident of the default configuration and those who use it
> > intentionally. By default, there will be no group id and the consumer 
will
> > not commit offsets. If a user has actually intentionally used the 
empty
> > group id, however, it will continue to work. I actually think there 
are
> > probably very few people doing this (maybe even no one), but I thought 
we
> > might err on the side of compatibility.
> >
> > The big incompatible change here is having brokers reject using
> > assign(...)
> > > with empty / null group.id.
> >
> >
> > This is not correct. In the proposal, the broker will only reject the
> > empty
> > group id for the new version of OffsetCommit. Older clients, which 
cannot
> > be changed, will continue to work because the old versions of the
> > OffsetCommit API still accept the empty group id. The null group id is
> > different from the empty group id: it is not allowed in any version of 
the
> > API. It is basically a way to indicate that the consumer has no 
dependence
> > on the coordinator at all, which we actually have a surprising number 
of
> > use cases for. Furthermore, if a user has an actual need for the empty
> > group id, it will still be allowed. We are just deprecating it.
> >
> > -Jason
> >
> > On Fri, Jul 27, 2018 at 9:56 AM, Colin McCabe  
wrote:
> >
> > > Sorry if this is a silly question, but what's the rationale for
> > switching
> > > to using null for the default group id, rather than the empty 
string?
> > > Continuing to use the empty string seems like less churn.  And after
> > all,
> > > we're not using the empty string group name for anything else.
> > >
> > > The big incompatible change here is having brokers reject using
> > > assign(...) with empty / null group.id.  If I understand correctly, 
the
> > > KIP proposes that this change be made on the brokers on the next
> > > incompatible Kafka release.  But that has noth

Re: KIP-352: Distinguish URPs caused by reassignment

[Ismael Juma]

Thanks Jason. This is definitely a pain point. I actually prefer the option
to redefine what under-replicated means (currently under rejected
alternatives). Also, do we need to make changes to what we store in ZK? If
so, that should be in the KIP too.

Ismael

On Thu, Aug 2, 2018 at 11:45 AM Jason Gustafson  wrote:

> Hey All,
>
> Another day, another KIP. This one is hopefully straightforward:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-352%3A+Distinguish+URPs+caused+by+reassignment
> .
> Have a look and let me know what you think!
>
> Thanks,
> Jason
>

[jira] [Created] (KAFKA-7239) Kafka Connect secret externalization not working

[satyanarayan komandur (JIRA)]

satyanarayan komandur created KAFKA-7239:


 Summary: Kafka Connect secret externalization not working
 Key: KAFKA-7239
 URL: https://issues.apache.org/jira/browse/KAFKA-7239
 Project: Kafka
  Issue Type: Bug
Reporter: satyanarayan komandur


I used the Kafka FileConfigProvider to externalize the properties like 
connection.user and connection.password for JDBC source connector. I noticed 
that the values in the connection properties are being replaced after the 
connector has attempted to establish a connection with original key/value pairs 
(untransformed). This is resulting a failure in connection. I am not sure if 
this issue belong to Kafka Connector framework or its an issue with JDBC Source 
Connector



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Discussion: New components in JIRA?

[Guozhang Wang]

Hello Ray,

I've added these two components. People should be able to use them creating
/ updating the JIRAs now.


Guozhang

On Wed, Aug 1, 2018 at 12:56 PM, Ray Chiang  wrote:

> I haven't seen any comments.  Let me know if/when you add the new
> components.  Thanks.
>
> -Ray
>
>
>
> On 7/27/18 9:54 PM, Guozhang Wang wrote:
>
>> Hello Ray,
>>
>> Any PMC member of the project can add more components in the JIRA system.
>> If there is no objection in the next 72 hours I can just go ahead and add
>> them.
>>
>>
>> Guozhang
>>
>>
>> On Thu, Jul 26, 2018 at 1:50 PM, Ray Chiang  wrote:
>>
>> Thanks Guozhang.  I'm good with the way the documentation is now.
>>>
>>> Is there any other procedure to follow to get "logging" and "mirrormaker"
>>> added as components or can we just request a JIRA admin to do that on
>>> this
>>> list?
>>>
>>> -Ray
>>>
>>>
>>> On 7/23/18 4:56 PM, Guozhang Wang wrote:
>>>
>>> I've just updated the web docs on http://kafka.apache.org/contributing
 accordingly.

 On Mon, Jul 23, 2018 at 3:30 PM, khaireddine Rezgui <
 khaireddine...@gmail.com> wrote:

 Good job Ray for the wiki, it's clear enough.

> Le 23 juil. 2018 10:17 PM, "Ray Chiang"  a écrit :
>
> Okay, I've created a wiki page Reporting Issues in Apache Kafka
> <
> https://cwiki.apache.org/confluence/display/KAFKA/
> Reporting+Issues+in+Apache+Kafka>.
>
> I'd appreciate any feedback.  If this is good enough, I can file a JIRA
> to change the link under "Bugs" in the "Project information" page.
>
>
> -Ray
>
>
> On 7/23/18 11:28 AM, Ray Chiang wrote:
>
> Good point.  I'll look into adding some JIRA guidelines to the
>> documentation/wiki.
>>
>> -Ray
>>
>> On 7/22/18 10:23 AM, Guozhang Wang wrote:
>>
>> Hello Ray,
>>>
>>> Thanks for brining this up. I'm generally +1 on the first two, while
>>> for
>>> the last category, personally I felt leaving them as part of `tools`
>>> is
>>> fine, but I'm also open for other opinions.
>>>
>>> A more general question though, is that today we do not have any
>>> guidelines
>>> to ask JIRA reporters to set the right component, i.e. it is purely
>>> best-effort, and we cannot disallow reporters to add any new
>>> component
>>> names. And so far the project does not really have a tradition to
>>> manage
>>> JIRA reports per-component, as the goal is to not "separate" the
>>> project
>>> into silos but recommending everyone to get hands on every aspect of
>>> the
>>> project.
>>>
>>>
>>> Guozhang
>>>
>>>
>>> On Fri, Jul 20, 2018 at 2:44 PM, Ray Chiang 
>>> wrote:
>>>
>>> I've been doing a little bit of component cleanup in JIRA.  What do
>>>
 people
 think of adding
 one or more of the following components?

 - logging: For any consumer/producer/broker logging (i.e. log4j).
 This
 should help disambiguate from the "log" component (i.e. Kafka
 messages).

 - mirrormaker: There are enough requests specific to MirrorMaker
 that it
 could be put into its own component.

 - scripts: I'm a little more ambivalent about this one, but any of
 the
 bin/*.sh script fixes could belong in their own category.  I'm not
 sure if
 other people feel strongly for how the "tools" component should be
 used
 w.r.t. the run scripts.

 Any thoughts?

 -Ray





>>
>


-- 
-- Guozhang

Re: Discussion: New components in JIRA?

[Ray Chiang]

Great.  Thanks!

-Ray

On 8/2/18 12:28 PM, Guozhang Wang wrote:

Hello Ray,

I've added these two components. People should be able to use them creating
/ updating the JIRAs now.


Guozhang

On Wed, Aug 1, 2018 at 12:56 PM, Ray Chiang  wrote:


I haven't seen any comments.  Let me know if/when you add the new
components.  Thanks.

-Ray



On 7/27/18 9:54 PM, Guozhang Wang wrote:


Hello Ray,

Any PMC member of the project can add more components in the JIRA system.
If there is no objection in the next 72 hours I can just go ahead and add
them.


Guozhang


On Thu, Jul 26, 2018 at 1:50 PM, Ray Chiang  wrote:

Thanks Guozhang.  I'm good with the way the documentation is now.

Is there any other procedure to follow to get "logging" and "mirrormaker"
added as components or can we just request a JIRA admin to do that on
this
list?

-Ray


On 7/23/18 4:56 PM, Guozhang Wang wrote:

I've just updated the web docs on http://kafka.apache.org/contributing

accordingly.

On Mon, Jul 23, 2018 at 3:30 PM, khaireddine Rezgui <
khaireddine...@gmail.com> wrote:

Good job Ray for the wiki, it's clear enough.


Le 23 juil. 2018 10:17 PM, "Ray Chiang"  a écrit :

Okay, I've created a wiki page Reporting Issues in Apache Kafka
<
https://cwiki.apache.org/confluence/display/KAFKA/
Reporting+Issues+in+Apache+Kafka>.

I'd appreciate any feedback.  If this is good enough, I can file a JIRA
to change the link under "Bugs" in the "Project information" page.


-Ray


On 7/23/18 11:28 AM, Ray Chiang wrote:

Good point.  I'll look into adding some JIRA guidelines to the

documentation/wiki.

-Ray

On 7/22/18 10:23 AM, Guozhang Wang wrote:

Hello Ray,

Thanks for brining this up. I'm generally +1 on the first two, while
for
the last category, personally I felt leaving them as part of `tools`
is
fine, but I'm also open for other opinions.

A more general question though, is that today we do not have any
guidelines
to ask JIRA reporters to set the right component, i.e. it is purely
best-effort, and we cannot disallow reporters to add any new
component
names. And so far the project does not really have a tradition to
manage
JIRA reports per-component, as the goal is to not "separate" the
project
into silos but recommending everyone to get hands on every aspect of
the
project.


Guozhang


On Fri, Jul 20, 2018 at 2:44 PM, Ray Chiang 
wrote:

I've been doing a little bit of component cleanup in JIRA.  What do


people
think of adding
one or more of the following components?

- logging: For any consumer/producer/broker logging (i.e. log4j).
This
should help disambiguate from the "log" component (i.e. Kafka
messages).

- mirrormaker: There are enough requests specific to MirrorMaker
that it
could be put into its own component.

- scripts: I'm a little more ambivalent about this one, but any of
the
bin/*.sh script fixes could belong in their own category.  I'm not
sure if
other people feel strongly for how the "tools" component should be
used
w.r.t. the run scripts.

Any thoughts?

-Ray

[VOTE] KIP-341: Update Sticky Assignor's User Data Protocol

[Vahid S Hashemian]

Hi everyone,

I believe the feedback on this KIP has been addressed so far. So I'd like 
to start a vote.
The KIP: 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-341%3A+Update+Sticky+Assignor%27s+User+Data+Protocol
Discussion thread: 
https://www.mail-archive.com/dev@kafka.apache.org/msg89733.html

Thanks!
--Vahid

[jira] [Created] (KAFKA-7240) -total metrics in Streams are incorrect

[Sam Lendle (JIRA)]

Sam Lendle created KAFKA-7240:
-

 Summary: -total metrics in Streams are incorrect
 Key: KAFKA-7240
 URL: https://issues.apache.org/jira/browse/KAFKA-7240
 Project: Kafka
  Issue Type: Bug
  Components: metrics, streams
Affects Versions: 2.0.0
Reporter: Sam Lendle


I noticed the values of total metrics for streams were decreasing periodically 
when viewed in JMX, for example process-total for each processor-node-id under 
stream-processor-node-metrics. 

 Looking at StreamsMetricsThreadImpl, I believe this behavior is due to using 
Count() as the Stat for the *-total metrics. Count() is a SampledStat, so the 
value it reports is the count in recent time windows, and the value decreases 
whenever a window is purged.



This explains the behavior I saw, but I think the issue is deeper. For example, 
processTimeSensor attempts to measure, process-latency-avg, 
process-latency-max, process-rate, and process-total. For that sensor, record 
is called like
streamsMetrics.processTimeSensor.record(computeLatency() / (double) processed, 
timerStartedMs);
so the value passed to record is average latency per processed message in this 
batch if I understand correctly. That gets pushed through to the call to 
Count#record, which increments it's count by 1, ignoring the value parameter. 
Whatever stat is recording the total would need to know is the number of 
messages processed. Because of that, I don't think it's possible for one Sensor 
to measure both latency and total.

That said, it's not clear to me how all the different Stats work and how 
exactly Sensors work, and I don't actually understand how the process-rate 
metric is working for similar reasons but that seems to be correct, so I may be 
missing something here. 
 

cc [~guozhang]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

ConsumerGroupCommand tool improvement?

[Vahid S Hashemian]

Hi all,

A requirement has been raised by a colleague and I wanted to see if there 
is any interest in the community in adding the functionality to Apache 
Kafka.

ConsumerGroupCommand tool in describe ('--describe' or '--describe 
--offsets') mode currently lists all topics the group has consumed from 
and all consumers with assigned partitions for a single group.
The idea is to allow filtering of topics, consumers (client ids), and even 
groups using regular expressions. This will allow the tool to handle use 
cases such as
What's the status of a particular consumer (or consumers) in all the 
groups they are consuming from? (for example to check if they are lagging 
behind in all groups)
What consumer groups are consuming from a topic (or topics) and what's the 
lag for each group?
Limit the existing result to the topics/consumers of interest (for groups 
with several topics/consumers)
...

This would potentially lead to enhancing the AdminClient API as well.

If the community also sees a value in this, I could start drafting a KIP.

Thanks for your feedback.
--Vahid

Build failed in Jenkins: kafka-trunk-jdk10 #357

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-7231; Ensure NetworkClient uses overridden request timeout 
(#5444)

--
[...truncated 1.97 MB...]
org.apache.kafka.streams.processor.internals.InternalTopologyBuilderTest > 
shouldNotAllowToAddGlobalStoreWithSourceNameEqualsProcessorName PASSED

org.apache.kafka.streams.processor.internals.InternalTopologyBuilderTest > 
shouldAddSourceWithOffsetReset STARTED

org.apache.kafka.streams.processor.internals.InternalTopologyBuilderTest > 
shouldAddSourceWithOffsetReset PASSED

org.apache.kafka.streams.processor.internals.InternalTopologyBuilderTest > 
testAddStateStoreWithSource STARTED

org.apache.kafka.streams.processor.internals.InternalTopologyBuilderTest > 
testAddStateStoreWithSource PASSED

org.apache.kafka.streams.processor.internals.InternalTopologyBuilderTest > 
shouldAddInternalTopicConfigForRepartitionTopics STARTED

org.apache.kafka.streams.processor.internals.InternalTopologyBuilderTest > 
shouldAddInternalTopicConfigForRepartitionTopics PASSED

org.apache.kafka.streams.processor.internals.InternalTopologyBuilderTest > 
shouldNotAllowNullProcessorNameWhenConnectingProcessorAndStateStores STARTED

org.apache.kafka.streams.processor.internals.InternalTopologyBuilderTest > 
shouldNotAllowNullProcessorNameWhenConnectingProcessorAndStateStores PASSED

org.apache.kafka.streams.processor.internals.InternalTopologyBuilderTest > 
testAddSourceWithSameName STARTED

org.apache.kafka.streams.processor.internals.InternalTopologyBuilderTest > 
testAddSourceWithSameName PASSED

org.apache.kafka.streams.processor.internals.InternalTopologyBuilderTest > 
testAddMoreThanOnePatternSourceNode STARTED

org.apache.kafka.streams.processor.internals.InternalTopologyBuilderTest > 
testAddMoreThanOnePatternSourceNode PASSED

org.apache.kafka.streams.processor.internals.InternalTopologyBuilderTest > 
shouldAddTimestampExtractorWithPatternPerSource STARTED

org.apache.kafka.streams.processor.internals.InternalTopologyBuilderTest > 
shouldAddTimestampExtractorWithPatternPerSource PASSED

org.apache.kafka.streams.processor.internals.InternalTopologyBuilderTest > 
testAddProcessorWithSameName STARTED

org.apache.kafka.streams.processor.internals.InternalTopologyBuilderTest > 
testAddProcessorWithSameName PASSED

org.apache.kafka.streams.processor.internals.InternalTopologyBuilderTest > 
shouldSetCorrectSourceNodesWithRegexUpdatedTopics STARTED

org.apache.kafka.streams.processor.internals.InternalTopologyBuilderTest > 
shouldSetCorrectSourceNodesWithRegexUpdatedTopics PASSED

org.apache.kafka.streams.processor.internals.InternalTopologyBuilderTest > 
shouldConnectRegexMatchedTopicsToStateStore STARTED

org.apache.kafka.streams.processor.internals.InternalTopologyBuilderTest > 
shouldConnectRegexMatchedTopicsToStateStore PASSED

org.apache.kafka.streams.processor.internals.InternalTopologyBuilderTest > 
testAddSourceWithSameTopic STARTED

org.apache.kafka.streams.processor.internals.InternalTopologyBuilderTest > 
testAddSourceWithSameTopic PASSED

org.apache.kafka.streams.processor.internals.InternalTopologyBuilderTest > 
testTopicGroupsByStateStore STARTED

org.apache.kafka.streams.processor.internals.InternalTopologyBuilderTest > 
testTopicGroupsByStateStore PASSED

org.apache.kafka.streams.processor.internals.InternalTopologyBuilderTest > 
shouldAddSourceWithoutOffsetReset STARTED

org.apache.kafka.streams.processor.internals.InternalTopologyBuilderTest > 
shouldAddSourceWithoutOffsetReset PASSED

org.apache.kafka.streams.processor.internals.InternalTopologyBuilderTest > 
testAddStateStoreWithDuplicates STARTED

org.apache.kafka.streams.processor.internals.InternalTopologyBuilderTest > 
testAddStateStoreWithDuplicates PASSED

org.apache.kafka.streams.processor.internals.InternalTopologyBuilderTest > 
shouldNotAllowNullStateStoreNameWhenConnectingProcessorAndStateStores STARTED

org.apache.kafka.streams.processor.internals.InternalTopologyBuilderTest > 
shouldNotAllowNullStateStoreNameWhenConnectingProcessorAndStateStores PASSED

org.apache.kafka.streams.processor.internals.InternalTopologyBuilderTest > 
shouldAddTimestampExtractorPerSource STARTED

org.apache.kafka.streams.processor.internals.InternalTopologyBuilderTest > 
shouldAddTimestampExtractorPerSource PASSED

org.apache.kafka.streams.processor.internals.InternalTopologyBuilderTest > 
testPatternSourceTopic STARTED

org.apache.kafka.streams.processor.internals.InternalTopologyBuilderTest > 
testPatternSourceTopic PASSED

org.apache.kafka.streams.processor.internals.InternalTopologyBuilderTest > 
shouldAssociateStateStoreNameWhenStateStoreSupplierIsExternal STARTED

org.apache.kafka.streams.processor.internals.InternalTopologyBuilderTest > 
shouldAssociateStateStoreNameWhenStateStoreSupplierIsExternal PASSED

org.apache.kafka.streams.processor.internals.InternalTopologyBuilderTest > 
testAddProcessorWithEmptyParents S

[jira] [Resolved] (KAFKA-7125) Calling StreamsBuilderbuilder.build().describe() causes java.util.NoSuchElementException: null

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-7125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang resolved KAFKA-7125.
--
Resolution: Won't Fix

> Calling StreamsBuilderbuilder.build().describe() causes 
> java.util.NoSuchElementException: null
> --
>
> Key: KAFKA-7125
> URL: https://issues.apache.org/jira/browse/KAFKA-7125
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 1.1.0
>Reporter: Jouni
>Assignee: Nikolay Izhikov
>Priority: Minor
>  Labels: beginner, newbie
>
> After adding a a processor and a sink to topology after a globalstore and 
> then calling StreamBuilder.build().describe() again (for debugging purposes 
> and to check I got the topology right), had the following exception and 
> stacktrace:
> {{Caused by: java.util.NoSuchElementException: null}}
>  {{    at java.util.HashMap$HashIterator.nextNode(HashMap.java:1444) 
> ~[na:1.8.0_171]}}
>  {{    at java.util.HashMap$KeyIterator.next(HashMap.java:1466) 
> ~[na:1.8.0_171]}}
>  {{    at 
> org.apache.kafka.streams.processor.internals.InternalTopologyBuilder.describeGlobalStore(InternalTopologyBuilder.java:1323)
>  ~[kafka-streams-1.1.0.jar:na]}}
>  {{    at 
> org.apache.kafka.streams.processor.internals.InternalTopologyBuilder.describe(InternalTopologyBuilder.java:1306)
>  ~[kafka-streams-1.1.0.jar:na]}}
>  {{    at org.apache.kafka.streams.Topology.describe(Topology.java:647) 
> ~[kafka-streams-1.1.0.jar:na]}}
> Snipped of code that caused this:
> {{    GlobalKTable jsonRoutesToServices}}
>  {{    = builder.globalTable("routes-to-services",}}
>  {{    Consumed.with(Serdes.String(), 
> jsonServiceListSerde),}}
>  {{    Materialized. KeyValueStore>as("routes-to-services"));}}
> {{    TopologyDescription td = builder.build().describe();}}
>  {{    String parent = null;}}
>  {{    // We get an iterator to a TreeSet sorted by processing order, and 
> just want the last one.}}
>  {{    for (TopologyDescription.GlobalStore store : td.globalStores()) {}}
>  {{    parent = store.processor().name();}}
>      }
>  {{    TopologyDescription tdtd = builder.build().describe();}}
>  {{    builder.build().addProcessor("ROUTES-TO-SERVICES-FORWARDER", new 
> UnneededCruftSupplier(), parent);}}
>  {{    builder.build().addSink("FST-ROUTES-TO-SERVICES", 
> "fst-routes-to-services", Serdes.String().serializer(), 
> fstServiceListSerde.serializer(), "ROUTES-TO-SERVICES-FORWARDER");}}
>  {{    TopologyDescription tdtdtd = builder.build().describe();}}
> Note that the exception is thrown on the last line of the code snippet, 
> calling describe again before adding anything works fine.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-7049) InternalTopicIntegrationTest sometimes fails

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-7049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang resolved KAFKA-7049.
--
Resolution: Cannot Reproduce

> InternalTopicIntegrationTest sometimes fails
> 
>
> Key: KAFKA-7049
> URL: https://issues.apache.org/jira/browse/KAFKA-7049
> Project: Kafka
>  Issue Type: Test
>  Components: streams, unit tests
>Reporter: Ted Yu
>Priority: Minor
>
> Saw the following based on commit fa1d0383902260576132e09bdf9efcc2784b55b4 :
> {code}
> org.apache.kafka.streams.integration.InternalTopicIntegrationTest > 
> shouldCompactTopicsForKeyValueStoreChangelogs FAILED
> java.lang.RuntimeException: Timed out waiting for completion. 
> lagMetrics=[0/2] totalLag=[0.0]
> at 
> org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitForCompletion(IntegrationTestUtils.java:227)
> at 
> org.apache.kafka.streams.integration.InternalTopicIntegrationTest.shouldCompactTopicsForKeyValueStoreChangelogs(InternalTopicIntegrationTest.java:164)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-7228) DeadLetterQueue throws a NullPointerException

[Ewen Cheslack-Postava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-7228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava resolved KAFKA-7228.
--
Resolution: Fixed

> DeadLetterQueue throws a NullPointerException
> -
>
> Key: KAFKA-7228
> URL: https://issues.apache.org/jira/browse/KAFKA-7228
> Project: Kafka
>  Issue Type: Task
>  Components: KafkaConnect
>Reporter: Arjun Satish
>Assignee: Arjun Satish
>Priority: Major
> Fix For: 2.0.1, 2.1.0
>
>
> Using the dead letter queue results in a NPE: 
> {code:java}
> [2018-08-01 07:41:08,907] ERROR WorkerSinkTask{id=s3-sink-yanivr-2-4} Task 
> threw an uncaught and unrecoverable exception 
> (org.apache.kafka.connect.runtime.WorkerTask)
> java.lang.NullPointerException
> at 
> org.apache.kafka.connect.runtime.errors.DeadLetterQueueReporter.report(DeadLetterQueueReporter.java:124)
> at 
> org.apache.kafka.connect.runtime.errors.ProcessingContext.report(ProcessingContext.java:137)
> at 
> org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execute(RetryWithToleranceOperator.java:108)
> at 
> org.apache.kafka.connect.runtime.WorkerSinkTask.convertAndTransformRecord(WorkerSinkTask.java:513)
> at 
> org.apache.kafka.connect.runtime.WorkerSinkTask.convertMessages(WorkerSinkTask.java:490)
> at 
> org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:321)
> at 
> org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:225)
> at 
> org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:193)
> at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:175)
> at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:219)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> [2018-08-01 07:41:08,908] ERROR WorkerSinkTask{id=s3-sink-yanivr-2-4} Task is 
> being killed and will not recover until manually restarted 
> (org.apache.kafka.connect.runtime.WorkerTask)
> {code}
> DLQ reporter does not get a {{errorHandlingMetrics}} object when initialized 
> through the WorkerSinkTask.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-4101) java.lang.IllegalStateException in org.apache.kafka.common.network.Selector.channelOrFail

[Ray Chiang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ray Chiang resolved KAFKA-4101.
---
Resolution: Duplicate

> java.lang.IllegalStateException in 
> org.apache.kafka.common.network.Selector.channelOrFail
> -
>
> Key: KAFKA-4101
> URL: https://issues.apache.org/jira/browse/KAFKA-4101
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.9.0.1
> Environment: Ubuntu 14.04, AWS deployment, under heavy network load
>Reporter: Andrey Savov
>Priority: Major
>
> {code}
>  at org.apache.kafka.common.network.Selector.channelOrFail(Selector.java:467)
> at org.apache.kafka.common.network.Selector.mute(Selector.java:347)
> at 
> kafka.network.Processor$$anonfun$run$11.apply(SocketServer.scala:434)
> at 
> kafka.network.Processor$$anonfun$run$11.apply(SocketServer.scala:421)
> at scala.collection.Iterator$class.foreach(Iterator.scala:742)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1194)
> at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> at kafka.network.Processor.run(SocketServer.scala:421)
> at java.lang.Thread.run(Thread.java:745)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: kafka-trunk-jdk10 #358

[Apache Jenkins Server]

See 

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on H31 (ubuntu xenial) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from 
https://github.com/apache/kafka.git
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:888)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1155)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186)
at hudson.scm.SCM.checkout(SCM.java:504)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
at hudson.model.Run.execute(Run.java:1794)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
Caused by: hudson.plugins.git.GitException: Command "git fetch --tags 
--progress https://github.com/apache/kafka.git 
+refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: error: Could not read 24d0d3351a43f52f6f688143672d7c57c5f19595
error: Could not read 1e98b2aa5027eb60ee1a112a7f8ec45b9c47
error: Could not read 689bac3c6fcae927ccee89c8d4fd54c2fae83be1
remote: Counting objects: 4186, done.
remote: Compressing objects:   3% (1/29)   remote: Compressing objects: 
  6% (2/29)   remote: Compressing objects:  10% (3/29)   
remote: Compressing objects:  13% (4/29)   remote: Compressing objects: 
 17% (5/29)   remote: Compressing objects:  20% (6/29)   
remote: Compressing objects:  24% (7/29)   remote: Compressing objects: 
 27% (8/29)   remote: Compressing objects:  31% (9/29)   
remote: Compressing objects:  34% (10/29)   remote: Compressing 
objects:  37% (11/29)   remote: Compressing objects:  41% (12/29)   
remote: Compressing objects:  44% (13/29)   remote: Compressing 
objects:  48% (14/29)   remote: Compressing objects:  51% (15/29)   
remote: Compressing objects:  55% (16/29)   remote: Compressing 
objects:  58% (17/29)   remote: Compressing objects:  62% (18/29)   
remote: Compressing objects:  65% (19/29)   remote: Compressing 
objects:  68% (20/29)   remote: Compressing objects:  72% (21/29)   
remote: Compressing objects:  75% (22/29)   remote: Compressing 
objects:  79% (23/29)   remote: Compressing objects:  82% (24/29)   
remote: Compressing objects:  86% (25/29)   remote: Compressing 
objects:  89% (26/29)   remote: Compressing objects:  93% (27/29)   
remote: Compressing objects:  96% (28/29)   remote: Compressing 
objects: 100% (29/29)   remote: Compressing objects: 100% (29/29), 
done.
Receiving objects:   0% (1/4186)   Receiving objects:   1% (42/4186)   
Receiving objects:   2% (84/4186)   Receiving objects:   3% (126/4186)   
Receiving objects:   4% (168/4186)   Receiving objects:   5% (210/4186)   
Receiving objects:   6% (252/4186)   Receiving objects:   7% (294/4186)   
Receiving objects:   8% (335/4186)   Receiving objects:   9% (377/4186)   
Receiving objects:  10% (419/4186)   Receiving objects:  11% (461/4186)   
Receiving objects:  12% (503/4186)   Receiving objects:  13% (545/4186)   
Receiving objects:  14% (587/4186)   Receiving objects:  15% (628/4186)   
Receiving objects:  16% (670/4186)   Receiving objects:  17% (712/4186)   
Receiving objects:  18% (754/4186)   Receiving objects:  19% (796/4186)   
Receiving objects:  20% (838/4186)   Receiving objects:  21% (880/4186)   
Receiving objects:  22% (921/4186)   Receiving objects:  23% (963/4186)   
Receiving objects:  24% (1005/4186)   Receiving objects:  25% (1047/4186)   
Receiving objects:  26% (1089/4186)   Receiving objects:  27% (1131/4186)   
Receiving objects:  28% (1173/4186)   Receiving objects:  29% (1214/4186)   
Receiving objects:  30% (1256/4186)   Receiving objects:  31% (1298/4186)   
Receiving objects:  32% (1340/4186)   Receiving objects:  33% (1382/4186)   
Receiving objects:  34% (1424/4186) 

Re: [VOTE] KIP-334 Include partitions in exceptions raised during consumer record deserialization/validation

[Stanislav Kozlovski]

Hi Jason and Ted,

@Jason
I did not oppose the idea as I'm not too familiar with Java conventions. I
agree it is a non-ideal way for the user to catch the exception so I
changed it back.

I've updated the KIP and PR with the latest changes. Now, there is only one
public exception `FaultyRecordException` which is thrown to the user in
both scenarios (corrupt record and deserialization error).
Please take a look again.

Best,
Stanislav

On Thu, Aug 2, 2018 at 5:25 PM Jason Gustafson  wrote:

> Hey Stanislav,
>
> I should have noticed it earlier from your example, but I just realized
> that interfaces don't mix well with exceptions. There is no way to catch
> the interface type, which means you have to depend on instanceof checks,
> which is not very conventional. At the moment, we raise
> SerializationException if there is a problem parsing the error, and we
> raise KafkaException if the record fails its CRC check. Since
> SerializationException extends KafkaExeption, it seems like we can handle
> both cases in a compatible way by handling both cases with a single type
> that extends SerializationException. Maybe something like
> RecordDeserializationException?
>
> Thanks,
> Jason
>
> On Thu, Aug 2, 2018 at 5:45 AM, Ted Yu  wrote:
>
> > +1
> >  Original message From: Stanislav Kozlovski <
> > stanis...@confluent.io> Date: 8/2/18  2:41 AM  (GMT-08:00) To:
> > dev@kafka.apache.org Subject: [VOTE] KIP-334 Include partitions in
> > exceptions raised during consumer record deserialization/validation
> > Hey everybody,
> >
> > I'd like to start a vote thread for KIP-334 Include partitions in
> > exceptions raised during consumer record deserialization/validation
> > <
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=87297793
> > >
> >
> > --
> > Best,
> > Stanislav
> >
>


-- 
Best,
Stanislav

Jenkins build is back to normal : kafka-2.0-jdk8 #99

[Apache Jenkins Server]

See 

Build failed in Jenkins: kafka-trunk-jdk10 #359

[Apache Jenkins Server]

See 

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on H30 (ubuntu xenial) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from 
https://github.com/apache/kafka.git
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:888)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1155)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186)
at hudson.scm.SCM.checkout(SCM.java:504)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
at hudson.model.Run.execute(Run.java:1794)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
Caused by: hudson.plugins.git.GitException: Command "git fetch --tags 
--progress https://github.com/apache/kafka.git 
+refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: error: Could not read 24d0d3351a43f52f6f688143672d7c57c5f19595
error: Could not read 1e98b2aa5027eb60ee1a112a7f8ec45b9c47
error: Could not read 689bac3c6fcae927ccee89c8d4fd54c2fae83be1
remote: Counting objects: 2772, done.
remote: Compressing objects:   3% (1/30)   remote: Compressing objects: 
  6% (2/30)   remote: Compressing objects:  10% (3/30)   
remote: Compressing objects:  13% (4/30)   remote: Compressing objects: 
 16% (5/30)   remote: Compressing objects:  20% (6/30)   
remote: Compressing objects:  23% (7/30)   remote: Compressing objects: 
 26% (8/30)   remote: Compressing objects:  30% (9/30)   
remote: Compressing objects:  33% (10/30)   remote: Compressing 
objects:  36% (11/30)   remote: Compressing objects:  40% (12/30)   
remote: Compressing objects:  43% (13/30)   remote: Compressing 
objects:  46% (14/30)   remote: Compressing objects:  50% (15/30)   
remote: Compressing objects:  53% (16/30)   remote: Compressing 
objects:  56% (17/30)   remote: Compressing objects:  60% (18/30)   
remote: Compressing objects:  63% (19/30)   remote: Compressing 
objects:  66% (20/30)   remote: Compressing objects:  70% (21/30)   
remote: Compressing objects:  73% (22/30)   remote: Compressing 
objects:  76% (23/30)   remote: Compressing objects:  80% (24/30)   
remote: Compressing objects:  83% (25/30)   remote: Compressing 
objects:  86% (26/30)   remote: Compressing objects:  90% (27/30)   
remote: Compressing objects:  93% (28/30)   remote: Compressing 
objects:  96% (29/30)   remote: Compressing objects: 100% (30/30)   
remote: Compressing objects: 100% (30/30), done.
Receiving objects:   0% (1/2772)   Receiving objects:   1% (28/2772)   
Receiving objects:   2% (56/2772)   Receiving objects:   3% (84/2772)   
Receiving objects:   4% (111/2772)   Receiving objects:   5% (139/2772)   
Receiving objects:   6% (167/2772)   Receiving objects:   7% (195/2772)   
Receiving objects:   8% (222/2772)   Receiving objects:   9% (250/2772)   
Receiving objects:  10% (278/2772)   Receiving objects:  11% (305/2772)   
Receiving objects:  12% (333/2772)   Receiving objects:  13% (361/2772)   
Receiving objects:  14% (389/2772)   Receiving objects:  15% (416/2772)   
Receiving objects:  16% (444/2772)   Receiving objects:  17% (472/2772)   
Receiving objects:  18% (499/2772)   Receiving objects:  19% (527/2772)   
Receiving objects:  20% (555/2772)   Receiving objects:  21% (583/2772)   
Receiving objects:  22% (610/2772)   Receiving objects:  23% (638/2772)   
Receiving objects:  24% (666/2772)   Receiving objects:  25% (693/2772)   
Receiving objects:  26% (721/2772)   Receiving objects:  27% (749/2772)   
Receiving objects:  28% (777/2772)   Receiving objects:  29% (804/2772)   
Receiving objects:  30% (832/2772)   Receiving objects:  31% (860/2772)   
Receiving objects:  32% (888/2772)   Receiving objects:  33% (915/2772

Build failed in Jenkins: kafka-2.0-jdk8 #100

[Apache Jenkins Server]

See 


Changes:

[me] KAFKA-7228: Set errorHandlingMetrics for dead letter queue

--
[...truncated 433.72 KB...]

kafka.controller.PartitionStateMachineTest > 
testInvalidNonexistentPartitionToOfflinePartitionTransition STARTED

kafka.controller.PartitionStateMachineTest > 
testInvalidNonexistentPartitionToOfflinePartitionTransition PASSED

kafka.controller.PartitionStateMachineTest > 
testOnlinePartitionToOnlineTransition STARTED

kafka.controller.PartitionStateMachineTest > 
testOnlinePartitionToOnlineTransition PASSED

kafka.controller.PartitionStateMachineTest > 
testNewPartitionToOnlinePartitionTransitionZkUtilsExceptionFromCreateStates 
STARTED

kafka.controller.PartitionStateMachineTest > 
testNewPartitionToOnlinePartitionTransitionZkUtilsExceptionFromCreateStates 
PASSED

kafka.controller.PartitionStateMachineTest > 
testInvalidNewPartitionToNonexistentPartitionTransition STARTED

kafka.controller.PartitionStateMachineTest > 
testInvalidNewPartitionToNonexistentPartitionTransition PASSED

kafka.controller.PartitionStateMachineTest > 
testNewPartitionToOnlinePartitionTransition STARTED

kafka.controller.PartitionStateMachineTest > 
testNewPartitionToOnlinePartitionTransition PASSED

kafka.controller.PartitionStateMachineTest > 
testInvalidOnlinePartitionToNewPartitionTransition STARTED

kafka.controller.PartitionStateMachineTest > 
testInvalidOnlinePartitionToNewPartitionTransition PASSED

kafka.controller.PartitionStateMachineTest > 
testOfflinePartitionToOnlinePartitionTransitionErrorCodeFromStateLookup STARTED

kafka.controller.PartitionStateMachineTest > 
testOfflinePartitionToOnlinePartitionTransitionErrorCodeFromStateLookup PASSED

kafka.controller.PartitionStateMachineTest > 
testOnlinePartitionToOnlineTransitionForControlledShutdown STARTED

kafka.controller.PartitionStateMachineTest > 
testOnlinePartitionToOnlineTransitionForControlledShutdown PASSED

kafka.controller.PartitionStateMachineTest > 
testOfflinePartitionToOnlinePartitionTransitionZkUtilsExceptionFromStateLookup 
STARTED

kafka.controller.PartitionStateMachineTest > 
testOfflinePartitionToOnlinePartitionTransitionZkUtilsExceptionFromStateLookup 
PASSED

kafka.controller.PartitionStateMachineTest > 
testInvalidOnlinePartitionToNonexistentPartitionTransition STARTED

kafka.controller.PartitionStateMachineTest > 
testInvalidOnlinePartitionToNonexistentPartitionTransition PASSED

kafka.controller.PartitionStateMachineTest > 
testInvalidOfflinePartitionToNewPartitionTransition STARTED

kafka.controller.PartitionStateMachineTest > 
testInvalidOfflinePartitionToNewPartitionTransition PASSED

kafka.controller.PartitionStateMachineTest > 
testOfflinePartitionToOnlinePartitionTransition STARTED

kafka.controller.PartitionStateMachineTest > 
testOfflinePartitionToOnlinePartitionTransition PASSED

kafka.controller.ControllerEventManagerTest > testEventThatThrowsException 
STARTED

kafka.controller.ControllerEventManagerTest > testEventThatThrowsException 
PASSED

kafka.controller.ControllerEventManagerTest > testSuccessfulEvent STARTED

kafka.controller.ControllerEventManagerTest > testSuccessfulEvent PASSED

kafka.controller.ControllerFailoverTest > testHandleIllegalStateException 
STARTED

kafka.controller.ControllerFailoverTest > testHandleIllegalStateException PASSED

kafka.network.SocketServerTest > testGracefulClose STARTED

kafka.network.SocketServerTest > testGracefulClose PASSED

kafka.network.SocketServerTest > 
testSendActionResponseWithThrottledChannelWhereThrottlingAlreadyDone STARTED

kafka.network.SocketServerTest > 
testSendActionResponseWithThrottledChannelWhereThrottlingAlreadyDone PASSED

kafka.network.SocketServerTest > controlThrowable STARTED

kafka.network.SocketServerTest > controlThrowable PASSED

kafka.network.SocketServerTest > testRequestMetricsAfterStop STARTED

kafka.network.SocketServerTest > testRequestMetricsAfterStop PASSED

kafka.network.SocketServerTest > testConnectionIdReuse STARTED

kafka.network.SocketServerTest > testConnectionIdReuse PASSED

kafka.network.SocketServerTest > testClientDisconnectionUpdatesRequestMetrics 
STARTED

kafka.network.SocketServerTest > testClientDisconnectionUpdatesRequestMetrics 
PASSED

kafka.network.SocketServerTest > testProcessorMetricsTags STARTED

kafka.network.SocketServerTest > testProcessorMetricsTags PASSED

kafka.network.SocketServerTest > testMaxConnectionsPerIp STARTED

kafka.network.SocketServerTest > testMaxConnectionsPerIp PASSED

kafka.network.SocketServerTest > testConnectionId STARTED

kafka.network.SocketServerTest > testConnectionId PASSED

kafka.network.SocketServerTest > 
testBrokerSendAfterChannelClosedUpdatesRequestMetrics STARTED

kafka.network.SocketServerTest > 
testBrokerSendAfterChannelClosedUpdatesRequestMetrics PASSED

kafka.network.SocketServerTest > testNoOpAction STARTED

kafka.network.SocketServerTest > testNoOpAction PA

Re: [VOTE] KIP-342 - Add support for custom SASL extensions in OAuthBearer authentication

[Jun Rao]

Hi, Stanislav,

Thanks for the KIP. +1

Just one minor comment. Since the JWT token supports customizable claim
fields, it would be useful to clarify when to use the SASL extension vs the
customized fields in JWT.

Jun

On Wed, Jul 25, 2018 at 10:03 AM, Stanislav Kozlovski <
stanis...@confluent.io> wrote:

> Hey everbody,
>
> I'd like to start a vote thread for KIP-342 Add support for custom SASL
> extensions in OAuthBearer authentication
>  342%3A+Add+support+for+Custom+SASL+extensions+in+
> OAuthBearer+authentication>
>
> --
> Best,
> Stanislav
>

Jenkins build is back to normal : kafka-trunk-jdk10 #360

[Apache Jenkins Server]

See 

[VOTE] KIP-332: Update AclCommand to use AdminClient API

[Manikumar]

Hi All,

I would like to start voting on KIP-332 which allows AclCommand to use
AdminClient API for acl management.

KIP:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-332%3A+Update+AclCommand+to+use+AdminClient+API

Discussion Thread:
https://www.mail-archive.com/dev@kafka.apache.org/msg90049.html

Thanks,
Manikumar

Re: [VOTE] KIP-332: Update AclCommand to use AdminClient API

[Ted Yu]

+1

On Thu, Aug 2, 2018 at 7:33 PM Manikumar  wrote:

> Hi All,
>
> I would like to start voting on KIP-332 which allows AclCommand to use
> AdminClient API for acl management.
>
> KIP:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-332%3A+Update+AclCommand+to+use+AdminClient+API
>
> Discussion Thread:
> https://www.mail-archive.com/dev@kafka.apache.org/msg90049.html
>
> Thanks,
> Manikumar
>

[jira] [Resolved] (KAFKA-7233) InMemoryKeyValueStore is not thread-safe for Interactive Queries

[Hashan Gayasri Udugahapattuwa (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-7233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hashan Gayasri Udugahapattuwa resolved KAFKA-7233.
--
Resolution: Invalid

The implementation is correct

> InMemoryKeyValueStore is not thread-safe for Interactive Queries
> 
>
> Key: KAFKA-7233
> URL: https://issues.apache.org/jira/browse/KAFKA-7233
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 1.1.0, 2.0.0
> Environment: Fedora 27
>Reporter: Hashan Gayasri Udugahapattuwa
>Priority: Critical
>
> When querying externally from via the 
> [ReadOnlyKeyValueStore|https://github.com/apache/kafka/blob/1.0/streams/src/main/java/org/apache/kafka/streams/state/ReadOnlyKeyValueStore.java]
>  API (Interactive Queries), the InMemoryKeyValueStore does not protect its 
> internal map from concurrent reads and writes which may produce incorrect 
> results to the  ReadOnlyKeyValueStore API. Note that reads(query thread) and 
> writes(stream thread) can happen concurrently.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-7083) maxTickMessages in ConsumerGroup option

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-7083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-7083.
--
Resolution: Information Provided

Closing the issue related to node.js client. Please contact client library 
project. Please reopen if any specific issue about Kafka 

> maxTickMessages in ConsumerGroup option
> ---
>
> Key: KAFKA-7083
> URL: https://issues.apache.org/jira/browse/KAFKA-7083
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Reporter: Rahul
>Priority: Major
>
> Hello,
> We are using Kafka v2.11.1.0.0. We have only one partition for our topic. 
> While consuming a record from the topic, I am setting maxTickMessages = 1 in 
> Kafka consumer group. It returns me 2 records. I am not getting why it is 
> giving me 1 extra record of the mentioned size. Whenever I increase the 
> number in maxTickMessages, it gives me one extra record.
> Can someone please suggest me a solution to this issue?
> Thanks & Regards,
> Rahul Singh



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-7226) kafka-console-consumer.sh doesn't use security.protocol provided in config file

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-7226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-7226.
--
Resolution: Information Provided

Closing the issue related to HDP Kafka distribution.

> kafka-console-consumer.sh doesn't use security.protocol provided in config 
> file
> ---
>
> Key: KAFKA-7226
> URL: https://issues.apache.org/jira/browse/KAFKA-7226
> Project: Kafka
>  Issue Type: Bug
>  Components: producer 
>Affects Versions: 1.0.0
>Reporter: Alexandre GRIFFAUT
>Priority: Minor
>
> It is confusing that kafka-console-consumer.sh doesn't read security.protocol 
> when provided in config file with --producer.config, wheras 
> kafka-console-consumer.sh does...
>  
> With debug activated:
> $ /usr/hdp/2.6.5.0-292/kafka/bin/kafka-console-producer.sh --broker-list 
> $(hostname):6668 --topic test --producer.config 
> /etc/kafka/ssl/kafka.client.properties
> [2018-08-01 14:17:18,505] INFO ProducerConfig values:
> ...
>     security.protocol = PLAINTEXT
> ...
> > abc
> ..
> java.io.EOFException
>     at 
> org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:147)
>     at 
> org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:93)
>     at 
> org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:235)
>     at 
> org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:196)
>     at 
> org.apache.kafka.common.network.Selector.attemptRead(Selector.java:538)
>     at 
> org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:482)
>     at org.apache.kafka.common.network.Selector.poll(Selector.java:412)
>     at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:460)
>     at 
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:239)
>     at 
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:163)
>     at java.lang.Thread.run(Thread.java:745)
> ...
>  
> The only way produce with SSL, is to use --security-protocol SSL
> kafka-console-consumer.sh correctly read security.protocol from config file



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] KIP-351: Add --critical-partitions option to describe topics command

[Jason Gustafson]

Hey Kevin,

Thanks for the KIP. I like Mickael's suggestion to
add --under-minisr-partitions since it fits with the metric we already
expose. It's also a good question whether there should be a separate
category for partitions which are right at min.isr. I'm reluctant to add
new categories, but I agree there might be a gap at the moment. Say you
have a replication factor of 3 and the min isr is set to 1. Our notion of
URP does not capture the difference between having an ISR down to a size of
1 and one down to a size of 2. The reason this might be significant is that
a shrink of the ISR down to 2 may just be caused by a rolling restart or a
transient network blip. A shrink to 1, on the other hand, might be
indicative of a more severe problem and could be cause for a call from
pagerduty.

-Jason

On Thu, Aug 2, 2018 at 9:28 AM, Kevin Lu  wrote:

> Hi Mickael,
>
> Thanks for the suggestion!
>
> Correct me if I am mistaken, but if a producer attempts to send to a
> partition that is under min ISR (and ack=all or -1) then the send will fail
> with a NotEnoughReplicas or NotEnoughReplicasAfterAppend exception? At this
> point, client-side has already suffered failure but the server-side is
> still fine for now?
>
> If the above is true, then this would be a FATAL case for producers.
>
> Would it be valuable to include the CRITICAL case where a topic partition
> has exactly min ISR so that Kafka operators can take action so it does not
> become FATAL? This could be in the same option or a new one.
>
> Thanks!
>
> Regards,
> Kevin
>
> On Thu, Aug 2, 2018 at 2:27 AM Mickael Maison 
> wrote:
>
> > What about also adding a --under-minisr-partitions option?
> >
> > That would match the
> > "kafka.server:type=ReplicaManager,name=UnderMinIsrPartitionCount"
> > broker metric and it's usually pretty relevant when investigating
> > issues
> >
> > On Thu, Aug 2, 2018 at 8:54 AM, Kevin Lu  wrote:
> > > Hi friends!
> > >
> > > This thread is to discuss KIP-351
> > > <
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-351%3A
> +Add+--critical-partitions+option+to+describe+topics+command
> > >
> > > !
> > >
> > > I am proposing to add a --critical-partitions option to the describe
> > topics
> > > command that will only list out topic partitions that have 1 ISR left
> > (RF >
> > > 1) as they would be in a critical state and need immediate
> > repartitioning.
> > >
> > > I wonder if the name "critical" is appropriate?
> > >
> > > Thoughts?
> > >
> > > Thanks!
> > >
> > > Regards,
> > > Kevin
> >
>

Build failed in Jenkins: kafka-trunk-jdk8 #2858

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-3514: Part II, Choose tasks with data on all partitions to 
process

--
[...truncated 2.47 MB...]
org.apache.kafka.streams.TopologyTest > 
multipleSourcesWithProcessorsShouldHaveDistinctSubtopologies STARTED

org.apache.kafka.streams.TopologyTest > 
multipleSourcesWithProcessorsShouldHaveDistinctSubtopologies PASSED

org.apache.kafka.streams.TopologyTest > shouldThrowOnUnassignedStateStoreAccess 
STARTED

org.apache.kafka.streams.TopologyTest > shouldThrowOnUnassignedStateStoreAccess 
PASSED

org.apache.kafka.streams.TopologyTest > 
shouldNotAllowNullProcessorNameWhenConnectingProcessorAndStateStores STARTED

org.apache.kafka.streams.TopologyTest > 
shouldNotAllowNullProcessorNameWhenConnectingProcessorAndStateStores PASSED

org.apache.kafka.streams.TopologyTest > 
kTableAnonymousMaterializedMapValuesShouldPreserveTopologyStructure STARTED

org.apache.kafka.streams.TopologyTest > 
kTableAnonymousMaterializedMapValuesShouldPreserveTopologyStructure PASSED

org.apache.kafka.streams.TopologyTest > 
sessionWindowZeroArgCountShouldPreserveTopologyStructure STARTED

org.apache.kafka.streams.TopologyTest > 
sessionWindowZeroArgCountShouldPreserveTopologyStructure PASSED

org.apache.kafka.streams.TopologyTest > 
tableAnonymousMaterializedCountShouldPreserveTopologyStructure STARTED

org.apache.kafka.streams.TopologyTest > 
tableAnonymousMaterializedCountShouldPreserveTopologyStructure PASSED

org.apache.kafka.streams.TopologyTest > shouldDescribeGlobalStoreTopology 
STARTED

org.apache.kafka.streams.TopologyTest > shouldDescribeGlobalStoreTopology PASSED

org.apache.kafka.streams.TopologyTest > 
kTableNonMaterializedMapValuesShouldPreserveTopologyStructure STARTED

org.apache.kafka.streams.TopologyTest > 
kTableNonMaterializedMapValuesShouldPreserveTopologyStructure PASSED

org.apache.kafka.streams.TopologyTest > 
kGroupedStreamAnonymousMaterializedCountShouldPreserveTopologyStructure STARTED

org.apache.kafka.streams.TopologyTest > 
kGroupedStreamAnonymousMaterializedCountShouldPreserveTopologyStructure PASSED

org.apache.kafka.streams.TopologyTest > 
shouldDescribeMultipleGlobalStoreTopology STARTED

org.apache.kafka.streams.TopologyTest > 
shouldDescribeMultipleGlobalStoreTopology PASSED

org.apache.kafka.streams.TopologyTest > shouldNotAllowToAddSinkWithSameName 
STARTED

org.apache.kafka.streams.TopologyTest > shouldNotAllowToAddSinkWithSameName 
PASSED

org.apache.kafka.streams.TopologyTest > shouldNotAllowToAddSinkWithNullParents 
STARTED

org.apache.kafka.streams.TopologyTest > shouldNotAllowToAddSinkWithNullParents 
PASSED

org.apache.kafka.streams.TopologyTest > 
shouldNotAllowNullProcessorSupplierWhenAddingProcessor STARTED

org.apache.kafka.streams.TopologyTest > 
shouldNotAllowNullProcessorSupplierWhenAddingProcessor PASSED

org.apache.kafka.streams.TopologyTest > 
timeWindowAnonymousMaterializedCountShouldPreserveTopologyStructure STARTED

org.apache.kafka.streams.TopologyTest > 
timeWindowAnonymousMaterializedCountShouldPreserveTopologyStructure PASSED

org.apache.kafka.streams.TopologyTest > shouldFailIfSinkIsParent STARTED

org.apache.kafka.streams.TopologyTest > shouldFailIfSinkIsParent PASSED

org.apache.kafka.streams.TopologyTest > shouldNotAllowToAddSourcesWithSameName 
STARTED

org.apache.kafka.streams.TopologyTest > shouldNotAllowToAddSourcesWithSameName 
PASSED

org.apache.kafka.streams.TopologyTest > 
sourceAndProcessorWithMultipleStatesShouldHaveSingleSubtopology STARTED

org.apache.kafka.streams.TopologyTest > 
sourceAndProcessorWithMultipleStatesShouldHaveSingleSubtopology PASSED

org.apache.kafka.streams.TopologyTest > shouldDescribeEmptyTopology STARTED

org.apache.kafka.streams.TopologyTest > shouldDescribeEmptyTopology PASSED

org.apache.kafka.streams.TopologyTest > shouldFailIfSinkIsItsOwnParent STARTED

org.apache.kafka.streams.TopologyTest > shouldFailIfSinkIsItsOwnParent PASSED

org.apache.kafka.streams.TopologyTest > 
testPatternMatchesAlreadyProvidedTopicSource STARTED

org.apache.kafka.streams.TopologyTest > 
testPatternMatchesAlreadyProvidedTopicSource PASSED

org.apache.kafka.streams.TopologyTest > 
singleSourcePatternShouldHaveSingleSubtopology STARTED

org.apache.kafka.streams.TopologyTest > 
singleSourcePatternShouldHaveSingleSubtopology PASSED

org.apache.kafka.streams.TopologyTest > 
kTableNonMaterializedFilterShouldPreserveTopologyStructure STARTED

org.apache.kafka.streams.TopologyTest > 
kTableNonMaterializedFilterShouldPreserveTopologyStructure PASSED

org.apache.kafka.streams.TopologyTest > singleSourceShouldHaveSingleSubtopology 
STARTED

org.apache.kafka.streams.TopologyTest > singleSourceShouldHaveSingleSubtopology 
PASSED

org.apache.kafka.streams.TopologyTest > 
shouldNotAllowNullTopicChooserWhenAddingSink STARTED

org.apache.kafka.streams.TopologyTest > 
shouldNotAllowNullTopicChooserWhenAddingSink PASSED

Re: [DISCUSS] KIP-289: Improve the default group id behavior in KafkaConsumer

[Jason Gustafson]

Hey Vahid,

I think we're in agreement. Perhaps since none of us feel too strongly, we
should go for the more minimal change? I'll vote for the KIP either way as
long as we can change the default group.id.

Thanks,
Jason

On Thu, Aug 2, 2018 at 11:55 AM, Vahid S Hashemian <
vahidhashem...@us.ibm.com> wrote:

> Hi Jason, Colin,
>
> Thanks for your feedback.
>
> If we are going to keep supporting the empty group id like before then it
> makes sense to do what you suggested.
>
> Though, IMHO if using the empty group id for committing offsets is a bad
> practice we should stop supporting it at some point; and it would be
> better to do it sooner before potentially more users adopt that.
> I understand that changing the default would reduce the chance of someone
> using it, but it doesn't stop them. Note that we did not receive any
> objections to dropping the support even right-away (a good sign).
> We could keep supporting the old API version in case someone must use this
> group id like before. But at least we won't expose new users to this bad
> practice.
>
> I, too, do now have a strong opinion about this. So if the current KIP
> should not move forward as is, I'm okay with just changing the default and
> deprecating the use on the client, as you suggested.
>
> Thanks.
> --Vahid
>
>
>
> From:   Colin McCabe 
> To: dev@kafka.apache.org
> Date:   08/02/2018 11:23 AM
> Subject:Re: [DISCUSS] KIP-289: Improve the default group id
> behavior in KafkaConsumer
>
>
>
> Thanks, Jason.  I don't have a very strong opinion on this.  But like you
> said, if we skip bumping the RPC versions, this would be a smaller change,
> which might be good.
>
> best,
> Colin
>
>
> On Wed, Aug 1, 2018, at 17:43, Jason Gustafson wrote:
> > Hey Vahid,
> >
> > I talked with Colin offline. I think specifically he felt the version
> bump
> > on the broker was overkill since the broker still has to support the
> empty
> > group id for older versions. I had thought that eventually we would be
> able
> > to remove those old versions, but it's true that this may not happen
> until
> > indefinitely far in the future. I think the main improvement here is
> > changing the default group.id to null instead of "". I could go either
> way
> > on whether bumping the protocol is useful. I do think it is helpful
> though
> > to signal clearly that it its use is deprecated and discouraged,
> especially
> > in light of the ACL problem. I guess we could just deprecate the use on
> the
> > client. What do you think?
> >
> > Thanks,
> > Jason
> >
> > On Wed, Aug 1, 2018 at 3:19 PM, Vahid S Hashemian
>  > > wrote:
> >
> > > Thanks Jason for responding to Colin's concerns.
> > >
> > > If there are no other comment / feedback / objection I'll start a vote
> > > soon.
> > >
> > > Thanks.
> > > --Vahid
> > >
> > >
> > >
> > > From:   Jason Gustafson 
> > > To: dev 
> > > Date:   07/27/2018 10:38 AM
> > > Subject:Re: [DISCUSS] KIP-289: Improve the default group id
> > > behavior in KafkaConsumer
> > >
> > >
> > >
> > > Hey Colin,
> > >
> > > The problem is both that the empty group id is the default value and
> that
> > > it is actually accepted by the broker for offset commits. Combine that
> > > with
> > > the fact that auto commit is enabled by default and you users get
> > > surprising behavior. If you look at a random Kafka cluster, you'll
> > > probably
> > > find a bunch of inadvertent offset commits for the empty group id. I
> was
> > > hoping we could distinguish between users who are using the empty
> group id
> > > as an accident of the default configuration and those who use it
> > > intentionally. By default, there will be no group id and the consumer
> will
> > > not commit offsets. If a user has actually intentionally used the
> empty
> > > group id, however, it will continue to work. I actually think there
> are
> > > probably very few people doing this (maybe even no one), but I thought
> we
> > > might err on the side of compatibility.
> > >
> > > The big incompatible change here is having brokers reject using
> > > assign(...)
> > > > with empty / null group.id.
> > >
> > >
> > > This is not correct. In the proposal, the broker will only reject the
> > > empty
> > > group id for the new version of OffsetCommit. Older clients, which
> cannot
> > > be changed, will continue to work because the old versions of the
> > > OffsetCommit API still accept the empty group id. The null group id is
> > > different from the empty group id: it is not allowed in any version of
> the
> > > API. It is basically a way to indicate that the consumer has no
> dependence
> > > on the coordinator at all, which we actually have a surprising number
> of
> > > use cases for. Furthermore, if a user has an actual need for the empty
> > > group id, it will still be allowed. We are just deprecating it.
> > >
> > > -Jason
> > >
> > > On Fri, Jul 27, 2018 at 9:56 AM, Colin McCabe 
> wrote:
> > >
> > > > Sorry if this is a silly question, but what's the