[jira] [Created] (KAFKA-13387) DescribeUserScramCredentialsResponse does not include entries from the request when there's an error

2021-10-20 Thread Mickael Maison (Jira) 
Mickael Maison created KAFKA-13387:
--

 Summary: DescribeUserScramCredentialsResponse does not include 
entries from the request when there's an error
 Key: KAFKA-13387
 URL: https://issues.apache.org/jira/browse/KAFKA-13387
 Project: Kafka
  Issue Type: Improvement
Reporter: Mickael Maison
Assignee: Mickael Maison


When building error response, we typically always include entries that were 
provided in the request. This is not the case for 
DescribeUserScramCredentialsResponse.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #530

2021-10-20 Thread Apache Jenkins Server 
See 


Changes:


--
[...truncated 495259 lines...]
[2021-10-20T08:50:38.492Z] 
[2021-10-20T08:50:38.492Z] ControllerIntegrationTest > 
testTopicIdCreatedOnUpgradeMultiBrokerScenario() STARTED
[2021-10-20T08:50:46.577Z] 
[2021-10-20T08:50:46.577Z] ControllerIntegrationTest > 
testTopicIdCreatedOnUpgradeMultiBrokerScenario() PASSED
[2021-10-20T08:50:46.577Z] 
[2021-10-20T08:50:46.577Z] ControllerIntegrationTest > 
testPreemptionWithCallbacks() STARTED
[2021-10-20T08:50:50.809Z] 
[2021-10-20T08:50:50.809Z] ControllerIntegrationTest > 
testPreemptionWithCallbacks() PASSED
[2021-10-20T08:50:50.809Z] 
[2021-10-20T08:50:50.809Z] ControllerIntegrationTest > 
testControllerDetectsBouncedBrokers() STARTED
[2021-10-20T08:50:58.354Z] 
[2021-10-20T08:50:58.354Z] ControllerIntegrationTest > 
testControllerDetectsBouncedBrokers() PASSED
[2021-10-20T08:50:58.354Z] 
[2021-10-20T08:50:58.354Z] ControllerIntegrationTest > testControlledShutdown() 
STARTED
[2021-10-20T08:51:04.383Z] 
[2021-10-20T08:51:04.383Z] ControllerIntegrationTest > testControlledShutdown() 
PASSED
[2021-10-20T08:51:04.383Z] 
[2021-10-20T08:51:04.383Z] ControllerIntegrationTest > 
testPreemptionOnControllerShutdown() STARTED
[2021-10-20T08:51:07.708Z] 
[2021-10-20T08:51:07.708Z] ControllerIntegrationTest > 
testPreemptionOnControllerShutdown() PASSED
[2021-10-20T08:51:07.708Z] 
[2021-10-20T08:51:07.708Z] ControllerIntegrationTest > 
testPartitionReassignmentWithOfflineReplicaHaltingProgress() STARTED
[2021-10-20T08:51:15.031Z] 
[2021-10-20T08:51:15.031Z] ControllerIntegrationTest > 
testPartitionReassignmentWithOfflineReplicaHaltingProgress() PASSED
[2021-10-20T08:51:15.031Z] 
[2021-10-20T08:51:15.031Z] ControllerIntegrationTest > 
testNoTopicIdPersistsThroughControllerReelection() STARTED
[2021-10-20T08:51:23.540Z] 
[2021-10-20T08:51:23.540Z] ControllerIntegrationTest > 
testNoTopicIdPersistsThroughControllerReelection() PASSED
[2021-10-20T08:51:23.540Z] 
[2021-10-20T08:51:23.540Z] ControllerIntegrationTest > 
testControllerEpochPersistsWhenAllBrokersDown() STARTED
[2021-10-20T08:51:24.759Z] 
[2021-10-20T08:51:24.759Z] ControllerIntegrationTest > 
testControllerEpochPersistsWhenAllBrokersDown() PASSED
[2021-10-20T08:51:24.759Z] 
[2021-10-20T08:51:24.759Z] ControllerIntegrationTest > testTopicIdsAreAdded() 
STARTED
[2021-10-20T08:51:30.114Z] 
[2021-10-20T08:51:30.114Z] ControllerIntegrationTest > testTopicIdsAreAdded() 
PASSED
[2021-10-20T08:51:30.114Z] 
[2021-10-20T08:51:30.114Z] ControllerIntegrationTest > 
testTopicCreationWithOfflineReplica() STARTED
[2021-10-20T08:51:34.614Z] 
[2021-10-20T08:51:34.614Z] ControllerIntegrationTest > 
testTopicCreationWithOfflineReplica() PASSED
[2021-10-20T08:51:34.614Z] 
[2021-10-20T08:51:34.614Z] ControllerIntegrationTest > 
testPartitionReassignmentResumesAfterReplicaComesOnline() STARTED
[2021-10-20T08:51:41.437Z] 
[2021-10-20T08:51:41.437Z] ControllerIntegrationTest > 
testPartitionReassignmentResumesAfterReplicaComesOnline() PASSED
[2021-10-20T08:51:41.437Z] 
[2021-10-20T08:51:41.437Z] ControllerIntegrationTest > 
testLeaderAndIsrWhenEntireIsrOfflineAndUncleanLeaderElectionDisabled() STARTED
[2021-10-20T08:51:46.761Z] 
[2021-10-20T08:51:46.761Z] ControllerIntegrationTest > 
testLeaderAndIsrWhenEntireIsrOfflineAndUncleanLeaderElectionDisabled() PASSED
[2021-10-20T08:51:46.761Z] 
[2021-10-20T08:51:46.761Z] ControllerIntegrationTest > 
testTopicIdMigrationAndHandlingWithOlderVersion() STARTED
[2021-10-20T08:51:50.008Z] 
[2021-10-20T08:51:50.008Z] ControllerIntegrationTest > 
testTopicIdMigrationAndHandlingWithOlderVersion() PASSED
[2021-10-20T08:51:50.008Z] 
[2021-10-20T08:51:50.008Z] ControllerIntegrationTest > 
testTopicPartitionExpansionWithOfflineReplica() STARTED
[2021-10-20T08:51:55.505Z] 
[2021-10-20T08:51:55.505Z] ControllerIntegrationTest > 
testTopicPartitionExpansionWithOfflineReplica() PASSED
[2021-10-20T08:51:55.505Z] 
[2021-10-20T08:51:55.505Z] ControllerIntegrationTest > 
testPartitionReassignmentToBrokerWithOfflineLogDir() STARTED
[2021-10-20T08:51:59.772Z] 
[2021-10-20T08:51:59.772Z] ControllerIntegrationTest > 
testPartitionReassignmentToBrokerWithOfflineLogDir() PASSED
[2021-10-20T08:51:59.772Z] 
[2021-10-20T08:51:59.772Z] ControllerIntegrationTest > 
testPreferredReplicaLeaderElectionWithOfflinePreferredReplica() STARTED
[2021-10-20T08:52:07.345Z] 
[2021-10-20T08:52:07.345Z] ControllerIntegrationTest > 
testPreferredReplicaLeaderElectionWithOfflinePreferredReplica() PASSED
[2021-10-20T08:52:07.345Z] 
[2021-10-20T08:52:07.345Z] ControllerIntegrationTest > 
testMetadataPropagationOnControlPlane() STARTED
[2021-10-20T08:52:10.562Z] 
[2021-10-20T08:52:10.562Z] ControllerIntegrationTest > 
testMetadataPropagationOnControlPlane() PASSED
[2021-10-20T08:52:10.562Z] 
[2021-10-20T08:52:10.562Z] ControllerIntegrationTest > 
testControllerFeatureZNodeSetupWhenFeatureVersioningIsEnabledWi

Re: [VOTE] KIP-764 Configurable backlog size for creating Acceptor

2021-10-20 Thread Haruki Okada 
Currently, KIP-764 got
+1 binding
+2 non-binding

votes. Could anyone help checking the KIP and join the vote?

Thanks,

2021年10月18日(月) 18:21 Haruki Okada :

> Hi Lucas.
>
> Thanks for the vote.
> As far as I understand, it should be enough to adjust `net.core.somaxconn`
> and `net.ipv4.tcp_max_syn_backlog` accordingly for allocating sufficient
> backlog size.
>
>
> 2021年10月18日(月) 17:37 Lucas Bradstreet :
>
>> The other related kernel config that I can recall is
>> net.ipv4.tcp_max_syn_backlog.
>>
>> On Mon, Oct 18, 2021 at 4:32 PM Lucas Bradstreet 
>> wrote:
>>
>> > Thank you for the KIP. +1 (non-binding)
>> >
>> > For the implementation can we ensure that any kernel parameters that may
>> > need to also be adjusted are documented in the config documentation
>> > (e.g. net.core.somaxconn)?
>> >
>> >
>> > On Mon, Oct 18, 2021 at 4:23 PM Haruki Okada 
>> wrote:
>> >
>> >> Hi Luke.
>> >>
>> >> Thank you for the vote.
>> >> Updated KIP to link to ServerSocket#bind javadoc.
>> >>
>> >>
>> >> 2021年10月18日(月) 17:00 Luke Chen :
>> >>
>> >> > Hi Okada,
>> >> > Thanks for the KIP.
>> >> > +1 (non-binding)
>> >> >
>> >> > One thing to add is that you should add ServerSocket#bind java doc
>> link
>> >> > into the KIP.
>> >> > I don't think everyone is familiar with the definition of the method
>> >> > parameters.
>> >> >
>> >> > Thank you.
>> >> > Luke
>> >> >
>> >> > On Mon, Oct 18, 2021 at 3:43 PM Haruki Okada 
>> >> wrote:
>> >> >
>> >> > > Hi Kafka.
>> >> > >
>> >> > > Let me bump this VOTE thread for the KIP.
>> >> > > We applied proposed changes in the KIP to our large Kafka cluster
>> by
>> >> > > building patched Kafka internally and confirmed it's working well.
>> >> > >
>> >> > > Please feel free to give your feedback if there's any points to be
>> >> > > clarified in the KIP.
>> >> > >
>> >> > > Thanks,
>> >> > >
>> >> > > 2021年8月9日(月) 11:25 Haruki Okada :
>> >> > >
>> >> > > > Thanks for your comment LI-san.
>> >> > > >
>> >> > > > Could anyone else review and vote for the KIP?
>> >> > > >
>> >> > > > I think the situation described in the KIP's motivation can
>> happen
>> >> in
>> >> > any
>> >> > > > large-scale Kafka deployment, so may be helpful for many users
>> while
>> >> > the
>> >> > > > proposed changes are small enough.
>> >> > > >
>> >> > > >
>> >> > > > Thanks,
>> >> > > >
>> >> > > > 2021年8月3日(火) 15:49 Xiangyuan LI :
>> >> > > >
>> >> > > >> Hi Haruki Okada:
>> >> > > >>   i read your comment, thx for your detail explain!
>> >> > > >>   add backlog parameter is a useful suggestion, hope it could
>> >> added to
>> >> > > >> kafka.
>> >> > > >>
>> >> > > >> Haruki Okada  于2021年8月2日周一 上午7:43写道：
>> >> > > >>
>> >> > > >> > Hi, Kafka.
>> >> > > >> >
>> >> > > >> > I would like to start a vote on KIP that makes SocketServer
>> >> > acceptor's
>> >> > > >> > backlog size configurable.
>> >> > > >> >
>> >> > > >> > KIP:
>> >> > > >> >
>> >> > > >> >
>> >> > > >>
>> >> > >
>> >> >
>> >>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-764%3A+Configurable+backlog+size+for+creating+Acceptor
>> >> > > >> >
>> >> > > >> > Discussion thread:
>> >> > > >> >
>> >> > > >> >
>> >> > > >>
>> >> > >
>> >> >
>> >>
>> https://lists.apache.org/thread.html/rd77469b7de0190d601dd37bd6894e1352a674d08038bcfe7ff68a1e0%40%3Cdev.kafka.apache.org%3E
>> >> > > >> >
>> >> > > >> > Thanks,
>> >> > > >> >
>> >> > > >> > --
>> >> > > >> > 
>> >> > > >> > Okada Haruki
>> >> > > >> > ocadar...@gmail.com
>> >> > > >> > 
>> >> > > >> >
>> >> > > >>
>> >> > > >
>> >> > > >
>> >> > > > --
>> >> > > > 
>> >> > > > Okada Haruki
>> >> > > > ocadar...@gmail.com
>> >> > > > 
>> >> > > >
>> >> > >
>> >> > >
>> >> > > --
>> >> > > 
>> >> > > Okada Haruki
>> >> > > ocadar...@gmail.com
>> >> > > 
>> >> > >
>> >> >
>> >>
>> >>
>> >> --
>> >> 
>> >> Okada Haruki
>> >> ocadar...@gmail.com
>> >> 
>> >>
>> >
>>
>
>
> --
> 
> Okada Haruki
> ocadar...@gmail.com
> 
>


-- 

Okada Haruki
ocadar...@gmail.com

Re: [VOTE] KIP-764 Configurable backlog size for creating Acceptor

2021-10-20 Thread Mickael Maison 
Thanks for the KIP!

I think the description/doc for the new setting needs some
improvements but we can follow that up in the PR.

+1 (binding)

On Wed, Oct 20, 2021 at 12:09 PM Haruki Okada  wrote:
>
> Currently, KIP-764 got
> +1 binding
> +2 non-binding
>
> votes. Could anyone help checking the KIP and join the vote?
>
> Thanks,
>
> 2021年10月18日(月) 18:21 Haruki Okada :
>
> > Hi Lucas.
> >
> > Thanks for the vote.
> > As far as I understand, it should be enough to adjust `net.core.somaxconn`
> > and `net.ipv4.tcp_max_syn_backlog` accordingly for allocating sufficient
> > backlog size.
> >
> >
> > 2021年10月18日(月) 17:37 Lucas Bradstreet :
> >
> >> The other related kernel config that I can recall is
> >> net.ipv4.tcp_max_syn_backlog.
> >>
> >> On Mon, Oct 18, 2021 at 4:32 PM Lucas Bradstreet 
> >> wrote:
> >>
> >> > Thank you for the KIP. +1 (non-binding)
> >> >
> >> > For the implementation can we ensure that any kernel parameters that may
> >> > need to also be adjusted are documented in the config documentation
> >> > (e.g. net.core.somaxconn)?
> >> >
> >> >
> >> > On Mon, Oct 18, 2021 at 4:23 PM Haruki Okada 
> >> wrote:
> >> >
> >> >> Hi Luke.
> >> >>
> >> >> Thank you for the vote.
> >> >> Updated KIP to link to ServerSocket#bind javadoc.
> >> >>
> >> >>
> >> >> 2021年10月18日(月) 17:00 Luke Chen :
> >> >>
> >> >> > Hi Okada,
> >> >> > Thanks for the KIP.
> >> >> > +1 (non-binding)
> >> >> >
> >> >> > One thing to add is that you should add ServerSocket#bind java doc
> >> link
> >> >> > into the KIP.
> >> >> > I don't think everyone is familiar with the definition of the method
> >> >> > parameters.
> >> >> >
> >> >> > Thank you.
> >> >> > Luke
> >> >> >
> >> >> > On Mon, Oct 18, 2021 at 3:43 PM Haruki Okada 
> >> >> wrote:
> >> >> >
> >> >> > > Hi Kafka.
> >> >> > >
> >> >> > > Let me bump this VOTE thread for the KIP.
> >> >> > > We applied proposed changes in the KIP to our large Kafka cluster
> >> by
> >> >> > > building patched Kafka internally and confirmed it's working well.
> >> >> > >
> >> >> > > Please feel free to give your feedback if there's any points to be
> >> >> > > clarified in the KIP.
> >> >> > >
> >> >> > > Thanks,
> >> >> > >
> >> >> > > 2021年8月9日(月) 11:25 Haruki Okada :
> >> >> > >
> >> >> > > > Thanks for your comment LI-san.
> >> >> > > >
> >> >> > > > Could anyone else review and vote for the KIP?
> >> >> > > >
> >> >> > > > I think the situation described in the KIP's motivation can
> >> happen
> >> >> in
> >> >> > any
> >> >> > > > large-scale Kafka deployment, so may be helpful for many users
> >> while
> >> >> > the
> >> >> > > > proposed changes are small enough.
> >> >> > > >
> >> >> > > >
> >> >> > > > Thanks,
> >> >> > > >
> >> >> > > > 2021年8月3日(火) 15:49 Xiangyuan LI :
> >> >> > > >
> >> >> > > >> Hi Haruki Okada:
> >> >> > > >>   i read your comment, thx for your detail explain!
> >> >> > > >>   add backlog parameter is a useful suggestion, hope it could
> >> >> added to
> >> >> > > >> kafka.
> >> >> > > >>
> >> >> > > >> Haruki Okada  于2021年8月2日周一 上午7:43写道：
> >> >> > > >>
> >> >> > > >> > Hi, Kafka.
> >> >> > > >> >
> >> >> > > >> > I would like to start a vote on KIP that makes SocketServer
> >> >> > acceptor's
> >> >> > > >> > backlog size configurable.
> >> >> > > >> >
> >> >> > > >> > KIP:
> >> >> > > >> >
> >> >> > > >> >
> >> >> > > >>
> >> >> > >
> >> >> >
> >> >>
> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-764%3A+Configurable+backlog+size+for+creating+Acceptor
> >> >> > > >> >
> >> >> > > >> > Discussion thread:
> >> >> > > >> >
> >> >> > > >> >
> >> >> > > >>
> >> >> > >
> >> >> >
> >> >>
> >> https://lists.apache.org/thread.html/rd77469b7de0190d601dd37bd6894e1352a674d08038bcfe7ff68a1e0%40%3Cdev.kafka.apache.org%3E
> >> >> > > >> >
> >> >> > > >> > Thanks,
> >> >> > > >> >
> >> >> > > >> > --
> >> >> > > >> > 
> >> >> > > >> > Okada Haruki
> >> >> > > >> > ocadar...@gmail.com
> >> >> > > >> > 
> >> >> > > >> >
> >> >> > > >>
> >> >> > > >
> >> >> > > >
> >> >> > > > --
> >> >> > > > 
> >> >> > > > Okada Haruki
> >> >> > > > ocadar...@gmail.com
> >> >> > > > 
> >> >> > > >
> >> >> > >
> >> >> > >
> >> >> > > --
> >> >> > > 
> >> >> > > Okada Haruki
> >> >> > > ocadar...@gmail.com
> >> >> > > 
> >> >> > >
> >> >> >
> >> >>
> >> >>
> >> >> --
> >> >> 
> >> >> Okada Haruki
> >> >> ocadar...@gmail.com
> >> >> 
> >> >>
> >> >
> >>
> >
> >
> > --
> > 
> > Okada Haruki
> > ocadar...@gmail.com
> > 
> >
>
>
> --
> 
> Okada Haruki
> ocadar...@gmail.com
>

Re: [VOTE] KIP-764 Configurable backlog size for creating Acceptor

2021-10-20 Thread David Jacot 
+1 (binding). Thanks for the KIP!

Best,
David

On Wed, Oct 20, 2021 at 12:09 PM Haruki Okada  wrote:

> Currently, KIP-764 got
> +1 binding
> +2 non-binding
>
> votes. Could anyone help checking the KIP and join the vote?
>
> Thanks,
>
> 2021年10月18日(月) 18:21 Haruki Okada :
>
> > Hi Lucas.
> >
> > Thanks for the vote.
> > As far as I understand, it should be enough to adjust
> `net.core.somaxconn`
> > and `net.ipv4.tcp_max_syn_backlog` accordingly for allocating sufficient
> > backlog size.
> >
> >
> > 2021年10月18日(月) 17:37 Lucas Bradstreet :
> >
> >> The other related kernel config that I can recall is
> >> net.ipv4.tcp_max_syn_backlog.
> >>
> >> On Mon, Oct 18, 2021 at 4:32 PM Lucas Bradstreet 
> >> wrote:
> >>
> >> > Thank you for the KIP. +1 (non-binding)
> >> >
> >> > For the implementation can we ensure that any kernel parameters that
> may
> >> > need to also be adjusted are documented in the config documentation
> >> > (e.g. net.core.somaxconn)?
> >> >
> >> >
> >> > On Mon, Oct 18, 2021 at 4:23 PM Haruki Okada 
> >> wrote:
> >> >
> >> >> Hi Luke.
> >> >>
> >> >> Thank you for the vote.
> >> >> Updated KIP to link to ServerSocket#bind javadoc.
> >> >>
> >> >>
> >> >> 2021年10月18日(月) 17:00 Luke Chen :
> >> >>
> >> >> > Hi Okada,
> >> >> > Thanks for the KIP.
> >> >> > +1 (non-binding)
> >> >> >
> >> >> > One thing to add is that you should add ServerSocket#bind java doc
> >> link
> >> >> > into the KIP.
> >> >> > I don't think everyone is familiar with the definition of the
> method
> >> >> > parameters.
> >> >> >
> >> >> > Thank you.
> >> >> > Luke
> >> >> >
> >> >> > On Mon, Oct 18, 2021 at 3:43 PM Haruki Okada 
> >> >> wrote:
> >> >> >
> >> >> > > Hi Kafka.
> >> >> > >
> >> >> > > Let me bump this VOTE thread for the KIP.
> >> >> > > We applied proposed changes in the KIP to our large Kafka cluster
> >> by
> >> >> > > building patched Kafka internally and confirmed it's working
> well.
> >> >> > >
> >> >> > > Please feel free to give your feedback if there's any points to
> be
> >> >> > > clarified in the KIP.
> >> >> > >
> >> >> > > Thanks,
> >> >> > >
> >> >> > > 2021年8月9日(月) 11:25 Haruki Okada :
> >> >> > >
> >> >> > > > Thanks for your comment LI-san.
> >> >> > > >
> >> >> > > > Could anyone else review and vote for the KIP?
> >> >> > > >
> >> >> > > > I think the situation described in the KIP's motivation can
> >> happen
> >> >> in
> >> >> > any
> >> >> > > > large-scale Kafka deployment, so may be helpful for many users
> >> while
> >> >> > the
> >> >> > > > proposed changes are small enough.
> >> >> > > >
> >> >> > > >
> >> >> > > > Thanks,
> >> >> > > >
> >> >> > > > 2021年8月3日(火) 15:49 Xiangyuan LI :
> >> >> > > >
> >> >> > > >> Hi Haruki Okada:
> >> >> > > >>   i read your comment, thx for your detail explain!
> >> >> > > >>   add backlog parameter is a useful suggestion, hope it could
> >> >> added to
> >> >> > > >> kafka.
> >> >> > > >>
> >> >> > > >> Haruki Okada  于2021年8月2日周一 上午7:43写道：
> >> >> > > >>
> >> >> > > >> > Hi, Kafka.
> >> >> > > >> >
> >> >> > > >> > I would like to start a vote on KIP that makes SocketServer
> >> >> > acceptor's
> >> >> > > >> > backlog size configurable.
> >> >> > > >> >
> >> >> > > >> > KIP:
> >> >> > > >> >
> >> >> > > >> >
> >> >> > > >>
> >> >> > >
> >> >> >
> >> >>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-764%3A+Configurable+backlog+size+for+creating+Acceptor
> >> >> > > >> >
> >> >> > > >> > Discussion thread:
> >> >> > > >> >
> >> >> > > >> >
> >> >> > > >>
> >> >> > >
> >> >> >
> >> >>
> >>
> https://lists.apache.org/thread.html/rd77469b7de0190d601dd37bd6894e1352a674d08038bcfe7ff68a1e0%40%3Cdev.kafka.apache.org%3E
> >> >> > > >> >
> >> >> > > >> > Thanks,
> >> >> > > >> >
> >> >> > > >> > --
> >> >> > > >> > 
> >> >> > > >> > Okada Haruki
> >> >> > > >> > ocadar...@gmail.com
> >> >> > > >> > 
> >> >> > > >> >
> >> >> > > >>
> >> >> > > >
> >> >> > > >
> >> >> > > > --
> >> >> > > > 
> >> >> > > > Okada Haruki
> >> >> > > > ocadar...@gmail.com
> >> >> > > > 
> >> >> > > >
> >> >> > >
> >> >> > >
> >> >> > > --
> >> >> > > 
> >> >> > > Okada Haruki
> >> >> > > ocadar...@gmail.com
> >> >> > > 
> >> >> > >
> >> >> >
> >> >>
> >> >>
> >> >> --
> >> >> 
> >> >> Okada Haruki
> >> >> ocadar...@gmail.com
> >> >> 
> >> >>
> >> >
> >>
> >
> >
> > --
> > 
> > Okada Haruki
> > ocadar...@gmail.com
> > 
> >
>
>
> --
> 
> Okada Haruki
> ocadar...@gmail.com
> 
>

Re: [DISCUSS] KIP-782: Expandable batch size in producer

2021-10-20 Thread Luke Chen 
Hi Ismael and all devs,
Is there any comments/suggestions to this KIP?
If no, I'm going to update the KIP based on my previous mail, and start a
vote tomorrow or next week.

Thank you.
Luke

On Mon, Oct 18, 2021 at 2:40 PM Luke Chen  wrote:

> Hi Ismael,
> Thanks for your comments.
>
> 1. Why do we have to reallocate the buffer? We can keep a list of buffers
> instead and avoid reallocation.
> -> Do you mean we allocate multiple buffers with "buffer.initial.size",
> and link them together (with linked list)?
> ex:
> a. We allocate 4KB initial buffer
> | 4KB |
>
> b. when new records reached and the remaining buffer is not enough for the
> records, we create another batch with "batch.initial.size" buffer
> ex: we already have 3KB of data in the 1st buffer, and here comes the 2KB
> record
>
> | 4KB (1KB remaining) |
> now, record: 2KB coming
> We fill the 1st 1KB into 1st buffer, and create new buffer, and linked
> together, and fill the rest of data into it
> | 4KB (full) | ---> | 4KB (3KB remaining) |
>
> Is that what you mean?
> If so, I think I like this idea!
> If not, please explain more detail about it.
> Thank you.
>
> 2. I think we should also consider tweaking the semantics of batch.size so
> that the sent batches can be larger if the batch is not ready to be sent
> (while still respecting max.request.size and perhaps a new max.batch.size).
>
> --> In the KIP, I was trying to make the "batch.size" as the upper bound
> of the batch size, and introduce a "batch.initial.size" as initial batch
> size.
> So are you saying that we can let "batch.size" as initial batch size and
> introduce a "max.batch.size" as upper bound value?
> That's a good suggestion, but that would change the semantics of
> "batch.size", which might surprise some users. I think my original proposal
> ("batch.initial.size") is safer for users. What do you think?
>
> Thank you.
> Luke
>
>
> On Mon, Oct 18, 2021 at 3:12 AM Ismael Juma  wrote:
>
>> I think we should also consider tweaking the semantics of batch.size so
>> that the sent batches can be larger if the batch is not ready to be sent
>> (while still respecting max.request.size and perhaps a new
>> max.batch.size).
>>
>> Ismael
>>
>> On Sun, Oct 17, 2021, 12:08 PM Ismael Juma  wrote:
>>
>> > Hi Luke,
>> >
>> > Thanks for the KIP. Why do we have to reallocate the buffer? We can
>> keep a
>> > list of buffers instead and avoid reallocation.
>> >
>> > Ismael
>> >
>> > On Sun, Oct 17, 2021, 2:02 AM Luke Chen  wrote:
>> >
>> >> Hi Kafka dev,
>> >> I'd like to start the discussion for the proposal: KIP-782: Expandable
>> >> batch size in producer.
>> >>
>> >> The main purpose for this KIP is to have better memory usage in
>> producer,
>> >> and also save users from the dilemma while setting the batch size
>> >> configuration. After this KIP, users can set a higher batch.size
>> without
>> >> worries, and of course, with an appropriate "batch.initial.size" and
>> >> "batch.reallocation.factor".
>> >>
>> >> Derailed description can be found here:
>> >>
>> >>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-782%3A+Expandable+batch+size+in+producer
>> >>
>> >> Any comments and feedback are welcome.
>> >>
>> >> Thank you.
>> >> Luke
>> >>
>> >
>>
>

Re: [VOTE] KIP-764 Configurable backlog size for creating Acceptor

2021-10-20 Thread Haruki Okada 
Hi all,

Thanks for participating in the vote.
With 3 binding +1 votes (Rajini Sivaram, Mickael Maison, David Jacot) and 2
non-binding +1 votes (Luke Chen, Lucas Bradstreet), the vote passes.

I will update the KIP status then raise a PR later.

Thanks,

2021年10月20日(水) 20:58 David Jacot :

> +1 (binding). Thanks for the KIP!
>
> Best,
> David
>
> On Wed, Oct 20, 2021 at 12:09 PM Haruki Okada  wrote:
>
> > Currently, KIP-764 got
> > +1 binding
> > +2 non-binding
> >
> > votes. Could anyone help checking the KIP and join the vote?
> >
> > Thanks,
> >
> > 2021年10月18日(月) 18:21 Haruki Okada :
> >
> > > Hi Lucas.
> > >
> > > Thanks for the vote.
> > > As far as I understand, it should be enough to adjust
> > `net.core.somaxconn`
> > > and `net.ipv4.tcp_max_syn_backlog` accordingly for allocating
> sufficient
> > > backlog size.
> > >
> > >
> > > 2021年10月18日(月) 17:37 Lucas Bradstreet :
> > >
> > >> The other related kernel config that I can recall is
> > >> net.ipv4.tcp_max_syn_backlog.
> > >>
> > >> On Mon, Oct 18, 2021 at 4:32 PM Lucas Bradstreet 
> > >> wrote:
> > >>
> > >> > Thank you for the KIP. +1 (non-binding)
> > >> >
> > >> > For the implementation can we ensure that any kernel parameters that
> > may
> > >> > need to also be adjusted are documented in the config documentation
> > >> > (e.g. net.core.somaxconn)?
> > >> >
> > >> >
> > >> > On Mon, Oct 18, 2021 at 4:23 PM Haruki Okada 
> > >> wrote:
> > >> >
> > >> >> Hi Luke.
> > >> >>
> > >> >> Thank you for the vote.
> > >> >> Updated KIP to link to ServerSocket#bind javadoc.
> > >> >>
> > >> >>
> > >> >> 2021年10月18日(月) 17:00 Luke Chen :
> > >> >>
> > >> >> > Hi Okada,
> > >> >> > Thanks for the KIP.
> > >> >> > +1 (non-binding)
> > >> >> >
> > >> >> > One thing to add is that you should add ServerSocket#bind java
> doc
> > >> link
> > >> >> > into the KIP.
> > >> >> > I don't think everyone is familiar with the definition of the
> > method
> > >> >> > parameters.
> > >> >> >
> > >> >> > Thank you.
> > >> >> > Luke
> > >> >> >
> > >> >> > On Mon, Oct 18, 2021 at 3:43 PM Haruki Okada <
> ocadar...@gmail.com>
> > >> >> wrote:
> > >> >> >
> > >> >> > > Hi Kafka.
> > >> >> > >
> > >> >> > > Let me bump this VOTE thread for the KIP.
> > >> >> > > We applied proposed changes in the KIP to our large Kafka
> cluster
> > >> by
> > >> >> > > building patched Kafka internally and confirmed it's working
> > well.
> > >> >> > >
> > >> >> > > Please feel free to give your feedback if there's any points to
> > be
> > >> >> > > clarified in the KIP.
> > >> >> > >
> > >> >> > > Thanks,
> > >> >> > >
> > >> >> > > 2021年8月9日(月) 11:25 Haruki Okada :
> > >> >> > >
> > >> >> > > > Thanks for your comment LI-san.
> > >> >> > > >
> > >> >> > > > Could anyone else review and vote for the KIP?
> > >> >> > > >
> > >> >> > > > I think the situation described in the KIP's motivation can
> > >> happen
> > >> >> in
> > >> >> > any
> > >> >> > > > large-scale Kafka deployment, so may be helpful for many
> users
> > >> while
> > >> >> > the
> > >> >> > > > proposed changes are small enough.
> > >> >> > > >
> > >> >> > > >
> > >> >> > > > Thanks,
> > >> >> > > >
> > >> >> > > > 2021年8月3日(火) 15:49 Xiangyuan LI :
> > >> >> > > >
> > >> >> > > >> Hi Haruki Okada:
> > >> >> > > >>   i read your comment, thx for your detail explain!
> > >> >> > > >>   add backlog parameter is a useful suggestion, hope it
> could
> > >> >> added to
> > >> >> > > >> kafka.
> > >> >> > > >>
> > >> >> > > >> Haruki Okada  于2021年8月2日周一 上午7:43写道：
> > >> >> > > >>
> > >> >> > > >> > Hi, Kafka.
> > >> >> > > >> >
> > >> >> > > >> > I would like to start a vote on KIP that makes
> SocketServer
> > >> >> > acceptor's
> > >> >> > > >> > backlog size configurable.
> > >> >> > > >> >
> > >> >> > > >> > KIP:
> > >> >> > > >> >
> > >> >> > > >> >
> > >> >> > > >>
> > >> >> > >
> > >> >> >
> > >> >>
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-764%3A+Configurable+backlog+size+for+creating+Acceptor
> > >> >> > > >> >
> > >> >> > > >> > Discussion thread:
> > >> >> > > >> >
> > >> >> > > >> >
> > >> >> > > >>
> > >> >> > >
> > >> >> >
> > >> >>
> > >>
> >
> https://lists.apache.org/thread.html/rd77469b7de0190d601dd37bd6894e1352a674d08038bcfe7ff68a1e0%40%3Cdev.kafka.apache.org%3E
> > >> >> > > >> >
> > >> >> > > >> > Thanks,
> > >> >> > > >> >
> > >> >> > > >> > --
> > >> >> > > >> > 
> > >> >> > > >> > Okada Haruki
> > >> >> > > >> > ocadar...@gmail.com
> > >> >> > > >> > 
> > >> >> > > >> >
> > >> >> > > >>
> > >> >> > > >
> > >> >> > > >
> > >> >> > > > --
> > >> >> > > > 
> > >> >> > > > Okada Haruki
> > >> >> > > > ocadar...@gmail.com
> > >> >> > > > 
> > >> >> > > >
> > >> >> > >
> > >> >> > >
> > >> >> > > --
> > >> >> > > 
> > >> >> > > Okada Haruki
> > >> >> > > ocadar...@gmail.com
> > >> >> > > 
> > >> >> > >
> > >> >> >
> > >> >>
> > >> >>
> > >> >> --

[jira] [Resolved] (KAFKA-7588) Rationalize configurations passed to pluggable APIs

2021-10-20 Thread Mickael Maison (Jira) 


 [ 
https://issues.apache.org/jira/browse/KAFKA-7588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mickael Maison resolved KAFKA-7588.
---
Fix Version/s: 2.5.0
   Resolution: Fixed

> Rationalize configurations passed to pluggable APIs
> ---
>
> Key: KAFKA-7588
> URL: https://issues.apache.org/jira/browse/KAFKA-7588
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Mickael Maison
>Assignee: Mickael Maison
>Priority: Major
> Fix For: 2.5.0
>
>
> There are a lot of extensions points both on the client and server sides. 
> Most of these pluggable APIs are configurable but the configurations they 
> receive are not the same.
> For example, Serializers, Deserializers, Partitioners, Assignors, 
> QuotaCallbacks are passed config.originals(). On the other hand LoginModules, 
> PrincipalBuilders and AuthenticationCallbackHandlers are passed 
> config.values().
> In practice, having access to originals() is nice as it allows to use custom 
> configurations by simply adding it to the client/server configuration.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #531

2021-10-20 Thread Apache Jenkins Server 
See

[jira] [Created] (KAFKA-13388) Kafka Producer has no timeout for nodes stuck in CHECKING_API_VERSIONS

2021-10-20 Thread David Hoffman (Jira) 
David Hoffman created KAFKA-13388:
-

 Summary: Kafka Producer has no timeout for nodes stuck in 
CHECKING_API_VERSIONS
 Key: KAFKA-13388
 URL: https://issues.apache.org/jira/browse/KAFKA-13388
 Project: Kafka
  Issue Type: Bug
  Components: core
Reporter: David Hoffman


I have been seeing expired batch errors in my app.
{code:java}
org.apache.kafka.common.errors.TimeoutException: Expiring 51 record(s) for 
xxx-17:120002 ms has passed since batch creation
{code}
 I would have assumed a request timout or connection timeout should have also 
been logged. I could not find any other associated errors. 

I added some instrumenting to my app and have traced this down to broker 
connections hanging in CHECKING_API_VERSIONS state. It appears there is no 
effective timeout for Kafka Producer broker connections in 
CHECKING_API_VERSIONS state.

In the code see the after the NetworkClient connects to a broker node it makes 
a request to check api versions, when it receives the response it marks the 
node as ready. I am seeing that sometimes a reply is not received for the check 
api versions request the connection just hangs in CHECKING_API_VERSIONS state 
until it is disposed I assume after the idle connection timeout.

I am guessing the connection setup timeout should be still in play for this, 
but it is not. 
There is a connectingNodes set that is consulted when checking timeouts and the 
node is removed 
when ClusterConnectionStates.checkingApiVersions(String id) is called to 
transition the node into CHECKING_API_VERSIONS



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-13389) Add to kafka shell scripts checks about server state

2021-10-20 Thread Seweryn Habdank-Wojewodzki (Jira) 
Seweryn Habdank-Wojewodzki created KAFKA-13389:
--

 Summary: Add to kafka shell scripts checks about server state
 Key: KAFKA-13389
 URL: https://issues.apache.org/jira/browse/KAFKA-13389
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 2.8.0
Reporter: Seweryn Habdank-Wojewodzki


Hello,

Within the discussion with Confluent included in the Confluent Support Ticket: 
[#71907|https://urldefense.com/v3/__https:/support.confluent.io/hc/requests/71907__;!!OMGRPR5eiCE28w!9-OfZd3vUrXgjEtagEYeB1O5tmebDaANKfi6c-VRV0RrdcFEnFlzb7pDwpSwJTZ7qFnigilEAPhGW1vS5XdsSkU$],
 we found out that according to "Eventually Consistency" Kafka shell scripts 
may deliver wrong information, for example when listing topics, the result 
might be empty even if topics are existing, but Server status is not in synch 
(e.g. when URP > 0).

To be concrete. This call below may return empty list, if server is not in 
synch.

{code}

$ ./bin/kafka-topics.sh --bootstrap-server= 
--list

{code}

 

Remark from Confluent engineers is: that before getting whose results, one have 
to check server status and in particular URP shall be 0, otherwise results 
might be wrong.

So in fact Kafka shell scripts contains bug delivering possibly broken results 
and not reporting error instead.

The proposal here is to add to all Kafka shell scripts check if server status 
is proper (e.g. URP is 0) and in case of having server not in good state, 
instead of returning possible wrong values, script shall return proper error 
code with message, that server is not in proper state.

Why in Kafka shell scripts and not on the user side?

Because Kafka Team knows all server conditions and can describe server status 
much better than any other user and checks will be done centrally for all 
users, who do not need to always implement the same. Also updates, when Kafka 
changes own API will be done synchronously.

 

Thanks in advance for adding those checks and best regards,

Seweryn.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)