Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #335

2021-07-19 Thread Apache Jenkins Server
See 




Apache Kafka, permission to contribute

2021-07-19 Thread Patrick Stuedi
Hi all,

I would like to request permission to contribute to Apache Kafka. My wiki
and jira id is "pstuedi".

Thanks!
-Patrick


[DISCUSS] KIP-763: Range queries with open endpoints

2021-07-19 Thread Patrick Stuedi
Hi everyone,

I would like to start the discussion for KIP-763: Range queries with open
endpoints.

The KIP can be found here:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-763%3A+Range+queries+with+open+endpoints

Any feedback will be highly appreciated.

Many thanks,
 Patrick


[jira] [Created] (KAFKA-13103) Should group admin handlers consider REBALANCE_IN_PROGRESS and GROUP_AUTHORIZATION_FAILED as retryable errors?

2021-07-19 Thread David Jacot (Jira)
David Jacot created KAFKA-13103:
---

 Summary: Should group admin handlers consider 
REBALANCE_IN_PROGRESS and GROUP_AUTHORIZATION_FAILED as retryable errors?
 Key: KAFKA-13103
 URL: https://issues.apache.org/jira/browse/KAFKA-13103
 Project: Kafka
  Issue Type: Improvement
Reporter: David Jacot


[~rajinisiva...@gmail.com] and I were discussing if we should consider 
REBALANCE_IN_PROGRESS and GROUP_AUTHORIZATION_FAILED as retryable errors in the 
group handlers. I think that this could make sense, especially for 
`REBALANCE_IN_PROGRESS`. `GROUP_AUTHORIZATION_FAILED` is more debatable as it 
means that the handler would retry until the request timeout is reached. It 
might be armful if the authorisation is really denied.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Resolved] (KAFKA-13055) Security without JAAS configuration

2021-07-19 Thread Alexey Kashavkin (Jira)


 [ 
https://issues.apache.org/jira/browse/KAFKA-13055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexey Kashavkin resolved KAFKA-13055.
--
Resolution: Workaround

I did it like this:
{code:bash}
listener.name.clients.scram-sha-256.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule
 required;
listener.name.clients-nossl.scram-sha-256.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule
 required;
{code}
Brokers work with it correctly.

> Security without JAAS configuration
> ---
>
> Key: KAFKA-13055
> URL: https://issues.apache.org/jira/browse/KAFKA-13055
> Project: Kafka
>  Issue Type: Bug
>  Components: security
>Affects Versions: 2.8.0
>Reporter: Alexey Kashavkin
>Priority: Major
>
> I think that this isn't a bug, but I don't know what type to choose for this 
> issue.
> I'm setting up kafka security with multiple listeners. Listener for broker 
> works as plaintext and I don't want to use authentication for them. My 
> cluster has configuration: 
> {code:bash}
> listeners=BROKERS://:9091,CLIENTS://:9092,CLIENTS-NOSSL://:9093
> inter.broker.listener.name=BROKERS
> sasl.enabled.mechanisms=SCRAM-SHA-256
> listener.security.protocol.map=BROKERS:PLAINTEXT,CLIENTS:SASL_SSL,CLIENTS-NOSSL:SASL_PLAINTEXT
> {code}
> For clients, I want use SCRAM and only for them (not for brokers). But when I 
> start broker, I see following message in log:
> {code:bash}
> ERROR [KafkaServer id=0] Fatal error during KafkaServer startup. Prepare to 
> shutdown (kafka.server.KafkaServer)
> java.lang.IllegalArgumentException: Could not find a 'KafkaServer' or 
> 'clients.KafkaServer' entry in the JAAS configuration. System property 
> 'java.security.auth.login.config' is not set
> {code}
> Why is this configuration so necessary, if I don't use authentication for 
> brokers and SCRAM mechanism stores credentials in zookeeper? Which user do I 
> need to add to {{sasl.jaas.config}}? I have created users who perform 
> operations as consumers and producers. I didn't create any admin users 
> because I don't need it. I didn't find any more detailed information in 
> [docs|http://kafka.apache.org/documentation/#security] about why this is so 
> necessary.
> For experiment:
> {code:bash}
> listener.name.clients.scram-sha-256.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule
>  required \
> username="alice" \
> password="alice-secret";
> --
> listener.name.clients-nossl.scram-sha-256.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule
>  required \
> username="alice" \
> password="alice-secret";
> {code}
> I specified user alice in {{sasl.jaas.config}}, but I didn't create this user 
> and Kafka starts correctly. What is the logic of this?



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #336

2021-07-19 Thread Apache Jenkins Server
See 




[jira] [Resolved] (KAFKA-13058) `AlterConsumerGroupOffsetsHandler` does not handle partition errors correctly.

2021-07-19 Thread David Jacot (Jira)


 [ 
https://issues.apache.org/jira/browse/KAFKA-13058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Jacot resolved KAFKA-13058.
-
  Reviewer: Rajini Sivaram
Resolution: Fixed

> `AlterConsumerGroupOffsetsHandler` does not handle partition errors correctly.
> --
>
> Key: KAFKA-13058
> URL: https://issues.apache.org/jira/browse/KAFKA-13058
> Project: Kafka
>  Issue Type: Bug
>Reporter: David Jacot
>Assignee: David Jacot
>Priority: Blocker
> Fix For: 3.0.0
>
>
> While reviewing https://github.com/apache/kafka/pull/10973, we have noticed 
> that `AlterConsumerGroupOffsetsHandler` does not handle the partition errors 
> correctly. The issue is that any partition error fails the entire future 
> returned in the results instead of failing only the future of the 
> corresponding partition. This is a regression that was introduced by KIP-699.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[DISCUSS] [KAFKA-7572] Producer should not send requests with negative partition id

2021-07-19 Thread Wenhao Ji
Hi everyone,

I would like to start a discussion about KAFKA-7572 [1] and its
corresponding pull request [2].

The ticket is about a bug that will happen if a user-defined
partitioner returns a negative partition number. KafkaProducer will
try to send this record to a nonexistent partition again and again
until timeout. And eventually, it will print a confusing error log.

The current problem of the PR is that whether we should
- throw an IllegalArgumentException
- or introduce another kind of KafkaException subclass (e.g.
InvalidPartitionException)
when a partitioner returns a negative integer.

As the pull request has not been reviewed for several months since the
last review and I cannot click the "Re-request review" on Github, I
would appreciate it if someone could help me review the code again and
also join us and share some thoughts about the problem.

[1] https://issues.apache.org/jira/browse/KAFKA-7572
[2] https://github.com/apache/kafka/pull/10525

Thanks,
Wenhao


[jira] [Resolved] (KAFKA-9555) Topic-based implementation for the RemoteLogMetadataManager

2021-07-19 Thread Jun Rao (Jira)


 [ 
https://issues.apache.org/jira/browse/KAFKA-9555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao resolved KAFKA-9555.

Resolution: Fixed

merged the PR to trunk.

> Topic-based implementation for the RemoteLogMetadataManager
> ---
>
> Key: KAFKA-9555
> URL: https://issues.apache.org/jira/browse/KAFKA-9555
> Project: Kafka
>  Issue Type: Sub-task
>  Components: core
>Reporter: Alexandre Dupriez
>Assignee: Satish Duggana
>Priority: Major
> Fix For: 3.1.0
>
>
> The purpose of this task is to implement a {{RemoteLogMetadataManager}} based 
> on an internal topic in Kafka. More details ar mentioned in the 
> [KIP|https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage#KIP405:KafkaTieredStorage-RemoteLogMetadataManagerimplementedwithaninternaltopic].
> Done means:
>  - Pull Request available for review and unit-tests.
> System and integration tests are out of scope of this task and will be part 
> of another task.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #337

2021-07-19 Thread Apache Jenkins Server
See 




Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.0 #43

2021-07-19 Thread Apache Jenkins Server
See 




[jira] [Created] (KAFKA-13104) Controller should notify the RaftClient when it resigns

2021-07-19 Thread Jose Armando Garcia Sancio (Jira)
Jose Armando Garcia Sancio created KAFKA-13104:
--

 Summary: Controller should notify the RaftClient when it resigns
 Key: KAFKA-13104
 URL: https://issues.apache.org/jira/browse/KAFKA-13104
 Project: Kafka
  Issue Type: Bug
  Components: controller, kraft
Reporter: Jose Armando Garcia Sancio
 Fix For: 3.0.0


{code:java}
  private Throwable handleEventException(String name,
 Optional 
startProcessingTimeNs,
 Throwable exception) {
  ...
  renounce();
  return new UnknownServerException(exception);
  }
 {code}
When the active controller encounters an event exception it attempts to 
renounce leadership. Unfortunately, this doesn't tell the {{RaftClient}} that 
it should attempt to give up leadership. This will result in inconsistent state 
with the {{RaftClient}} as leader but with the controller as inactive.

We should change this implementation so that the active controller asks the 
{{RaftClient}} to resign. The active controller waits until 
{{handleLeaderChange}} before calling {{renounce()}}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #338

2021-07-19 Thread Apache Jenkins Server
See 




Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.0 #44

2021-07-19 Thread Apache Jenkins Server
See 




[jira] [Created] (KAFKA-13105) Expose a method in KafkaConfig to write the configs to a logger

2021-07-19 Thread Colin McCabe (Jira)
Colin McCabe created KAFKA-13105:


 Summary: Expose a method in KafkaConfig to write the configs to a 
logger
 Key: KAFKA-13105
 URL: https://issues.apache.org/jira/browse/KAFKA-13105
 Project: Kafka
  Issue Type: Bug
Reporter: Colin McCabe


We should expose a method in KafkaConfig to write the configs to a logger. 
Currently there is no good way to write them out except creating a new 
KafkaConfig object with doLog = true, which is unintuitive.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Resolved] (KAFKA-13090) Improve cluster snapshot integration test

2021-07-19 Thread Konstantine Karantasis (Jira)


 [ 
https://issues.apache.org/jira/browse/KAFKA-13090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantine Karantasis resolved KAFKA-13090.

Resolution: Fixed

Resolving given that the PR got merged and cherry-picked to 3.0: 
https://github.com/apache/kafka/pull/11054

> Improve cluster snapshot integration test
> -
>
> Key: KAFKA-13090
> URL: https://issues.apache.org/jira/browse/KAFKA-13090
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Jose Armando Garcia Sancio
>Assignee: Jose Armando Garcia Sancio
>Priority: Major
>  Labels: kip-500
> Fix For: 3.0.0
>
>
> Extends the test in RaftClusterSnapshotTest to verify that both the 
> controllers and brokers are generating snapshots.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Resolved] (KAFKA-12803) Support reassigning partitions when in KRaft mode

2021-07-19 Thread Konstantine Karantasis (Jira)


 [ 
https://issues.apache.org/jira/browse/KAFKA-12803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantine Karantasis resolved KAFKA-12803.

Resolution: Fixed

Resolving as Fixed given that the PR got merged and cherry-picked to 3.0:
https://github.com/apache/kafka/pull/10753

> Support reassigning partitions when in KRaft mode
> -
>
> Key: KAFKA-12803
> URL: https://issues.apache.org/jira/browse/KAFKA-12803
> Project: Kafka
>  Issue Type: Improvement
>  Components: controller
>Affects Versions: 2.8.0
>Reporter: Colin McCabe
>Assignee: Colin McCabe
>Priority: Major
>  Labels: kip-500
> Fix For: 3.0.0
>
>
> Support reassigning partitions when in KRaft mode



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #339

2021-07-19 Thread Apache Jenkins Server
See 


Changes:


--
[...truncated 485711 lines...]
[2021-07-19T21:40:12.671Z] 
[2021-07-19T21:40:12.671Z] DeleteOffsetsConsumerGroupCommandIntegrationTest > 
testDeleteOffsetsOfStableConsumerGroupWithUnknownTopicOnly() PASSED
[2021-07-19T21:40:12.671Z] 
[2021-07-19T21:40:12.671Z] TopicCommandIntegrationTest > 
testAlterPartitionCount() STARTED
[2021-07-19T21:40:16.198Z] 
[2021-07-19T21:40:16.198Z] TopicCommandIntegrationTest > 
testAlterPartitionCount() PASSED
[2021-07-19T21:40:16.198Z] 
[2021-07-19T21:40:16.198Z] TopicCommandIntegrationTest > 
testCreatePartitionsDoesNotRetryThrottlingQuotaExceededException() STARTED
[2021-07-19T21:40:19.924Z] 
[2021-07-19T21:40:19.924Z] TopicCommandIntegrationTest > 
testCreatePartitionsDoesNotRetryThrottlingQuotaExceededException() PASSED
[2021-07-19T21:40:19.924Z] 
[2021-07-19T21:40:19.924Z] TopicCommandIntegrationTest > 
testAlterWhenTopicDoesntExistWithIfExists() STARTED
[2021-07-19T21:40:24.586Z] 
[2021-07-19T21:40:24.586Z] TopicCommandIntegrationTest > 
testAlterWhenTopicDoesntExistWithIfExists() PASSED
[2021-07-19T21:40:24.586Z] 
[2021-07-19T21:40:24.586Z] TopicCommandIntegrationTest > 
testCreateWithDefaultReplication() STARTED
[2021-07-19T21:40:27.257Z] 
[2021-07-19T21:40:27.257Z] TopicCommandIntegrationTest > 
testCreateWithDefaultReplication() PASSED
[2021-07-19T21:40:27.257Z] 
[2021-07-19T21:40:27.257Z] TopicCommandIntegrationTest > 
testDescribeAtMinIsrPartitions() STARTED
[2021-07-19T21:40:35.869Z] 
[2021-07-19T21:40:35.869Z] TopicCommandIntegrationTest > 
testDescribeAtMinIsrPartitions() PASSED
[2021-07-19T21:40:35.869Z] 
[2021-07-19T21:40:35.869Z] TopicCommandIntegrationTest > 
testCreateWithNegativeReplicationFactor() STARTED
[2021-07-19T21:40:38.458Z] 
[2021-07-19T21:40:38.458Z] TopicCommandIntegrationTest > 
testCreateWithNegativeReplicationFactor() PASSED
[2021-07-19T21:40:38.458Z] 
[2021-07-19T21:40:38.458Z] TopicCommandIntegrationTest > 
testCreateWithInvalidReplicationFactor() STARTED
[2021-07-19T21:40:43.050Z] 
[2021-07-19T21:40:43.050Z] TopicCommandIntegrationTest > 
testCreateWithInvalidReplicationFactor() PASSED
[2021-07-19T21:40:43.050Z] 
[2021-07-19T21:40:43.050Z] TopicCommandIntegrationTest > 
testDeleteTopicDoesNotRetryThrottlingQuotaExceededException() STARTED
[2021-07-19T21:40:46.032Z] 
[2021-07-19T21:40:46.032Z] TopicCommandIntegrationTest > 
testDeleteTopicDoesNotRetryThrottlingQuotaExceededException() PASSED
[2021-07-19T21:40:46.032Z] 
[2021-07-19T21:40:46.032Z] TopicCommandIntegrationTest > 
testListTopicsWithExcludeInternal() STARTED
[2021-07-19T21:40:50.584Z] 
[2021-07-19T21:40:50.584Z] TopicCommandIntegrationTest > 
testListTopicsWithExcludeInternal() PASSED
[2021-07-19T21:40:50.584Z] 
[2021-07-19T21:40:50.584Z] TopicCommandIntegrationTest > 
testDescribeUnderReplicatedPartitionsWhenReassignmentIsInProgress() STARTED
[2021-07-19T21:40:57.622Z] 
[2021-07-19T21:40:57.622Z] TopicCommandIntegrationTest > 
testDescribeUnderReplicatedPartitionsWhenReassignmentIsInProgress() PASSED
[2021-07-19T21:40:57.622Z] 
[2021-07-19T21:40:57.622Z] TopicCommandIntegrationTest > 
testCreateWithNegativePartitionCount() STARTED
[2021-07-19T21:41:01.040Z] 
[2021-07-19T21:41:01.040Z] TopicCommandIntegrationTest > 
testCreateWithNegativePartitionCount() PASSED
[2021-07-19T21:41:01.040Z] 
[2021-07-19T21:41:01.040Z] TopicCommandIntegrationTest > 
testAlterWhenTopicDoesntExist() STARTED
[2021-07-19T21:41:04.684Z] 
[2021-07-19T21:41:04.684Z] TopicCommandIntegrationTest > 
testAlterWhenTopicDoesntExist() PASSED
[2021-07-19T21:41:04.684Z] 
[2021-07-19T21:41:04.684Z] TopicCommandIntegrationTest > 
testCreateAlterTopicWithRackAware() STARTED
[2021-07-19T21:41:10.362Z] 
[2021-07-19T21:41:10.363Z] TopicCommandIntegrationTest > 
testCreateAlterTopicWithRackAware() PASSED
[2021-07-19T21:41:10.363Z] 
[2021-07-19T21:41:10.363Z] TopicCommandIntegrationTest > 
testListTopicsWithIncludeList() STARTED
[2021-07-19T21:41:16.036Z] 
[2021-07-19T21:41:16.036Z] TopicCommandIntegrationTest > 
testListTopicsWithIncludeList() PASSED
[2021-07-19T21:41:16.036Z] 
[2021-07-19T21:41:16.036Z] TopicCommandIntegrationTest > testTopicDeletion() 
STARTED
[2021-07-19T21:41:19.603Z] 
[2021-07-19T21:41:19.603Z] TopicCommandIntegrationTest > testTopicDeletion() 
PASSED
[2021-07-19T21:41:19.603Z] 
[2021-07-19T21:41:19.603Z] TopicCommandIntegrationTest > 
testCreateWithDefaults() STARTED
[2021-07-19T21:41:24.152Z] 
[2021-07-19T21:41:24.152Z] TopicCommandIntegrationTest > 
testCreateWithDefaults() PASSED
[2021-07-19T21:41:24.152Z] 
[2021-07-19T21:41:24.152Z] TopicCommandIntegrationTest > 
testDescribeReportOverriddenConfigs() STARTED
[2021-07-19T21:41:27.851Z] 
[2021-07-19T21:41:27.851Z] TopicCommandIntegrationTest > 
testDescribeReportOverriddenConfigs() PASSED
[2021-07-19T21:41:27.851Z] 
[2021-07-19T21:41:27.851Z] TopicCommandIntegrationTest > 
testDescribeWhenTopicDoesntExist()

[jira] [Resolved] (KAFKA-12588) Remove deprecated --zookeeper in shell commands

2021-07-19 Thread Ismael Juma (Jira)


 [ 
https://issues.apache.org/jira/browse/KAFKA-12588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-12588.
-
Resolution: Fixed

> Remove deprecated --zookeeper in shell commands
> ---
>
> Key: KAFKA-12588
> URL: https://issues.apache.org/jira/browse/KAFKA-12588
> Project: Kafka
>  Issue Type: Task
>Reporter: Luke Chen
>Assignee: Luke Chen
>Priority: Blocker
> Fix For: 3.0.0
>
>
> At first check, there are still 4 commands existing *--zookeeper* option. 
> Should be removed in V3.0.0
>  
> _preferredReplicaLeaderElectionCommand_
> _ConfigCommand_
> _ReassignPartitionsCommand_
> _TopicCommand_



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


Re: [DISCUSS] Apache Kafka 3.0.0 release plan with new updated dates

2021-07-19 Thread Konstantine Karantasis
Hi all,

Since last week, we have reached the stage of Code Freeze for the 3.0.0
Apache Kafka release.

>From this point forward and until the official release of 3.0.0, only
critical fixes for blocker issues should be merged to the 3.0 release
branch.

The release plan currently includes ten (10) such known blockers.

https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.0.0

Besides these issues, any new issue that potentially gets discovered will
need to be reported on dev@kafka.apache.org (under this thread) and be
evaluated as a release blocker. At this point, the bar for such issues is
high; they need to be regressions or critical issues without an acceptable
workaround to be considered as release blockers.

Exceptions of changes that may be merged to the 3.0 branch without a
mention on the dev mailing list are fixes for test failures that will help
stabilize the build and small documentation changes.

If by the end of this week we are down to zero blockers and have green
builds and passing system tests, I will attempt to generate the first
Release Candidate (RC) for 3.0.0 on Friday.

Thank you all for the hard work so far.
Konstantine


On Mon, Jul 12, 2021 at 1:59 PM Konstantine Karantasis
 wrote:

> Hi all,
>
> This is a reminder that Code Freeze for Apache Kafka 3.0 is coming up this
> week and is set to take place by the end of day Wednesday, July 14th.
>
> Currently in the project we have 22 blocker issues for 3.0, out of 41 total
> tickets targeting 3.0.
>
> You may find the list of open issues in the release plan page:
> https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.0.0
>
> Thanks for all the hard work so far and for reducing the number of open
> issues in the recent days.
> Please take another look and help us resolve all the blockers for this
> upcoming major release.
>
> Best,
> Konstantine
>
> On Mon, Jul 12, 2021 at 1:57 PM Konstantine Karantasis <
> konstant...@confluent.io> wrote:
>
> >
> > Thanks for the update Levani,
> >
> > KIP-708 is now on the list of postponed KIPs.
> >
> > Konstantine
> >
> > On Thu, Jul 1, 2021 at 10:48 PM Levani Kokhreidze <
> levani.co...@gmail.com>
> > wrote:
> >
> >> Hi Konstantine,
> >>
> >> FYI, I don’t think we will be able to have KIP-708 ready on time.
> >> Feel free to remove it from the release plan.
> >>
> >> Best,
> >> Levani
> >>
> >> > On 1. Jul 2021, at 01:27, Konstantine Karantasis <
> >> konstant...@confluent.io.INVALID> wrote:
> >> >
> >> > Hi all,
> >> >
> >> > Today we have reached the Feature Freeze milestone for Apache Kafka
> 3.0.
> >> > Exciting!
> >> >
> >> > I'm going to allow for any pending changes to settle within the next
> >> couple
> >> > of days.
> >> > I trust that we all approve and merge adopted features and changes
> >> which we
> >> > consider to be in good shape for 3.0.
> >> >
> >> > Given the 4th of July holiday in the US, the 3.0 release branch will
> >> appear
> >> > sometime on Tuesday, July 6th.
> >> > Until then, please keep merging to trunk only the changes you intend
> to
> >> > include in Apache Kafka 3.0.
> >> >
> >> > Regards,
> >> > Konstantine
> >> >
> >> >
> >> > On Wed, Jun 30, 2021 at 3:25 PM Konstantine Karantasis <
> >> > konstant...@confluent.io> wrote:
> >> >
> >> >>
> >> >> Done. Thanks Luke!
> >> >>
> >> >> On Tue, Jun 29, 2021 at 6:39 PM Luke Chen  wrote:
> >> >>
> >> >>> Hi Konstantine,
> >> >>> We've decided that the KIP-726 will be released in V3.1, not V3.0.
> >> >>> KIP-726: Make the "cooperative-sticky, range" as the default
> assignor
> >> >>>
> >> >>> Could you please remove this KIP from the 3.0 release plan wiki
> page?
> >> >>>
> >> >>> Thank you.
> >> >>> Luke
> >> >>>
> >> >>> On Wed, Jun 30, 2021 at 8:23 AM Konstantine Karantasis
> >> >>>  wrote:
> >> >>>
> >>  Thanks for the update Colin.
> >>  They are now both in the release plan.
> >> 
> >>  Best,
> >>  Konstantine
> >> 
> >>  On Tue, Jun 29, 2021 at 2:55 PM Colin McCabe 
> >> >>> wrote:
> >> 
> >> > Hi Konstantine,
> >> >
> >> > Can you please add two KIPs to the 3.0 release plan wiki page?
> >> >
> >> > I'm thinking of:
> >> >KIP-630: Kafka Raft Snapshots
> >> >KIP-746: Revise KRaft Metadata Records
> >> >
> >> > These are marked as 3.0 on the KIP page but I guess we don't have
> >> >>> them on
> >> > the page yet.
> >> >
> >> > Many thanks.
> >> > Colin
> >> >
> >> >
> >> > On Tue, Jun 22, 2021, at 06:29, Josep Prat wrote:
> >> >> Hi there,
> >> >>
> >> >> As the feature freeze date is approaching, I just wanted to
> kindly
> >> >>> ask
> >> > for
> >> >> some reviews on the already submitted PR (
> >> >> https://github.com/apache/kafka/pull/10840) that implements the
> >>  approved
> >> >> KIP-744 (https://cwiki.apache.org/confluence/x/XIrOCg). The PR
> has
> >>  been
> >> >> ready for review for 2 weeks, and I simply want to make sure
> there
> >> >>> is

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #340

2021-07-19 Thread Apache Jenkins Server
See 




Jenkins build is unstable: Kafka » Kafka Branch Builder » 3.0 #46

2021-07-19 Thread Apache Jenkins Server
See 




Re: [DISCUSS] KIP-761: Add total blocked time metric to streams

2021-07-19 Thread Rohan Desai
Thanks for the review Guozhang! responding to your feedback inline:

> 1) I agree that the current ratio metrics is just "snapshot in point", and
more flexible metrics that would allow reporters to calculate based on
window intervals are better. However, the current mechanism of the proposed
metrics assumes the thread->clients mapping as of today, where each thread
would own exclusively one main consumer, restore consumer, producer and an
admin client. But this mapping may be subject to change in the future. Have
you thought about how this metric can be extended when, e.g. the embedded
clients and stream threads are de-coupled?

Of course this depends on how exactly we refactor the runtime - assuming
that we plan to factor out consumers into an "I/O" layer that is
responsible for receiving records and enqueuing them to be processed by
processing threads, then I think it should be reasonable to count the time
we spend blocked on this internal queue(s) as blocked. The main concern
there to me is that the I/O layer would be doing something expensive like
decompression that shouldn't be counted as "blocked". But if that really is
so expensive that it starts to throw off our ratios then it's probably
indicative of a larger problem that the "i/o layer" is a bottleneck and it
would be worth refactoring so that decompression (or insert other expensive
thing here) can also be done on the processing threads.

> 2) [This and all below are minor comments] The "flush-time-total" may
better be a producer client metric, as "flush-wait-time-total", than a
streams metric, though the streams-level "total-blocked" can still leverage
it. Similarly, I think "txn-commit-time-total" and
"offset-commit-time-total" may better be inside producer and consumer
clients respectively.

Good call - I'll update the KIP

> 3) The doc was not very clear on how "thread-start-time" would be needed
when calculating streams utilization along with total-blocked time, could
you elaborate a bit more in the KIP?

Yes, will do.

> For "txn-commit-time-total" specifically, besides producer.commitTxn.
other txn-related calls may also be blocking, including
producer.beginTxn/abortTxn, I saw you mentioned "txn-begin-time-total"
later in the doc, but did not include it as a separate metric, and
similarly, should we have a `txn-abort-time-total` as well? If yes, could
you update the KIP page accordingly.

Ack.

On Mon, Jul 12, 2021 at 11:29 PM Rohan Desai 
wrote:

> Hello All,
>
> I'd like to start a discussion on the KIP linked above which proposes some
> metrics that we would find useful to help measure whether a Kafka Streams
> application is saturated. The motivation section in the KIP goes into some
> more detail on why we think this is a useful addition to the metrics
> already implemented. Thanks in advance for your feedback!
>
> Best Regards,
>
> Rohan
>
> On Mon, Jul 12, 2021 at 12:00 PM Rohan Desai 
> wrote:
>
>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-761%3A+Add+Total+Blocked+Time+Metric+to+Streams
>>
>