REMINDER - Travel Assistance available for ApacheCon NA New Orleans 2022

[Gavin McDonald]

Hi All Contributors and Committers,

This is a first reminder email that travel
assistance applications for ApacheCon NA 2022 are now open!

We will be supporting ApacheCon North America in New Orleans, Louisiana,
on October 3rd through 6th, 2022.

TAC exists to help those that would like to attend ApacheCon events, but
are unable to do so for financial reasons. This year, We are supporting
both committers and non-committers involved with projects at the
Apache Software Foundation, or open source projects in general.

For more info on this year's applications and qualifying criteria, please
visit the TAC website at http://www.apache.org/travel/
Applications are open and will close on the 1st of July 2022.

Important: Applicants have until the closing date above to submit their
applications (which should contain as much supporting material as required
to efficiently and accurately process their request), this will enable TAC
to announce successful awards shortly afterwards.

As usual, TAC expects to deal with a range of applications from a diverse
range of backgrounds. We therefore encourage (as always) anyone thinking
about sending in an application to do so ASAP.

Why should you attend as a TAC recipient? We encourage you to read stories
from
past recipients at https://apache.org/travel/stories/ . Also note that
previous TAC recipients have gone on to become Committers, PMC Members, ASF
Members, Directors of the ASF Board and Infrastructure Staff members.
Others have gone from Committer to full time Open Source Developers!

How far can you go! - Let TAC help get you there.

[GitHub] [kafka-site] mimaison merged pull request #379: MINOR: Update coding guide to mention need to maintain public client API compatibility

[GitBox]

mimaison merged PR #379:
URL: https://github.com/apache/kafka-site/pull/379


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.2 #42

[Apache Jenkins Server]

See 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #903

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-827: Expose logdirs total and usable space via Kafka API

[Divij Vaidya]

I understand your point but I think we could potentially improve the user
experience here by sending different error codes for the following two
different situations.
Situation 1: "when broker hits an Exception accessing a logdir" ->
UNKNOWN_SPACE
Situation 2: "when feature is not supported" -> UNSUPPORTED
Different error codes will help the customer understand whether the failure
was due to a server/client mismatch (situation 2) or was it a genuine
exception while trying to find the value (situation 1).

Or, are you saying that we don't have a way to distinguish between
situation 1 and situation 2 from code perspective and hence, we will send
UNKNOWN in both cases?


Divij Vaidya



On Mon, May 2, 2022 at 6:24 PM Mickael Maison 
wrote:

> Hi Divij,
>
> The new fields default to -1 in the protocol too. So in case a broker
> hits an Exception accessing a logdir and sends an error response back,
> the client will get -1. For this reason I think UNKNOWN_SPACE is
> slightly better than UNSUPPORTED.
>
> Thanks,
> Mickael
>
> On Fri, Apr 22, 2022 at 9:35 AM Tom Bentley  wrote:
> >
> > Hi Mickael,
> >
> > Thanks for the KIP, I can see this would be useful.
> >
> > I guess you could have used optional tagged fields, rather than bumping
> the
> > version, but then again I don't see it being particularly advantageous in
> > this case either.
> >
> > Kind regards,
> >
> > Tom
> >
> > On Tue, 19 Apr 2022 at 10:23, Divij Vaidya 
> wrote:
> >
> > > I have a minor suggestion below but overall KIP looks good to me to
> start a
> > > vote.
> > >
> > > *Reg#6* Would you consider replacing UNKNOWN_SPACE with UNSUPPORTED?
> > > UNSUPPORTED tells the user explicitly that the value is missing due to
> > > client/server version mismatch whereas with UNKNOWN_SPACE, the user is
> left
> > > wondering whether it is a problem with underlying storage not providing
> > > space information or something else.
> > >
> > > Divij Vaidya
> > >
> > >
> > >
> > > On Fri, Apr 15, 2022 at 3:40 PM Mickael Maison <
> mickael.mai...@gmail.com>
> > > wrote:
> > >
> > > > Hi Luke,
> > > >
> > > > 7. I've updated the KIP to clarify these sizes are in bytes.
> > > >
> > > > Thanks,
> > > > Mickael
> > > >
> > > > On Fri, Apr 15, 2022 at 12:16 PM Luke Chen 
> wrote:
> > > > >
> > > > > Hi Mickael,
> > > > >
> > > > > Thanks for the KIP!
> > > > > This is a good improvement.
> > > > >
> > > > > (3) +1 for not adding the number of files in the directory.
> Counting
> > > the
> > > > > file numbers should be slow.
> > > > > (7) Could you make the fields clear in `DescribeLogDirsResponse`,
> to
> > > > > mention the returned number is size in Byte (or not?)
> > > > >
> > > > > Thank you.
> > > > > Luke
> > > > >
> > > > > On Fri, Apr 15, 2022 at 5:27 PM Mickael Maison <
> > > mickael.mai...@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > Thanks for the feedback.
> > > > > >
> > > > > > 3. Yes that's right. Also the number of file descriptors is
> really
> > > not
> > > > > > a property of log directories. Administrators typically tracked
> that
> > > > > > count per process and for the whole operating system.
> > > > > >
> > > > > > 5. That's a good point, I've updated the KIP to mention sizes
> will be
> > > > > > capped to Long.MAX_VALUE even if the actual storage is larger.
> > > > > >
> > > > > > 6. Brokers would never return UNKNOWN_SPACE. When new clients
> query
> > > > > > older brokers via the admin API, the admin client will use
> > > > > > UNKNOWN_SPACE to indicate these values weren't provided by
> brokers.
> > > > > >
> > > > > > Thanks,
> > > > > > Mickael
> > > > > >
> > > > > > On Fri, Apr 8, 2022 at 5:00 PM Divij Vaidya <
> divijvaidy...@gmail.com
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > Thanks for replying. I still have a few lingering
> > > questions/comments.
> > > > > > >
> > > > > > > *Reg#1* Understood. I checked and the underlying system call is
> > > > statvfs
> > > > > > for
> > > > > > > unix systems which should be ok to call here.
> > > > > > > *Reg#2* Fair point. I checked again and yes, log.dir always
> means
> > > > local
> > > > > > > storage even when tiered storage is enabled.
> > > > > > > *Reg#3* The rationale for adding these new (size) fields to the
> > > > > > > `DescribeLogDirs` is to allow the administrator to monitor or
> > > perhaps
> > > > > > take
> > > > > > > automated action based on results. Doesn't monitoring the
> number of
> > > > file
> > > > > > > descriptors fall in the same category of use cases? I am
> assuming
> > > > that we
> > > > > > > want to add the size information in the API response because
> JVM
> > > > makes it
> > > > > > > possible to get this information in a platform agnostic manner
> > > which
> > > > is
> > > > > > not
> > > > > > > true for open file descriptors, correct?
> > > > > > > *Reg#4* Agree.
> > > > > > > *New#5*: As an FYI, Java FileStore API breaks on large storage
> > > sizes.
> > > > > > See:
> > > > > > > https://b

[VOTE] 3.2.0 RC1

[Bruno Cadonna]

Hello Kafka users, developers and client-developers,

This is the second candidate for release of Apache Kafka 3.2.0.

* log4j 1.x is replaced with reload4j (KAFKA-9366)
* StandardAuthorizer for KRaft (KIP-801)
* Send a hint to the partition leader to recover the partition (KIP-704)
* Top-level error code field in DescribeLogDirsResponse (KIP-784)
* kafka-console-producer writes headers and null values (KIP-798 and 
KIP-810)

* JoinGroupRequest and LeaveGroupRequest have a reason attached (KIP-800)
* Static membership protocol lets the leader skip assignment (KIP-814)
* Rack-aware standby task assignment in Kafka Streams (KIP-708)
* Interactive Query v2 (KIP-796, KIP-805, and KIP-806)
* Connect APIs list all connector plugins and retrieve their 
configuration (KIP-769)

* TimestampConverter SMT supports different unix time precisions (KIP-808)
* Connect source tasks handle producer exceptions (KIP-779)


Release notes for the 3.2.0 release:
https://home.apache.org/~cadonna/kafka-3.2.0-rc1/RELEASE_NOTES.html

*** Please download, test and vote by Tuesday, May 10th, 9am PDT

Kafka's KEYS file containing PGP keys we use to sign the release:
https://kafka.apache.org/KEYS

* Release artifacts to be voted upon (source and binary):
https://home.apache.org/~cadonna/kafka-3.2.0-rc1/

* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/org/apache/kafka/

* Javadoc:
https://home.apache.org/~cadonna/kafka-3.2.0-rc1/javadoc/

* Tag to be voted upon (off 3.2 branch) is the 3.2.0 tag:
https://github.com/apache/kafka/releases/tag/3.2.0-rc1

* Documentation:
https://kafka.apache.org/32/documentation.html

* Protocol:
https://kafka.apache.org/32/protocol.html

* Successful Jenkins builds for the 3.2 branch:
Unit/integration tests: I'll share a link once the builds complete
System tests: https://jenkins.confluent.io/job/system-test-kafka/job/3.2/30/

/**

Thanks,
Bruno

Re: [VOTE] 3.1.1 RC1

[Jakub Scholz]

+1 (non-binding). I used the Scala 2.13 binaries and staged Maven artifacts
and ran various tests with them. Thanks for doing the release.

Jakub

On Fri, Apr 29, 2022 at 8:16 PM Mickael Maison  wrote:

> Hi Tom,
>
> Thanks for running this release!
>
> I've done the following:
> - Checked signatures and checksums
> - Checked javadocs/maven artifacts
> - Built from source and run all tests with Java 11
> - Ran quickstart on Scala 2.13 artifact with Java 11
>
> It looks like the website has not been updated yet, I still only see
> 3.1.0. When you'll add 3.1.1, let's make sure we mention reload4j in
> the notable changes section.
>
> +1 (binding)
>
> Thanks,
> Mickael
>
>
>
> On Fri, Apr 29, 2022 at 11:12 AM Tom Bentley  wrote:
> >
> > Hello Kafka users, developers and client-developers,
> >
> > This is the first candidate for release of Apache Kafka 3.1.1.
> >
> > Apache Kafka 3.1.1 is a bugfix release and 30 issues have been fixed
> > since 3.1.0.
> >
> > Release notes for the 3.1.1 release:
> > https://home.apache.org/~tombentley/kafka-3.1.1-rc1/RELEASE_NOTES.html
> >
> > *** Please download, test and vote by 09:00 UTC, Friday 6th May
> >
> > Kafka's KEYS file containing PGP keys we use to sign the release:
> > https://kafka.apache.org/KEYS
> >
> > * Release artifacts to be voted upon (source and binary):
> > https://home.apache.org/~tombentley/kafka-3.1.1-rc1/
> >
> > * Maven artifacts to be voted upon:
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> >
> > * Javadoc:
> > https://home.apache.org/~tombentley/kafka-3.1.1-rc1/javadoc/
> >
> > * Tag to be voted upon (off 3.1 branch) is the 3.1.1 tag:
> > https://github.com/apache/kafka/releases/tag/3.1.1-rc1
> >
> > * Documentation:
> > https://kafka.apache.org/31/documentation.html
> >
> > * Protocol:
> > https://kafka.apache.org/31/protocol.html
> >
> > * Successful Jenkins builds for the 3.1 branch:
> > I will share a link one the build is complete.
> >
> > /**
> >
> > Thanks,
> > Tom
>

Re: [DISCUSS] KIP-660: Pluggable ReplicaPlacer

[Divij Vaidya]

Nice proposal. I can definitely see the value add it could bring based on
varied custom use cases.

1. When exposing an interface external to Kafka, could we strive to ensure
that the implementation can not mutate the incoming parameters? This would
ensure that the implementations don't un-intentionally change some internal
state of the cluster. Also, this would be a two way door, so later if we
decide to relax the constraint to allow implementations to mutate certain
things, we should be able to do that. This immutability can be achieved for
example by passing a clone of the list of usable brokers to the interface.
2. Why is the interface Closeable? What resource does the interface expect
to close? Please add a comment on the class documentation about this.
3. The iterator that is being passed as an argument to `place()`. Do we
expect the place() method to mutate the input? If not, then an iterator may
not be necessary? Could it be an unmodifiable collection instead?
4. Please add documentation on behaviour expected out of the place()
function such as, should it be idempotent? Is it called in some latency
sensitive path? What should be the performance expectation from this
function? etc.
5. Please add documentation on behaviour expected out of the place()
function for error scenario's e.g. would it be able to set a message in the
ReplicaPlacementException?
6. The interface should be extensible in the future, e.g. if we want to add
a new parameter to the place() function in future, we shouldn't have to
require code changes to all concrete implementations. This can be achieved
by grouping the parameters of place() into higher abstractions, say,
PlacementSpec (which already exist). The advantage is that a new parameter
could be added to PlacementSpec with a setter/getter in future but it
wouldn't require any code modification to the custom implementations of
ReplicaPlacer.

Divij Vaidya



On Mon, May 2, 2022 at 6:28 PM Mickael Maison 
wrote:

> Hi,
>
> If there are no further comments, I'll start a vote in the next few days.
>
> Thanks,
> Mickael
>
> On Wed, Mar 30, 2022 at 3:51 AM Luke Chen  wrote:
> >
> > Hi Mickael,
> >
> > Thanks for the update.
> > It answered my questions!
> >
> > Thank you.
> > Luke
> >
> > On Wed, Mar 30, 2022 at 12:09 AM Mickael Maison <
> mickael.mai...@gmail.com>
> > wrote:
> >
> > > Hi Luke,
> > >
> > > Thanks for the feedback.
> > >
> > > 1. Thanks, fixed!
> > > 2. Yes that's right. It's the same behavior for topic policies
> > > 3. I've added details about how the mentioned scenarios could be
> > > addressed. The information required to make such decisions is not part
> > > of the Kafka cluster metadata so an external input is necessary. This
> > > KIP does not propose a specific mechanism for doing it.
> > >
> > > I hope this answers your questions.
> > >
> > > Thanks,
> > > Mickael
> > >
> > >
> > > On Tue, Mar 29, 2022 at 5:42 PM Mickael Maison <
> mickael.mai...@gmail.com>
> > > wrote:
> > > >
> > > > Hi Ryanne,
> > > >
> > > > That's a good point!
> > > >
> > > > There's no value in having all implementations perform the same
> sanity
> > > > checks. If the replication factor is < 1 or larger than the current
> > > > number of registered brokers, the controller should directly throw
> > > > InvalidReplicationFactorException and not call the ReplicaPlacer.
> I've
> > > > updated the KIP so the place() method now only throws
> > > > ReplicaPlacementException.
> > > >
> > > > Thanks,
> > > > Mickael
> > > >
> > > >
> > > >
> > > > On Mon, Mar 28, 2022 at 6:20 PM Ryanne Dolan 
> > > wrote:
> > > > >
> > > > > Wondering about InvalidReplicationFactorException. Why would an
> > > > > implementation throw this? Given the information passed to the
> method,
> > > > > seems like this could only be thrown if there were obviously
> invalid
> > > > > arguments, like a negative number or zero. Can we just guarantee
> such
> > > > > invalid arguments aren't passed in?
> > > > >
> > > > > Ryanne
> > > > >
> > > > > On Sat, Mar 26, 2022, 8:51 AM Luke Chen  wrote:
> > > > >
> > > > > > Hi Mickael,
> > > > > >
> > > > > > Thanks for the KIP!
> > > > > > It's indeed a pain point for the Kafka admins.
> > > > > >
> > > > > > I have some comments:
> > > > > > 1. Typo in motivation section: When administrators [when to]
> remove
> > > brokers
> > > > > > from a cluster,
> > > > > > 2. If different `replica.placer.class.name` configs are set in
> all
> > > > > > controllers, I think only the config for  "active controller"
> will
> > > take
> > > > > > effect, right?
> > > > > > 3. Could you explain more about how the proposal fixes some
> > > scenarios you
> > > > > > listed, ex: the new added broker case. How could we know the
> broker
> > > is new
> > > > > > added? I guess it's by checking the broker load via some metrics
> > > > > > dynamically, right?
> > > > > >
> > > > > >
> > > > > > Thank you.
> > > > > > Luke
> > > > > >
> > > > > > On Fri, Mar 18, 2022 at 10:30 AM Ryanne Dolan <

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.2 #43

[Apache Jenkins Server]

See 

Re: [VOTE] 3.2.0 RC1

[Bill Bejeck]

Hi Bruno,

Thanks for running the 3.2 release!

I did the following:

   1. Built from source
   2. Ran the unit tests
   3. Validated keys and checksums
   4. Lightly checked the Javadoc
   5. Ran the quickstart and the Kafka Streams demo app
   6. Looked over the docs
   7. Ran the KRaft quickstart including the dump-log and metadata-shell

In the docs, I found that in section 1.3 on the main page the  `Kafka
Streams Demo` and `app development tutorial` still point to version 2.5.
But since this is a document-only change only we can directly update the
kafka-site repo.

+1(binding)

Thanks,
Bill


On Tue, May 3, 2022 at 10:07 AM Bruno Cadonna  wrote:

> Hello Kafka users, developers and client-developers,
>
> This is the second candidate for release of Apache Kafka 3.2.0.
>
> * log4j 1.x is replaced with reload4j (KAFKA-9366)
> * StandardAuthorizer for KRaft (KIP-801)
> * Send a hint to the partition leader to recover the partition (KIP-704)
> * Top-level error code field in DescribeLogDirsResponse (KIP-784)
> * kafka-console-producer writes headers and null values (KIP-798 and
> KIP-810)
> * JoinGroupRequest and LeaveGroupRequest have a reason attached (KIP-800)
> * Static membership protocol lets the leader skip assignment (KIP-814)
> * Rack-aware standby task assignment in Kafka Streams (KIP-708)
> * Interactive Query v2 (KIP-796, KIP-805, and KIP-806)
> * Connect APIs list all connector plugins and retrieve their
> configuration (KIP-769)
> * TimestampConverter SMT supports different unix time precisions (KIP-808)
> * Connect source tasks handle producer exceptions (KIP-779)
>
>
> Release notes for the 3.2.0 release:
> https://home.apache.org/~cadonna/kafka-3.2.0-rc1/RELEASE_NOTES.html
>
> *** Please download, test and vote by Tuesday, May 10th, 9am PDT
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> https://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> https://home.apache.org/~cadonna/kafka-3.2.0-rc1/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>
> * Javadoc:
> https://home.apache.org/~cadonna/kafka-3.2.0-rc1/javadoc/
>
> * Tag to be voted upon (off 3.2 branch) is the 3.2.0 tag:
> https://github.com/apache/kafka/releases/tag/3.2.0-rc1
>
> * Documentation:
> https://kafka.apache.org/32/documentation.html
>
> * Protocol:
> https://kafka.apache.org/32/protocol.html
>
> * Successful Jenkins builds for the 3.2 branch:
> Unit/integration tests: I'll share a link once the builds complete
> System tests:
> https://jenkins.confluent.io/job/system-test-kafka/job/3.2/30/
>
> /**
>
> Thanks,
> Bruno
>

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #904

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-13868) Website updates to satisfy Apache privacy policies

[Mickael Maison (Jira)]

Mickael Maison created KAFKA-13868:
--

 Summary: Website updates to satisfy Apache privacy policies
 Key: KAFKA-13868
 URL: https://issues.apache.org/jira/browse/KAFKA-13868
 Project: Kafka
  Issue Type: Bug
Reporter: Mickael Maison


The ASF has updated its privacy policy and all websites must be compliant.

The full guidelines can be found in 
[https://privacy.apache.org/faq/committers.html]

The Kafka website has a few issues, including:
- It's missing a link to the privacy policy: 
[https://privacy.apache.org/policies/privacy-policy-public.html]
- It's using Google Analytics
- It's using Google Fonts
- It's using scripts hosted on Cloudflare CDN
- Embedded videos don't have an image placeholder





--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Created] (KAFKA-13869) Update quota callback metadata in KRaft

[Jason Gustafson (Jira)]

Jason Gustafson created KAFKA-13869:
---

 Summary: Update quota callback metadata in KRaft
 Key: KAFKA-13869
 URL: https://issues.apache.org/jira/browse/KAFKA-13869
 Project: Kafka
  Issue Type: Bug
Reporter: Jason Gustafson
Assignee: Jason Gustafson


The `ClientQuotaCallback` interface implements a method 
`updateClusterMetadata`, which allows the callback to take partition 
assignments into account when assigning quotas. For zk , this method is called 
after receiving `UpdateMetadata` requests from the controller. We do not yet 
have this implemented in KRaft for updates from the metadata log.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Re: [DISCUSS] KIP-660: Pluggable ReplicaPlacer

[Colin McCabe]

Hi Mickael,

We did discuss this earlier, and I remember not being too enthusiastic about a 
pluggable policy here :)

There have been several changes to the placement code in the last few weeks. 
(These are examples of the kind of changes that are impossible to do once an 
API is established, by the way.) Can you please revise the KIP to take these 
into account?

I'd also like to understand a little bit better why we need this API when we 
have the explicit placement API for createTopics and createPartitions. Can you 
give me a few scenarios where the manual placement API would be insufficient?

best,
Colin


On Mon, May 2, 2022, at 09:28, Mickael Maison wrote:
> Hi,
>
> If there are no further comments, I'll start a vote in the next few days.
>
> Thanks,
> Mickael
>
> On Wed, Mar 30, 2022 at 3:51 AM Luke Chen  wrote:
>>
>> Hi Mickael,
>>
>> Thanks for the update.
>> It answered my questions!
>>
>> Thank you.
>> Luke
>>
>> On Wed, Mar 30, 2022 at 12:09 AM Mickael Maison 
>> wrote:
>>
>> > Hi Luke,
>> >
>> > Thanks for the feedback.
>> >
>> > 1. Thanks, fixed!
>> > 2. Yes that's right. It's the same behavior for topic policies
>> > 3. I've added details about how the mentioned scenarios could be
>> > addressed. The information required to make such decisions is not part
>> > of the Kafka cluster metadata so an external input is necessary. This
>> > KIP does not propose a specific mechanism for doing it.
>> >
>> > I hope this answers your questions.
>> >
>> > Thanks,
>> > Mickael
>> >
>> >
>> > On Tue, Mar 29, 2022 at 5:42 PM Mickael Maison 
>> > wrote:
>> > >
>> > > Hi Ryanne,
>> > >
>> > > That's a good point!
>> > >
>> > > There's no value in having all implementations perform the same sanity
>> > > checks. If the replication factor is < 1 or larger than the current
>> > > number of registered brokers, the controller should directly throw
>> > > InvalidReplicationFactorException and not call the ReplicaPlacer. I've
>> > > updated the KIP so the place() method now only throws
>> > > ReplicaPlacementException.
>> > >
>> > > Thanks,
>> > > Mickael
>> > >
>> > >
>> > >
>> > > On Mon, Mar 28, 2022 at 6:20 PM Ryanne Dolan 
>> > wrote:
>> > > >
>> > > > Wondering about InvalidReplicationFactorException. Why would an
>> > > > implementation throw this? Given the information passed to the method,
>> > > > seems like this could only be thrown if there were obviously invalid
>> > > > arguments, like a negative number or zero. Can we just guarantee such
>> > > > invalid arguments aren't passed in?
>> > > >
>> > > > Ryanne
>> > > >
>> > > > On Sat, Mar 26, 2022, 8:51 AM Luke Chen  wrote:
>> > > >
>> > > > > Hi Mickael,
>> > > > >
>> > > > > Thanks for the KIP!
>> > > > > It's indeed a pain point for the Kafka admins.
>> > > > >
>> > > > > I have some comments:
>> > > > > 1. Typo in motivation section: When administrators [when to] remove
>> > brokers
>> > > > > from a cluster,
>> > > > > 2. If different `replica.placer.class.name` configs are set in all
>> > > > > controllers, I think only the config for  "active controller" will
>> > take
>> > > > > effect, right?
>> > > > > 3. Could you explain more about how the proposal fixes some
>> > scenarios you
>> > > > > listed, ex: the new added broker case. How could we know the broker
>> > is new
>> > > > > added? I guess it's by checking the broker load via some metrics
>> > > > > dynamically, right?
>> > > > >
>> > > > >
>> > > > > Thank you.
>> > > > > Luke
>> > > > >
>> > > > > On Fri, Mar 18, 2022 at 10:30 AM Ryanne Dolan > > >
>> > > > > wrote:
>> > > > >
>> > > > > > Thanks Mickael, this makes sense to me! I've been wanting
>> > something like
>> > > > > > this in order to decommission a broker without new partitions
>> > getting
>> > > > > > accidentally assigned to it.
>> > > > > >
>> > > > > > Ryanne
>> > > > > >
>> > > > > > On Thu, Mar 17, 2022, 5:56 AM Mickael Maison <
>> > mickael.mai...@gmail.com>
>> > > > > > wrote:
>> > > > > >
>> > > > > > > Hi,
>> > > > > > >
>> > > > > > > I'd like to start a new discussion on KIP-660. I originally
>> > wrote this
>> > > > > > > KIP in 2020 and the initial discussion
>> > > > > > > (
>> > https://lists.apache.org/thread/xn7xyb74nyt281brto4x28r9rzxm4lp9)
>> > > > > > > raised some concerns especially around KRaft (which did not
>> > exist at
>> > > > > > > that time) and scalability.
>> > > > > > >
>> > > > > > > Since then, we got a new KRaft controller so I've been able to
>> > revisit
>> > > > > > > this KIP. I kept the KIP number as it's essentially the same
>> > idea, but
>> > > > > > > the proposal is significantly different:
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-660%3A+Pluggable+ReplicaPlacer
>> > > > > > >
>> > > > > > > Please take a look and let me know if you have any feedback.
>> > > > > > >
>> > > > > > > Thanks,
>> > > > > > > Mickael
>> > > > > > >
>> > > > > >
>> > > > >
>> >

Re: [DISCUSS] KIP-827: Expose logdirs total and usable space via Kafka API

[Colin McCabe]

Hi Mickael,

Thanks for the KIP.

In the API, I would suggest using an OptionalLong rather than a "magic value" 
of -1.

best,
Colin


On Thu, Apr 7, 2022, at 02:16, Mickael Maison wrote:
> Hi,
>
> I wrote a small KIP to expose the total and usable space of logdirs
> via the DescribeLogDirs API:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-827%3A+Expose+logdirs+total+and+usable+space+via+Kafka+API
>
> Please take a look and let me know if you have any feedback.
>
> Thanks,
> Mickael

[DISCUSS] KIP-833: Mark KRaft as Production Ready

[Colin McCabe]

Hi all,

I've written a KIP for marking KRaft as production ready. Please take a look if 
you have a chance:

https://cwiki.apache.org/confluence/x/8xKhD

thanks,
Colin

Re: [DISCUSS] KIP-833: Mark KRaft as Production Ready

[Luke Chen]

Hi Colin,

So exciting to see the KIP to mark the Kraft production ready!

Just one comment: We should make sure the period between ZK deprecation
(i.e. v3.4.0) to ZK removal (i.e. v4.0.0) is not too short.
Do we have any expectation for the deprecation period?
After all, this is not a small feature change, and users need more time to
do the migration.
But to be honest, I don't know how long is long enough.
Maybe at least half a year? Or more? Thoughts?


Thank you.
Luke

On Wed, May 4, 2022 at 9:10 AM Colin McCabe  wrote:

> Hi all,
>
> I've written a KIP for marking KRaft as production ready. Please take a
> look if you have a chance:
>
> https://cwiki.apache.org/confluence/x/8xKhD
>
> thanks,
> Colin
>

Re: [DISCUSS] KIP-833: Mark KRaft as Production Ready

[Israel Ekpo]

Hi Colin,

Thanks for this KIP. I am very excited to see the proposal.

A lot in the community have been asking when KRaft mode will be ready and I
think this KIP will provide a lot of the answers and guidance desperately
needed.

Thanks for working on it.

I also have some of the questions from Luke regarding the duration of the
deprecation period and the release where ZK will be completely dropped.

As per our time-based release plan [1], the release should be every 4
months, but it has been approximately 5 months between releases from the
downloads page [2]

So, right now we are about to push out 3.2.0 and 3.4.0 is approximately 10
months from now based on the schedule/plan.

I wonder if this will be enough time to complete all the features needed to
bring KRaft mode to parity with Legacy/Zookeeper mode.

The missing features you have highlighted and any other critical features
not working in KRaft mode should be implemented fully before marking KRaft
mode as production ready and deprecating Legacy mode.

The main question here is would we have Kafka versions 3.6.0, 3.7.0, 3.8.0
and 3.9.0 because this will add approximately 20 additional months after
3.4.0 which should be enough time to have ZK-mode deprecated and tested
thoroughly before removal completely. I feel we should not mark ZK mode as
deprecated until all the features in KRaft mode are at parity with ZK-mode.

If we are going to have 20 months after 3.4 to work with a deprecated
ZK-mode, then it could be possible to even drop ZK fully before 4.0

Let's move the boat closer to the dock before we jump out of the boat. It
will be very stressful if we jump out of the boat and have to swim several
miles to shore.

[1]
https://cwiki.apache.org/confluence/display/KAFKA/Time+Based+Release+Plan
[2] https://kafka.apache.org/downloads


Israel Ekpo
Lead Instructor, IzzyAcademy.com
https://www.youtube.com/c/izzyacademy
https://izzyacademy.com/


On Tue, May 3, 2022 at 10:32 PM Luke Chen  wrote:

> Hi Colin,
>
> So exciting to see the KIP to mark the Kraft production ready!
>
> Just one comment: We should make sure the period between ZK deprecation
> (i.e. v3.4.0) to ZK removal (i.e. v4.0.0) is not too short.
> Do we have any expectation for the deprecation period?
> After all, this is not a small feature change, and users need more time to
> do the migration.
> But to be honest, I don't know how long is long enough.
> Maybe at least half a year? Or more? Thoughts?
>
>
> Thank you.
> Luke
>
> On Wed, May 4, 2022 at 9:10 AM Colin McCabe  wrote:
>
> > Hi all,
> >
> > I've written a KIP for marking KRaft as production ready. Please take a
> > look if you have a chance:
> >
> > https://cwiki.apache.org/confluence/x/8xKhD
> >
> > thanks,
> > Colin
> >
>

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #905

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 613973 lines...]
[2022-05-04T05:32:14.666Z] 
[2022-05-04T05:32:14.666Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorManyStandbys STARTED
[2022-05-04T05:32:17.585Z] 
[2022-05-04T05:32:17.585Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorManyStandbys PASSED
[2022-05-04T05:32:17.585Z] 
[2022-05-04T05:32:17.585Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorManyStandbys STARTED
[2022-05-04T05:32:31.843Z] > Task :streams:compileTestJava
[2022-05-04T05:32:31.843Z] > Task :streams:testClasses
[2022-05-04T05:32:31.843Z] > Task :streams:testJar
[2022-05-04T05:32:31.843Z] > Task :streams:testSrcJar
[2022-05-04T05:32:31.843Z] > Task 
:streams:publishMavenJavaPublicationToMavenLocal
[2022-05-04T05:32:31.843Z] > Task :streams:publishToMavenLocal
[2022-05-04T05:32:31.843Z] 
[2022-05-04T05:32:31.843Z] Deprecated Gradle features were used in this build, 
making it incompatible with Gradle 8.0.
[2022-05-04T05:32:31.843Z] 
[2022-05-04T05:32:31.843Z] You can use '--warning-mode all' to show the 
individual deprecation warnings and determine if they come from your own 
scripts or plugins.
[2022-05-04T05:32:31.843Z] 
[2022-05-04T05:32:31.843Z] See 
https://docs.gradle.org/7.4.2/userguide/command_line_interface.html#sec:command_line_warnings
[2022-05-04T05:32:31.843Z] 
[2022-05-04T05:32:31.843Z] Execution optimizations have been disabled for 2 
invalid unit(s) of work during this build to ensure correctness.
[2022-05-04T05:32:31.843Z] Please consult deprecation warnings for more details.
[2022-05-04T05:32:31.843Z] 
[2022-05-04T05:32:31.843Z] BUILD SUCCESSFUL in 4m 48s
[2022-05-04T05:32:31.843Z] 78 actionable tasks: 37 executed, 41 up-to-date
[Pipeline] sh
[2022-05-04T05:32:36.421Z] + grep ^version= gradle.properties
[2022-05-04T05:32:36.421Z] + cut -d= -f 2
[Pipeline] dir
[2022-05-04T05:32:37.113Z] Running in 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/quickstart
[Pipeline] {
[Pipeline] sh
[2022-05-04T05:32:37.478Z] 
[2022-05-04T05:32:37.478Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorManyStandbys PASSED
[2022-05-04T05:32:37.478Z] 
[2022-05-04T05:32:37.478Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorLargeNumConsumers STARTED
[2022-05-04T05:32:37.478Z] 
[2022-05-04T05:32:37.478Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorLargeNumConsumers PASSED
[2022-05-04T05:32:37.478Z] 
[2022-05-04T05:32:37.478Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testStickyTaskAssignorLargeNumConsumers STARTED
[2022-05-04T05:32:38.428Z] 
[2022-05-04T05:32:38.428Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testStickyTaskAssignorLargeNumConsumers PASSED
[2022-05-04T05:32:39.560Z] + mvn clean install -Dgpg.skip
[2022-05-04T05:32:41.325Z] [INFO] Scanning for projects...
[2022-05-04T05:32:41.325Z] [INFO] 

[2022-05-04T05:32:41.325Z] [INFO] Reactor Build Order:
[2022-05-04T05:32:41.325Z] [INFO] 
[2022-05-04T05:32:41.325Z] [INFO] Kafka Streams :: Quickstart   
 [pom]
[2022-05-04T05:32:41.325Z] [INFO] streams-quickstart-java   
 [maven-archetype]
[2022-05-04T05:32:41.325Z] [INFO] 
[2022-05-04T05:32:41.325Z] [INFO] < 
org.apache.kafka:streams-quickstart >-
[2022-05-04T05:32:42.267Z] [INFO] Building Kafka Streams :: Quickstart 
3.3.0-SNAPSHOT[1/2]
[2022-05-04T05:32:42.267Z] [INFO] [ pom 
]-
[2022-05-04T05:32:42.267Z] [INFO] 
[2022-05-04T05:32:42.267Z] [INFO] --- maven-clean-plugin:3.0.0:clean 
(default-clean) @ streams-quickstart ---
[2022-05-04T05:32:42.267Z] [INFO] 
[2022-05-04T05:32:42.267Z] [INFO] --- maven-remote-resources-plugin:1.5:process 
(process-resource-bundles) @ streams-quickstart ---
[2022-05-04T05:32:43.237Z] [INFO] 
[2022-05-04T05:32:43.237Z] [INFO] --- maven-site-plugin:3.5.1:attach-descriptor 
(attach-descriptor) @ streams-quickstart ---
[2022-05-04T05:32:43.651Z] 
[2022-05-04T05:32:43.651Z] FAILURE: Build failed with an exception.
[2022-05-04T05:32:43.651Z] 
[2022-05-04T05:32:43.651Z] * What went wrong:
[2022-05-04T05:32:43.651Z] Execution failed for task ':storage:unitTest'.
[2022-05-04T05:32:43.651Z] > Process 'Gradle Test Executor 142' finished with 
non-zero exit value 1
[2022-05-04T05:32:43.651Z]   This problem might be caused by incorrect test 
process configuration.
[2022-05-04T05:

[jira] [Created] (KAFKA-13870) support both Suppressed untilTimeLimit and maxBytes without using emitEarlyWhenFull()

[Anil (Jira)]

Anil created KAFKA-13870:


 Summary: support both Suppressed untilTimeLimit and maxBytes 
without using emitEarlyWhenFull()
 Key: KAFKA-13870
 URL: https://issues.apache.org/jira/browse/KAFKA-13870
 Project: Kafka
  Issue Type: New Feature
Reporter: Anil


My use case is to use  ** *untilTimeLimit* with *maxBytes,* but when the buffer 
is full, the application is breaking, but with using *{{emitEarlyWhenFull}}* 
{{{}application is not breaking but{}}}{*}{{}}{*} it sends out the same key 
record multiple times in a particular window when the buffer exceeds max bytes 

for eg:-
*Suppressed.untilTimeLimit(Duration.ofMinutes(15),Suppressed.BufferConfig.maxBytes(1).emitEarlyWhenFull())*
 
messages flow : (A,1) (A,2) (A,3) -> aggregation result : (A,6) . suppose here, 
the buffer is full, (A,6) will be sent downstream. Let's suppose (A,4) comes 
now in the same tumbling window.
 
current response:- the aggregation will continue and eventually *(A,10)* will 
be emitted
 
but our application expected *(A,4),*  so the request for the feature is that 
window should be happening with window time(untilTimeLimit) or Buffer(maxByte) 
should full, in either of these two conditions met, a new window should be 
created and data should be emitted 
 
 



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Re: [DISCUSS] KIP-833: Mark KRaft as Production Ready

[Colin McCabe]

On Tue, May 3, 2022, at 20:48, Israel Ekpo wrote:
> Hi Colin,
>
> Thanks for this KIP. I am very excited to see the proposal.
>
> A lot in the community have been asking when KRaft mode will be ready and I
> think this KIP will provide a lot of the answers and guidance desperately
> needed.
>
> Thanks for working on it.
>
> I also have some of the questions from Luke regarding the duration of the
> deprecation period and the release where ZK will be completely dropped.
>
> As per our time-based release plan [1], the release should be every 4
> months, but it has been approximately 5 months between releases from the
> downloads page [2]
>
> So, right now we are about to push out 3.2.0 and 3.4.0 is approximately 10
> months from now based on the schedule/plan.
>

Hi Israel,

Sorry, I meant to write "the next release, 3.3" not "the next release, 3.4." I 
fixed the KIP text now.

As you noted, 3.3 should be about 4 or 5 months from now.

> I wonder if this will be enough time to complete all the features needed to
> bring KRaft mode to parity with Legacy/Zookeeper mode.
>
> The missing features you have highlighted and any other critical features
> not working in KRaft mode should be implemented fully before marking KRaft
> mode as production ready and deprecating Legacy mode.
>

As I wrote in "potential objections and responses," I don't think we need full 
feature parity before marking KRaft mode as production ready. It's customary 
for Kafka to bring new features to production incrementally.

>
> The main question here is would we have Kafka versions 3.6.0, 3.7.0, 3.8.0
> and 3.9.0 because this will add approximately 20 additional months after
> 3.4.0 which should be enough time to have ZK-mode deprecated and tested
> thoroughly before removal completely. I feel we should not mark ZK mode as
> deprecated until all the features in KRaft mode are at parity with ZK-mode.
>
> If we are going to have 20 months after 3.4 to work with a deprecated
> ZK-mode, then it could be possible to even drop ZK fully before 4.0
>
> Let's move the boat closer to the dock before we jump out of the boat. It
> will be very stressful if we jump out of the boat and have to swim several
> miles to shore.
>

That is a lot of Kafka versions! I don't think anyone wants 3.x to go on that 
long, or the ZooKeeper metadata implementation either. Ultimately I believe 
such a very long multi-year transition period would make the code quality and 
project stability worse, since we'd have to split our efforts across the two 
implementations.

To be clear, the proposal here is to have the bridge release be 3.4 and then 
move on to a ZK-free 4.0. With a 3.5 release as an option (but not requirement) 
if we can't finish everything in time after 3.4. So that would be two more 
releases with ZK, or dropping ZK a little bit less than a year from now.

best,
Colin

>
> [1]
> https://cwiki.apache.org/confluence/display/KAFKA/Time+Based+Release+Plan
> [2] https://kafka.apache.org/downloads
>
>
> Israel Ekpo
> Lead Instructor, IzzyAcademy.com
> https://www.youtube.com/c/izzyacademy
> https://izzyacademy.com/
>
>
> On Tue, May 3, 2022 at 10:32 PM Luke Chen  wrote:
>
>> Hi Colin,
>>
>> So exciting to see the KIP to mark the Kraft production ready!
>>
>> Just one comment: We should make sure the period between ZK deprecation
>> (i.e. v3.4.0) to ZK removal (i.e. v4.0.0) is not too short.
>> Do we have any expectation for the deprecation period?
>> After all, this is not a small feature change, and users need more time to
>> do the migration.
>> But to be honest, I don't know how long is long enough.
>> Maybe at least half a year? Or more? Thoughts?
>>
>>
>> Thank you.
>> Luke
>>
>> On Wed, May 4, 2022 at 9:10 AM Colin McCabe  wrote:
>>
>> > Hi all,
>> >
>> > I've written a KIP for marking KRaft as production ready. Please take a
>> > look if you have a chance:
>> >
>> > https://cwiki.apache.org/confluence/x/8xKhD
>> >
>> > thanks,
>> > Colin
>> >
>>

Re: [DISCUSS] KIP-833: Mark KRaft as Production Ready

[Colin McCabe]

On Tue, May 3, 2022, at 19:32, Luke Chen wrote:
> Hi Colin,
>
> So exciting to see the KIP to mark the Kraft production ready!
>
> Just one comment: We should make sure the period between ZK deprecation
> (i.e. v3.4.0) to ZK removal (i.e. v4.0.0) is not too short.
> Do we have any expectation for the deprecation period?
> After all, this is not a small feature change, and users need more time to
> do the migration.
> But to be honest, I don't know how long is long enough.
> Maybe at least half a year? Or more? Thoughts?
>

Hi Luke,

Like we discussed below, in chronological terms a Kafka release is about 4 or 5 
months. So if we have 3.3, followed by a 4.0 where we drop ZK, that would be 
about 8 to 10 months. If we need more time we can add a 3.4 which will add 
another 4-5 months in between.

best,
Colin

>
> Thank you.
> Luke
>
> On Wed, May 4, 2022 at 9:10 AM Colin McCabe  wrote:
>
>> Hi all,
>>
>> I've written a KIP for marking KRaft as production ready. Please take a
>> look if you have a chance:
>>
>> https://cwiki.apache.org/confluence/x/8xKhD
>>
>> thanks,
>> Colin
>>

Re: [DISCUSS] KIP-833: Mark KRaft as Production Ready

[Colin McCabe]

On Tue, May 3, 2022, at 23:16, Colin McCabe wrote:
>
> To be clear, the proposal here is to have the bridge release be 3.4 and 
> then move on to a ZK-free 4.0. With a 3.5 release as an option (but not 
> requirement) if we can't finish everything in time after 3.4. So that 
> would be two more releases with ZK, or dropping ZK a little bit less 
> than a year from now.
>

Sigh. This should read "To be clear, the proposal here is to have the bridge 
release be 3.3 and then move on to a ZK-free 4.0. With a 3.4 release as an 
option (but not requirement) if we can't finish everything in time after 3.3." 
So much for being clear, I guess :)

cheers,
Colin


> best,
> Colin
>
>>
>> [1]
>> https://cwiki.apache.org/confluence/display/KAFKA/Time+Based+Release+Plan
>> [2] https://kafka.apache.org/downloads
>>
>>
>> Israel Ekpo
>> Lead Instructor, IzzyAcademy.com
>> https://www.youtube.com/c/izzyacademy
>> https://izzyacademy.com/
>>
>>
>> On Tue, May 3, 2022 at 10:32 PM Luke Chen  wrote:
>>
>>> Hi Colin,
>>>
>>> So exciting to see the KIP to mark the Kraft production ready!
>>>
>>> Just one comment: We should make sure the period between ZK deprecation
>>> (i.e. v3.4.0) to ZK removal (i.e. v4.0.0) is not too short.
>>> Do we have any expectation for the deprecation period?
>>> After all, this is not a small feature change, and users need more time to
>>> do the migration.
>>> But to be honest, I don't know how long is long enough.
>>> Maybe at least half a year? Or more? Thoughts?
>>>
>>>
>>> Thank you.
>>> Luke
>>>
>>> On Wed, May 4, 2022 at 9:10 AM Colin McCabe  wrote:
>>>
>>> > Hi all,
>>> >
>>> > I've written a KIP for marking KRaft as production ready. Please take a
>>> > look if you have a chance:
>>> >
>>> > https://cwiki.apache.org/confluence/x/8xKhD
>>> >
>>> > thanks,
>>> > Colin
>>> >
>>>