Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #92

[Apache Jenkins Server]

See 

Request to be added the contributor list

[Josep Prat]

Hi,

I would like to start contributing to Apache Kafka, could you add my
username (josep.prat) to the contributors list for the Apache Kafka project?

Many thanks,

-- 

Josep Prat

*Aiven Deutschland GmbH*

Immanuelkirchstraße 26, 10405 Berlin

Amtsgericht Charlottenburg, HRB 209739 B

Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen

*w:* aiven.io

*e:* josep.p...@aiven.io

Re: Request to be added the contributor list

[Bruno Cadonna]

Hi Josep,

Thank you for your interest in Apache Kafka.

I cannot find your user name neither in Jira nor in the wiki.

Did you register yourself in Jira and Confluence?

Best,
Bruno

On 04.05.21 14:29, Josep Prat wrote:

Hi,

I would like to start contributing to Apache Kafka, could you add my
username (josep.prat) to the contributors list for the Apache Kafka project?

Many thanks,

Re: [VOTE] KIP-690: Add additional configuration to control MirrorMaker 2 internal topics naming convention

[Igor Soarez]

Another +1 here, also non-binding.

Thank you Omnia!

--
Igor


On Fri, Apr 30, 2021, at 3:15 PM, Ryanne Dolan wrote:
> +1 (non-binding), thanks!
> 
> On Thu, Jan 21, 2021, 4:31 AM Omnia Ibrahim  wrote:
> 
>> Hi
>> Can I get a vote on this, please?
>> 
>> Best
>> Omnia
>> 
>> On Tue, Dec 15, 2020 at 12:16 PM Omnia Ibrahim 
>> wrote:
>> 
>>> If anyone interested in reading the discussions you can find it here
>>> https://www.mail-archive.com/dev@kafka.apache.org/msg113373.html
>>> 
>>> On Tue, Dec 8, 2020 at 4:01 PM Omnia Ibrahim 
>>> wrote:
>>> 
 Hi everyone,
 I’m proposing a new KIP for MirrorMaker 2 to add the ability to control
 internal topics naming convention. The proposal details are here
 https://cwiki.apache.org/confluence/display/KAFKA/KIP-690%3A+Add+additional+configuration+to+control+MirrorMaker+2+internal+topics+naming+convention
 
 Please vote in this thread.
 Thanks
 Omnia
 
>>> 
> 


smime.p7s
Description: S/MIME cryptographic signature

Re: Request to be added the contributor list

[Josep Prat]

Hi Bruno,
I just double checked now, and I'm logged in at https://issues.apache.org
and the username is josep.prat. I registered myself a couple of hours ago.

Thanks in advance,

On Tue, May 4, 2021, 15:34 Bruno Cadonna  wrote:

> Hi Josep,
>
> Thank you for your interest in Apache Kafka.
>
> I cannot find your user name neither in Jira nor in the wiki.
>
> Did you register yourself in Jira and Confluence?
>
> Best,
> Bruno
>
> On 04.05.21 14:29, Josep Prat wrote:
> > Hi,
> >
> > I would like to start contributing to Apache Kafka, could you add my
> > username (josep.prat) to the contributors list for the Apache Kafka
> project?
> >
> > Many thanks,
> >
>

Re: Request to be added the contributor list

[Josep Prat]

Hi again Bruno,

Now I realize that every time I log in, I get the welcome page. Maybe
something went wrong with the registration process?

Best,

On Tue, May 4, 2021, 16:49 Josep Prat  wrote:

> Hi Bruno,
> I just double checked now, and I'm logged in at https://issues.apache.org
> and the username is josep.prat. I registered myself a couple of hours ago.
>
> Thanks in advance,
>
> On Tue, May 4, 2021, 15:34 Bruno Cadonna  wrote:
>
>> Hi Josep,
>>
>> Thank you for your interest in Apache Kafka.
>>
>> I cannot find your user name neither in Jira nor in the wiki.
>>
>> Did you register yourself in Jira and Confluence?
>>
>> Best,
>> Bruno
>>
>> On 04.05.21 14:29, Josep Prat wrote:
>> > Hi,
>> >
>> > I would like to start contributing to Apache Kafka, could you add my
>> > username (josep.prat) to the contributors list for the Apache Kafka
>> project?
>> >
>> > Many thanks,
>> >
>>
>

Re: [DISCUSS] KIP-730: Producer ID generation in KRaft mode

[David Arthur]

I've updated the KIP with a section on idempotence to reflect Ron's
comments in this thread. I'm going to open the vote thread shortly.

Thanks!
David

On Fri, Apr 16, 2021 at 7:04 PM Ron Dagostino  wrote:

> Thanks, David.  Yeah, I agree.  I was more bringing it up to make sure we
> explicitly discussed it.
>
> Ron
>
> > On Apr 16, 2021, at 2:15 PM, David Arthur  wrote:
> >
> > Guozhang / Ismael, yes agreed on the plurality of the naming. I've
> updated
> > the KIP.
> >
> > Ron, idempotent allocations are certainly possible, but as you pointed
> out
> > it might not be needed. It would require some additional book-keeping by
> > the controller to recall what was the last producer ID block allocated
> for
> > each broker. It would also open us up to bugs where the same ID block
> could
> > be given out repeatedly in the case of a broker providing some wrong
> > information in its request. It also doesn't help in the case of a broker
> > restarting since the broker won't know what its last block was (unless it
> > adds some local state management). Generally, I think the added
> complexity
> > is not worth it. The RPC rate limiting should be sufficient to protect us
> > from exhausting the ID space. If you agree, I can update the discussion
> > section of the KIP with this conclusion.
> >
> > Thanks for the feedback so far, everyone!
> >
> > -David
> >
> >> On Wed, Apr 14, 2021 at 4:37 PM Ismael Juma  wrote:
> >>
> >> Hi Guozhang,
> >>
> >> That was my original suggestion, so I am naturally +1 :)
> >>
> >> Ismael
> >>
> >>> On Wed, Apr 14, 2021 at 11:44 AM Guozhang Wang 
> wrote:
> >>>
> >>> Hi David,
> >>>
> >>> Just putting my paranoid hat here :) Could we name the req/resp name as
> >>> "AllocateProducerIds" instead of "AllocateProducerId"? Otherwise, LGTM!
> >>>
> >>> Guozhang
> >>>
>  On Thu, Apr 8, 2021 at 2:23 PM Ron Dagostino 
> wrote:
> 
>  Hi David.  I'm wondering if it might be a good idea to have the broker
>  send information about the last block it successfully received when it
>  requests a new block.  As the RPC stands right now it can't be
>  idempotent -- it just tells the controller "provide me a new block,
>  please".  One case where it might be useful for the RPC to be
>  idempotent is if the broker never receives the response from the
>  controller such that it asks again.  That would result in the burning
>  of the block that the controller provided but that the broker never
>  received.  Now, granted, the ID space is 64 bits, so we would have to
>  make ~2^54 requests to burn the entire space, and that isn't going to
>  happen.  So whether this is actually needed is questionable.  And it
>  might not be worth it to write the controller side code to make it act
>  idempotently even if we added the request field to make it possible.
>  But I figured this is worth mentioning even if we explicitly decide to
>  reject it.
> 
>  Ron
> 
>  On Thu, Apr 8, 2021 at 3:16 PM Ron Dagostino 
> >> wrote:
> >
> > Oh, I see.  Yes, my mistake -- I read it wrong.  You are right that
> > all we need in the metadata log is the latest value allocated.
> >
> > Ron
> >
> > On Thu, Apr 8, 2021 at 11:21 AM David Arthur 
> >> wrote:
> >>
> >> Ron -- I considered making the RPC response and record use the same
> >>> (or
> >> very similar) fields, but the use case is slightly different. A
> >>> broker
> >> handling the RPC needs to know the bounds of the block since it has
> >>> no
>  idea
> >> what the block size is. Also, the brokers will normally see
>  non-contiguous
> >> blocks.
> >>
> >> For the metadata log, we can just keep track of the latest producer
> >>> Id
>  that
> >> was allocated. It's kind of like a high watermark for producer IDs.
>  This
> >> actually saves us from needing an extra field in the record (the
> >> KIP
>  has
> >> just ProducerIdEnd => int64 in the record).
> >>
> >> Does that make sense?
> >>
> >> On Wed, Apr 7, 2021 at 8:44 AM Ron Dagostino 
>  wrote:
> >>
> >>> Thanks for the KIP, David.
> >>>
> >>> With the RPC returning a start and length, should the record in
> >> the
> >>> metadata log do the same thing for consistency and to save the
> >> byte
> >>> per record?
> >>>
> >>> Ron
> >>>
> >>>
> >>> On Tue, Apr 6, 2021 at 11:06 PM Ismael Juma 
>  wrote:
> 
>  Great, thanks. Instead of calling it "bridge release", can we
> >> say
>  3.0?
> 
>  Ismael
> 
>  On Tue, Apr 6, 2021 at 7:48 PM David Arthur 
>  wrote:
> 
> > Thanks for the feedback, Ismael. Renaming the RPC and using
>  start+len
> > instead of start+end sounds fine.
> >
> > And yes, the controller will allocate the IDs in ZK mode for
> >>> the
>  bridge
> > release.
> >

[VOTE] KIP-730: Producer ID generation in KRaft mode

[David Arthur]

Hello everyone, I'd like to start the vote on KIP-730 which adds a new RPC
for producer ID generation in KRaft mode.

https://cwiki.apache.org/confluence/display/KAFKA/KIP-730%3A+Producer+ID+generation+in+KRaft+mode



-- 
David Arthur

Re: Request to be added the contributor list

[Josep Prat]

Hi,
If for whatever reason it doesn't work with that username (work account), I
also have a personal account whose username is jlprat.

Thanks again for your help,


On Tue, May 4, 2021, 16:52 Josep Prat  wrote:

> Hi again Bruno,
>
> Now I realize that every time I log in, I get the welcome page. Maybe
> something went wrong with the registration process?
>
> Best,
>
> On Tue, May 4, 2021, 16:49 Josep Prat  wrote:
>
>> Hi Bruno,
>> I just double checked now, and I'm logged in at https://issues.apache.org
>> and the username is josep.prat. I registered myself a couple of hours ago.
>>
>> Thanks in advance,
>>
>> On Tue, May 4, 2021, 15:34 Bruno Cadonna  wrote:
>>
>>> Hi Josep,
>>>
>>> Thank you for your interest in Apache Kafka.
>>>
>>> I cannot find your user name neither in Jira nor in the wiki.
>>>
>>> Did you register yourself in Jira and Confluence?
>>>
>>> Best,
>>> Bruno
>>>
>>> On 04.05.21 14:29, Josep Prat wrote:
>>> > Hi,
>>> >
>>> > I would like to start contributing to Apache Kafka, could you add my
>>> > username (josep.prat) to the contributors list for the Apache Kafka
>>> project?
>>> >
>>> > Many thanks,
>>> >
>>>
>>

Re: Request to be added the contributor list

[Bruno Cadonna]

Hi Josep,

Now I found you and added you to the contributors.
Sorry, my bad!

Thank you again for your interest in Apache Kafka.

Best,
Bruno

On 04.05.21 16:52, Josep Prat wrote:

Hi again Bruno,

Now I realize that every time I log in, I get the welcome page. Maybe
something went wrong with the registration process?

Best,

On Tue, May 4, 2021, 16:49 Josep Prat  wrote:


Hi Bruno,
I just double checked now, and I'm logged in at https://issues.apache.org
and the username is josep.prat. I registered myself a couple of hours ago.

Thanks in advance,

On Tue, May 4, 2021, 15:34 Bruno Cadonna  wrote:


Hi Josep,

Thank you for your interest in Apache Kafka.

I cannot find your user name neither in Jira nor in the wiki.

Did you register yourself in Jira and Confluence?

Best,
Bruno

On 04.05.21 14:29, Josep Prat wrote:

Hi,

I would like to start contributing to Apache Kafka, could you add my
username (josep.prat) to the contributors list for the Apache Kafka

project?


Many thanks,

Re: Request to be added the contributor list

[Josep Prat]

Hi Bruno,

Great! Thanks again.


On Tue, May 4, 2021, 19:30 Bruno Cadonna  wrote:

> Hi Josep,
>
> Now I found you and added you to the contributors.
> Sorry, my bad!
>
> Thank you again for your interest in Apache Kafka.
>
> Best,
> Bruno
>
> On 04.05.21 16:52, Josep Prat wrote:
> > Hi again Bruno,
> >
> > Now I realize that every time I log in, I get the welcome page. Maybe
> > something went wrong with the registration process?
> >
> > Best,
> >
> > On Tue, May 4, 2021, 16:49 Josep Prat  wrote:
> >
> >> Hi Bruno,
> >> I just double checked now, and I'm logged in at
> https://issues.apache.org
> >> and the username is josep.prat. I registered myself a couple of hours
> ago.
> >>
> >> Thanks in advance,
> >>
> >> On Tue, May 4, 2021, 15:34 Bruno Cadonna  wrote:
> >>
> >>> Hi Josep,
> >>>
> >>> Thank you for your interest in Apache Kafka.
> >>>
> >>> I cannot find your user name neither in Jira nor in the wiki.
> >>>
> >>> Did you register yourself in Jira and Confluence?
> >>>
> >>> Best,
> >>> Bruno
> >>>
> >>> On 04.05.21 14:29, Josep Prat wrote:
>  Hi,
> 
>  I would like to start contributing to Apache Kafka, could you add my
>  username (josep.prat) to the contributors list for the Apache Kafka
> >>> project?
> 
>  Many thanks,
> 
> >>>
> >>
> >
>

[jira] [Created] (KAFKA-12748) Explore new RocksDB options to consider enabling by default

[A. Sophie Blee-Goldman (Jira)]

A. Sophie Blee-Goldman created KAFKA-12748:
--

 Summary: Explore new RocksDB options to consider enabling by 
default
 Key: KAFKA-12748
 URL: https://issues.apache.org/jira/browse/KAFKA-12748
 Project: Kafka
  Issue Type: Improvement
  Components: streams
Reporter: A. Sophie Blee-Goldman


With the rocksdb version bump comes a lot of new options, some of which look 
interesting enough to explore for usage in Streams. We should try setting these 
as default options and run the benchmarks to look for any performance benefit 
(or decrease). See javadocs for all Options 
[here|https://javadoc.io/doc/org.rocksdb/rocksdbjni/latest/org/rocksdb/Options.html]


Options.setAvoidUnnecessaryBlockingIO: 
- As the name suggest, avoids blocking/long-latency tasks by scheduling a 
background job to do it

 Options.setBestEffortsRecovery: 
- Interesting feature to allow recovering missing files without the use of 
the WAL. Could be useful if the on-disk state is corrupted (eg user deletes a 
file) without needing to rebuild state from scratch. Though I'd want to dig in 
further to understand what exactly it does and does not do. Not a performance 
improvement but we should run the benchmarks to make sure it doesn't make the 
performance worse.

Options.setWriteDbidToManifest:
- Should be set to true if/when we ever need to rely on the DB id eg for 
backups. Also not a performance improvement but we should still benchmark this.


Options.optimizeForSmallDb:
- This one is definitely not something we should set by default, as "small" 
here means under 1GB. But it's probably worth at least calling out in the docs 
for those users who know their data set size (per store) is under a GB



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] KIP-618: Atomic commit of source connector records and offsets

[Chris Egerton]

Hi all,

Good news everyone! I've reworked the design one more time and hopefully
some of the improvements here should make the proposal more palatable.
TL;DR:

- Rolling upgrades are now possible, in at most two phases; workers will
first have to be given a binary upgrade to 3.0 (the targeted version for
this feature) which can be a rolling upgrade, and then a rolling upgrade to
enable exactly-once source support in a cluster should be possible with no
anticipated downtime for source connectors or their tasks
- Offset topic migration is completely removed in favor of fallback to the
global offsets topic (much simpler!)
- One backwards-incompatible change is introduced: the leader will be
required to use a transactional producer for writes to the config topic
regardless of whether exactly-once support is enabled on the cluster.
Technically we could gate this behind a config property but since the
benefits of a transactional producer actually extend beyond exactly-once
source support (we can now ensure that there's only one writer to the
config topic at any given time, which isn't guaranteed with the current
model) and the cost to accommodate it is fairly low (a handful of
well-defined and limited-scope ACLs), I erred on the side of keeping things
simple

Looking forward to the next round of review and really hoping we can get
the ball rolling in time for this to land with 3.0!

Cheers,

Chris

On Mon, Apr 12, 2021 at 7:51 AM Chris Egerton  wrote:

> Hi Randall,
>
> After thinking things over carefully, I've done some reworking of the
> design. Instead of performing zombie fencing during rebalance, the leader
> will expose an internal REST endpoint that will allow workers to request a
> round of zombie fencing on demand, at any time. Workers will then hit this
> endpoint after starting connectors and after task config updates for
> connectors are detected; the precise details of this are outlined in the
> KIP. If a round of fencing should fail for any reason, the worker will be
> able to mark its Connector failed and, if the user wants to retry, they can
> simply restart the Connector via the REST API (specifically, the POST
> /connectors/{connector}/restart endpoint).
>
> The idea I'd been playing with to allow workers to directly write to the
> config topic seemed promising at first, but it allowed things to get pretty
> hairy for users if any kind of rebalancing bug took place and two workers
> believed they owned the same Connector object.
>
> I hope this answers any outstanding questions and look forward to your
> thoughts.
>
> Cheers,
>
> Chris
>
> On Mon, Mar 22, 2021 at 4:38 PM Chris Egerton  wrote:
>
>> Hi Randall,
>>
>> No complaints about email size from me. Let's dive in!
>>
>> 1. Sure! Especially important in my mind is that this is already possible
>> with Connect as it is today, and users can benefit from this with or
>> without the expanded exactly-once souce support we're trying to add with
>> this KIP. I've added that info to the "Motivation" section and included a
>> brief overview of the idempotent producer in the "Background and
>> References" section.
>>
>> 2. I actually hadn't considered enabling exactly-once source support by
>> default. Thinking over it now, I'm a little hesitant to do so just because,
>> even with the best testing out there, it's a pretty large change and it
>> seems safest to try to make it opt-in in case there's unanticipated
>> fallout. Then again, when incremental cooperative rebalancing was
>> introduced, it was made opt-out instead of opt-in. However, ICR came with
>> lower known risk of breaking existing users' setups; we know for a fact
>> that, if you haven't granted your worker or connector principals some ACLs
>> on Kafka, your connectors will fail. In an ideal world people would
>> carefully read upgrade notes and either grant those permissions or disable
>> the feature before upgrading their Connect cluster to 3.0, but if they
>> don't, they'll be in for a world of hurt. Overall I'd be more comfortable
>> letting this feature incubate for a little bit to let everyone get familiar
>> with it before possibly enabling it in 4.0 by default; what do you think?
>>
>> 3. I didn't think too long about the name for the offsets topic property;
>> it seemed pretty unlikely that it'd conflict with existing connector
>> property names. One alternative could be to follow the idiom established by
>> KIP-458 and include the word "override" in there somewhere, but none of the
>> resulting names seem very intuitive ("offsets.storage.override.topic" seems
>> the best to me but even that isn't super clear). Happy to field suggestions
>> if anyone has any alternatives they'd like to propose.
>>
>> 4. I _really_ wanted to enable per-connector toggling of this feature for
>> the exact reasons you've outlined. There's already a couple cases where
>> some piece of functionality was introduced that users could only control at
>> the worker config level and, later, effort had to

[jira] [Created] (KAFKA-12749) Changelog topic config on suppressed KTable lost

[Philip Bourke (Jira)]

Philip Bourke created KAFKA-12749:
-

 Summary: Changelog topic config on suppressed KTable lost
 Key: KAFKA-12749
 URL: https://issues.apache.org/jira/browse/KAFKA-12749
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 2.6.2, 2.8.0, 2.6.1, 2.7.0, 2.6.0
Reporter: Philip Bourke


When trying to set the changelog configuration on a suppressed KTable, the 
config is lost if either {{emitEarlyWhenFull}} or {{shutDownWhenFull}} is set 
after the logging config.

This works - 
{code:java}
.suppress(Suppressed.untilTimeLimit(Duration.ofMillis(maxIdleIntervalMs), 
BufferConfig.maxRecords(
 
maxBufferRecords).emitEarlyWhenFull().withLoggingEnabled(changelogConfig)){code}
but not if you set {{emitEarlyWhenFull}} last.

See comments in https://issues.apache.org/jira/browse/KAFKA-8147

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] KIP-699: Update FindCoordinator to resolve multiple Coordinators at a time

[Sanjana Kaundinya]

Hi Mickael,

Just wanted to ping the thread to inquire if you’ve made any progress on the 
implementation. I was planning on doing the implementation for KIP-709 and 
seeing how KIP-699 is something that’s tightly coupled and related to KIP-709 I 
thought I’d ask what the status of this KIP was.

Thanks,
Sanjana
On Apr 13, 2021, 8:44 AM -0700, Rajini Sivaram , wrote:
> Thanks Mickael!
>
> Regards,
>
> Rajini
>
> On Tue, Apr 13, 2021 at 4:40 PM Mickael Maison 
> wrote:
>
> > Thanks Rajini!
> > Yes it would be good to get it into 3.0. I've got some of it
> > implemented already, I'll try to get a PR out in the next couple of
> > weeks.
> >
> > I'm closing this vote. This KIP has been accepted with 3 +1 binding
> > votes from David, Tom and Rajini.
> >
> > On Tue, Apr 13, 2021 at 1:49 PM Rajini Sivaram 
> > wrote:
> > >
> > > Hi Mickael,
> > >
> > > +1 (binding)
> > >
> > > Thanks for the KIP!. Looks like this KIP has sufficient votes. It will be
> > > good to get this into 3.0 along with KIP-709. Will you have time to work
> > on
> > > this? Please let us know if we can help with the implementation or
> > reviews.
> > >
> > > Thank you,
> > >
> > > Rajini
> > >
> > >
> > > On Thu, Mar 18, 2021 at 4:00 PM Tom Bentley  wrote:
> > >
> > > > Hi Mickael,
> > > >
> > > > I'd like to re-cast my vote as +1 (binding) now I'm a committer.
> > > >
> > > > Thanks again,
> > > >
> > > > Tom
> > > >
> > > > On Tue, Mar 2, 2021 at 9:46 AM David Jacot 
> > wrote:
> > > >
> > > > > Thanks for the KIP, Mickael. +1 (binding)
> > > > >
> > > > > On Mon, Mar 1, 2021 at 11:53 AM Tom Bentley 
> > wrote:
> > > > >
> > > > > > +1 (non-binding), thanks Mickael.
> > > > > >
> > > > > > On Thu, Feb 25, 2021 at 6:32 PM Mickael Maison <
> > mimai...@apache.org>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > I'd like to start a vote on KIP-699 to support resolving multiple
> > > > > > > coordinators at a time:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-699%3A+Update+FindCoordinator+to+resolve+multiple+Coordinators+at+a+time
> > > > > > >
> > > > > > > Thanks
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >