Re: [ANNOUNCE] New committer: Justine Olshan

[Tom Bentley]

Congratulations!

On Mon, 2 Jan 2023 at 18:25, Rajini Sivaram  wrote:

> Congratulations, Justine, well deserved!
>
> Regards,
>
> Rajini
>
> On Mon, Jan 2, 2023 at 5:29 PM Ismael Juma  wrote:
>
> > Congratulations Justine!
> >
> > Ismael
> >
> > On Thu, Dec 29, 2022, 12:58 PM David Jacot  wrote:
> >
> > > Hi all,
> > >
> > > The PMC of Apache Kafka is pleased to announce a new Kafka committer
> > > Justine
> > > Olshan.
> > >
> > > Justine has been contributing to Kafka since June 2019. She contributed
> > 53
> > > PRs including the following KIPs.
> > >
> > > KIP-480: Sticky Partitioner
> > > KIP-516: Topic Identifiers & Topic Deletion State Improvements
> > > KIP-854: Separate configuration for producer ID expiry
> > > KIP-890: Transactions Server-Side Defense (in progress)
> > >
> > > Congratulations, Justine!
> > >
> > > Thanks,
> > >
> > > David (on behalf of the Apache Kafka PMC)
> > >
> >
>

Re: [DISCUSS] KIP-892: Transactional Semantics for StateStores

[Nick Telford]

Hi Lucas,

Thanks for looking over my KIP.

A) The bound is per-instance, not per-Task. This was a typo in the KIP that
I've now corrected. It was originally per-Task, but I changed it to
per-instance for exactly the reason you highlighted.
B) It's worth noting that transactionality is only enabled under EOS, and
in the default mode of operation (ALOS), there should be no change in
behavior at all. I think, under EOS, we can mitigate the impact on users by
sufficiently low default values for the memory bound configuration. I
understand your hesitation to include a significant change of behaviour,
especially in a minor release, but I suspect that most users will prefer
the memory impact (under EOS) to the existing behaviour of frequent state
restorations! If this is a problem, the changes can wait until the next
major release. I'll be running a patched version of streams in production
with these changes as soon as they're ready, so it won't disrupt me :-D
C) The main purpose of this sentence was just to note that some changes
will need to be made to the way Segments are handled in order to ensure
they also benefit from transactions. At the time I wrote it, I hadn't
figured out the specific changes necessary, so it was deliberately vague.
This is the one outstanding problem I'm currently working on, and I'll
update this section with more detail once I have figured out the exact
changes required.
D) newTransaction() provides the necessary isolation guarantees. While the
RocksDB implementation of transactions doesn't technically *need* read-only
users to call newTransaction(), other implementations (e.g. a hypothetical
PostgresStore) may require it. Calling newTransaction() when no transaction
is necessary is essentially free, as it will just return this.

I didn't do any profiling of the KIP-844 PoC, but I think it should be
fairly obvious where the performance problems stem from: writes under
KIP-844 require 3 extra memory-copies: 1 to encode it with the
tombstone/record flag, 1 to decode it from the tombstone/record flag, and 1
to copy the record from the "temporary" store to the "main" store, when the
transaction commits. The different approach taken by KIP-869 should perform
much better, as it avoids all these copies, and may actually perform
slightly better than trunk, due to batched writes in RocksDB performing
better than non-batched writes.[1]

Regards,
Nick

1: https://github.com/adamretter/rocksjava-write-methods-benchmark#results

On Mon, 2 Jan 2023 at 16:18, Lucas Brutschy 
wrote:

> Hi Nick,
>
> I'm just starting to read up on the whole discussion about KIP-892 and
> KIP-844. Thanks a lot for your work on this, I do think
> `WriteBatchWithIndex` may be the way to go here. I do have some
> questions about the latest draft.
>
>  A) If I understand correctly, you propose to put a bound on the
> (native) memory consumed by each task. However, I wonder if this is
> sufficient if we have temporary imbalances in the cluster. For
> example, depending on the timing of rebalances during a cluster
> restart, it could happen that a single streams node is assigned a lot
> more tasks than expected. With your proposed change, this would mean
> that the memory required by this one node could be a multiple of what
> is required during normal operation. I wonder if it wouldn't be safer
> to put a global bound on the memory use, across all tasks.
>  B) Generally, the memory concerns still give me the feeling that this
> should not be enabled by default for all users in a minor release.
>  C) In section "Transaction Management": the sentence "A similar
> analogue will be created to automatically manage `Segment`
> transactions.". Maybe this is just me lacking some background, but I
> do not understand this, it would be great if you could clarify what
> you mean here.
>  D) Could you please clarify why IQ has to call newTransaction(), when
> it's read-only.
>
> And one last thing not strictly related to your KIP: if there is an
> easy way for you to find out why the KIP-844 PoC is 20x slower (e.g.
> by providing a flame graph), that would be quite interesting.
>
> Cheers,
> Lucas
>
> On Thu, Dec 22, 2022 at 8:30 PM Nick Telford 
> wrote:
> >
> > Hi everyone,
> >
> > I've updated the KIP with a more detailed design, which reflects the
> > implementation I've been working on:
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-892%3A+Transactional+Semantics+for+StateStores
> >
> > This new design should address the outstanding points already made in the
> > thread.
> >
> > Please let me know if there are areas that are unclear or need more
> > clarification.
> >
> > I have a (nearly) working implementation. I'm confident that the
> remaining
> > work (making Segments behave) will not impact the documented design.
> >
> > Regards,
> >
> > Nick
> >
> > On Tue, 6 Dec 2022 at 19:24, Colt McNealy  wrote:
> >
> > > Nick,
> > >
> > > Thank you for the reply; that makes sense. I was hoping that, since
> reading
> > > u

[jira] [Created] (KAFKA-14560) Remove old client protocol API versions in Kafka 4.0 (KIP-896)

[Ismael Juma (Jira)]

Ismael Juma created KAFKA-14560:
---

 Summary: Remove old client protocol API versions in Kafka 4.0 
(KIP-896)
 Key: KAFKA-14560
 URL: https://issues.apache.org/jira/browse/KAFKA-14560
 Project: Kafka
  Issue Type: Improvement
Reporter: Ismael Juma
Assignee: Ismael Juma


Please see KIP for details:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-896%3A+Remove+old+client+protocol+API+versions+in+Kafka+4.0



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[DISCUSS] KIP-896: Remove old client protocol API versions in Kafka 4.0

[Ismael Juma]

Hi all,

I would like to start a discussion regarding the removal of very old client
protocol API versions in Apache Kafka 4.0 to improve maintainability &
supportability of Kafka. Please take a look at the proposal:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-896%3A+Remove+old+client+protocol+API+versions+in+Kafka+4.0

Ismael

Re: [ANNOUNCE] New committer: Josep Prat

[Bill Bejeck]

Congratulations, Josep!

-Bill

On Tue, Dec 20, 2022 at 9:03 PM Luke Chen  wrote:

> Congratulations, Josep!
>
> Luke
>
> On Wed, Dec 21, 2022 at 6:26 AM Viktor Somogyi-Vass
>  wrote:
>
> > Congrats Josep!
> >
> > On Tue, Dec 20, 2022, 21:56 Matthias J. Sax  wrote:
> >
> > > Congrats!
> > >
> > > On 12/20/22 12:01 PM, Josep Prat wrote:
> > > > Thank you all!
> > > >
> > > > ———
> > > > Josep Prat
> > > >
> > > > Aiven Deutschland GmbH
> > > >
> > > > Immanuelkirchstraße 26, 10405 Berlin
> > > >
> > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > >
> > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > >
> > > > m: +491715557497
> > > >
> > > > w: aiven.io
> > > >
> > > > e: josep.p...@aiven.io
> > > >
> > > > On Tue, Dec 20, 2022, 20:42 Bill Bejeck  wrote:
> > > >
> > > >> Congratulations Josep!
> > > >>
> > > >> -Bill
> > > >>
> > > >> On Tue, Dec 20, 2022 at 1:11 PM Mickael Maison <
> > > mickael.mai...@gmail.com>
> > > >> wrote:
> > > >>
> > > >>> Congratulations Josep!
> > > >>>
> > > >>> On Tue, Dec 20, 2022 at 6:55 PM Bruno Cadonna 
> > > >> wrote:
> > > 
> > >  Congrats, Josep!
> > > 
> > >  Well deserved!
> > > 
> > >  Best,
> > >  Bruno
> > > 
> > >  On 20.12.22 18:40, Kirk True wrote:
> > > > Congrats Josep!
> > > >
> > > > On Tue, Dec 20, 2022, at 9:33 AM, Jorge Esteban Quilcate Otoya
> > wrote:
> > > >> Congrats Josep!!
> > > >>
> > > >> On Tue, 20 Dec 2022, 17:31 Greg Harris,
> > > >>  > > 
> > > >> wrote:
> > > >>
> > > >>> Congratulations Josep!
> > > >>>
> > > >>> On Tue, Dec 20, 2022 at 9:29 AM Chris Egerton <
> > > >>> fearthecel...@gmail.com>
> > > >>> wrote:
> > > >>>
> > >  Congrats Josep! Well-earned.
> > > 
> > >  On Tue, Dec 20, 2022, 12:26 Jun Rao  >
> > > >>> wrote:
> > > 
> > > > Hi, Everyone,
> > > >
> > > > The PMC of Apache Kafka is pleased to announce a new Kafka
> > > >>> committer
> > >  Josep
> > > >Prat.
> > > >
> > > > Josep has been contributing to Kafka since May 2021. He
> > > >>> contributed 20
> > >  PRs
> > > > including the following 2 KIPs.
> > > >
> > > > KIP-773 Differentiate metric latency measured in ms and ns
> > > > KIP-744: Migrate TaskMetadata and ThreadMetadata to an
> > interface
> > > >>> with
> > > > internal implementation
> > > >
> > > > Congratulations, Josep!
> > > >
> > > > Thanks,
> > > >
> > > > Jun (on behalf of the Apache Kafka PMC)
> > > >
> > > 
> > > >>>
> > > >>
> > > >
> > > >>>
> > > >>
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-858: Handle JBOD broker disk failure in KRaft

[Igor Soarez]

Hi Jun,

Thank you for having another look.

11. That is correct. I have updated the KIP in an attempt to make this clearer.
I think the goal should be to try to minimize the chance that a log directory
may happen while the metadata is incorrect about the log directory assignment,
but also have a fallback safety mechanism to indicate to the controller that
some replica was missed in case of a bad race.

13. Ok, I think I have misunderstood this. Thank you for correcting me.
In this case the broker can update the existing meta.properties and create
new meta.properties in the new log directories.
This also means that the update-directories subcommand in kafka-storage.sh
is not necessary.
I have updated the KIP to reflect this.

Please have another look.


Thank you,

--
Igor


> On 22 Dec 2022, at 00:25, Jun Rao  wrote:
> 
> Hi, Igor,
> 
> Thanks for the reply.
> 
> 11. Yes, your proposal could work. Once the broker receives confirmation of
> the metadata change, I guess it needs to briefly block appends to the old
> replica, make sure the future log fully catches up and then make the switch?
> 
> 13 (b). The kafka-storage.sh is only required in KIP-631 for a brand new
> KRaft cluster. If the cluster already exists and one just wants to add a
> log dir, it seems inconvenient to have to run the kafka-storage.sh tool
> again.
> 
> Thanks,
> 
> Jun

[jira] [Created] (KAFKA-14561) Improve transactions experience for older clients by ensuring ongoing transaction

[Justine Olshan (Jira)]

Justine Olshan created KAFKA-14561:
--

 Summary: Improve transactions experience for older clients by 
ensuring ongoing transaction
 Key: KAFKA-14561
 URL: https://issues.apache.org/jira/browse/KAFKA-14561
 Project: Kafka
  Issue Type: Sub-task
Reporter: Justine Olshan


This is part 3 of KIP-890:

3. *To cover older clients, we will ensure a transaction is ongoing before we 
write to a transaction. We can do this by querying the transaction coordinator 
and caching the result.*

See KIP-890 for more details: 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-890%3A+Transactions+Server-Side+Defense



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14562) Implement epoch bump after every transaction

[Justine Olshan (Jira)]

Justine Olshan created KAFKA-14562:
--

 Summary: Implement epoch bump after every transaction
 Key: KAFKA-14562
 URL: https://issues.apache.org/jira/browse/KAFKA-14562
 Project: Kafka
  Issue Type: Sub-task
Reporter: Justine Olshan
Assignee: Justine Olshan


This is part 1 of KIP-890


 # *Uniquely identify transactions by bumping the producer epoch after every 
commit/abort marker. That way, each transaction can be identified by (producer 
id, epoch).* 



See KIP-890 for more information: 
[https://cwiki.apache.org/confluence/display/KAFKA/KIP-890%3A+Transactions+Server-Side+Defense]
 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14563) Remove AddPartitionsToTxn call for newer clients as optimization

[Justine Olshan (Jira)]

Justine Olshan created KAFKA-14563:
--

 Summary: Remove AddPartitionsToTxn call for newer clients as 
optimization
 Key: KAFKA-14563
 URL: https://issues.apache.org/jira/browse/KAFKA-14563
 Project: Kafka
  Issue Type: Sub-task
Reporter: Justine Olshan
Assignee: Justine Olshan


This is part 2 of KIP-890:

{*}2. Remove the addPartitionsToTxn call and implicitly just add partitions to 
the transaction on the first produce request during a transaction{*}.

See KIP-890 for more information: 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-890%3A+Transactions+Server-Side+Defense



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-10550) Update AdminClient and kafka-topics.sh to support topic IDs

[Justine Olshan (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-10550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Justine Olshan resolved KAFKA-10550.

Resolution: Fixed

I think the scope of the kip – describe and delete has been completed so I will 
mark this as resolved for now.

> Update AdminClient and kafka-topics.sh to support topic IDs
> ---
>
> Key: KAFKA-10550
> URL: https://issues.apache.org/jira/browse/KAFKA-10550
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Justine Olshan
>Assignee: Deng Ziming
>Priority: Major
>
> Change describe topics AdminClient method to expose and support topic IDs 
>  
>  Make changes to kafka-topics.sh --describe so a user can specify a topic 
> name to describe with the --topic parameter, or alternatively the user can 
> supply a topic ID with the --topic_id parameter



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [ANNOUNCE] New committer: Justine Olshan

[Matthias J. Sax]

Congrats!

On 12/29/22 6:47 PM, ziming deng wrote:

Congratulations Justine!
—
Best,
Ziming


On Dec 30, 2022, at 10:06, Luke Chen  wrote:

Congratulations, Justine!
Well deserved!

Luke

On Fri, Dec 30, 2022 at 9:15 AM Ron Dagostino  wrote:


Congratulations, Justine!Well-deserved., and I’m very happy for you.

Ron


On Dec 29, 2022, at 6:13 PM, Israel Ekpo  wrote:

Congratulations Justine!



On Thu, Dec 29, 2022 at 5:05 PM Greg Harris



wrote:

Congratulations Justine!


On Thu, Dec 29, 2022 at 1:37 PM Bill Bejeck  wrote:

Congratulations Justine!


-Bill


On Thu, Dec 29, 2022 at 4:36 PM Philip Nee 

wrote:



wow congrats!

On Thu, Dec 29, 2022 at 1:05 PM Chris Egerton <

fearthecel...@gmail.com



wrote:


Congrats, Justine!

On Thu, Dec 29, 2022, 15:58 David Jacot  wrote:


Hi all,

The PMC of Apache Kafka is pleased to announce a new Kafka

committer

Justine
Olshan.

Justine has been contributing to Kafka since June 2019. She

contributed

53

PRs including the following KIPs.

KIP-480: Sticky Partitioner
KIP-516: Topic Identifiers & Topic Deletion State Improvements
KIP-854: Separate configuration for producer ID expiry
KIP-890: Transactions Server-Side Defense (in progress)

Congratulations, Justine!

Thanks,

David (on behalf of the Apache Kafka PMC)

Re: [VOTE] 3.3.2 RC1

[Manikumar]

Hi Chris,

+1 (binding)

- verified the signatures, artifacts
- ran the tests on the source archive
- verified the quickstarts

Thanks for running the release!

Thanks,
Manikumar


On Fri, Dec 23, 2022 at 4:44 PM Federico Valeri  wrote:
>
> Hi, I did the following to validate the release:
>
> - Checksums and signatures ok
> - Build from source using Java 17 and Scala 2.13 ok
> - Unit and integration tests ok
> - Quickstart in both ZK and KRaft modes ok
> - Test app with staging Maven artifacts ok
>
> +1 (non binding)
>
> Thanks
> Fede
>
> On Wed, Dec 21, 2022 at 11:21 PM Chris Egerton  
> wrote:
> >
> > Hello Kafka users, developers and client-developers,
> >
> > This is the second candidate for release of Apache Kafka 3.3.2.
> >
> > This is a bugfix release with several fixes since the release of 3.3.1. A
> > few of the major issues include:
> >
> > * KAFKA-14358 Users should not be able to create a regular topic name
> > __cluster_metadata
> > KAFKA-14379 Consumer should refresh preferred read replica on update
> > metadata
> > * KAFKA-13586 Prevent exception thrown during connector update from
> > crashing distributed herder
> >
> >
> > Release notes for the 3.3.2 release:
> > https://home.apache.org/~cegerton/kafka-3.3.2-rc1/RELEASE_NOTES.html
> >
> > *** Please download, test and vote by Friday, January 6, 2023, 10pm UTC
> > (this date is chosen to accommodate the various upcoming holidays that
> > members of the community will be taking and give everyone enough time to
> > test out the release candidate, without unduly delaying the release)
> >
> > Kafka's KEYS file containing PGP keys we use to sign the release:
> > https://kafka.apache.org/KEYS
> >
> > * Release artifacts to be voted upon (source and binary):
> > https://home.apache.org/~cegerton/kafka-3.3.2-rc1/
> >
> > * Maven artifacts to be voted upon:
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> >
> > * Javadoc:
> > https://home.apache.org/~cegerton/kafka-3.3.2-rc1/javadoc/
> >
> > * Tag to be voted upon (off 3.3 branch) is the 3.3.2 tag:
> > https://github.com/apache/kafka/releases/tag/3.3.2-rc1
> >
> > * Documentation:
> > https://kafka.apache.org/33/documentation.html
> >
> > * Protocol:
> > https://kafka.apache.org/33/protocol.html
> >
> > The most recent build has had test failures. These all appear to be due to
> > flakiness, but it would be nice if someone more familiar with the failed
> > tests could confirm this. I may update this thread with passing build links
> > if I can get one, or start a new release vote thread if test failures must
> > be addressed beyond re-running builds until they pass.
> >
> > Unit/integration tests:
> > https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.3/142/testReport/
> >
> > José, would it be possible to re-run the system tests for 3.3 on the latest
> > commit for 3.3 (e3212f2), and share the results on this thread?
> >
> > Cheers,
> >
> > Chris

Re: [ANNOUNCE] New committer: Josep Prat

[Josep Prat]

Thanks all again! :)

On Tue, Jan 3, 2023 at 6:19 PM Bill Bejeck 
wrote:

> Congratulations, Josep!
>
> -Bill
>
> On Tue, Dec 20, 2022 at 9:03 PM Luke Chen  wrote:
>
> > Congratulations, Josep!
> >
> > Luke
> >
> > On Wed, Dec 21, 2022 at 6:26 AM Viktor Somogyi-Vass
> >  wrote:
> >
> > > Congrats Josep!
> > >
> > > On Tue, Dec 20, 2022, 21:56 Matthias J. Sax  wrote:
> > >
> > > > Congrats!
> > > >
> > > > On 12/20/22 12:01 PM, Josep Prat wrote:
> > > > > Thank you all!
> > > > >
> > > > > ———
> > > > > Josep Prat
> > > > >
> > > > > Aiven Deutschland GmbH
> > > > >
> > > > > Immanuelkirchstraße 26, 10405 Berlin
> > > > >
> > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > >
> > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > > >
> > > > > m: +491715557497
> > > > >
> > > > > w: aiven.io
> > > > >
> > > > > e: josep.p...@aiven.io
> > > > >
> > > > > On Tue, Dec 20, 2022, 20:42 Bill Bejeck  wrote:
> > > > >
> > > > >> Congratulations Josep!
> > > > >>
> > > > >> -Bill
> > > > >>
> > > > >> On Tue, Dec 20, 2022 at 1:11 PM Mickael Maison <
> > > > mickael.mai...@gmail.com>
> > > > >> wrote:
> > > > >>
> > > > >>> Congratulations Josep!
> > > > >>>
> > > > >>> On Tue, Dec 20, 2022 at 6:55 PM Bruno Cadonna <
> cado...@apache.org>
> > > > >> wrote:
> > > > 
> > > >  Congrats, Josep!
> > > > 
> > > >  Well deserved!
> > > > 
> > > >  Best,
> > > >  Bruno
> > > > 
> > > >  On 20.12.22 18:40, Kirk True wrote:
> > > > > Congrats Josep!
> > > > >
> > > > > On Tue, Dec 20, 2022, at 9:33 AM, Jorge Esteban Quilcate Otoya
> > > wrote:
> > > > >> Congrats Josep!!
> > > > >>
> > > > >> On Tue, 20 Dec 2022, 17:31 Greg Harris,
> > > > >>  > > > 
> > > > >> wrote:
> > > > >>
> > > > >>> Congratulations Josep!
> > > > >>>
> > > > >>> On Tue, Dec 20, 2022 at 9:29 AM Chris Egerton <
> > > > >>> fearthecel...@gmail.com>
> > > > >>> wrote:
> > > > >>>
> > > >  Congrats Josep! Well-earned.
> > > > 
> > > >  On Tue, Dec 20, 2022, 12:26 Jun Rao
>  > >
> > > > >>> wrote:
> > > > 
> > > > > Hi, Everyone,
> > > > >
> > > > > The PMC of Apache Kafka is pleased to announce a new Kafka
> > > > >>> committer
> > > >  Josep
> > > > >Prat.
> > > > >
> > > > > Josep has been contributing to Kafka since May 2021. He
> > > > >>> contributed 20
> > > >  PRs
> > > > > including the following 2 KIPs.
> > > > >
> > > > > KIP-773 Differentiate metric latency measured in ms and ns
> > > > > KIP-744: Migrate TaskMetadata and ThreadMetadata to an
> > > interface
> > > > >>> with
> > > > > internal implementation
> > > > >
> > > > > Congratulations, Josep!
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun (on behalf of the Apache Kafka PMC)
> > > > >
> > > > 
> > > > >>>
> > > > >>
> > > > >
> > > > >>>
> > > > >>
> > > > >
> > > >
> > >
> >
>


-- 
[image: Aiven] 

*Josep Prat*
Open Source Engineering Director, *Aiven*
josep.p...@aiven.io   |   +491715557497
aiven.io    |   
     
*Aiven Deutschland GmbH*
Immanuelkirchstraße 26, 10405 Berlin
Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
Amtsgericht Charlottenburg, HRB 209739 B

[jira] [Created] (KAFKA-14564) Upgrade Netty to 4.1.86 to fix CVEs

[Brendan Ribera (Jira)]

Brendan Ribera created KAFKA-14564:
--

 Summary: Upgrade Netty to 4.1.86 to fix CVEs
 Key: KAFKA-14564
 URL: https://issues.apache.org/jira/browse/KAFKA-14564
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 3.3.1
Reporter: Brendan Ribera


4.1.86 fixes two CVEs:
 * [https://nvd.nist.gov/vuln/detail/CVE-2022-41881]
 * [https://nvd.nist.gov/vuln/detail/CVE-2022-41915]

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14565) Add A No Implementation Default Open Method To Consumer and Producer Interceptor Interfaces

[Terry Beard (Jira)]

Terry Beard created KAFKA-14565:
---

 Summary: Add A No Implementation Default Open Method To Consumer 
and Producer Interceptor Interfaces
 Key: KAFKA-14565
 URL: https://issues.apache.org/jira/browse/KAFKA-14565
 Project: Kafka
  Issue Type: Improvement
  Components: clients
Reporter: Terry Beard
Assignee: Terry Beard


h2. PROBLEM

The Consumer and Producer interceptor interfaces and their corresponding Kafka 
Consumer and Producer constructors do not adequately support cleanup of 
underlying interceptor resources. [More 
colors|https://issues.apache.org/jira/secure/CreateIssue.jspa#]

Currently within the Kafka Consumer and Kafka Producer constructors,  the 
AbstractConfig.getConfiguredInstances()  is delagated responsibilty for both 
creating and configuring each interceptor listed in the interceptor.classes 
property and returns a configured  List> interceptors.
h2. Kafka Consumer Constructor

 
{code:java}
try {

List> interceptorList = (List) 
config.getConfiguredInstances(
        ConsumerConfig.INTERCEPTOR_CLASSES_CONFIG,
        ConsumerInterceptor.class,
        Collections.singletonMap(ConsumerConfig.CLIENT_ID_CONFIG, clientId));
 {code}
 

 
h2. Kafka Producer Constructor
{code:java}
try {

List> interceptorList = (List) 
config.getConfiguredInstances(
        ProducerConfig.INTERCEPTOR_CLASSES_CONFIG,
        ProducerInterceptor.class,
        Collections.singletonMap(ProducerConfig.CLIENT_ID_CONFIG, clientId));
 {code}
This dual responsibility for both creation and configuration is problematic 
when it involves multiple interceptors where at least one interceptor's 
configure method implementation creates and/or depends on objects which creates 
threads, connections or other resources which requires clean up and the 
subsequent interceptor's configure method raises a runtime exception.  This 
raising of the runtime exception results produces a resource leakage in the 
first interceptor as the interceptor container i.e. 
ConsumerInterceptors/ProducerInterceptors are never created and therefore the 
first interceptor's and really any interceptor's close method are never called. 
 
h2. KafkaConsumer Constructor
{code:java}
try {

List> interceptorList = (List) 
config.getConfiguredInstances(
        ConsumerConfig.INTERCEPTOR_CLASSES_CONFIG,
        ConsumerInterceptor.class,
        Collections.singletonMap(ConsumerConfig.CLIENT_ID_CONFIG, clientId));
 {code}
If the above line results in a runtime exception, the below this.interceptors 
is never created. 
{code:java}
this.interceptors = new ConsumerInterceptors<>(interceptorList); {code}
h2. Kafka Producer{color:#172b4d} Constructor{color}
{code:java}
try {

List> interceptorList = (List) 
config.getConfiguredInstances(
        ProducerConfig.INTERCEPTOR_CLASSES_CONFIG,
        ProducerInterceptor.class,
        Collections.singletonMap(ProducerConfig.CLIENT_ID_CONFIG, clientId)); 
{code}
If the above line results in a runtime exception, the below this.interceptors 
is never created. 
{code:java}
if (interceptors != null)
    this.interceptors = interceptors;
else
    this.interceptors = new ProducerInterceptors<>(interceptorList);
 {code}
 

Although, both Kafka Consumer and Kafka Producer constructors try/catch 
implement  close for resource clean up, 
{code:java}
...
catch (Throwable t) {
    // call close methods if internal objects are already constructed; this is 
to prevent resource leak. see KAFKA-2121
    // we do not need to call `close` at all when `log` is null, which means no 
internal objects were initialized.
    if (this.log != null) {
        close(0, true);
    }
    // now propagate the exception
    throw new KafkaException("Failed to construct kafka consumer", t);
} {code}
their respective close implementation located in the catch above never calls 
the respective container interceptor close method below as the 
{color:#172b4d}*{color:#ffab00}this{color}.{color:#403294}interceptors{color}*{color}{color:#403294}
 {color}was never created.
{code:java}
private void close(long timeoutMs, boolean swallowException) {
   
Utils.closeQuietly(interceptors, "consumer interceptors", firstException);
   {code}
This problem is magnified within a webserver cluster i.e. Confluent's REST 
Proxy server where thousands of requests containing interceptor configuration 
failures can occur in seconds resulting in an inadvertent DDoS attack as 
cluster resources are quickly exhausted, disrupting all service activities.   
h2. PROPOSAL

To help ensure the respective container interceptors are able to invoke their 
respective interceptor close methods for proper resource clean up, I propose 
defining a default open method with no implementation and a check exception on 
the respective Consumer/Producer interceptor interfaces.  This open method will 
be responsible for creating threads an

[jira] [Created] (KAFKA-14566) Add A No Implementation Default Open Method To Consumer and Producer Interceptor Interfaces

[Terry Beard (Jira)]

Terry Beard created KAFKA-14566:
---

 Summary: Add A No Implementation Default Open Method To Consumer 
and Producer Interceptor Interfaces
 Key: KAFKA-14566
 URL: https://issues.apache.org/jira/browse/KAFKA-14566
 Project: Kafka
  Issue Type: Improvement
  Components: clients
Reporter: Terry Beard


h2. PROBLEM

The Consumer and Producer interceptor interfaces and their corresponding Kafka 
Consumer and Producer constructors do not adequately support cleanup of 
underlying interceptor resources. 

Currently within the Kafka Consumer and Kafka Producer constructors,  the 
AbstractConfig.getConfiguredInstances()  is delegated responsibility for both 
creating and configuring each interceptor listed in the interceptor.classes 
property and returns a configured  *List>* 
interceptors.
h2. Kafka Consumer Constructor
{code:java}
try {

List> interceptorList = (List) 
config.getConfiguredInstances(
        ConsumerConfig.INTERCEPTOR_CLASSES_CONFIG,
        ConsumerInterceptor.class,
        Collections.singletonMap(ConsumerConfig.CLIENT_ID_CONFIG, clientId));
Kafka Producer Constructor
{code}
h2. Kafka Producer Constructor
{code:java}
try {

List> interceptorList = (List) 
config.getConfiguredInstances(
        ProducerConfig.INTERCEPTOR_CLASSES_CONFIG,
        ProducerInterceptor.class,
        Collections.singletonMap(ProducerConfig.CLIENT_ID_CONFIG, 
clientId));{code}
 

This dual responsibility for both creation and configuration is problematic 
when it involves multiple interceptors where at least one interceptor's 
configure method implementation creates and/or depends on objects which creates 
threads, connections or other resources which requires clean up and the 
subsequent interceptor's configure method raises a runtime exception.  This 
raising of the runtime exception results produces a resource leakage in the 
first interceptor as the interceptor container i.e. 
ConsumerInterceptors/ProducerInterceptors are never created and therefore the 
first interceptor's and really any interceptor's close method are never called.
h2.  Kafka Consumer Constructor
{code:java}
try {

List> interceptorList = (List) 
config.getConfiguredInstances(
        ConsumerConfig.INTERCEPTOR_CLASSES_CONFIG,
        ConsumerInterceptor.class,
        Collections.singletonMap(ConsumerConfig.CLIENT_ID_CONFIG, clientId));
{code}
If the above line results in a runtime exception, the below 
{*}{color:#ffab00}this{color}{*}.{*}{color:#403294}interceptors{color}{*} is 
never created. 

 
{code:java}
this.interceptors = new ConsumerInterceptors<>(interceptorList);{code}
 
h2. Kafka Producer Constructor
{code:java}
try {

List> interceptorList = (List) 
config.getConfiguredInstances(
        ProducerConfig.INTERCEPTOR_CLASSES_CONFIG,
        ProducerInterceptor.class,
        Collections.singletonMap(ProducerConfig.CLIENT_ID_CONFIG, clientId));
{code}
If the above line results in a runtime exception, the below this.interceptors 
is never created. 
{code:java}
if (interceptors != null)
    this.interceptors = interceptors;
else
    this.interceptors = new ProducerInterceptors<>(interceptorList);
 {code}
Although, both Kafka Consumer and Kafka Producer constructors try/catch 
implement  close for resource clean up, 
{code:java}
catch (Throwable t) {
    // call close methods if internal objects are already constructed; this is 
to prevent resource leak. see KAFKA-2121
    // we do not need to call `close` at all when `log` is null, which means no 
internal objects were initialized.
    if (this.log != null) {
        close(0, true);
    }
    // now propagate the exception
    throw new KafkaException("Failed to construct kafka consumer", t);
} {code}
their respective close implementation located in the catch above never calls 
the respective container interceptor close method below as the 
{*}{color:#ffab00}this{color}{*}.{*}{color:#403294}interceptors{color}{*} was 
never created.
{code:java}
private void close(long timeoutMs, boolean swallowException) {
   
Utils.closeQuietly(interceptors, "consumer interceptors", firstException);
 {code}
 

This problem is magnified within a webserver cluster i.e. Confluent's REST 
Proxy server where thousands of requests containing interceptor configuration 
failures can occur in seconds resulting in an inadvertent DDoS attack as 
cluster resources are quickly exhausted, disrupting all service activities.
h2. PROPOSAL

To help ensure the respective container interceptors are able to invoke their 
respective interceptor close methods for proper resource clean up, I propose 
defining a default open method with no implementation and a check exception on 
the respective Consumer/Producer interceptor interfaces.  This open method will 
be responsible for creating threads and/or objects which utilizes threads, 
connections or other resource which requires clea

Re: [ANNOUNCE] New committer: Justine Olshan

[John Roesler]

Congrats, Justine!
-John

On Tue, Jan 3, 2023, at 13:03, Matthias J. Sax wrote:
> Congrats!
>
> On 12/29/22 6:47 PM, ziming deng wrote:
>> Congratulations Justine!
>> —
>> Best,
>> Ziming
>> 
>>> On Dec 30, 2022, at 10:06, Luke Chen  wrote:
>>>
>>> Congratulations, Justine!
>>> Well deserved!
>>>
>>> Luke
>>>
>>> On Fri, Dec 30, 2022 at 9:15 AM Ron Dagostino  wrote:
>>>
 Congratulations, Justine!Well-deserved., and I’m very happy for you.

 Ron

> On Dec 29, 2022, at 6:13 PM, Israel Ekpo  wrote:
>
> Congratulations Justine!
>
>
>> On Thu, Dec 29, 2022 at 5:05 PM Greg Harris
 
>> wrote:
>>
>> Congratulations Justine!
>>
>>> On Thu, Dec 29, 2022 at 1:37 PM Bill Bejeck  wrote:
>>>
>>> Congratulations Justine!
>>>
>>>
>>> -Bill
>>>
 On Thu, Dec 29, 2022 at 4:36 PM Philip Nee 
 wrote:
>>>
 wow congrats!

 On Thu, Dec 29, 2022 at 1:05 PM Chris Egerton <
 fearthecel...@gmail.com
>>>
 wrote:

> Congrats, Justine!
>
> On Thu, Dec 29, 2022, 15:58 David Jacot  wrote:
>
>> Hi all,
>>
>> The PMC of Apache Kafka is pleased to announce a new Kafka
>> committer
>> Justine
>> Olshan.
>>
>> Justine has been contributing to Kafka since June 2019. She
>>> contributed
> 53
>> PRs including the following KIPs.
>>
>> KIP-480: Sticky Partitioner
>> KIP-516: Topic Identifiers & Topic Deletion State Improvements
>> KIP-854: Separate configuration for producer ID expiry
>> KIP-890: Transactions Server-Side Defense (in progress)
>>
>> Congratulations, Justine!
>>
>> Thanks,
>>
>> David (on behalf of the Apache Kafka PMC)
>>
>

>>>
>>

>> 
>>

[jira] [Created] (KAFKA-14567) Kafka Streams crashes after ProducerFencedException

[Matthias J. Sax (Jira)]

Matthias J. Sax created KAFKA-14567:
---

 Summary: Kafka Streams crashes after ProducerFencedException
 Key: KAFKA-14567
 URL: https://issues.apache.org/jira/browse/KAFKA-14567
 Project: Kafka
  Issue Type: Bug
  Components: streams
Reporter: Matthias J. Sax


Running a Kafka Streams application with EOS-v2.

After a thread crashed, we re-spanned a new thread what implies that the 
thread-index number was re-used, resulting in an `transactional.id` reuse, that 
lead to a `ProducerFencedException`.

After the fencing, the fenced thread crashed resulting in a non-recoverable 
error:
{quote}[2022-12-22 13:49:13,423] ERROR [i-0c291188ec2ae17a0-StreamThread-3] 
stream-thread [i-0c291188ec2ae17a0-StreamThread-3] Failed to process stream 
task 1_2 due to the following error: 
(org.apache.kafka.streams.processor.internals.TaskExecutor)
org.apache.kafka.streams.errors.StreamsException: Exception caught in process. 
taskId=1_2, processor=KSTREAM-SOURCE-05, topic=node-name-repartition, 
partition=2, offset=539776276, stacktrace=java.lang.IllegalStateException: 
TransactionalId stream-soak-test-72b6e57c-c2f5-489d-ab9f-fdbb215d2c86-3: 
Invalid transition attempted from state FATAL_ERROR to state ABORTABLE_ERROR
at 
org.apache.kafka.clients.producer.internals.TransactionManager.transitionTo(TransactionManager.java:974)
at 
org.apache.kafka.clients.producer.internals.TransactionManager.transitionToAbortableError(TransactionManager.java:394)
at 
org.apache.kafka.clients.producer.internals.TransactionManager.maybeTransitionToErrorState(TransactionManager.java:620)
at 
org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:1079)
at 
org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:959)
at 
org.apache.kafka.streams.processor.internals.StreamsProducer.send(StreamsProducer.java:257)
at 
org.apache.kafka.streams.processor.internals.RecordCollectorImpl.send(RecordCollectorImpl.java:207)
at 
org.apache.kafka.streams.processor.internals.RecordCollectorImpl.send(RecordCollectorImpl.java:162)
at 
org.apache.kafka.streams.processor.internals.SinkNode.process(SinkNode.java:85)
at 
org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forwardInternal(ProcessorContextImpl.java:290)
at 
org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:269)
at 
org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:228)
at 
org.apache.kafka.streams.kstream.internals.KStreamKTableJoinProcessor.process(KStreamKTableJoinProcessor.java:88)
at 
org.apache.kafka.streams.processor.internals.ProcessorNode.process(ProcessorNode.java:157)
at 
org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forwardInternal(ProcessorContextImpl.java:290)
at 
org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:269)
at 
org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:228)
at 
org.apache.kafka.streams.processor.internals.SourceNode.process(SourceNode.java:84)
at 
org.apache.kafka.streams.processor.internals.StreamTask.lambda$doProcess$1(StreamTask.java:791)
at 
org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl.maybeMeasureLatency(StreamsMetricsImpl.java:867)
at 
org.apache.kafka.streams.processor.internals.StreamTask.doProcess(StreamTask.java:791)
at 
org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:722)
at 
org.apache.kafka.streams.processor.internals.TaskExecutor.processTask(TaskExecutor.java:95)
at 
org.apache.kafka.streams.processor.internals.TaskExecutor.process(TaskExecutor.java:76)
at 
org.apache.kafka.streams.processor.internals.TaskManager.process(TaskManager.java:1645)
at 
org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:788)
at 
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:607)
at 
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:569)
at 
org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:748)
at 
org.apache.kafka.streams.processor.internals.TaskExecutor.processTask(TaskExecutor.java:95)
at 
org.apache.kafka.streams.processor.internals.TaskExecutor.process(TaskExecutor.java:76)
at 
org.apache.kafka.streams.processor.internals.TaskManager.process(TaskManager.java:1645)
at 
org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:788)
at 
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamTh