[jira] [Resolved] (KAFKA-14819) Publish a single kafka (aka core) Maven artifact in Apache Kafka 4.0 (KIP-897)

[Ismael Juma (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-14819.
-
Fix Version/s: (was: 4.0.0)
   Resolution: Won't Fix

As described in the KIP, we're taking a different approach.

> Publish a single kafka (aka core) Maven artifact in Apache Kafka 4.0 (KIP-897)
> --
>
> Key: KAFKA-14819
> URL: https://issues.apache.org/jira/browse/KAFKA-14819
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Ismael Juma
>Assignee: Ismael Juma
>Priority: Blocker
>
> Please see KIP for details:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-897%3A+Publish+a+single+kafka+%28aka+core%29+Maven+artifact+in+Apache+Kafka+4.0



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15855) RFC 9266: Channel Bindings for TLS 1.3 support | SCRAM-SHA-*-PLUS variants

[Neustradamus (Jira)]

Neustradamus created KAFKA-15855:


 Summary: RFC 9266: Channel Bindings for TLS 1.3 support | 
SCRAM-SHA-*-PLUS variants
 Key: KAFKA-15855
 URL: https://issues.apache.org/jira/browse/KAFKA-15855
 Project: Kafka
  Issue Type: Bug
  Components: connect, core, security
Reporter: Neustradamus


Dear Apache, and Kafka teams,

Can you add the support of RFC 9266: Channel Bindings for TLS 1.3?
- [https://datatracker.ietf.org/doc/html/rfc9266]

Little details, to know easily:
- tls-unique for TLS =< 1.2
- tls-server-end-point
- tls-exporter for TLS = 1.3

It is needed for SCRAM-SHA-*-PLUS variants.
Note: Some SCRAM-SHA are already supported.

I think that you have seen the jabber.ru MITM and Channel Binding is the 
solution:
- [https://notes.valdikss.org.ru/jabber.ru-mitm/]
- [https://snikket.org/blog/on-the-jabber-ru-mitm/]
- [https://www.devever.net/~hl/xmpp-incident]
- [https://blog.jmp.chat/b/certwatch]

IETF links:

SCRAM-SHA-1(-PLUS):
- RFC5802: Salted Challenge Response Authentication Mechanism (SCRAM) SASL and 
GSS-API Mechanisms: [https://tools.ietf.org/html/rfc5802] // July 2010
- RFC6120: Extensible Messaging and Presence Protocol (XMPP): Core: 
[https://tools.ietf.org/html/rfc6120] // March 2011

SCRAM-SHA-256(-PLUS):
- RFC7677: SCRAM-SHA-256 and SCRAM-SHA-256-PLUS Simple Authentication and 
Security Layer (SASL) Mechanisms: [https://tools.ietf.org/html/rfc7677] // 
2015-11-02
- RFC8600: Using Extensible Messaging and Presence Protocol (XMPP) for Security 
Information Exchange: [https://tools.ietf.org/html/rfc8600] // 2019-06-21: 
[https://mailarchive.ietf.org/arch/msg/ietf-announce/suJMmeMhuAOmGn_PJYgX5Vm8lNA]

SCRAM-SHA-512(-PLUS):
- [https://tools.ietf.org/html/draft-melnikov-scram-sha-512]

SCRAM-SHA3-512(-PLUS):
- [https://tools.ietf.org/html/draft-melnikov-scram-sha3-512]

SCRAM BIS: Salted Challenge Response Authentication Mechanism (SCRAM) SASL and 
GSS-API Mechanisms:
- [https://tools.ietf.org/html/draft-melnikov-scram-bis]

-PLUS variants:
- RFC5056: On the Use of Channel Bindings to Secure Channels: 
[https://tools.ietf.org/html/rfc5056] // November 2007
- RFC5929: Channel Bindings for TLS: [https://tools.ietf.org/html/rfc5929] // 
July 2010
- Channel-Binding Types: 
[https://www.iana.org/assignments/channel-binding-types/channel-binding-types.xhtml]
- RFC9266: Channel Bindings for TLS 1.3: [https://tools.ietf.org/html/rfc9266] 
// July 2022

IMAP:
- RFC9051: Internet Message Access Protocol (IMAP) - Version 4rev2: 
[https://tools.ietf.org/html/rfc9051] // August 2021

LDAP:
- RFC5803: Lightweight Directory Access Protocol (LDAP) Schema for Storing 
Salted: Challenge Response Authentication Mechanism (SCRAM) Secrets: 
[https://tools.ietf.org/html/rfc5803] // July 2010

HTTP:
- RFC7804: Salted Challenge Response HTTP Authentication Mechanism: 
[https://tools.ietf.org/html/rfc7804] // March 2016

JMAP:
- RFC8621: The JSON Meta Application Protocol (JMAP) for Mail: 
[https://tools.ietf.org/html/rfc8621] // August 2019

2FA:
- Extensions to Salted Challenge Response (SCRAM) for 2 factor authentication: 
[https://tools.ietf.org/html/draft-ietf-kitten-scram-2fa]

Thanks in advance.

Linked to:
- [https://github.com/scram-sasl/info/issues/1]

Note: This ticket can be for other Apache projects too.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #2394

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-954: expand default DSL store configuration to custom types

[Almog Gavra]

Hello Guozhang,

Thanks for the feedback! For 1 there are tests verifying this and I did so
manually as well, it does not reveal anything about the store types -- just
the names, so I think we're good there. I've put an example at the bottom
of this reply for people following the conversation.

I'm not sure I understand your question about 2. What's the integration
point with the actual store for this external component? What does that
have to do with this PR/how does it differ from what's available today
(with the default.dsl.store configuration)? In either scenario, getting the
actual instantiated store supplier must be done only after the topology is
built and rewritten (it can be passed in either via
Materialized/StreamJoined in the DSL code, via TopologyConfig overrides or
in the global StreamsConfig passed in to KafkaStreams). Today, AFAIK, this
isn't possible (you can't get from the built topology the instantiated
store supplier).

Thanks,
Almog



Topologies:
   Sub-topology: 0
Source: KSTREAM-SOURCE-00 (topics: [test_topic])
  --> KSTREAM-TRANSFORMVALUES-01
Processor: KSTREAM-TRANSFORMVALUES-01 (stores: [])
  --> Aggregate-Prepare
  <-- KSTREAM-SOURCE-00
Processor: Aggregate-Prepare (stores: [])
  --> KSTREAM-AGGREGATE-03
  <-- KSTREAM-TRANSFORMVALUES-01
Processor: KSTREAM-AGGREGATE-03 (stores:
[Aggregate-Aggregate-Materialize])
  --> Aggregate-Aggregate-ToOutputSchema
  <-- Aggregate-Prepare
Processor: Aggregate-Aggregate-ToOutputSchema (stores: [])
  --> Aggregate-Project
  <-- KSTREAM-AGGREGATE-03
Processor: Aggregate-Project (stores: [])
  --> KTABLE-TOSTREAM-06
  <-- Aggregate-Aggregate-ToOutputSchema
Processor: KTABLE-TOSTREAM-06 (stores: [])
  --> KSTREAM-SINK-07
  <-- Aggregate-Project
Sink: KSTREAM-SINK-07 (topic: S2)
  <-- KTABLE-TOSTREAM-06

On Sat, Nov 18, 2023 at 6:05 PM Guozhang Wang 
wrote:

> Hello Almog,
>
> I left a comment in the PR before I got to read the newest updates
> from this thread. My 2c:
>
> 1. I liked the idea of delaying the instantiation of StoreBuiler from
> suppliers after the Topology is created. It has been a bit annoying
> for many other features we were trying back then. The only thing is,
> we need to check when we call Topology.describe() which gets a
> TopologyDescription, does that reveal anything about the source of
> truth store impl types already; if it does not, then we are good to
> go.
>
> 2. I originally thought (and commented in the PR) that maybe we can
> just add this new func "resolveDslStoreSuppliers" into StreamsConfig
> directly and mark it as EVOLVING, because I was not clear that we are
> trying to do 1) above. Now I'm leaning more towards what you proposed.
> But I still have a question in mind: even after we've done
> https://github.com/apache/kafka/pull/14548 later, don't we still need
> some interface that user's can call to get the actual instantiated
> store supplier for cases where some external custom logic, like an
> external controller / scheduler which is developed by a different
> group of people rather than the Streams app developers themselves,
> that can only turn on certain features after learning the actual store
> impl suppliers used?
>
> Guozhang
>
> On Sat, Nov 18, 2023 at 2:46 PM Almog Gavra  wrote:
> >
> > Hello Everyone - one more minor change to the KIP that came up during
> > implementation (reflected in the KIP itself). I will be adding the method
> > below to TopologyConfig. This allows us to determine whether or not the
> > DslStoreSuppliers was explicitly passed in via either
> > DSL_STORE_SUPPLIERS_CLASS_CONFIG or DEFAULT_DSL_STORE_CONFIG (if it was
> not
> > explicitly passed in, we will use the one that is configured in
> > StreamsConfig, or the default value of RocksDBDslStoreSuppliers).
> >
> > See the discussion on the PR (
> > https://github.com/apache/kafka/pull/14648#discussion_r1394939779) for
> more
> > context. Ideally this would be an internal utility method but there's no
> > clean way to get that done now, so the goal was to minimize the surface
> > area of what's being exposed (as opposed to exposing a generic method
> like
> > isOverridden(String config).
> >
> > /**
> >  * @return the DslStoreSuppliers if the value was explicitly configured
> > (either by
> >  * {@link StreamsConfig#DEFAULT_DSL_STORE} or {@link
> > StreamsConfig#DSL_STORE_SUPPLIERS_CLASS_CONFIG})
> >  */
> > public Optional resolveDslStoreSuppliers();
> >
> > On Fri, Nov 3, 2023 at 10:48 AM Matthias J. Sax 
> wrote:
> >
> > > Thanks. Will take a look into the PR.
> > >
> > > I don't have any objection to the goal; contrary! It's very annoying
> > > what we have right now, and if we can improve it, I am totally in favor
> > > of it.
> > >
> > >
> > > -Matthias
> > >
> > > On 11/3/23 8:47 AM, Almog Gavra wrote:
> > > > G

[jira] [Created] (KAFKA-15856) Add integration tests for JoinGroup API and SyncGroup API

[Dongnuo Lyu (Jira)]

Dongnuo Lyu created KAFKA-15856:
---

 Summary: Add integration tests for JoinGroup API and SyncGroup API
 Key: KAFKA-15856
 URL: https://issues.apache.org/jira/browse/KAFKA-15856
 Project: Kafka
  Issue Type: Sub-task
Reporter: Dongnuo Lyu






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2395

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-974 Docker Image for GraalVM based Native Kafka Broker

[Krishna Agarwal]

Hi,
Thanks for the insightful feedback on this KIP.
As there are no ongoing discussions, I'm considering moving into the voting
process.
Your continued input is greatly appreciated!

Regards,
Krishna

On Fri, Sep 8, 2023 at 12:47 PM Krishna Agarwal <
krishna0608agar...@gmail.com> wrote:

> Hi,
> I want to submit a KIP to deliver an experimental Apache Kafka docker
> image.
> The proposed docker image can launch brokers with sub-second startup time
> and minimal memory footprint by leveraging a GraalVM based native Kafka
> binary.
>
> KIP-974: Docker Image for GraalVM based Native Kafka Broker
> 
>
> Regards,
> Krishna
>

[jira] [Resolved] (KAFKA-15854) Move Java classes from kafka.server to the server module

[Ismael Juma (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-15854.
-
Fix Version/s: 3.7.0
   Resolution: Fixed

> Move Java classes from kafka.server to the server module
> 
>
> Key: KAFKA-15854
> URL: https://issues.apache.org/jira/browse/KAFKA-15854
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Ismael Juma
>Assignee: Ismael Juma
>Priority: Major
> Fix For: 3.7.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[VOTE] KIP-974: Docker Image for GraalVM based Native Kafka Broker

[Krishna Agarwal]

Hi,
I'd like to call a vote on KIP-974 which aims to publish a docker image for
GraalVM based Native Kafka Broker.

KIP -
https://cwiki.apache.org/confluence/display/KAFKA/KIP-974%3A+Docker+Image+for+GraalVM+based+Native+Kafka+Broker

Discussion thread -
https://lists.apache.org/thread/98wnx4w92fqj5wymkqlqyjsvzxz277hk

Regards,
Krishna

Re: [VOTE] KIP-997: Partition-Level Throughput Metrics

[Qichao Chu]

@Matthias: yeah it should be 977, sorry for the confusion.
Btw, do you want to cast another binding vote for it?

Best,
Qichao Chu


On Fri, Nov 17, 2023 at 12:45 AM Matthias J. Sax  wrote:

> This is KIP-977, right? Not as the subject says.
>
> Guess we won't be able to fix this now. Hope it does not cause confusion
> down the line...
>
>
> -Matthias
>
> On 11/16/23 4:43 AM, Kamal Chandraprakash wrote:
> > +1 (non-binding). Thanks for the KIP!
> >
> > On Thu, Nov 16, 2023 at 9:00 AM Satish Duggana  >
> > wrote:
> >
> >> Thanks Qichao for the KIP.
> >>
> >> +1 (binding)
> >>
> >> ~Satish.
> >>
> >> On Thu, 16 Nov 2023 at 02:20, Jorge Esteban Quilcate Otoya
> >>  wrote:
> >>>
> >>> Qichao, thanks again for leading this proposal!
> >>>
> >>> +1 (non-binding)
> >>>
> >>> Cheers,
> >>> Jorge.
> >>>
> >>> On Wed, 15 Nov 2023 at 19:17, Divij Vaidya 
> >> wrote:
> >>>
>  +1 (binding)
> 
>  I was involved in the discussion thread for this KIP and support it in
> >> its
>  current form.
> 
>  --
>  Divij Vaidya
> 
> 
> 
>  On Wed, Nov 15, 2023 at 10:55 AM Qichao Chu 
>  wrote:
> 
> > Hi all,
> >
> > I'd like to call a vote for KIP-977: Partition-Level Throughput
> >> Metrics.
> >
> > Please take a look here:
> >
> >
> 
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-977%3A+Partition-Level+Throughput+Metrics
> >
> > Best,
> > Qichao Chu
> >
> 
> >>
> >
>

How Kafka handle partition leader change?

[De Gao]

Hi all I have a interesting question here.

Let's say we have 2 broker B1 B2, controller C and producer P1, P2...Pn. 
Currently B1 holds the partition leader and Px is constantly producing messages 
to B1. We want to move the partition leadership to B2. How does the leadership 
change synced between B1, B2, C, and Px that it is guaranteed that all the 
parties acknowledged the leadership change in the right order? Was there a 
break of produce flow in between? Any chance of  message lost?

Thanks

De Gao