Re: [DISCUSS] KIP-1015: Limit number of ssl connections in brokers

[JimmyWang]

Hi Colin,
Hi all,
Thanks for your feedback!

>Do you really think you'll have both a lot of PLAINTEXT and a lot of SSL 
>connections?
Yes, and here is our use-case:
Generally we have four listeners for users:

MGR_CHANNEL: For administrators management (SSL)
REPLICA: Inter broker replication(PLAINTEXT)
CLIENT: Inner VPC connections (SASL_PLAINTEXT/SASL_SSL)
PUBLIC_CLIENT : Public network connection (SASL_PLAINTEXT/SASL_SSL)

Users will use the CLIENT listener for connections under same VPC subnet, while 
using the PUBLIC_CLIENT port for public  or cross-VPC access. 
When user configure SASL_SSL protocol for both the two listeners, there will be 
a lot of PLAINTEXT connections and SSL connections at the same time. 
 
>limit the SSL connections precisely
To elaborate further, by setting connection limits at the listener level, it is 
possible to limit SSL connections. However, determining the appropriate value 
can be challenging, especially when considering multiple SSL listeners in the 
future. For example, if we have four listeners, the maximum value for the 
connection count of each listener may be set to 2500. 
This implementation may result in that the actual limit imposed much lower than 
what we desired to be when the connections of some listeners are relatively low.



Original

From:"Colin McCabe"< cmcc...@apache.org >;

Date:2024/1/9 4:37

To:"dev"< dev@kafka.apache.org >;

Subject:Re: [DISCUSS] KIP-1015: Limit number of ssl connections in brokers


Hi zw,

As you yourself wrote in the rejected alternatives section, the existing 
listener-specific connection limit already lets administrators limit the number 
of SSL connections (assuming that one listener is SSL and another is not).

I don't understand the objection to just using that capability. You mention 
that "the protocol of the listener may change dynamically, and the limit of the 
listener also needs to be modified." But the connection limit is also dynamic, 
so that doesn't seem to be a problem. I didn't understand the objection that a 
per-listener limit doesn't allow you to "limit the ssl connections precisely" 
-- can you explain more?

Just as a general comment, I think the biggest use-case for mixed clusters that 
support both PLAINTEXT and SSL is when you have replication using PLAINTEXT and 
external connections using SSL. In that case, it's hardly worth having a 
separate limit for SSL, since the number of plaintext connections is bounded, 
and low. But perhaps your use-case is different. Do you really think you'll 
have both a lot of PLAINTEXT and a lot of SSL connections?

cheers,
Colin


On Sat, Jan 6, 2024, at 23:17, zw wrote:
> Hi all,
> I'd like to begin discussion on KIP-1015 which proposes to add new 
> configuration max.ssl.connections to limit number of ssl connections in 
> brokers.
> KIP link：
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1015%3A+Limit+number+of+ssl+connections+in+brokers
>
>
> Best,
> Jimmy Wang
a

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.7 #61

[Apache Jenkins Server]

See 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2570

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-16123) KStreamKStreamJoinProcessor forwards null records for left/outer joins unconditionally of the join window.

[Florin Akermann (Jira)]

Florin Akermann created KAFKA-16123:
---

 Summary: KStreamKStreamJoinProcessor forwards null records for 
left/outer joins unconditionally of the join window.
 Key: KAFKA-16123
 URL: https://issues.apache.org/jira/browse/KAFKA-16123
 Project: Kafka
  Issue Type: Bug
Reporter: Florin Akermann
Assignee: Florin Akermann


As part of KIP-962 the non-null key requirements have been relaxed for left and 
outer joins.
However, the implementation forwards null records for left/outer joins 
unconditionally of
the join window.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] 3.7.0 RC2

[Jakub Scholz]

Hi,

I was trying the RC2 and run into the following issue ... when I run
3.7.0-RC2 KRaft cluster with metadata version set to 3.6-IV2 metadata
version, I seem to be getting repeated errors like this in the controller
logs:

2024-01-13 16:58:01,197 INFO [QuorumController id=0] assignReplicasToDirs:
event failed with UnsupportedVersionException in 15 microseconds.
(org.apache.kafka.controller.QuorumController)
[quorum-controller-0-event-handler]
2024-01-13 16:58:01,197 ERROR [ControllerApis nodeId=0] Unexpected error
handling request RequestHeader(apiKey=ASSIGN_REPLICAS_TO_DIRS,
apiVersion=0, clientId=1000, correlationId=14, headerVersion=2) --
AssignReplicasToDirsRequestData(brokerId=1000, brokerEpoch=5,
directories=[DirectoryData(id=w_uxN7pwQ6eXSMrOKceYIQ,
topics=[TopicData(topicId=bvAKLSwmR7iJoKv2yZgygQ,
partitions=[PartitionData(partitionIndex=2),
PartitionData(partitionIndex=1)]),
TopicData(topicId=uNe7f5VrQgO0zST6yH1jDQ,
partitions=[PartitionData(partitionIndex=0)])])]) with context
RequestContext(header=RequestHeader(apiKey=ASSIGN_REPLICAS_TO_DIRS,
apiVersion=0, clientId=1000, correlationId=14, headerVersion=2),
connectionId='172.16.14.219:9090-172.16.14.217:53590-7', clientAddress=/
172.16.14.217, principal=User:CN=my-cluster-kafka,O=io.strimzi,
listenerName=ListenerName(CONTROLPLANE-9090), securityProtocol=SSL,
clientInformation=ClientInformation(softwareName=apache-kafka-java,
softwareVersion=3.7.0), fromPrivilegedListener=false,
principalSerde=Optional[org.apache.kafka.common.security.authenticator.DefaultKafkaPrincipalBuilder@71004ad2])
(kafka.server.ControllerApis) [quorum-controller-0-event-handler]
java.util.concurrent.CompletionException:
org.apache.kafka.common.errors.UnsupportedVersionException: Directory
assignment is not supported yet.

 at
java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332)
 at
java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347)
 at
java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:636)
 at
java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
 at
java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
 at
org.apache.kafka.controller.QuorumController$ControllerWriteEvent.complete(QuorumController.java:880)
 at
org.apache.kafka.controller.QuorumController$ControllerWriteEvent.handleException(QuorumController.java:871)
 at
org.apache.kafka.queue.KafkaEventQueue$EventContext.completeWithException(KafkaEventQueue.java:148)
 at
org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:137)
 at
org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
 at
org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
 at java.base/java.lang.Thread.run(Thread.java:840)

Caused by: org.apache.kafka.common.errors.UnsupportedVersionException:
Directory assignment is not supported yet.

Is that expected? I guess with the metadata version set to 3.6-IV2, it
makes sense that the request is not supported. But shouldn't then the
request not be sent at all by the brokers? (I did not opened a JIRA for it,
but I can open one if you agree this is not expected)

Thanks & Regards
Jakub

On Sat, Jan 13, 2024 at 8:03 AM Luke Chen  wrote:

> Hi Stanislav,
>
> I commented in the "Apache Kafka 3.7.0 Release" thread, but maybe you
> missed it.
> cross-posting here:
>
> There is a bug KAFKA-16101
>  reporting that "Kafka
> cluster will be unavailable during KRaft migration rollback".
> The impact for this issue is that if brokers try to rollback to ZK mode
> during KRaft migration process, there will be a period of time the cluster
> is unavailable.
> Since ZK migrating to KRaft feature is a production ready feature, I think
> this should be addressed soon.
> Do you think this is a blocker for v3.7.0?
>
> Thanks.
> Luke
>
> On Sat, Jan 13, 2024 at 8:36 AM Chris Egerton 
> wrote:
>
> > Thanks, Kirk!
> >
> > @Stanislav--do you believe that this warrants a new RC?
> >
> > On Fri, Jan 12, 2024, 19:08 Kirk True  wrote:
> >
> > > Hi Chris/Stanislav,
> > >
> > > I'm working on the 'Unable to find FetchSessionHandler' log problem
> > > (KAFKA-16029) and have put out a draft PR (
> > > https://github.com/apache/kafka/pull/15186). I will use the quickstart
> > > approach as a second means to reproduce/verify while I wait for the
> PR's
> > > Jenkins job to finish.
> > >
> > > Thanks,
> > > Kirk
> > >
> > > On Fri, Jan 12, 2024, at 11:31 AM, Chris Egerton wrote:
> > > > Hi Stanislav,
> > > >
> > > >
> > > > Thanks for running this release!
> > > >
> > > > To verify, I:
> > > > - Built from source using Java 11 with both:
> > > > - - the 3.7.0-rc2 tag on GitHub
> > > > - - the kafka-3.7.0-src.tgz artifact from
> > > > https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
> > > > - Ch

[jira] [Resolved] (KAFKA-15958) Add tests to validate telemetry requests with different version

[Apoorv Mittal (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Apoorv Mittal resolved KAFKA-15958.
---
Resolution: Won't Do

> Add tests to validate telemetry requests with different version
> ---
>
> Key: KAFKA-15958
> URL: https://issues.apache.org/jira/browse/KAFKA-15958
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Apoorv Mittal
>Assignee: Apoorv Mittal
>Priority: Major
>
> Details: https://github.com/apache/kafka/pull/14724#discussion_r1412530561



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-15952) Create public doc for telemetry state transition

[Apoorv Mittal (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Apoorv Mittal resolved KAFKA-15952.
---
Resolution: Won't Do

> Create public doc for telemetry state transition
> 
>
> Key: KAFKA-15952
> URL: https://issues.apache.org/jira/browse/KAFKA-15952
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Apoorv Mittal
>Assignee: Apoorv Mittal
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Permission to contribute

[Rowland Smith]

I would like permission to contribute to Kafka. I have created Wiki and
Jira ID's 'rowls'.

I will be working with a KIP for XA support.

-- 
*Rowland E. Smith*
P: (862) 260-4163
M: (201) 396-3842