Re: Issue - Kafka server shutting down automatically

[Akash Jain]

Hi Brijendra,

I suspect that the path you are using for log.dir is being
updated/modified/deleted etc.
Can you ensure that this path -> D:\kafka\kafka_logs is not being
cleaned/modified in anyway (manually/automatically/by some application)?
Are you going to use windows server or linux in production? If you going to
use linux do not configure brokers to use /tmp path.
log.dir path should exclusively be for Kafka broker process only.

Let me know if you have any findings.

Thanks,
Akash

On Thu, Sep 5, 2024 at 8:26 PM Brijendra Suman
 wrote:

> Dear Kafka Team,
>
>
>
> We are developing one application in which we are utilising Kafka for some
> use cases.
>
>
>
> We have created a Topic and retention period we have set is 1 hour for a
> topic. It has been observed that when Kafka starts deleting the messages
> from the topics then below error pops up and servers shuts down
> automatically.
>
>
>
> [2024-09-05 10:31:06,964] INFO [ReplicaFetcherManager on broker 1] Removed
> fetcher for partitions Set(candidates_identity-0)
> (kafka.server.ReplicaFetcherManager)
>
> [2024-09-05 10:31:06,965] INFO [ReplicaAlterLogDirsManager on broker 1]
> Removed fetcher for partitions Set(candidates_identity-0)
> (kafka.server.ReplicaAlterLogDirsManager)
>
> [2024-09-05 10:31:06,969] WARN [ReplicaManager broker=1] Broker 1 stopped
> fetcher for partitions candidates_identity-0 and stopped moving logs for
> partitions  because they are in the failed log directory
> D:\kafka\kafka_logs. (kafka.server.ReplicaManager)
>
> [2024-09-05 10:31:06,970] WARN Stopping serving logs in dir
> D:\kafka\kafka_logs (kafka.log.LogManager)
>
> [2024-09-05 10:31:06,972] ERROR Shutdown broker because all log dirs in
> D:\kafka\kafka_logs have failed (kafka.log.LogManager)
>
>
>
>
>
> Please help as we are planning to utilize Kafka in production but with
> this issue we are stuck.
>
>
>
>
>
>
> Thanks & Regards,
>
> Brijendra Kumar Suman
>
> Scientist-B
>
> NRA Division, NIC
>
> Mobile  - 9754661383, IP Phone number - 5399

KRaft combined mode usage in production

[Jouni Lammi]

Hi,

I'd have a question about using the KRaft combined mode
(process.roles='broker,controller')
 in
production. Despite my searches I fail to find very much info of its usage.
Confluent says in its documentation that the feature is in early access,
but I'd like to hear your take on this matter.

Currently I'm running a multi-broker Kafka cluster (multiple ZooKeepers,
multiple Kafka brokers, ZooKeeper server count = Kafka broker server
count), which for my use case has worked well.

- If the combined mode could be used, could I use less servers for the same
performance?
- If the combined mode isn't suitable for production usage, what
combination of controllers and broker amounts do you suggest for similar
performance than with ZooKeeper?
- If the combined mode isn't suitable for production usage, is there any
roadmap on when it could be production ready, or is this even a thing that
would come in the future?

Best regards,
Jouni Lammi

Query regarding Extended Key Usage (EKU) in SSL certificates

[TUSHAR PATIL]

Hello,

I would like to know if there have been any changes related to Extended Key
Usage (EKU) in SSL certificates starting from Apache Kafka version 3.3.2.
Recently, I attempted to upgrade from version 3.3.1 to 3.3.2, but
encountered the following error:
"ERROR [BrokerServer id=1] Fatal error during broker startup. Prepare to
shutdown (kafka.server.BrokerServer)
org.apache.kafka.common.config.ConfigException: Invalid value
javax.net.ssl.SSLHandshakeException: Extended key usage does not permit use
for TLS client authentication for configuration A client SSLEngine created
with the provided settings can't connect to a server SSLEngine created with
those settings."

The current certificate I’m using for SSL has EKU set to serverAuth only
(no clientAuth). While this certificate works without issues in Kafka
version 3.3.1, it triggers above error in version 3.3.2.

Is having EKU configured for both serverAuth and clientAuth mandatory in
certificates for Kafka version 3.3.2 and later? I checked the release notes
but couldn't find any details regarding changes in certificate EKU
requirements. Could you clarify this?

Regards,
Tushar Patil.