Reg: Dual Kafka Clients - Issues with jar version conflicts

Hello, As per our team requirements, we are supposed to use and communicate between two different Kafka Clients, provided and managed by two separate teams). Mentioned below are the client versions Version 1 - kafka-clients-2.4.0.jar Version 2 - kafka-clients-0.10.0.1.jar Please suggest if possi

Re: High write operations rate on disk

Our typical IOPS stays at ~10K write ops/min, but it goes to 37K write ops/min (which is where AWS throttles). The spike in write ops isn't accompanied by any spike in write throughput or produce requests (except for the first few minutes of catch up). The write ops spike stays up (persistently for

Re: High write operations rate on disk

ST1 EBS fit only for sequential rights and reads. Once you have many partitions on EBS it will be mostly random. Interesting to monitor random vs sequential... We tested kafka on ST1 with 1xx partitions on each EBS and it was constantly lagging. BR On Tue, Apr 7, 2020 at 10:06 AM Soumyajit Sahu

Re: High write operations rate on disk

We have used st1 volumes and we never saw any issue. Yes, we are using m-series. Even t-series worked for us :D During those spikes, do you observe any background operations going on? Check server logs, controller logs. On Tue, Apr 7, 2020 at 12:49 PM Seva Feldman wrote: > ST1 EBS fit only for

Re: High write operations rate on disk

@Suman, thanks for confirming. I will dig more then. The instances are dedicated to running Kafka, and so is the mounted volume. @Seva, thanks for the insight. I guess if nothing works, then we will move from st1 to gp2 volumes. On Tue, Apr 7, 2020 at 12:28 AM Suman B N wrote: > We have used st

Re: High write operations rate on disk

We are using mainly ephemeral instances like i3en as our pattern is more fit for it. On Tue, Apr 7, 2020 at 10:40 AM Soumyajit Sahu wrote: > @Suman, thanks for confirming. I will dig more then. The instances are > dedicated to running Kafka, and so is the mounted volume. > > @Seva, thanks for th

How to set custom properties and message expiration for Kafka topic message.

Hello Friends, Hope you are all doing good! I am bit new to Kafka messaging system. We are using Kafka as messaging broker. Could someone let me know how to set custom properties and message expiration while sending message to Kafka topic and how to read customer properties in consumer end? Th

EOL

Hello, I am trying to find the End of Life (EOL) of the current version of Kafka. I found links that showed tables and probably answered the question, but I found them to be a bit confusing. Can someone just give me a version number and the EOL for that version? If there is a set time period

RE: Brokers session terminated and removed

Hi All, Can somebody help me on this issue? Thanks, Pradeep V.B. -Original Message- From: Pradeep Badiger Sent: Monday, March 30, 2020 12:58 PM To: users@kafka.apache.org Subject: Brokers session terminated and removed Hi, I am running into an issue where the Kafka brokers (0.10.2.1)

Adding new connector Jars dynamically in Kafka Connect

Hi, I have a use case where new connectors will keep on adding to existing running Kafka Connect cluster. Is there any way in Kafka Connect to submit the new connector jar dynamically without restarting the Connect process? Thanks, Nitin

Re: EOL

Hello Tim, The following link should be the source of truth for EOL policy: https://cwiki.apache.org/confluence/display/KAFKA/Time+Based+Release+Plan#TimeBasedReleasePlan-WhatIsOurEOLPolicy? Guozhang On Tue, Apr 7, 2020 at 5:54 AM Gaudet, Timothy wrote: > Hello, > > I am trying to find the En

RE: EOL

I saw that and to me it sounded like the EOL was one year. Timothy Gaudet Linux Engineer General Dynamics Information Technology _Intelligence Solututions Div. (ISD) US BICES-X and Intelligence Programs Tampa, FL. 813-826-0110 001282659 BICES timothy.gau...@gdit.com timothy.d.gaudet@socom.mil

Re: EOL

After reviewing the documentation, it appears the document focuses more on release schedule and bug fixes for the 3 most recent major release versions. It is not really clear when the support for specific versions end and when they become deprecated. We need an official deprecation and EOL policy

[VOTE] 2.5.0 RC3

Hello Kafka users, developers and client-developers, This is the forth candidate for release of Apache Kafka 2.5.0. * TLS 1.3 support (1.2 is now the default) * Co-groups for Kafka Streams * Incremental rebalance for Kafka Consumer * New metrics for better operational insight * Upgrade Zookeeper

Oauthbearer - Azure

Hi, I'm looking to implement authentication using Oauthbearer mechanism in my Kafka cluster. My Oauth server will be Azure AD. If anyone have implemented similar setup on Azure and can share some tips on how to proceed? Thanks AA