Following up to check if someone can help. And is this the right approach
or should i open a Jira ticket for same.
Thanks,
Amit
On Fri, Jul 5, 2024 at 9:42 AM Amit Chopra wrote:
> But 20K topics is also not a realistic assumption to have.
> This is our existing scale with AWS kinesis 20k shards
ge001.png@01DACEDE.1DAE3A70]
>
>
> From: Brebner, Paul
> Date: Thursday, 4 July 2024 at 3:44 PM
> To: users@kafka.apache.org
> Subject: Re: Kafka 20k topics metadata update taking long time
> EXTERNAL EMAIL - USE CAUTION when clicking links or attachments
>
>
>
>
But 20K topics is also not a realistic assumption to have.
This is our existing scale with AWS kinesis 20k shards. And we are moving
over to kafka and thus testing with equivalent scale. We are looking at an
ingestion rate of 2.5 GBps.
I don't see an alarming difference in the latency results from
OK so repeating with Java Kafka producer there is no problem – it’s specific to
the Kafka CLI Producer! Paul
From: Brebner, Paul
Date: Friday, 5 July 2024 at 1:21 PM
To: users@kafka.apache.org
Subject: Re: Kafka 20k topics metadata update taking long time
EXTERNAL EMAIL - USE CAUTION when
: Sabarish Sasidharan
Date: Thursday, 4 July 2024 at 3:28 PM
To: users@kafka.apache.org
Subject: Re: Kafka 20k topics metadata update taking long time
[You don't often get email from sabarish@gmail.com. Learn why this is
important at https://aka.ms/LearnAboutSenderIdentification ]
EXTERNAL
@kafka.apache.org
Subject: Re: Kafka 20k topics metadata update taking long time
[You don't often get email from sabarish@gmail.com. Learn why this is
important at https://aka.ms/LearnAboutSenderIdentification ]
EXTERNAL EMAIL - USE CAUTION when clicking links or attachments
This is interesting
This is interesting. But 20K topics is also not a realistic assumption to
have. I don't see an alarming difference in the latency results from the
two scenarios. Also even when cached, the metadata needs to be refreshed
anyway.
On Thu, Jul 4, 2024 at 4:37 AM Amit Chopra
wrote:
> Hey,
>
> I wante
Hey,
I wanted to raise a performance issue that I encountered while testing on a
large scale with Kafka. The issue pertains to low throughput with 20k
topics (each topic with 1 partition) compared to good throughput when using
1 topic with 20k partitions.
Scenario: I have a setup of kafka with 20
