Re: Kafka 20k topics metadata update taking long time

2024-07-08 Thread Amit Chopra
Following up to check if someone can help. And is this the right approach or should i open a Jira ticket for same. Thanks, Amit On Fri, Jul 5, 2024 at 9:42 AM Amit Chopra wrote: > But 20K topics is also not a realistic assumption to have. > This is our existing scale with AWS kines

Re: Kafka 20k topics metadata update taking long time

2024-07-05 Thread Amit Chopra
ERNAL EMAIL - USE CAUTION when clicking links or attachments > > > > > This is interesting. But 20K topics is also not a realistic assumption to > have. I don't see an alarming difference in the latency results from the > two scenarios. Also even when cached, the metadata ne

Re: Kafka 20k topics metadata update taking long time

2024-07-05 Thread Amit Chopra
haran wrote: > This is interesting. But 20K topics is also not a realistic assumption to > have. I don't see an alarming difference in the latency results from the > two scenarios. Also even when cached, the metadata needs to be refreshed > anyway. > > On Thu, Jul 4, 2024 at 4

Kafka 20k topics metadata update taking long time

2024-07-03 Thread Amit Chopra
Hey, I wanted to raise a performance issue that I encountered while testing on a large scale with Kafka. The issue pertains to low throughput with 20k topics (each topic with 1 partition) compared to good throughput when using 1 topic with 20k partitions. Scenario: I have a setup of kafka with 20