Hi – interesting, I had maybe similar problems today when “testing” the limits
of a Kafka cluster for max partitions – I could create a topic with lots of
partitions (ok so more than sensible, taking into account RF=3 over 1M
partitions) – but trying to send a message failed with a meta-data tim
This is interesting. But 20K topics is also not a realistic assumption to
have. I don't see an alarming difference in the latency results from the
two scenarios. Also even when cached, the metadata needs to be refreshed
anyway.
On Thu, Jul 4, 2024 at 4:37 AM Amit Chopra
wrote:
> Hey,
>
> I wante
Hey Burton,
Thanks for your question and bug report.
The exception you included does not indicate that your connectors are
overloaded. The primary way of diagnosing an overloaded connector is the
consumer lag metric, and if you're seeing acceptable lag, that should
indicate that your connectors a
Hey,
I wanted to raise a performance issue that I encountered while testing on a
large scale with Kafka. The issue pertains to low throughput with 20k
topics (each topic with 1 partition) compared to good throughput when using
1 topic with 20k partitions.
Scenario: I have a setup of kafka with 20
Hi,
I have 100+ sink connectors running with 100+ topics each with roughly 3
partitions per topic. There are running on K8s on 10 pods with 6 cpus and
32Gig mem. The connector in question is Snowflake's sink connector v2.2.0.
This worked in the mini batch mode SNOWPIPE, but once i switched over to
