Hi, I have a few blogs on Kafka partitions performance for background if you
are interested:
1.
https://www.instaclustr.com/blog/the-power-of-kafka-partitions-how-to-get-the-most-out-of-your-kafka-cluster/
The Power of Apache Kafka® Partitions: How to Get the Most out of Your Kafka
Cluster<
bW-E0XiPhBxvmC0s-I]<https://www.linkedin.com/pulse/complete-guide-apache-kafka-developers-everything-i-know-paul-brebner/>
A Complete Guide to Apache Kafka for Developers (or, everything I know about
Kafka in one
place)<https://www.linkedin.com/pulse/complete-guide-apache-kafka-developers-eve
I have a recent 3 part blog series on Kraft (expanded version of ApacheCon 2022
talk):
https://www.instaclustr.com/blog/apache-kafka-kraft-abandons-the-zookeeper-part-1-partitions-and-data-performance/
https://www.instaclustr.com/blog/apache-kafka-kraft-abandons-the-zookeeper-part-2-partitions-an
nd in Kafka 3.5.0.
Cheers,
David
On Mon, Mar 27, 2023 at 6:32 PM Brebner, Paul
wrote:
> I have a recent 3 part blog series on Kraft (expanded version of ApacheCon
> 2022 talk):
>
>
>
>
> https://www.instaclustr.com/blog/apache-kafka-kraft-abandons-the-zookeeper-part-1-parti
Hi – as each topic can have 1 or more consumers (depending on the number of
partitions), it’s certainly possible to have a 1-1 relationship between topics
and consumers. The downside of this is typically that if the single consumer
fails (and you don’t have some automatic approach to restart it
Hi Alexandre, looks interesting. Would you consider submitting something
(related to performance) to the Community over Code Performance Engineering
track please?
https://www.linkedin.com/pulse/call-papers-2nd-performance-engineering-track-over-code-brebner/
Thanks, Paul
From: Alexandre Strapa
Hi Edvard, interesting problem – I’ve had similar problems with high fan out
use cases, but only for demo applications where I’m more interested in scale
than order – e.g. have a look at this list of blogs, examples include Anomalia
Machina for Kafka+Cassandra, and Kongo, Kafka+IoT.
https://www
Oh, the Kafka parallel consumer may help potentially?
https://www.instaclustr.com/blog/improving-apache-kafka-performance-and-scalability-with-the-parallel-consumer-part-2/
Paul
From: Edvard Fagerholm
Date: Tuesday, 30 May 2023 at 6:55 am
To: users@kafka.apache.org
Subject: Patterns for gener
more ideas and links
including the CPF submission page:
https://www.linkedin.com/pulse/call-papers-2nd-performance-engineering-track-over-code-brebner/
- Paul Brebner and Roger Abelenda
Yes I found out just in time to mention this during my Community over Code talk
today, paul
From: Edoardo Comar
Date: Monday, 3 June 2024 at 2:50 PM
To: users@kafka.apache.org
Subject: Franz Kafka 100th
[You don't often get email from eco...@uk.ibm.com. Learn why this is important
at https://a
Hi – interesting, I had maybe similar problems today when “testing” the limits
of a Kafka cluster for max partitions – I could create a topic with lots of
partitions (ok so more than sensible, taking into account RF=3 over 1M
partitions) – but trying to send a message failed with a meta-data tim
dedicated Kraft
controllers.
Curious if there is a time out setting somewhere for the client meta-data
request? Paul
[cid:image001.png@01DACEDE.1DAE3A70]
From: Brebner, Paul
Date: Thursday, 4 July 2024 at 3:44 PM
To: users@kafka.apache.org
Subject: Re: Kafka 20k topics metadata update taking
OK so repeating with Java Kafka producer there is no problem – it’s specific to
the Kafka CLI Producer! Paul
From: Brebner, Paul
Date: Friday, 5 July 2024 at 1:21 PM
To: users@kafka.apache.org
Subject: Re: Kafka 20k topics metadata update taking long time
EXTERNAL EMAIL - USE CAUTION when
Hi – just curious if anyone can suggest why the following occurs:
1 – try to create a topic with 10,000 partitions with Kafka CLI
(kafka-topics.sh)
Fails with ERROR org.apache.kafka.common.errors.PolicyViolationException:
Unable to perform excessively large batch operation.
2- create a topic wi
duced by
KIP-599, it may be triggered for non-existing topics but not for the
existing resources.
Hope this help you track it down.
OSB
On Thu, Jul 11, 2024, 08:04 Brebner, Paul
wrote:
> Hi – just curious if anyone can suggest why the following occurs:
>
> 1 – try to create a top
Hi Om, I asked some of our techops people about this, and their general advice
is that increasing the log level from the default (INFO?) is likely to increase
the I/O (the amount depending on a variety of factors including the cluster
traffic etc) – and my take on this is that assuming there is
And the flip side, if the cluster is heavily loaded then increasing log levels
is likely to have a detectable impact on performance! Paul
From: Brebner, Paul
Date: Saturday, 2 November 2024 at 9:50 am
To: users@kafka.apache.org , om22sh...@gmail.com
Subject: Re: Q: Does Kafka log level affect
We recommend (and also provide as a managed service) the open source schema
registry Karapace, https://www.karapace.io/
Looking at the docs here https://github.com/Aiven-Open/karapace it looks like
it can be installed with Docker or a “Source install” which may work for you?
Regards, Paul Brebne
Hi Karan, good question! I’ve asked our Kafka dev team and they think it may be
possible in theory, but depending on if you are using a managed Kafka service
maybe not supported – e.g. NetApp Instaclustr managed Kafka supports Karapace,
but not with multiple clusters.
Good luck, Paul Brebner
F
than 87, you
have recommended.
wrt to the number of partitions 35 seems to be OK as our consumers should
rarely scale beyond 10 or something like that.
Please let me know if this sounds OK given our current utilization rates.
Thanks
Sachin
On Mon, Feb 3, 2025 at 7:51 AM Brebner, Paul
wrote:
Hi Sachin,
I’m not an “operational” Kafka person but do have some limited experience with
Kafka benchmarking etc, so here are a few ideas.
I’m playing around with a Kafka tiered storage sizing model at present,
designed to predict min IO and/or network with local and tiered storage
enabled. Th
Hi John,
I’m not a Kafka streams expert but have experimented a few times – I recall
that Kafka Streams does need to create/use “internal topics” – and security has
to be set on clients correctly from memory.
This may help?
https://kafka.apache.org/23/documentation/streams/developer-guide/manag
Hi – short answer is consumers can read from a specific partition, but in
general for a consumer group you want to balance the partitions across the
available consumers for high throughput – if a consumer fails or is kicked off
the group because it times out etc then the remainder of the consume
Sounds “interesting” – in theory it could work, just remember that segment size
will impact latency – records are stored in segments on local/remote storage
(with tiering enabled), bigger segments improve throughput, but smaller
segments may improve read latency, Paul Brebner
From: Greg Harris
em
> be saved locally by consumer or Redis, etc.
>
> The concept of consumer group shifts the responsibility of partition
> assignment to broker because only broker knows the number of partitions.
>
> Best regards.
>
> On Thu, 23 Jan, 2025, 06:25 Brebner, Paul, .invalid>
> w
Hi Ernar, I don’t think anyone responded yet so here’s my 2 cents worth (I’m
not a Kafka ops expert, but I did ask our Kafka techops people – the following
are suggestions however, not professional advice – which we do also offer 😉):
Looks like there is more traffic at nigh and cluster struggles
And after a 2nd (and 3rd) opinion it does look like replica.lag.time.max.ms is
below the default value, so maybe try increasing as a first step, Paul
From: Brebner, Paul
Date: Thursday, 13 February 2025 at 2:33 pm
To: users@kafka.apache.org
Subject: Re: Under replicated partition
EXTERNAL
Hi Kafka people!
There's still time (7 days and counting down) to submit talks for Community
Over Code NA 2025 - if you have something performance related here's the CFP
for the 7th C/C Performance Engineering track https://lnkd.in/gR5wv3RD (it
doesn’t need to be Kafka specific),
Regards, Paul
Hi Omer,
We (NetApp Instaclustr) are pretty happy with Karapace (we contributed to it),
and provide it as a managed service along with our Apache Kafka offering.
I did a blog series on it a while back if you are interested in reading about
my experiences with it, starting here with part 1
http
Well spotted I think – I was briefly puzzled with the time retention behaviour,
as segments seemed to live longer than advertised – until I realised it was min
time, deletion is lazy – can occur at some (distant?) time in the future (and
is async I think) – this was particularly noticeable for
Hi all, time for me to ask a silly question please!
I'm puzzled about the transition from KIP-429
https://cwiki.apache.org/confluence/display/KAFKA/KIP-429%3A+Kafka+Consumer+Incremental+Rebalance+Protocol
to KIP-848
https://cwiki.apache.org/confluence/display/KAFKA/KIP-848%3A+The+Next+Generatio
Sorry error:
429 appeared in Kafka 2.4.0, and 848 appeared in 3.7
Paul
From: Brebner, Paul
Date: Tuesday, 17 June 2025 at 12:32 pm
To: Kafka Users , dev
Subject: KIP-429 vs. KIP 848?
EXTERNAL EMAIL - USE CAUTION when clicking links or attachments
Hi all, time for me to ask a silly
Hi Anana,
Typically in Kafka, it is useful for Consumers to know about the number of
partitions (as the number of consumers must be <= partitions).
So one way for Consumers to find partitions is the KafkaConsumer class using
the partitionsFor(topic) method,
https://kafka.apache.org/40/javadoc/o
1...@qq.com
-- 原始邮件 --
发件人: "Brebner, Paul"https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/KAFKA/KIP-429*3A*Kafka*Consumer*Incremental*Rebalance*Protocol**BKIP-848https:/*cwiki.apache.org/confluence/display/KAFKA/KIP-848*3A*The*Next*Gener
Hi,
Curious if anyone has thought about using Kafka Queues (KIP-932) with Kafka
Connect in the future? I.e. autoscaling connect tasks?
There could be some Connect use cases where order isn’t as important as keeping
up with load spikes, maybe.
Regards, Paul Brebner
NetApp Instaclustr Technology
35 matches
Mail list logo
