In your producer application, you could write a logic for key1 & key2 to be
produced to separate topics with appropriate number of partitions for that
key according to your throughput requirements. For the consumer application
you can configure it to consumer from multiple topics or can have instan
Thanks Akash Jain for your detailed explanation. I have answered your
queries.
Yes as you said the key 1 and key 2 doing more produces. So the skew is
happening.
In which case you will have to get creative with your topic
> design
Can you please guide on how to do dynamic segregation of such ke
Hi Karthick,
The choice has to be yours depending on what you want to achieve. I
understand you want to achieve even distribution of messages across your
partitions. This depends on the following factors:
- The frequency of keys
- Hashing logic itself
What you can control is the hashing lo
Hi Akash Jain
Thanks for the reply seeking help for the same to choose hashing logics.
Please refer/suggest any.
On Sat, Aug 17, 2024 at 10:21 AM Akash Jain wrote:
> Hi Karthick. You could implement your own custom partitioner.
>
> On Saturday, August 17, 2024, Karthick wrote:
>
> > Hi Team,
>
Hi Karthick. You could implement your own custom partitioner.
On Saturday, August 17, 2024, Karthick wrote:
> Hi Team,
>
> I'm using Kafka partitioning to maintain field-based ordering across
> partitions, but I'm experiencing data skewness among the partitions. I have
> 96 partitions, and I'm s
